Web design · Development · SEO
Context Window (AI)
How much information an AI can use at once
Editorially reviewed ·
Clear definition
The context window is the limited amount of information an AI language model can process during a request. Its size is measured in tokens. It covers the inputs being used and space for generation. The precise limits depend on the model and application.
Imagine a desk holding your question, documents and notes from the conversation so far. You also need space to work on the answer. A larger desk lets you bring more material. It does not automatically ensure that every note is interpreted correctly. This picture can help you manage long AI conversations more deliberately.
What occupies the shared workspace
Your latest message is only part of the input. The application may also supply instructions, previous messages, file contents and tool results. Technical message structures occupy space as well. The application determines which information actually reaches the model.
An attachment visible in a chat therefore need not be included in full in every request. An application may extract text or retrieve selected passages. Images and audio follow model-dependent counting rules. When planning a budget, use the complete request and the provider’s usage data rather than the length of your latest question alone.
At a glance
A simple budgeting example
Assume a fictional context window of 10,000 tokens. Your complete input occupies 6,000. You reserve 3,000 for generation and leave a buffer of 1,000. That adds up to 10,000. The graphic shows this allocation. The buffer is a planning reserve, not a special token category.
Also check the model’s separate output limit. Unused context does not automatically permit an arbitrarily long answer. For reasoning models, the generation budget may include internal reasoning tokens; at OpenAI, these share the output limit with the visible response. These figures illustrate planning rather than describing a particular product.
What happens when the window fills up?
There is no single response across all AI products. A request may fail at a limit. An application may omit older material or condense the current state. An interrupted answer may instead result from the output limit. Check the specific message rather than describing every problem as forgetting.
Compaction turns earlier context into a smaller representation so that work can continue. This need not be a readable summary: OpenAI also documents a representation that is not intended for people to read. What matters for your work is whether goals, decisions and open questions remain available afterwards. Check important details against the original document.
Distinguishing context, memory and training
Chat history is the conversation an application stores or displays. Active context is the material available for the current processing step. A memory feature may store selected information across conversations and supply it again later. These are three different things.
The knowledge an LLM acquired during training is different again. Putting a document into context does not automatically change the trained model parameters. Whether a provider uses content for future training depends on its data policies and your settings. Context size does not answer that question.
Why larger does not automatically mean more reliable
A large capacity initially tells you how much can fit. How well a model uses the relevant information needs to be evaluated for your task. The 2023 study “Lost in the Middle” found that the position of relevant information affected results for the models and tasks it examined. This is not a universal measurement of today’s models, but it is a useful reason to test actual performance.
For a document comparison, you could first record three known differences, then check whether the AI finds them correctly. Ask for specific source locations. This evaluates quality against your own material instead of inferring it from an impressive token count.
Keeping longer tasks manageable
Start by defining the result you need and the conditions it must meet. Separate current requirements from discarded ideas. When starting a new conversation, a short handover can capture the goal, confirmed decisions, important sources and open questions. Read that handover yourself before relying on it.
For larger knowledge collections, a RAG system can retrieve relevant passages and supply them as context. It selects material rather than enlarging the window. Check that selection, too: if an essential source is missing, a well-written answer may still be incomplete. Sometimes splitting a task into verifiable steps is more useful than putting everything into one request.
Practical code example
Check a fictional budget
Original arithmetic example without an API call. All values are fictional and already expressed in tokens. This code neither tokenizes text nor validates a real model. Its output limit and counting rules require separate checks.
context_limit = 10000
input_tokens = 6000
output_budget = 3000
safety_buffer = 1000
required = input_tokens + output_budget + safety_buffer
print(required <= context_limit) # True
print(context_limit - input_tokens) # 4000Sources and further reading
- Documentation Conversation state: Managing the context window OpenAI · Checked
- Documentation Counting tokens OpenAI · Checked
- Documentation Reasoning models OpenAI · Checked
- Documentation Compaction OpenAI · Checked
- Documentation Retrieval OpenAI · Checked
- Research Lost in the Middle: How Language Models Use Long Contexts Nelson F. Liu et al. · arXiv / TACL · Checked
- Documentation Prompt caching OpenAI · Checked
