A context window is the maximum number of tokens a language model can process in a single request, including the system instructions, the prompt, any supplied documents, the conversation history and, in most systems, the tokens the model generates in its answer. In language models, it acts as the model’s working memory: anything outside the window is invisible to the model during that request.
Context window size is measured in tokens, not words or characters. For English text, a token averages roughly three quarters of a word, but the ratio varies with language, formatting and the tokenizer, and code or non-Latin scripts often use more tokens per word.
How the context window works
The context window works as a fixed-length sequence of token positions that a Transformer-based LLM processes together. During generation, each new token is produced by attending to all earlier tokens in the window, then appended to it, so the input and output share the same budget.
The model has no memory between requests beyond what is placed back into the window. Chat applications create the appearance of memory by resending previous turns with every message, which means a long conversation steadily consumes the available space until older content must be dropped or summarized.
The table below shows what typically occupies the window in a single request.
| Component | Typical content | Effect on available space |
|---|---|---|
| System instructions | Role, rules, output format | Fixed cost on every request |
| Conversation history | Previous user and model turns | Grows with each exchange |
| Supplied documents | Retrieved passages, files, code | Often the largest share |
| Tool definitions and results | Function schemas, returned data | Can be large in agent workflows |
| Generated output | The model’s answer | Limited by a separate output cap |
Context window sizes and the limits behind them
Context window sizes have grown from a few thousand tokens in early large language models to hundreds of thousands of tokens, and in some models more than a million. Many models also set a maximum output length far smaller than the total window.
The main constraint is the cost of self-attention, which in its standard form grows quadratically with sequence length: doubling the context roughly quadruples the attention computation. Memory for the key-value cache, which stores intermediate results for every token, grows linearly with context length and often dominates serving cost.
| Factor | Why it matters | Common technique |
|---|---|---|
| Attention computation | Quadratic growth with length | Optimized attention kernels, sparse attention |
| Key-value cache memory | Stored for every token and layer | Grouped-query attention, cache quantization |
| Position encoding | Must represent distant positions | Rotary embeddings with scaling |
| Training data length | Model must learn to use long inputs | Long-context fine-tuning stages |
| Latency and price | More input tokens take longer and cost more | Prompt caching, shorter prompts |
Effective context versus advertised context
The advertised window states how much a model accepts, not how well it uses every part of it. Evaluations repeatedly show that retrieval and reasoning quality can degrade as inputs grow longer, and that information placed in the middle of a long context is sometimes used less reliably than information near the beginning or end.
Simple needle-in-a-haystack tests, which hide one fact in a long document, measure only lookup. Tasks that require combining many scattered facts, tracking state across a long transcript or reasoning over the whole input are harder, so the effective context for demanding tasks is usually smaller than the advertised maximum. The size of that gap varies by model and task and should be measured rather than assumed.
Context window compared with related concepts
The context window is closely tied to other language model concepts, and confusing them leads to wrong expectations about what a model can remember or know.
| Concept | What it is | Relation to the context window |
|---|---|---|
| Token | Unit of text the model processes | The unit in which the window is measured |
| Tokenization | Splitting text into tokens | Determines how much text fits |
| Prompt | Input given to the model | Occupies part of the window |
| Training data knowledge | Patterns learned in the weights | Available without using the window |
| Retrieval-augmented generation | Fetching relevant passages per query | Selects what enters the window |
| Hallucination | Fluent but unsupported output | More likely when needed facts fall outside |
A large window does not update the model’s knowledge; it only provides material for one request. Settings such as temperature affect how tokens are sampled, not how much context the model can see.
Working within the context window in practice
Practical systems treat the window as a scarce budget to be allocated deliberately. Sending everything available is rarely optimal: it raises cost and latency and can dilute the relevant information.
| Situation | Recommended strategy | Reason |
|---|---|---|
| Large document collection | Retrieval of the most relevant chunks | Keeps the window focused |
| Long chat sessions | Rolling summaries of older turns | Preserves key facts in less space |
| Repeated long instructions | Prompt caching where supported | Reduces cost and latency |
| Single long document analysis | Full document with precise questions | Avoids losing cross-references |
| Agent workflows with many tools | Trim tool output, store state externally | Prevents the window from overflowing |
Developers should count tokens with the model’s own tokenizer, reserve room for the expected output and handle overflow explicitly. When a request exceeds the limit, an API typically returns an error or truncates input, and silent truncation of the oldest content is a frequent source of puzzling model behavior.
Put simply
A context window is how much text a language model can keep in view at one time. Think of a desk. Everything on the desk can be read and used. Anything that does not fit stays in the filing cabinet, and the model cannot see it.
A bigger desk holds more pages, such as a whole contract or a long chat. Still, a crowded desk can make the important page harder to find. The useful lesson is to give the model the right material, not simply the most material.
In practice
A legal operations team uses a language model to review supplier contracts against a 40-page internal policy. Each contract runs to roughly 30,000 tokens, and the policy to about 25,000. Sending the full policy and contract together fits comfortably within a 200,000-token context window, but answers about specific clauses are occasionally inconsistent.
The team restructures the workflow. The policy is split into sections, and only the three or four sections relevant to each question are retrieved and placed before the contract text. The cached system instructions stay at the start of every request. Each request now uses about 45,000 tokens instead of 55,000, responses arrive faster, and reviewers find that the model cites the correct policy clause far more consistently.