Long context is the capability of language models to accept and reason over very large inputs in a single request, typically from about 100,000 tokens to several million tokens, instead of the few thousand tokens that early systems could process. In artificial intelligence practice, a long-context model can read an entire codebase, a full contract set or hours of transcripts at once and answer questions that depend on details spread across the whole input.
The term describes more than a large number on a specification sheet. A model has genuine long context only when it can retrieve, connect and reason about information located anywhere in that input, which is why advertised window size and effective context length are treated as separate measurements.
How language models achieve long context
Long context is achieved by combining positional encoding changes, efficient attention kernels and targeted training on long documents. A transformer processes every token in relation to other tokens through self-attention, and the standard form of that operation has a cost that grows quadratically with sequence length. Doubling the input therefore roughly quadruples the attention computation, which made long inputs impractical until several engineering advances matured.
The first obstacle is position representation. Most current models use rotary position embeddings (RoPE), which encode token order as rotations. A model trained on 8,000-token sequences does not generalize well to positions it has never seen, so developers apply position interpolation, frequency rescaling methods such as YaRN, or larger base frequencies, and then continue training on long sequences so the model adapts to the stretched positions.
The second obstacle is memory. During generation, the model stores keys and values for every previous token in the KV cache, and that cache grows linearly with context length. At hundreds of thousands of tokens, the cache can exceed the memory used by the model weights themselves. The table below summarizes the main techniques and the bottleneck each one addresses.
| Technique | Bottleneck addressed | Main trade-off |
|---|---|---|
| Position interpolation and YaRN | Unseen token positions | Needs extra long-sequence training |
| FlashAttention | Memory traffic during attention | Exact result, but cost remains quadratic |
| Sliding-window attention | Quadratic compute | Distant tokens are reached only indirectly |
| Grouped-query attention | KV cache size | Small quality loss in some models |
| Ring or sequence parallelism | Single-accelerator memory limit | Requires multi-device infrastructure |
| KV cache quantization | Cache memory footprint | Possible precision loss on fine details |
Advertised window versus effective context
The context window is the maximum number of tokens a model accepts, while the effective context length is the span over which it still performs reliably. Evaluations such as needle-in-a-haystack tests, RULER and LongBench consistently show that many models degrade well before their advertised limit, especially on tasks that require combining several facts rather than finding one.
A well-documented failure pattern is lost in the middle: models recall information near the beginning and end of the input more accurately than information buried in the center. Simple retrieval tests also overstate ability, because finding one planted sentence is far easier than multi-hop reasoning across distant passages. The following table compares common evaluation types.
| Evaluation type | What it measures | Limitation |
|---|---|---|
| Needle in a haystack | Retrieval of one planted fact | Too easy; saturates quickly |
| Multi-needle retrieval | Recall of several scattered facts | Still mostly lexical matching |
| Variable tracking and aggregation | Following state across the input | Synthetic, not real documents |
| Long-document question answering | Comprehension of natural text | Answers may leak from training data |
| Repository-level coding tasks | Use of cross-file dependencies | Expensive to build and grade |
Typical context sizes and what they hold
Context sizes are measured in tokens, and in English one token averages roughly three quarters of a word, although the ratio depends on the tokenizer, the language and the content type. Source code, tables and non-Latin scripts usually consume more tokens per word than plain English prose, so capacity estimates should always be checked with the actual tokenizer.
| Window size | Approximate English words | Typical content that fits |
|---|---|---|
| 8,000 tokens | About 6,000 | A long article or short report |
| 32,000 tokens | About 24,000 | A detailed technical specification |
| 128,000 tokens | About 96,000 | A full-length novel or contract bundle |
| 200,000 tokens | About 150,000 | A medium codebase or several books |
| 1,000,000 tokens | About 750,000 | A large repository or hours of transcripts |
Long context compared with retrieval-augmented generation
Long context and retrieval-augmented generation (RAG) are complementary strategies for giving a model information it does not hold in its parametric knowledge. Long context places all material directly in the prompt, while RAG selects a small set of relevant passages through search and inserts only those. The choice depends on corpus size, cost, latency and the kind of reasoning required.
| Situation | Preferred approach | Reason |
|---|---|---|
| Single large document, holistic questions | Long context | Answers need the whole structure |
| Corpus far larger than any window | Retrieval | No window can hold millions of pages |
| Frequent queries over stable material | Long context with prompt caching | Cached prefix lowers repeated cost |
| Strict latency or budget limits | Retrieval | Short prompts are faster and cheaper |
| Cross-file code changes | Long context or hybrid | Dependencies are hard to retrieve piecemeal |
In practice, many production systems use a hybrid design: retrieval narrows a huge corpus to a few hundred thousand tokens, and a long-context model then reasons over that selection. This approach keeps the ranking benefit of search while avoiding the fragmentation that occurs when retrieved chunks are too small.
Costs, risks and common mistakes
Long context carries real cost and reliability penalties that teams often underestimate. Pricing is usually per input token, so a million-token request can cost more than thousands of short ones, and time to first token rises noticeably because the model must process the entire prompt before generating output.
- Filling the window by default: irrelevant material dilutes attention and can lower answer quality.
- Trusting the advertised limit: reliability should be tested at the actual input sizes used in production.
- Placing the question only at the start: restating the task after the documents usually improves results.
- Ignoring prompt injection: long untrusted inputs give attackers more room to hide instructions.
- Assuming perfect recall: long context still produces hallucinations when facts are ambiguous or absent.
Good practice is to structure the input with clear document labels, ask the model to quote relevant passages before answering, and use prompt caching when the same large prefix is reused across many requests.
Explained without jargon
Long context means an AI system can read a very large amount of text at once. That could be a whole book or an entire software project. Older systems saw only a few pages at a time. People had to cut documents into pieces.
Picture one person who sees a single page of a binder. Another person spreads the whole binder across a big table. The second person can compare an early page with a late one. Still, a big table does not guarantee careful reading. Details in the middle can still be missed. Size and reading quality are two different things.
A typical use case
A legal team at a mid-sized company must review a supplier agreement bundle of about 140,000 tokens: the master agreement, nine amendments and several annexes written over six years. Earlier attempts with a retrieval tool returned isolated clauses but missed how a later amendment quietly overrode a liability cap in the original contract.
The team loads the complete bundle into a model with a 200,000-token window, labels each document with its date, and places the questions at the end of the prompt. The model identifies three conflicting provisions, quotes each one with its document label, and explains which amendment controls. A lawyer verifies every quoted passage, and the review that previously took two working days is completed in about three hours, with the human check remaining the final authority.