Long context

Also known as: long-context model

On this page 7

Long context is the capability of language models to accept and reason over very large inputs in a single request, typically from about 100,000 tokens to several million tokens, instead of the few thousand tokens that early systems could process. In artificial intelligence practice, a long-context model can read an entire codebase, a full contract set or hours of transcripts at once and answer questions that depend on details spread across the whole input.

The term describes more than a large number on a specification sheet. A model has genuine long context only when it can retrieve, connect and reason about information located anywhere in that input, which is why advertised window size and effective context length are treated as separate measurements.

How language models achieve long context

Long context is achieved by combining positional encoding changes, efficient attention kernels and targeted training on long documents. A transformer processes every token in relation to other tokens through self-attention, and the standard form of that operation has a cost that grows quadratically with sequence length. Doubling the input therefore roughly quadruples the attention computation, which made long inputs impractical until several engineering advances matured.

The first obstacle is position representation. Most current models use rotary position embeddings (RoPE), which encode token order as rotations. A model trained on 8,000-token sequences does not generalize well to positions it has never seen, so developers apply position interpolation, frequency rescaling methods such as YaRN, or larger base frequencies, and then continue training on long sequences so the model adapts to the stretched positions.

The second obstacle is memory. During generation, the model stores keys and values for every previous token in the KV cache, and that cache grows linearly with context length. At hundreds of thousands of tokens, the cache can exceed the memory used by the model weights themselves. The table below summarizes the main techniques and the bottleneck each one addresses.

TechniqueBottleneck addressedMain trade-off
Position interpolation and YaRNUnseen token positionsNeeds extra long-sequence training
FlashAttentionMemory traffic during attentionExact result, but cost remains quadratic
Sliding-window attentionQuadratic computeDistant tokens are reached only indirectly
Grouped-query attentionKV cache sizeSmall quality loss in some models
Ring or sequence parallelismSingle-accelerator memory limitRequires multi-device infrastructure
KV cache quantizationCache memory footprintPossible precision loss on fine details
Techniques that enable long context

Advertised window versus effective context

The context window is the maximum number of tokens a model accepts, while the effective context length is the span over which it still performs reliably. Evaluations such as needle-in-a-haystack tests, RULER and LongBench consistently show that many models degrade well before their advertised limit, especially on tasks that require combining several facts rather than finding one.

A well-documented failure pattern is lost in the middle: models recall information near the beginning and end of the input more accurately than information buried in the center. Simple retrieval tests also overstate ability, because finding one planted sentence is far easier than multi-hop reasoning across distant passages. The following table compares common evaluation types.

Evaluation typeWhat it measuresLimitation
Needle in a haystackRetrieval of one planted factToo easy; saturates quickly
Multi-needle retrievalRecall of several scattered factsStill mostly lexical matching
Variable tracking and aggregationFollowing state across the inputSynthetic, not real documents
Long-document question answeringComprehension of natural textAnswers may leak from training data
Repository-level coding tasksUse of cross-file dependenciesExpensive to build and grade
Ways to measure long-context ability

Typical context sizes and what they hold

Context sizes are measured in tokens, and in English one token averages roughly three quarters of a word, although the ratio depends on the tokenizer, the language and the content type. Source code, tables and non-Latin scripts usually consume more tokens per word than plain English prose, so capacity estimates should always be checked with the actual tokenizer.

Window sizeApproximate English wordsTypical content that fits
8,000 tokensAbout 6,000A long article or short report
32,000 tokensAbout 24,000A detailed technical specification
128,000 tokensAbout 96,000A full-length novel or contract bundle
200,000 tokensAbout 150,000A medium codebase or several books
1,000,000 tokensAbout 750,000A large repository or hours of transcripts
Approximate capacity of common window sizes

Long context compared with retrieval-augmented generation

Long context and retrieval-augmented generation (RAG) are complementary strategies for giving a model information it does not hold in its parametric knowledge. Long context places all material directly in the prompt, while RAG selects a small set of relevant passages through search and inserts only those. The choice depends on corpus size, cost, latency and the kind of reasoning required.

SituationPreferred approachReason
Single large document, holistic questionsLong contextAnswers need the whole structure
Corpus far larger than any windowRetrievalNo window can hold millions of pages
Frequent queries over stable materialLong context with prompt cachingCached prefix lowers repeated cost
Strict latency or budget limitsRetrievalShort prompts are faster and cheaper
Cross-file code changesLong context or hybridDependencies are hard to retrieve piecemeal
Choosing between long context and retrieval

In practice, many production systems use a hybrid design: retrieval narrows a huge corpus to a few hundred thousand tokens, and a long-context model then reasons over that selection. This approach keeps the ranking benefit of search while avoiding the fragmentation that occurs when retrieved chunks are too small.

Costs, risks and common mistakes

Long context carries real cost and reliability penalties that teams often underestimate. Pricing is usually per input token, so a million-token request can cost more than thousands of short ones, and time to first token rises noticeably because the model must process the entire prompt before generating output.

  • Filling the window by default: irrelevant material dilutes attention and can lower answer quality.
  • Trusting the advertised limit: reliability should be tested at the actual input sizes used in production.
  • Placing the question only at the start: restating the task after the documents usually improves results.
  • Ignoring prompt injection: long untrusted inputs give attackers more room to hide instructions.
  • Assuming perfect recall: long context still produces hallucinations when facts are ambiguous or absent.

Good practice is to structure the input with clear document labels, ask the model to quote relevant passages before answering, and use prompt caching when the same large prefix is reused across many requests.

Explained without jargon

Long context means an AI system can read a very large amount of text at once. That could be a whole book or an entire software project. Older systems saw only a few pages at a time. People had to cut documents into pieces.

Picture one person who sees a single page of a binder. Another person spreads the whole binder across a big table. The second person can compare an early page with a late one. Still, a big table does not guarantee careful reading. Details in the middle can still be missed. Size and reading quality are two different things.

A typical use case

A legal team at a mid-sized company must review a supplier agreement bundle of about 140,000 tokens: the master agreement, nine amendments and several annexes written over six years. Earlier attempts with a retrieval tool returned isolated clauses but missed how a later amendment quietly overrode a liability cap in the original contract.

The team loads the complete bundle into a model with a 200,000-token window, labels each document with its date, and places the questions at the end of the prompt. The model identifies three conflicting provisions, quotes each one with its document label, and explains which amendment controls. A lawyer verifies every quoted passage, and the review that previously took two working days is completed in about three hours, with the human check remaining the final authority.