Lost in the middle

Also known as: context rot

On this page 8

Lost in the middle is the tendency of language models to use information placed at the beginning or end of a long input far more reliably than information buried in its middle. In artificial intelligence practice, the effect means that a fact can sit inside the context window and still be ignored, so accuracy depends not only on what the prompt contains but on where it appears.

The name comes from a 2023 study that measured a U-shaped accuracy curve: retrieval and question answering were strongest when the relevant passage came first or last and weakest when it sat in the center. The broader decline of quality as inputs grow longer is often called context rot, and the two ideas are closely related.

How the U-shaped position effect appears

The effect appears as a dip in accuracy for mid-context positions when the same question is asked with the answer moved through a long input. Researchers typically place one relevant document among many distractors, for example 20 retrieved passages, and shift its position from first to last while keeping everything else constant.

The resulting curve resembles the serial position effect known from human memory research: a primacy advantage at the start, a recency advantage at the end and a trough in between. In the original experiments, some models answered worse with the relevant passage in the middle than with no documents at all, which shows that a long context can actively harm performance.

The following table summarizes the typical pattern across positions in a long prompt.

Position in the inputTypical reliabilityMain reason
First 10 to 20 percentHighPrimacy bias and strong attention to early tokens
Middle 40 to 60 percentLowestWeak attention and competing distractor text
Last 10 to 20 percentHighRecency bias near the question being answered
Immediately before the questionUsually highestShortest distance to the generated answer
Typical recall by position of the relevant information

Why language models lose mid-context information

No single cause explains the effect; it emerges from architecture, training data and positional encoding acting together. Researchers still debate the relative weight of each factor, and the size of the dip varies considerably between model families and versions.

  • Training data distribution: documents used for pretraining and instruction tuning tend to put key content at the start, such as titles and summaries, or at the end, such as conclusions and the latest turn of a dialogue.
  • Positional encoding behavior: schemes such as rotary position embeddings introduce a long-range decay that favors nearby tokens, which helps the end of the prompt.
  • Attention sinks: models learn to place disproportionate attention on the first few tokens, which inflates the weight of the beginning.
  • Causal attention: in a decoder-only model, every later token can attend to the first tokens, so early content is processed more often across layers.

These mechanisms explain why the problem is not a capacity limit. A model with a 128,000-token window can store a whole report but still assign insufficient weight to a paragraph in the middle of it.

Lost in the middle is one of several long context failure modes, and it is often confused with simpler limits on input length. The table below separates it from neighboring concepts.

ConceptWhat failsKey difference
Lost in the middleUse of mid-context factsDepends on position, not total length alone
Context rotOverall quality as input growsGradual decline even with good placement
Context window limitTokens beyond the maximumText is truncated and never seen
HallucinationFactual groundingMay result when the relevant fact is missed
Distractor interferenceChoice among similar passagesSimilar but wrong text pulls the answer
Related long context concepts

The distinction matters for diagnosis. Truncation is fixed by a larger window, while lost in the middle requires better ordering or shorter inputs, because the information was technically available to the model.

How the effect is measured

The effect is measured with controlled position sweeps that change only the location of the target information. Three test designs dominate current evaluation practice.

Test designWhat it measuresLimitation
Multi-document question answeringUse of one relevant passage among distractorsDepends on retrieval quality of the test set
Key-value retrievalExact lookup in synthetic dataEasier than real reasoning tasks
Needle in a haystackFinding one inserted sentenceOften saturated by newer models
Multi-hop long context tasksCombining several scattered factsExpensive to build and score
Common evaluation methods

A model that passes a needle in a haystack test may still show the dip on harder tasks. Single-fact retrieval is now handled well by many frontier models, whereas reasoning across several mid-context facts remains noticeably weaker, so evaluation should match the real workload.

Practical ways to reduce the effect

Mitigation focuses on placing critical information where models attend best and on sending less text overall. The most effective techniques are simple changes to prompt construction rather than changes to the model.

TechniqueHow it helpsTrade-off
Reranking retrieved passagesPuts the best evidence first or lastAdds a reranker call and latency
Fewer, better chunksShrinks the weak middle regionRisk of dropping useful context
Question at the endUses recency near the answerInstructions at the start may weaken
Repeating key instructionsRestates rules after long documentsUses extra tokens
Map-reduce summarizationProcesses sections in short callsMore calls and possible detail loss
Mitigation techniques and their trade-offs

In retrieval-augmented generation, a robust pattern is to retrieve generously, rerank aggressively and pass only the top five to ten passages. Placing the most relevant passage last, directly before the question, often gives a further gain, although the best ordering should be confirmed on the specific model.

Common misconceptions about long context recall

  • A larger window solves it: a longer window increases capacity but can widen the weak middle zone.
  • The effect is fixed for all models: newer models trained on long documents show a flatter curve, yet few eliminate it on complex tasks.
  • Perfect needle scores prove robustness: synthetic retrieval is easier than synthesis across scattered facts.
  • Only retrieval systems are affected: long chat histories, codebases and agent tool logs suffer the same position bias.

The simple explanation

Imagine handing someone a thick stack of papers and asking one question. They read the first pages carefully and remember the last pages because they just saw them, but the pages in the middle blur together. Language models behave in a similar way.

The model can technically see every page, yet it pays most attention to the beginning and the end. If the answer is hidden in the middle, the model may miss it and answer from guesswork instead. The practical lesson is simple: put the most important material at the start or the end, and do not send more text than the task needs.

How it is used in practice

A legal technology team builds an assistant that answers questions about commercial contracts. The first version retrieves 30 contract clauses for each question and pastes them into the prompt in the order returned by the search index. Testing shows that answers are correct about 85 percent of the time when the decisive clause lands near the top, but noticeably less often when it falls between positions 10 and 20.

The team adds a reranking step, keeps only the eight strongest clauses and places the highest-scoring clause directly before the question. They also restate the answer format after the documents. On the same internal test set, accuracy becomes far more stable across positions, prompt costs fall because fewer tokens are sent, and lawyers report fewer answers that overlook an exception clause buried in a long agreement.