Lost in the middle is the tendency of language models to use information placed at the beginning or end of a long input far more reliably than information buried in its middle. In artificial intelligence practice, the effect means that a fact can sit inside the context window and still be ignored, so accuracy depends not only on what the prompt contains but on where it appears.
The name comes from a 2023 study that measured a U-shaped accuracy curve: retrieval and question answering were strongest when the relevant passage came first or last and weakest when it sat in the center. The broader decline of quality as inputs grow longer is often called context rot, and the two ideas are closely related.
How the U-shaped position effect appears
The effect appears as a dip in accuracy for mid-context positions when the same question is asked with the answer moved through a long input. Researchers typically place one relevant document among many distractors, for example 20 retrieved passages, and shift its position from first to last while keeping everything else constant.
The resulting curve resembles the serial position effect known from human memory research: a primacy advantage at the start, a recency advantage at the end and a trough in between. In the original experiments, some models answered worse with the relevant passage in the middle than with no documents at all, which shows that a long context can actively harm performance.
The following table summarizes the typical pattern across positions in a long prompt.
| Position in the input | Typical reliability | Main reason |
|---|---|---|
| First 10 to 20 percent | High | Primacy bias and strong attention to early tokens |
| Middle 40 to 60 percent | Lowest | Weak attention and competing distractor text |
| Last 10 to 20 percent | High | Recency bias near the question being answered |
| Immediately before the question | Usually highest | Shortest distance to the generated answer |
Why language models lose mid-context information
No single cause explains the effect; it emerges from architecture, training data and positional encoding acting together. Researchers still debate the relative weight of each factor, and the size of the dip varies considerably between model families and versions.
- Training data distribution: documents used for pretraining and instruction tuning tend to put key content at the start, such as titles and summaries, or at the end, such as conclusions and the latest turn of a dialogue.
- Positional encoding behavior: schemes such as rotary position embeddings introduce a long-range decay that favors nearby tokens, which helps the end of the prompt.
- Attention sinks: models learn to place disproportionate attention on the first few tokens, which inflates the weight of the beginning.
- Causal attention: in a decoder-only model, every later token can attend to the first tokens, so early content is processed more often across layers.
These mechanisms explain why the problem is not a capacity limit. A model with a 128,000-token window can store a whole report but still assign insufficient weight to a paragraph in the middle of it.
How lost in the middle compares with related limitations
Lost in the middle is one of several long context failure modes, and it is often confused with simpler limits on input length. The table below separates it from neighboring concepts.
| Concept | What fails | Key difference |
|---|---|---|
| Lost in the middle | Use of mid-context facts | Depends on position, not total length alone |
| Context rot | Overall quality as input grows | Gradual decline even with good placement |
| Context window limit | Tokens beyond the maximum | Text is truncated and never seen |
| Hallucination | Factual grounding | May result when the relevant fact is missed |
| Distractor interference | Choice among similar passages | Similar but wrong text pulls the answer |
The distinction matters for diagnosis. Truncation is fixed by a larger window, while lost in the middle requires better ordering or shorter inputs, because the information was technically available to the model.
How the effect is measured
The effect is measured with controlled position sweeps that change only the location of the target information. Three test designs dominate current evaluation practice.
| Test design | What it measures | Limitation |
|---|---|---|
| Multi-document question answering | Use of one relevant passage among distractors | Depends on retrieval quality of the test set |
| Key-value retrieval | Exact lookup in synthetic data | Easier than real reasoning tasks |
| Needle in a haystack | Finding one inserted sentence | Often saturated by newer models |
| Multi-hop long context tasks | Combining several scattered facts | Expensive to build and score |
A model that passes a needle in a haystack test may still show the dip on harder tasks. Single-fact retrieval is now handled well by many frontier models, whereas reasoning across several mid-context facts remains noticeably weaker, so evaluation should match the real workload.
Practical ways to reduce the effect
Mitigation focuses on placing critical information where models attend best and on sending less text overall. The most effective techniques are simple changes to prompt construction rather than changes to the model.
| Technique | How it helps | Trade-off |
|---|---|---|
| Reranking retrieved passages | Puts the best evidence first or last | Adds a reranker call and latency |
| Fewer, better chunks | Shrinks the weak middle region | Risk of dropping useful context |
| Question at the end | Uses recency near the answer | Instructions at the start may weaken |
| Repeating key instructions | Restates rules after long documents | Uses extra tokens |
| Map-reduce summarization | Processes sections in short calls | More calls and possible detail loss |
In retrieval-augmented generation, a robust pattern is to retrieve generously, rerank aggressively and pass only the top five to ten passages. Placing the most relevant passage last, directly before the question, often gives a further gain, although the best ordering should be confirmed on the specific model.
Common misconceptions about long context recall
- A larger window solves it: a longer window increases capacity but can widen the weak middle zone.
- The effect is fixed for all models: newer models trained on long documents show a flatter curve, yet few eliminate it on complex tasks.
- Perfect needle scores prove robustness: synthetic retrieval is easier than synthesis across scattered facts.
- Only retrieval systems are affected: long chat histories, codebases and agent tool logs suffer the same position bias.
The simple explanation
Imagine handing someone a thick stack of papers and asking one question. They read the first pages carefully and remember the last pages because they just saw them, but the pages in the middle blur together. Language models behave in a similar way.
The model can technically see every page, yet it pays most attention to the beginning and the end. If the answer is hidden in the middle, the model may miss it and answer from guesswork instead. The practical lesson is simple: put the most important material at the start or the end, and do not send more text than the task needs.
How it is used in practice
A legal technology team builds an assistant that answers questions about commercial contracts. The first version retrieves 30 contract clauses for each question and pastes them into the prompt in the order returned by the search index. Testing shows that answers are correct about 85 percent of the time when the decisive clause lands near the top, but noticeably less often when it falls between positions 10 and 20.
The team adds a reranking step, keeps only the eight strongest clauses and places the highest-scoring clause directly before the question. They also restate the answer format after the documents. On the same internal test set, accuracy becomes far more stable across positions, prompt costs fall because fewer tokens are sent, and lawyers report fewer answers that overlook an exception clause buried in a long agreement.