Lost in the middle

Hand a model twenty documents and ask it a question. It will reliably use the first few and the last few — and accuracy on anything buried in the middle falls away.

The curve is U-shaped, and it persists in models explicitly built for long contexts. A bigger window is not the same thing as better recall, which is the whole argument for curating context rather than dumping it.

The argument rests on a study by Nelson Liu and colleagues, circulated as a preprint in 2023 and published in Transactions of the Association for Computational Linguistics in 2024. The design is plain enough to repeat with your own documents, and it tests an ordinary job: answering a question from a stack of sources.

What was tested

The main task was multi-document question answering. The questions came from Natural Questions, a set of real queries once typed into Google, each paired with an answer found in Wikipedia. The model received a question and a stack of 10, 20 or 30 passages, exactly one of which held the answer. The rest were distractors, chosen by a retrieval system for being relevant to the question while not containing the answer — so the hay looked a good deal like the needle.

The researchers then moved the answer-bearing passage to different positions in the stack, from first to last, and varied how many distractors surrounded it. Accuracy was highest with the answer at the very start or the very end, and sagged in between. In the worst cases — GPT-3.5-Turbo with the answer in the middle of 20 or 30 passages — the model did worse than when it was given no passages at all and had to answer from its training alone. The document it needed was in front of it, and it would have been better off without the pile.

A second task removed meaning altogether. The model received a long list of key–value pairs, every key and value a random string, and was asked for the value belonging to one key. No reasoning, no paraphrase, only lookup. One model family did this almost perfectly at every length tested; the others struggled as the list grew, and again their accuracy depended on where the key sat. Even exact-match retrieval, the simplest thing you can ask of a text, was subject to position.

The finding about models built for long contexts came from pairs. GPT-3.5-Turbo was tested alongside a 16K-token version of itself, Claude 1.3 alongside a 100K version. When the input fitted inside both windows, each pair performed nearly identically. The longer window let a model accept more text. It did not make the model better at using the text it accepted.

A familiar curve, loosely

The U will look familiar to anyone who has read about human memory. In free-recall experiments — Murdock’s 1962 study is the classic — people are given a list of words one at a time and then recall as many as they can, in any order. The first few and the last few come back best; the middle sags. Psychologists call the two ends primacy and recency, and Liu and colleagues borrow the terms, pointing out the parallel themselves.

It is a parallel of shape, not of mechanism. Human recall unfolds in time: words arrive one after another and have to be held while more arrive. A model receives its whole input at once, and, as the authors note, the attention mechanism underneath is technically as capable of retrieving one token as any other. That is what made the result surprising. Nothing in that design obviously calls for a sag in the middle, yet the models tested produced one.

So resist the conclusion that models ‘forget like we do’. The safer lesson is narrower: where something sits changes whether it gets used, and when you assemble a model’s input, position is something you control.

Accepted is not used

A context window is a limit on what a model can accept, not a promise about what it will use. The study made this concrete in one more experiment. Feeding a model more retrieved passages kept raising the chance that the answer was somewhere in its input, but the model’s accuracy levelled off long before that chance did. Going from 20 passages to 50 improved accuracy only marginally. The extra material was present; it was not doing much.

Later work points the same way. The well-known ‘needle in a haystack’ test hides one sentence in a long document and asks for it back. Hsieh and colleagues (2024) found that models scoring nearly perfectly on that test still showed large drops across a broader set of tasks — including following chains of references and aggregating information spread through the text — as inputs grew longer. Finding one planted sentence is the easiest version of long-context work.

None of this fails loudly. Picture someone planning a kitchen renovation who pastes forty contractor emails, three quotes and a summary of the building regulations into one prompt, then asks for the agreed start date. The date that was finally agreed sits in email twenty-three. The model answers fluently — perhaps with the date proposed in the first quote. Nothing about the reply signals that it rests on the wrong source.

One caution about all of these numbers. They describe the models tested at the time — GPT-3.5-Turbo, Claude 1.3 and open models of 2023 — not the one you use today. Long-context behaviour has been a busy area of work since, and a current model may show a flatter curve on some tasks and a familiar sag on others. The 2023 figures neither measure your model nor prove the problem has gone. What carries over is the test: move the thing that matters and see whether the answer changes.

Curating what goes in

If both volume and position cost something, the first remedy is to send less. A student revising for an exam could paste a whole term’s lecture transcripts and ask how two theories differ. Or they could pick the two lectures where the theories are introduced and the chapter that compares them. The second prompt is shorter, and it leaves far less middle for the decisive passage to get lost in.

Order is the second lever. Liu and colleagues suggest reranking retrieved material so the most relevant passages sit near the start, and cutting the list short when fewer documents will do. The same applies by hand: put the source you most need the model to use first, or last beside the question, not halfway down a long paste. If you cannot decide which source matters most, the question is probably too broad.

Restating the question helps less than you might hope. In the study, putting the question both before and after the material made the key lookup nearly perfect but barely changed the pattern in question answering. It is cheap, it tells the model what to look for before it starts reading, and it is not a cure.

Splitting a big question is the last lever. An engineer writing an incident report might paste three days of logs, a deployment timeline and a chat transcript, then ask what happened. A better sequence asks narrow questions, each with only its own material: when the configuration change shipped, from the deploy log; when the first alert fired, from the monitoring export; who picked it up, from a short stretch of chat. Each context is small enough that nothing sits deep inside it, and the engineer, not the model, assembles the timeline.

Checking what an answer rests on

Curation lowers the odds that a buried source is skipped; it does not remove them. The remaining defence is to check which sources an answer actually used. Ask the model to quote the passage behind each claim, then check each quote against the source itself. If every quote comes from the first and last documents, and the one you know matters sat in between, treat the answer as unproven. Then move that source to the top and ask again. If the answer changes, position was doing the work.

When producing text is cheap, the bottleneck moves to deciding what goes in and checking what comes out. Both are easier when sources stay attached to findings. A long chat thread buries what the model was given under everything said since. On a PlansFlow canvas, a web search answer lands as a block with its sources attached, and when you generate a draft you choose which blocks inform it — so the material behind an answer stays visible, short and easy to question.

In practice

1. 01

   Give the model the few sources a question needs rather than everything you have; if you cannot say why a document is included, leave it out.

2. 02

   Put the source that matters most first, or last beside the question — never deep inside a long paste.

3. 03

   State the question before the material and again after it, but treat that as a cheap habit rather than a fix.

4. 04

   Split a broad question into narrow ones, each with only its own material, and assemble the answers yourself.

5. 05

   For any answer you will rely on, ask which passage supports each claim and check the quotes against the sources. If in doubt, move the key source to the top and ask again.