Buying a retrieval system usually means being shown one number: how often it fetches the right document. The number goes by different names on different slides (recall, Hit@k, MRR), and they all measure the same thing, whether the system found the right material and ranked it near the top. A high figure reads as proof the system is accurate. A team from Amazon Web Services and Cisco ran nine retrieval setups against a medical knowledge base and found that number describes the wrong step. The system found 83.5% of the correct answers, and the model wrote fewer than half of them into its reply (Chen et al., 2026).

Retrieval-augmented generation, or RAG, is the standard way to keep a language model grounded in facts it was never trained on. It has two jobs. First it fetches the passages relevant to a question. Then the model reads those passages and writes the answer. The score on the vendor slide almost always measures the first job. The user experiences the second.

The two jobs stopped agreeing with each other

Those jobs used to move together. Fetch better, answer better. The paper breaks that assumption with a clean measurement: it grades what the model put in its answer, not what the system pulled up to work from. On the run where the system found 83.5% of the correct answers, the model used 47.9% of them in what it wrote (Chen et al., 2026). The right facts were in front of the model. It left more than half of them on the floor.

The team then tried the obvious fix. Fetch more.

It did not work. As they pulled in more, the share of correct answers found kept rising while answer quality stayed flat, stuck between 45% and 48% no matter how much more they added (Chen et al., 2026). The run that found the most correct answers of all, 94.4%, gave the worst first-answer accuracy in the study (Chen et al., 2026). More material in front of the model, same answer, sometimes a worse one.

Four ways to build the fetch step

The fetch step can be built four ways, and a buyer will recognize all four from vendor pitches. In plain terms:

• Regular RAG looks up documents by meaning and hands them to the model. It is cheap, fast, and the default. It struggles when a question needs facts spread across several documents that no single passage holds.
• GraphRAG builds a map of entities and how they connect, then walks that map to answer questions that hop between related facts. It shines on "how is A linked to B" questions and costs more to build and maintain. On its own, without document text alongside it, it can miss badly.
• Modular RAG runs fixed steps in order: reword the question, fetch, re-rank the results, then answer. Adding steps buys predictability, which is the point for teams that need repeatable behavior.
• Agentic RAG lets the model run its own retrieval: search, read, search again in different words, stop when satisfied. It handles messy open-ended questions best, and it is the slowest and most expensive, since the model can loop and pile up context.

The study built nine setups across these four families to compare them fairly, and fancier did not track with better (Chen et al., 2026). The plainest method, regular RAG with each document's direct relationships stapled on, beat the full graph pipeline on the headline metric, and a stripped-down agent with a single search tool scored best of all (Chen et al., 2026). What every family shared was the gap: pushed to fetch more, each widened the distance between what it retrieved and what the model used.

Where the missing half bites, and where it hides

The gap does not hurt every job equally. It depends on whether the question has one right answer or many.

• Customer support: one good passage usually closes the ticket, so dropped facts rarely show. The gap hides here.
• Internal knowledge search: the employee reads the top hit and moves on. A partial answer passes for a complete one.
• Drug and diagnosis lookup, the paper's own domain: the question has many correct answers and the model returns the familiar one. Asked which disease fits a set of gum symptoms, it named the common terms and skipped the subtypes sitting nearby (Chen et al., 2026).
• Legal and compliance review: "find every clause that applies" is the exhaustive-answer task the model underperforms. High stakes, high exposure.
• Financial and research synthesis: pulling one figure is fine; assembling all the relevant ones into a single answer is where facts get left out.

Single-answer jobs mask the gap. Complete-the-list jobs expose it. That split is the fastest way to tell which of your own deployments is quietly running on half.

Why the model drops what it was handed

The authors traced three reasons. The first is position. The model reads the facts you hand it the way a tired reader works down a page, taking in almost everything at the top and little at the bottom. Facts in the first tenth of the page were used 85.5% of the time; past the two-thirds mark, the rate hit zero (Chen et al., 2026). Bury a correct answer near the bottom and it may as well not be there.

The second is preference. Asked which disease matches a set of symptoms, the model named the familiar term and skipped the specialist subtypes sitting right next to it, behaving like a clinician offering a short differential rather than a list that checks every box (Chen et al., 2026). The third is phrasing. A question worded "what is an" drug pushed the model to return one answer even when fourteen correct ones were present (Chen et al., 2026).

None of these show up in a retrieval score. All three change the answer.

The scoreboard rewards the step where sophistication looks best

Read the primer against the vendor cycle. Every quarter a more elaborate retrieval setup arrives with a higher benchmark attached, and that benchmark measures fetching, the step where added machinery shows its best face. The gap this paper names sits one step later, when the model writes the answer, where the same additions can wash out or reverse. A buyer comparing two systems on their fetch scores is reading the half of the machine that flatters the fancier product.

Two limits keep this from being a universal law. The work runs on a single dataset in one domain, precision medicine, and on a single model (Chen et al., 2026). The authors are testing a stack their own employers sell. The direction of the finding is what carries: a retrieval score and an answer are not the same measurement, and only one of them reaches your user.

The bandage, and why it is not a cure

Two things help. Put the facts that matter most at the top of the page you hand the model, where it is paying attention. Keep that page short, so good facts are not pushed down by filler the model stops reading. Trimming the filler also cut the computing cost 19% to 53%, which buys room for the facts that count (Chen et al., 2026).

Neither fixes the underlying behavior. The model still drops correct facts even when they sit right at the top where it is reading closely, because habit and question-wording keep steering it toward short answers. Placement and trimming stop you making the gap worse. They do not close it.

Which points at the one thing a buyer can act on now. Stop reading retrieval scores as answer quality. They are a measurement of the first half of a two-half machine.