Why would adding retrieval make a model worse on questi...

Adding a retrieval component to an LLM does not strictly improve performance; in some cases, it degrades accuracy on queries the parametric model previously answered correctly. This happens because retrieval forces the generator to trade off its parametric prior (internal memory) for passage-local context.

For popular entities and well-represented subjects in the pre-training data, the model’s internal prior is usually accurate. However, if the retriever returns an off-target, noisy, or irrelevant context passage, the fine-tuned instruction-following behavior of the model forces it to faithfully copy information from the provided context. The distracting passage overrides the model’s correct parametric knowledge—an effect observed in around 10% of queries on benchmarks like PopQA.

Blaming the embedding retriever and attempting to replace it is often the wrong fix, as even a retriever with perfect recall can return distracting text. The proper way to analyze this phenomenon is to measure accuracy stratified by entity popularity, identifying the exact popularity crossover point where retrieval assistance transitions from beneficial to detrimental.