← All questions
EvaluationHard· multi-document QA· synthetic key-value retrieval· control experiment

Why include the synthetic key-value retrieval task alon...

Multi‑document QA can be confounded by the fact that middle chunks are harder to reason about, so a performance drop might be due to reasoning difficulty rather than a positional bias (the U). By adding a synthetic key‑value retrieval task, which requires no reasoning at all, the experiment controls for this confound. Observing the same U pattern in the synthetic task demonstrates that the effect is purely positional, not driven by the increased difficulty of reasoning over middle passages. This control experiment therefore validates that the observed U is not an artifact of reasoning complexity.

Code playground (Python)