Retrieval made an internal QA assistant worse on questi...

Formalize it before reaching for a heuristic. Let p(q) be the probability a closed-book answer is correct and r(q) the probability a retrieval-augmented answer is correct, with gate g(q) deciding which branch runs. Always-retrieve achieves E[r], never-retrieve achieves E[p], and a perfect oracle gate achieves E[max(p, r)]. Partition queries into four buckets: both branches right (α), only closed-book right (β), only retrieval right (γ), both wrong (δ). Always-retrieve’s accuracy is α+γ, and the oracle’s ceiling above that is exactly β — the share of queries where retrieval actively breaks an answer the model already had right. That single fact reframes the whole design problem: a good gate has to separate β from γ, not separate easy queries from hard ones, because easy queries mostly live safely in α, where gating changes nothing but cost.

Three practical proxies attempt that separation. Classifier gates train a small model on the query or on frozen hidden states to predict retrieve-or-not. Confidence/competence gates threshold a scalar like entity popularity or perplexity — cheap and label-free, but blunt. Self-reflective gates let the model emit its own control signal mid-generation (Self-RAG’s retrieve token, FLARE’s low-confidence trigger). The tempting shortcut — gate on the generator’s own next-token confidence — fails twice: it measures fluency under the model’s distribution, not truth, so a confidently wrong continuation looks identical to a confidently right one; and the signal doesn’t exist until a full closed-book decode has already run, so you pay generation cost on every query regardless of the gate’s decision. All three families share the same failure mode out of distribution: a gate trained on observed β/γ splits has no signal for query types where neither branch has been seen to fail.