The retrieval team just shipped a reranker taking recall@5 from 80% to 92%. One engineer wants to retrain with P = 0.92; another says P no longer matters, ship the always-golden model. Adjudicate.

The choice between an ‘always-golden’ model and a RAFT-trained model depends on the recall threshold. By solving for the intersection of the two models’ accuracies, we find that above approximately 91% recall, the always-golden model is superior. However, this calculation is sensitive; a reranker changes which distractors survive, meaning the previous measurements for g and b are no longer accurate. The decision requires re-measuring conditional accuracies and checking if the performance margin exceeds the noise of the evaluation split.