We replaced BM25 with a fine-tuned dense retriever last quarter. End-to-end quality moved two points and latency is unchanged. Where did the rest of the win go?

The retriever shifted the recall curve up by roughly ten points at fixed k. Because depth was held constant, this showed up as a small ceiling improvement instead of a large latency refund. The action is to find the depth at which the new curve matches the old system’s ceiling and cut to it. To mitigate risk, hold the old depth in a canary and compare recall@m rather than end-to-end accuracy, rolling forward only if the gap is within noise.