← All questions
Retrieval & EmbeddingsHard· Debugging· Evaluation· Root Cause Analysis

How do you adjudicate a disagreement between engineers regarding why a specific document is not being retrieved?

Do not accept proposed fixes until the failure is localized. By pulling the document by ID and checking the retriever’s output, you can determine if the bug is a retrieval failure or a generation failure. If the document never appears in the top-k, the gap is in the indexing or query generation process. If it appears in the top-k but the generator ignores it, the problem is entirely downstream of retrieval, and neither engineer’s proposed fix is relevant.

Code playground (Python)