← All questions
RAGHard· Latency· Context Window· Evaluation

One engineer wants aggressive pruning (k = 10); another...

Flat offline accuracy metrics can be misleading if the model is subject to positional bias, effectively ignoring the majority of the provided context. Paying a 3x latency penalty for unused information is inefficient. The decision should be monitored by tracking the recall of the answer-bearing triple within the top-k results rather than relying on end-to-end accuracy, which fails to distinguish between missing information and model ignorance. A change in strategy would only be warranted if the workload involves deep, multi-hop reasoning where the extra context is proven to be load-bearing.

Code playground (Python)