179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Reranking· Caching· Cost-Benefit Analysis
Your lead wants to replace the production cross-encoder...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe swap only makes sense if there is sufficient cache hit concentration to justify the overhead of a larger decoder model. If 95% of documents are queried only once, the system pays the penalty of the larger model without gaining the performance benefits of caching. The team should instead instrument retrieval hit-rate concentration to see if a small, hot subset of documents accounts for a disproportionate share of traffic before committing to the architecture change.