179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Throughput· Context Window· Inference Optimization
Retrieval wants to raise the context from 8k to 32k to...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansIncreasing the context from 8k to 32k results in a 4x increase in KV cache per request and a significant increase in prefill FLOPs due to the quadratic nature of the attention mechanism. This leads to a substantial drop in concurrency and throughput. Instead of immediately scaling hardware, one should evaluate if a reranker can solve the truncation issue. If failures persist after reranking, then the hardware upgrade is justified, as no amount of ranking can recover information that was never provided to the model.