179 questions
No questions match those filters.
← All questions
Inference & ServingMedium· Long Context· KV Cache· Latency
Our knowledge base is 200k tokens and the model takes 2...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansPrefill compute costs scale with context length, and at 210k tokens, the system is well past the crossover point where retrieval becomes more efficient. Beyond compute, the memory footprint is a major constraint; 27.5 GB of KV cache for a single request limits the system to only two concurrent users per GPU. Retrieval is necessary to maintain performance, concurrency, and cost-efficiency.