179 questions
No questions match those filters.
← All questions
EvaluationVery Hard· Throughput· Prefix Caching· Performance Modeling
Your prefix cache has an 80% hit rate on a 4K shared sy...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTo quantify the impact, calculate the effective KV footprint. A hit sequence uses 0.78 GB (unique 1K KV), while a miss uses 3.9 GB (full 5K KV). With an 80% hit rate, the average per-sequence KV is (0.8 * 0.78) + (0.2 * 3.9) = 1.40 GB. Adding the 3.1 GB for the cached prefix pinned in GPU memory, the effective Bmax becomes 36, compared to 13 without caching. This results in a 2.8x throughput improvement.