179 questions
No questions match those filters.
← All questions
Inference & ServingMedium· KV Cache· VRAM· GQA
Your KV cache is overflowing VRAM at the batch size you...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe most effective architectural lever is GQA, which reduces the number of KV heads from N to N/G (typically G=8), shrinking the cache by 8x with minimal accuracy impact. Other effective strategies include KV quantization, such as using INT8 to halve the cache size, or applying sliding-window attention on non-critical layers to limit the context stored in VRAM.