179 questions
No questions match those filters.
← All questions
Inference & ServingMedium· KV Cache· GQA· Memory Optimization
Why does LLaMA-2 70B use GQA instead of standard multi-...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansStandard Multi-Head Attention (MHA) is memory-intensive because it stores keys and values for every head. At a 70B parameter scale, the KV cache requirements for MHA make it impossible to fit the model and cache on a single H100 GPU. GQA-8 shares keys and values across groups of query heads, reducing the memory overhead by a factor of 8 without sacrificing benchmark performance.