179 questions
No questions match those filters.
← All questions
LLM FoundationsHard· Attention Mechanisms· Model Architecture· GQA
How do you design attention hyperparameters for a 70B model regarding head count, KV heads, and head dimension?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe head dimension is generally set to 128, which aligns with modern GPU warp sizes and Flash Attention block requirements. With a model dimension of 8192 for a 70B model, this results in 64 attention heads. To optimize for serving, Multi-Head Attention is replaced by Grouped-Query Attention (GQA), where KV heads are reduced to 8 or fewer. This configuration allows each KV head to be shared by multiple query heads, providing an 8x reduction in KV cache memory usage while maintaining performance.