179 questions
No questions match those filters.
Your cluster uses TP-8. You want a 70B model with dmode...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTo determine the head count, divide the model dimension (8192) by the target head dimension (128) to get 64 heads. When using a Tensor Parallel (TP) degree of 8, each GPU shard must handle an integer number of heads. With 64 total heads, each shard gets 8 heads. For Grouped Query Attention (GQA), the number of KV heads must also be divisible by the TP degree. GQA-8 (8 KV heads) and GQA-4 (16 KV heads) are both compatible with TP-8, as they result in 1 and 2 KV heads per shard respectively. A configuration like GQA-6 would be invalid because 6 is not divisible by 8.