179 questions
No questions match those filters.
← All questions
Inference & ServingHard· KV Cache· Tensor Parallelism· HBM
You are serving a 70B GQA-8 model (nq = 64, nkv = 8, dh...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTo calculate the KV cache, compute bytes per token per layer: 2 (for K and V) * 8 (KV heads) * 128 (dhead) * 2 (bytes per float16) = 4,096 bytes. Total cache = 4,096 * 8192 (context) * 32 (batch) * 80 (layers) ≈ 85.9 GB. With TP-4, the cache per GPU is ~21.5 GB. Adding the sharded model weights (140 GB / 4 = 35 GB), the total memory footprint is ~56.5 GB, leaving ~23.5 GB for activations, which is viable on an 80 GB H100.