179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Memory Estimation· KV Cache· Model Quantization
Estimate the memory footprint of a 70B model during inf...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTo estimate the footprint, start by calculating the bytes required for parameters based on the chosen dtype. Then, add the KV cache term, which is proportional to the batch size, context length, number of key-value heads, and layer count. Note that switching from BF16 to FP8 halves the parameter footprint but does not affect the KV cache unless the cache itself is quantized. Grouped-query attention can be used to directly reduce the KV cache term.