179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Throughput Optimization· Quantization· KV Cache
Your batch is already at memory capacity. What are your...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansWhen memory capacity is reached, you can increase TPS by reducing the memory footprint per request. First, use GQA or MLA to reduce the number of KV heads, which shrinks the KV cache size (Cseq) and allows for a larger batch size (Bmax). Second, apply quantization to the model weights (e.g., INT8); this reduces the weight-sweep cost and shifts the crossover point where KV cache dominates, effectively allowing for a larger batch size at the same per-step latency. These strategies are multiplicative.