179 questions
No questions match those filters.
← All questions
Inference & ServingMedium· Inference Optimization· Quantization· Memory Bandwidth
Why is INT8 inference faster than BF16 inference at bat...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansAt a batch size of 1, the primary bottleneck for LLM inference is the time taken to load weights from HBM, not the arithmetic throughput of the compute units. Because INT8 weights are half the size of BF16 weights, the GPU can load the same number of parameters in half the time. The higher arithmetic throughput of INT8 tensor cores is secondary here, as the compute units are already waiting for data from HBM; therefore, the reduction in weight-loading time is the dominant factor in speeding up per-token decode latency.