Why is BF16 faster than FP32 for LLM training?

BF16 provides two primary mechanisms for speedup over FP32. First, by halving the bytes per element, it reduces HBM traffic for memory-bound operations like activations, norms, and elementwise operations, effectively halving their runtime. Second, BF16 inputs allow the use of H100 tensor cores, which offer a massive peak TFLOPs advantage over general FP32 ALUs. Since matrix multiplications dominate the total FLOPs in LLM training, this architectural shift results in a significant overall performance gain.