How does increasing batch size help, and when does it s...

Increasing batch size (B) improves efficiency by allowing the model to load weight matrices once and apply them to multiple activation vectors simultaneously, which raises the arithmetic intensity for MLP layers toward the roofline.

However, this approach hits a ceiling because attention layers do not see a similar benefit; each sequence’s KV cache is independent, so the intensity for attention remains near 1. The hard limit is GPU VRAM: once the combined size of the weights and the KV cache for all sequences in the batch exceeds the available memory, the batch size can no longer be increased.