Explain why token generation is memory-bandwidth-bound...

The memory-bandwidth bottleneck in token generation arises from the nature of the decode step. Because each step processes a single new token, the operation is essentially a matrix-vector multiplication rather than a matrix-matrix multiplication.

Consequently, the arithmetic intensity is limited by the ratio of the batch size to the hidden dimension. In most practical scenarios, this value remains below the GPU’s compute-bandwidth ridge point, meaning the performance is constrained by how quickly data can be moved from memory to the compute units rather than the raw compute capability of the GPU.