Why is transformer generation memory-bound, and what ar...

Transformer generation is memory-bound because the attention arithmetic intensity is roughly 1; batching requests does not amortize the cost because both FLOPs and memory bandwidth scale linearly with the batch size. The architectural solution is to replace the growing KV cache with a fixed-size recurrent state, as seen in State Space Models (SSMs), which keeps the generation memory cost constant regardless of context length.