How does KV cache size constrain maximum batch size at...

The KV cache for B sequences of length T occupies significant VRAM, often exhausting capacity before the GPU’s compute capability is fully utilized. Paged attention addresses this by storing KV blocks in non-contiguous VRAM pages, similar to virtual memory. This approach enables fine-grained memory sharing, such as prefix caching, and eliminates memory fragmentation caused by variable-length sequences.