How does GQA improve long-context throughput?

Grouped Query Attention (GQA) improves throughput by reducing the memory footprint of the KV cache. By reducing the number of K and V heads by a factor g, the KV cache size (Cseq) is reduced by the same factor. This allows the system to fit more requests into the same HBM budget, increasing the maximum batch size (Bmax) and, consequently, the total TPS. This effect is most pronounced at long context lengths, where the KV cache is the binding constraint.