If the next GPU generation doubles peak FLOPs but keeps memory bandwidth the same, does token generation speed improve?

Since token generation is already memory-bound, doubling peak FLOPs provides no benefit to throughput. The ridge point on the roofline model shifts higher, making the memory-bound regime even more pronounced. Improvements in generation speed can only be achieved by increasing arithmetic intensity, such as through larger batch sizes, reducing the number of KV-cache bytes per token via quantization or GQA, or utilizing architectures with constant-size state.