GPU utilization is at 80% but token generation is still...

When GPU utilization is high but generation is slow, it is critical to recognize that compute utilization is not the same as memory bandwidth utilization. During generation, the GPU is frequently waiting on HBM reads. Therefore, the bottleneck is typically memory bandwidth. The path to improvement is not to add more FLOPs, but rather to reduce memory traffic.