Why should most FLOPs in an LLM be in matrix multiplica...

Matrix multiplications (matmuls) are unique because their arithmetic intensity (AI) is N/3 for an N x N matrix, meaning they become compute-bound as the size increases. Other common operations in LLMs have constant AI, meaning they are always limited by memory bandwidth. Furthermore, tensor cores provide significantly higher FLOPs/s than general ALUs, and only matmul operations can effectively utilize this specialized hardware.