Walk me through the arithmetic intensity of a LayerNorm...

The roofline model highlights that LayerNorm is bandwidth-rooflined, characterized by low arithmetic intensity. Consequently, reducing memory accesses in these kernels leads to proportional improvements in wall time. In contrast, matrix multiplications at large batch sizes are compute-rooflined, meaning that reducing FLOPs in those operations results in a much smaller real-world speedup. Optimizing memory-bound kernels like normalization provides super-linear returns relative to their share of total FLOPs.