Why is the naive PyTorch GeLU 7× slower than the fused version?

The naive PyTorch GeLU implementation is slower because it consists of six separate CUDA kernels. Each kernel reads and writes the full tensor to HBM. For a 256 MB tensor at 2 TB/s, each round-trip costs approximately 128 μs. Six round-trips plus six kernel launch overheads result in roughly 8 ms of latency. In contrast, the fused kernel performs only one read and one write, reducing the HBM cost to approximately 0.26 ms.