How does an SM hide memory latency?

Streaming Multiprocessors (SMs) hide latency by keeping a large number of warps in flight. When a warp issues a memory request and must wait hundreds of cycles for data, the SM switches to a different warp that is ready to execute. Because each warp’s register state is stored directly in the register file, this switch happens with zero overhead. Consequently, high occupancy—the ratio of active warps to the maximum—is essential for keeping the SM busy during long-latency operations.