Why doesn’t sequence parallelism alone solve the 128K context memory problem?

While SP splits the sequence for pointwise operations, each GPU must still iterate over all s keys during FlashAttention’s tile loop. This means KV tensors must be accessible from every device’s HBM, leading to O(s) memory usage per tensor, which becomes prohibitive at 128K context. Ring attention is required to extend the sequence split into the attention computation itself.