How would you design a training system for 128K context?

Within an 8-GPU NVLink node, apply TP=8 for weight sharding and SP=8 for LayerNorm sharding. For attention, apply CP=8, rotating KV in a ring while each GPU holds 16K queries and uses online-softmax tile accumulation. Pipeline parallelism handles model depth across nodes. The CP ring adds roughly one AllGather-equivalent of communication per layer, which can be masked by overlapping ring sends with computation.