How does CLA interact with tensor parallelism and disaggregated KV caching?

With tensor parallelism, each device holds a shard of the KV heads, and downstream layers in a group read the shared shard instead of computing fresh projections, reducing communication overhead by a factor of G. In disaggregated serving architectures, where the KV cache is stored in remote memory or on the CPU, CLA reduces the data transfer volume per decode step by G. Since this transfer is often the primary bottleneck, the latency benefits are disproportionately large.