How would you benchmark whether your pipeline-parallel...

To determine the bottleneck in pipeline parallelism, instrument the code with torch.cuda.Event timers to track compute time and send/recv latency separately on each rank. A job is compute-bound if intermediate ranks maintain near 100% GPU utilization while the first and last ranks are idle, indicating the pipeline is saturated. It is communication-bound if the pipeline bubble shrinks when the volume of data sent between stages is reduced, suggesting that the time spent waiting for data transfer is the primary constraint.