Describe zero-bubble pipelining and what it costs to im...

The key insight is that the backward pass has two independent phases: B (activation gradients, required by earlier stages) and W (weight gradients, executable anytime after the forward pass). Implementing this requires intercepting autograd at the layer level, maintaining a deferred queue for W work, and tracking intermediate activations. The memory overhead for deferred activations can offset efficiency gains, and the scheduling logic is notoriously complex, as seen in implementations like DualPipe.