179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Tensor Parallelism· Sequence Parallelism· Communication
Walk me through the exact communication ops for one tra...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe steps are: (1) Input arrives as s/t rows. (2) AllGather before QKV/FFN projection to get full s x h activation. (3) Column-parallel matmul (no communication). (4) Row-parallel matmul (partial sums). (5) ReduceScatter to merge sums. (6) LayerNorm and Dropout on s/t rows. (7) Residual add. This uses two collective ops per MLP and attention block, maintaining identical byte counts to the replaced AllReduce.