179 questions
No questions match those filters.
← All questions
System DesignMedium· Context Parallelism· Attention· Memory Optimization
Why does Llama 3 use context parallelism only for the l...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansContext parallelism (CP) distributes attention computation by passing KV blocks across devices. At standard pre-training lengths like 4096 tokens, existing techniques like Tensor Parallelism (TP) and Sequence Parallelism (SP) keep activation memory within limits. However, because attention memory scales quadratically with sequence length, CP becomes necessary at extreme lengths like 128K tokens to prevent OOM errors.