179 questions
No questions match those filters.
← All questions
LLM FoundationsHard· Model Architecture· Parallel Layers· Inference Optimization
Some models run attention and MLP in parallel instead o...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansParallel layout fuses both sets of projections against the same normalized input in a single large matrix multiply, which saves memory round-trips and can yield roughly a 15% wall-clock speedup when kernels are properly written. However, the trade-off is that the MLP sees the pre-attention representation rather than the post-attention one, meaning they cannot interact within the same layer. Because of this, serial is the safer default unless custom kernel support is available and quality impact has been ablated.