179 questions
No questions match those filters.
← All questions
LLM FoundationsMedium· MoE· Scaling Laws· Model Architecture
Why does a sparse MoE outperform a dense model trained...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansAt equal FLOPs, the activated experts in an MoE perform the same arithmetic as a dense FFN, but the model retains additional capacity in dormant experts. This extra capacity is empirically productive for memorization across diverse tasks, allowing the model to be parameter-rich while remaining compute-efficient during training.