179 questions
No questions match those filters.
← All questions
LLM FoundationsHard· MoE· Compute-Optimal· Scaling Laws
How does the MoE FLOPs advantage interact with compute-...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansCompute-optimal reasoning applies to training FLOPs and activated parameters. Because dormant MoE parameters do not require multiply-accumulate operations, they are essentially free during training. This allows an MoE to be compute-optimal regarding activated parameters while maintaining a large total parameter count. However, this efficiency advantage flips at inference time, where serving costs scale with total parameters.