179 questions
No questions match those filters.
← All questions
LLM FoundationsMedium· MoE· Efficiency· Architecture
What is the parameter efficiency gain from fine-grained...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansVanilla MoE configurations typically use N=8 full-size experts with K=2, yielding 8x FFN parameters at 2x FFN FLOPs. By moving to a fine-grained approach (e.g., m=4, N=64, K=6), you can achieve approximately 16x FFN parameters while maintaining the same 2x FFN FLOPs. This significantly improves the parameter-to-activated-FLOP ratio, allowing for larger models with the same inference compute cost.