How would you design the FFN structure for a 7B active...

To design a 7B active parameter MoE, one should anchor the design on fine-grained experts. By setting m=4, you can have 64 total routed experts at 1/4-size, with K=6 active routed experts plus 1-2 shared experts. This results in a FLOP arithmetic of roughly 8 * 2 * dmodel * dexpert * 2, which equates to about 2x the dense FFN FLOPs but at approximately 16x the FFN parameters. This configuration is modeled after the DeepSeek MoE architecture.