Why would you add a shared expert to a MoE layer?

The hypothesis behind adding a shared expert is that certain MLP-level transformations are universal across all token types. By factoring these into a dedicated always-on expert, the routed experts can focus on more specialized tasks. Furthermore, the shared expert receives a dense gradient on every token at every step, which helps it converge faster and provides a stable baseline during the early stages of training when routed experts are still underfit.