OlMoE found no gain from shared experts but DeepSeek di...

The discrepancy between OlMoE and DeepSeek results can be reconciled by looking at scale and training budget. DeepSeek models are significantly larger (16B-671B parameters) compared to OlMoE (7B). At larger scales, each routed expert encounters fewer tokens relative to its capacity, which allows a shared expert to absorb more useful gradient. At the 7B scale, routed experts converge fast enough that the shared component adds no net value. The crossover point is estimated to be between 7B and 16B total parameters.