Why does the optimal learning rate decrease as you scale up a transformer?

In standard transformer training, as the width n increases, the Adam update norm per layer scales at a rate of O(√n) when using a global learning rate. This results in larger effective updates per activation. Maximal Update Parameterization (μP) addresses this by pre-correcting each layer’s learning rate by 1/n, ensuring that the update size remains constant regardless of the model’s width.