← All questions
Training & Fine-tuningHard· Maximal Update Parameterization· Transformer Architecture

Walk me through the μP recipe for a transformer.

To implement μP in a transformer, non-embedding weights should be initialized at 1/√nin. The Adam learning rate for each layer must be scaled by 1/nin relative to a base rate determined on a small proxy model. Additionally, the attention scale should be changed from 1/√dhead to 1/dhead, while the embedding learning rate is held constant at the base rate.

Code playground (Python)