179 questions
No questions match those filters.
← All questions
Training & Fine-tuningHard· Maximal Update Parameterization· Transformer Architecture
Walk me through the μP recipe for a transformer.
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTo implement μP in a transformer, non-embedding weights should be initialized at 1/√nin. The Adam learning rate for each layer must be scaled by 1/nin relative to a base rate determined on a small proxy model. Additionally, the attention scale should be changed from 1/√dhead to 1/dhead, while the embedding learning rate is held constant at the base rate.