How do weight decay and cosine LR decay interact to imp...

Weight decay and cosine LR decay work together by managing the loss landscape. Weight decay constrains parameter magnitudes, which reduces the dominant eigenvalue of the Hessian, effectively creating a smoother loss landscape. When the learning rate is high, gradient direction dominates. However, as the learning rate decays, the optimizer becomes sensitive to local curvature. By keeping the model in a smoother neighborhood, weight decay allows the optimizer to descend more efficiently into low-sharpness minima during the final stages of training.