If RMSNorm drops mean-centering, doesn’t that hurt mode...

The concern regarding the lack of mean-centering in RMSNorm is mitigated by the pre-norm architecture. Because the residual stream entering each normalization layer is already regulated by upstream norms, the empirical mean remains close to zero at initialization and throughout the training process. Ablation studies confirm that in this regime, explicit mean-centering provides negligible correction to the model’s performance.