Why does scaling weights by activation magnitude before...

The per-token quantization error for a channel j is defined as ϵj = xj · Δwj, where Δwj is the quantization step size (approximately ‖wj‖∞/127). By multiplying the weight wj by a scale factor sj > 1 before quantization, the post-dequantization error becomes Δw’j = Δwj / sj. This results in a reduced output error of ϵ’j = xj · Δwj / sj. By setting sj proportional to the activation magnitude (specifically ¯sj^α with α ≈ 0.5), we can balance the error reduction for important channels against the distortion introduced to less salient channels. This optimization is typically performed via a cheap 1D grid search over the calibration set.