What are the systemic side effects of RLHF on model beh...

RLHF often introduces length bias, where verbosity is spuriously rewarded because human annotators conflate length with quality. Additionally, it causes calibration degradation, as RL optimizes a scalar reward rather than a distributional likelihood, which destroys the probabilistic interpretation of output probabilities. Both effects are empirically replicated and tend to worsen with more RL steps.