DPO is often observed to reduce output diversity and pr...

The mechanism behind DPO’s reduced diversity is that the gradient update forces the policy to increase the likelihood of chosen responses while decreasing that of rejected ones. If the dataset’s chosen responses are systematically shorter or less varied, the model converges toward that mode. SimPO mitigates this by incorporating a length normalization term into the reward calculation. IPO (Identity Preference Optimization) modifies the objective by using an identity link instead of the standard Bradley-Terry sigmoid, which helps prevent the model from becoming over-confident on pairs that are nearly equal in quality.