Why would DPO underperform PPO in some experimental set...

DPO suffers when the policy drifts significantly from the distribution used to collect the preference data, causing the implied rewards to become stale and the gradients to provide incorrect signals. PPO avoids this by using on-policy rollouts, which ensures the gradient signal remains current. While iterated DPO can close this gap by re-collecting preferences, it incurs a compute cost similar to PPO.