When would you use PPO instead of DPO?

PPO is the better choice when you have verifiable rewards, such as in code generation or mathematical problem solving, or when you need to perform on-policy exploration to discover high-reward completions not present in your initial offline dataset. Because PPO is on-policy, it can escape the limitations of a static dataset. DPO, by contrast, is bounded by the quality of the provided preference pairs, making it less effective if the preference dataset is small or noisy.