When would you choose DPO over training an explicit rew...

DPO reparameterizes the reward in terms of the policy’s own log-ratio, which removes the need for a separate reward model and the associated RL training loop. It is the optimal choice for simpler pipelines, static datasets where no new on-policy preferences are collected, and policies under approximately 30B parameters. Conversely, an explicit reward model paired with PPO is required when you need live scoring of model-generated outputs (on-policy rollouts) that do not exist in precomputed datasets, or if the reward model needs to be reused across different policy variants.