Derive DPO from first principles.

To derive DPO, start with the KL-regularized RL objective. First, find the non-parametric optimum for the policy. Second, solve for the implied reward function that leads to this optimal policy. Finally, substitute this implied reward into the Bradley-Terry preference model. Through this substitution, the partition function Z(x) cancels out, resulting in the DPO loss function which depends only on the policy and the reference model.