Explain the RLHF pipeline: what does the reward model l...

In the RLHF pipeline, human raters compare pairs of model responses, and a Bradley-Terry model is used to fit a scalar reward function that predicts which response is preferred. The policy is then optimized using Proximal Policy Optimization (PPO) to maximize the expected reward minus a KL divergence term relative to a reference policy. This KL term is critical as it prevents the policy from collapsing into ‘reward-hacking’ behaviors that are out-of-distribution for the reward model. Implementing PPO is memory-intensive, as it requires maintaining four model copies: the current policy, the old policy for importance sampling, the reference policy for the KL penalty, and the value function.