Walk me through the PPO loop for a language model.

The PPO loop for a language model requires managing four distinct models. The process begins with a rollout step where the policy generates completions, followed by reward computation and the construction of a KL penalty to ensure the policy does not drift too far from the reference model. Advantage estimation is performed using Generalized Advantage Estimation (GAE), and the policy is updated using a clipped surrogate loss. It is critical to define the clip range epsilon and the KL coefficient beta, as omitting the value model or the KL shaping is a common mistake.