Describe the three policy roles in GRPO and when each o...

In GRPO, πθ is the primary optimization target and updates every inner step. The πold policy acts as the importance weight anchor and is updated only once per outer iteration after rollout generation is complete. The πref policy serves as a behavioral stability anchor and remains fixed or is updated very infrequently throughout the training process.