PPO for an LLM is actually a contextual bandit, not a f...

Treating an LLM as a contextual bandit rather than a full Markov Decision Process (MDP) means that there are no intermediate state transitions; the prompt is the context, the response is the action, and the reward is provided once at the end. This simplifies credit assignment, as multi-step Bellman backups are unnecessary. Consequently, the discount factor gamma can be set to 1, and the value function only needs to estimate the expected return of the entire completion. The KL penalty is applied as a token-by-token reward shaping term rather than a state-transition constraint, which allows methods like GRPO to function without a value network.