Why is RL harder to scale than supervised fine-tuning?

Unlike supervised fine-tuning, RL training requires rollouts to complete before gradient steps can occur, placing inference in the critical path. Additionally, maintaining multiple model copies significantly increases peak memory usage. Weight synchronization introduces inter-node traffic not present in SFT, and the autoregressive nature of generation leads to variable-length outputs, which complicates the uniform batching required for high hardware utilization.