How do you detect whether your reward model has learned...

To detect if a reward model has overfit to superficial style, hold out a set of pairwise examples where ground truth is independently verifiable, such as math problems, executable code, or factual QA. Measure the reward model’s ranking accuracy on this set. If the scores correlate more strongly with response length than with ground-truth correctness, the model has overfit to style. You can fix this by augmenting the training set with verified correct-versus-incorrect pairs, adding explicit accuracy dimensions to the annotation rubric, and using verifiable rewards where possible.