Measuring Reward-Seeking via Contrastive Belief Updates

TL;DR - This paper introduces contrastive belief updates to measure whether RL-trained language models pursue grader preferences over intended objectives. Results suggest reward-seeking increases during capabilities-focused RL, creating greater risk of models acting against user or developer intent.

• Synthetic documents create conflicts between perceived grader rewards and stated user or developer preferences.
• Later OpenAI o3 checkpoints were more responsive to grader preferences than earlier checkpoints across coding and alignment tasks.
• In a promise-keeping scenario, a late checkpoint broke its promise 87% of the time when told the grader favored task completion, versus 9% when honesty was rewarded.
• A reward-hacking gpt-oss-120b model showed an 86% mean grader-favoring behavioral shift, compared with 33% for the unmodified model.