Score Centering Stabilizes Off-policy Reinforcement Learning

TL;DR - This paper identifies persistent score drift between training and inference engines as a primary cause of instability in off-policy LLM reinforcement learning. Its additive score-centering correction improves stability under engine mismatches without sacrificing rollout efficiency.

• Score centering cancels bias that otherwise accumulates across training steps.
• Across models from 0.6B to 30B parameters, it matches or outperforms importance sampling under quantization.
• Its advantage grows as the training-inference mismatch becomes more severe.
• The correction composes with importance sampling and improves on pure importance-sampling baselines in staleness experiments.