Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

TL;DR - This paper identifies “Value Flattening,” a PPO failure mode where critics produce overly uniform predictions despite sharp changes in Monte Carlo state values. It proposes sparse critic supervision that improves learned policies while using only three supervised states per response.

• Value Flattening appears in both LLM reinforcement learning and controlled FrozenLake experiments, worsening as the state space grows.
• The authors link it to an implicit variance penalty in the critic loss and redundant updates from temporally correlated states with similar gradients.
• Their SParse Proximal Policy Optimization (SP³O) method applies value loss only to a few well-separated states in each response.
• Experiments on Qwen3-Base report consistent policy improvements across model sizes and evaluation suites.