‹ back to 2026-09-17

Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

Research LLMs & Foundation Models

Ranking

Overall 83

Content 90

Popularity 67

Observed public metrics from 1 member.

Merged summary

TL;DR - This paper identifies “Value Flattening,” a PPO failure mode where critics produce overly uniform predictions despite sharp changes in Monte Carlo state values. It proposes sparse critic supervision that improves learned policies while using only three supervised states per response.

Value Flattening appears in both LLM reinforcement learning and controlled FrozenLake experiments, worsening as the state space grows.
The authors link it to an implicit variance penalty in the critic loss and redundant updates from temporally correlated states with similar gradients.
Their SParse Proximal Policy Optimization (SP³O) method applies value loss only to a few well-separated states in each response.
Experiments on Qwen3-Base report consistent policy improvements across model sizes and evaluation suites.

Sources (1)

Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

arXiv cs.LG Yizhuo Li, Jianhao Yan, Yun Luo, Zhi Wang, Futing Wang, Rong-Xi Tan, Kanghui Tian, Ganqu Cui, Ning Ding, Peilin Zhao, Yafu Li, Yu Cheng 2026-09-16 arXiv:2609.18708

Public signals Hugging Face upvotes 79

Providers: Hugging Face · Upvotes 79 OpenAlex · N/A Publisher · N/A Semantic Scholar · N/A X · N/A Fetched 2026-09-25 14:19:00.461671 UTC

Value Flattening appears in both LLM reinforcement learning and controlled FrozenLake experiments, worsening as the state space grows.
The authors link it to an implicit variance penalty in the critic loss and redundant updates from temporally correlated states with similar gradients.
Their SParse Proximal Policy Optimization (SP³O) method applies value loss only to a few well-separated states in each response.
Experiments on Qwen3-Base report consistent policy improvements across model sizes and evaluation suites.

item →