‹ back to 2026-08-04

Start Classifying: Categorical Critics for LLM Reinforcement Learning

arXiv cs.LG LLMs & Foundation Models Zhijian Zhou, Long Li, Xuan Zhang, Zongkai Liu, Yulei Qin, Ke Li, Xing Sun, Xiaoyu Tan, Chao Qu, Yuan Qi 2026-08-03

TL;DR - HL-Gauss PPO replaces the PPO critic's scalar MSE regression head with a categorical predictor over a discretized value support trained via cross-entropy on smoothed targets, improving LLM RL with verifiable rewards (RLVR). It matters because sparse binary rewards make critic calibration errors directly distort PPO advantages.

The critic outputs a distribution over value bins and is decoded to a scalar expectation, so GAE and the actor update remain standard PPO — the method is not distributional on the actor side.
Gains hold across mathematical reasoning, tool-augmented math, and Search-R1, on both Qwen2.5 and Qwen3 backbones, beating strong PPO and DAPO baselines.
Ablations with one-hot, two-hot, and Bernoulli two-bin critics rule out "bigger output head" or "binary classification alone" as the explanation.
On shared reasoning prefixes, HL-Gauss improves Brier score and calibration error and produces more symmetric, lower-variance advantages, supporting the claim that categorical value learning acts as an optimization surrogate.

view merged work →