The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and What Actually Works
Merged summary
TL;DR - Dense prediction rewards can collapse GRPO-trained LLM agents because groupwise standard-deviation normalization amplifies reward variance in all-fail groups. Delivering the same signal through an auxiliary loss instead improved ALFWorld performance by roughly 20 points.
- Across Qwen3-1.7B/4B/8B, prediction accuracy reached 1.0 while task success fell to 0% and episodes hit the maximum horizon.
- Removing only GRPO’s standard-deviation normalization restored performance to baseline parity.
- The authors propose that dense signals are safe when their within-group variance declines as the model masters them.
- Results are currently single-seed; replication and group-size controls are preregistered and ongoing.
Sources (1)
The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and What Actually Works
TL;DR - Dense prediction rewards can collapse GRPO-trained LLM agents because groupwise standard-deviation normalization amplifies reward variance in all-fail groups. Delivering the same signal through an auxiliary loss instead improved ALFWorld performance by roughly 20 points.
- Across Qwen3-1.7B/4B/8B, prediction accuracy reached 1.0 while task success fell to 0% and episodes hit the maximum horizon.
- Removing only GRPO’s standard-deviation normalization restored performance to baseline parity.
- The authors propose that dense signals are safe when their within-group variance declines as the model masters them.
- Results are currently single-seed; replication and group-size controls are preregistered and ongoing.