Learning Latent Reasoning Traces for Scalar Reward Models End-to-End
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - LatentRM is a reward-modeling framework that treats intermediate reasoning traces as discrete latent variables optimized end-to-end to maximize scalar reward likelihood, bridging the gap between interpretable scalar RMs and robust generative RMs. It matters because reward model generalization is a key bottleneck for RLHF-based alignment on complex and out-of-distribution tasks.
- Frames the problem as a mismatch: scalar RMs are efficient and probabilistically interpretable but latch onto superficial cues; generative RMs reason well but produce natural-language scores lacking numerical flexibility.
- Argues existing hybrids using off-policy multi-task learning optimize reasoning and scoring in parallel, with no guarantee the generated traces actually help downstream scalar prediction.
- Core method: discrete latent reasoning traces optimized on-policy end-to-end, explicitly coupling reasoning-based evaluation to precise scoring.
- Reported to outperform scalar, generative, and hybrid RMs on in-distribution and OOD preference modeling plus RLHF policy alignment, spanning open-ended conversation to complex reasoning (no specific numbers given in the abstract).
Sources (1)
Learning Latent Reasoning Traces for Scalar Reward Models End-to-End
TL;DR - LatentRM is a reward-modeling framework that treats intermediate reasoning traces as discrete latent variables optimized end-to-end to maximize scalar reward likelihood, bridging the gap between interpretable scalar RMs and robust generative RMs. It matters because reward model generalization is a key bottleneck for RLHF-based alignment on complex and out-of-distribution tasks.
- Frames the problem as a mismatch: scalar RMs are efficient and probabilistically interpretable but latch onto superficial cues; generative RMs reason well but produce natural-language scores lacking numerical flexibility.
- Argues existing hybrids using off-policy multi-task learning optimize reasoning and scoring in parallel, with no guarantee the generated traces actually help downstream scalar prediction.
- Core method: discrete latent reasoning traces optimized on-policy end-to-end, explicitly coupling reasoning-based evaluation to precise scoring.
- Reported to outperform scalar, generative, and hybrid RMs on in-distribution and OOD preference modeling plus RLHF policy alignment, spanning open-ended conversation to complex reasoning (no specific numbers given in the abstract).