‹ back to 2026-08-07

RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

arXiv cs.LG LLMs & Foundation Models Chenglong Wang, Ziming Zhu, Yifu Huo, Bei Li, Qiaozhi He, Yan Ding, Xiaoyang Hao, Yuxin Gao, Tianhua Zhou, Xiaojia Chang, Tongran Liu, Jingbo Zhu 2026-08-06

TL;DR - An arXiv preprint proposing Ranking-based Reward Construction (RRC), which converts generative reward models' comparative judgments into usable RL learning signals. It matters because generative reward models rank responses well but underperform in RL due to a mismatch with scalar-score-based RL algorithms.

Diagnoses the core problem: generative reward models are inherently comparative, while existing RL algorithms consume scalar rewards — this mismatch limits their RL effectiveness.
RRC derives rewards from relative preference rankings instead of absolute scores, via two complementary strategies.
Self-competitive ranking compares among sampled responses; anchor-guided ranking scales reward construction using a small set of reference responses.
Reported consistent gains over existing reward construction methods on open-ended chat and reasoning benchmarks; code released at github.com/wangclnlp/RRC.

view merged work →