‹ back to 2026-09-22

Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use

arXiv cs.LG LLM Agents Zixiang Chen, Wenting Zhao, Zhepeng Cen, Akshara Prabhakar, Jielin Qiu, Jianguo Zhang, Zhiwei Liu, Tulika Manoj Awalgaonkar, Liangwei Yang, Shelby Heinecke, Silvio Savarese, Huan Wang 2026-09-21

TL;DR - Critical-State RL diagnoses which model calls in multi-turn tool-use trajectories are actually trainable by separating action-dependent reward variation from downstream noise. Training only at selected states improved tool-use performance, including roughly 14 percentage points on a missing-function task.

The method checks whether local rewards reflect an action’s effect on success and whether improvement over a reference policy is feasible.
Nested sampling distinguishes reward variation caused by the current action from randomness in later interactions.
Selected states are optimized with contextual-bandit training rather than indiscriminately training every turn.
On BFCL v4, training diagnostic-selected responses outperformed training alternative states, which produced flat or worse results.

view merged work →