‹ back to 2026-09-23

PACT: From Credit Assignment to Critic Alignment

arXiv cs.LG LLM Agents Jiayan Fu, Hang Xu, Yong Zhang, Zhaokai Luo, Yao Hu, Dongyan Zhao, Mu Chuan 2026-09-22

TL;DR - PACT formalizes token-level credit assignment for LLM reinforcement learning and introduces an actor-then-critic update that better aligns the critic with the updated policy. It substantially improves agentic mathematical reasoning and software-engineering benchmark performance over several established RL methods.

Three regularity conditions—Completeness, Prefix Consistency, and Neutrality—uniquely characterize token-level credit.
The analysis connects token-level credit to On-Policy Distillation and explains why response-level RLOO can produce the same expected policy-gradient contribution.
PACT uses importance-sampling correction during critic training to address policy mismatch and critic errors associated with GAE.
PACT reaches 72.87% average accuracy across four math benchmarks and 67.4% on SWE-bench Verified, outperforming the reported PPO, GRPO, and SAO baselines.

view merged work →