Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

TL;DR - Q-Planning enables large behavior-cloned robot policies to improve from successful and failed deployment rollouts by training a small off-policy Q-function while keeping the original policy frozen. It delivers stable gains in simulation and on contact-rich real-robot tasks without further human demonstrations.

• The Q-function guides inference using a single-step, Q-weighted average over actions sampled from the behavior-cloned policy.
• After ten self-improvement iterations, success rose from 93% to 99% on LIBERO-10 and from 83.8% to 91.4% on bimanual RoboTwin.
• In five real-robot iterations without human intervention, stack-cups improved from 40% to 90% and insert-wallet from 25% to 80%.
• Under the same online data budget, Q-Planning was the only tested method to improve stably from failures without training an auxiliary actor.