Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
Ranking
Overall
87
Content
95
Popularity
68
Observed public metrics from 1 member.
Merged summary
TL;DR - Q-Planning enables large behavior-cloned robot policies to improve from successful and failed deployment rollouts by training a small off-policy Q-function while keeping the original policy frozen. It delivers stable gains in simulation and on contact-rich real-robot tasks without further human demonstrations.
- The Q-function guides inference using a single-step, Q-weighted average over actions sampled from the behavior-cloned policy.
- After ten self-improvement iterations, success rose from 93% to 99% on LIBERO-10 and from 83.8% to 91.4% on bimanual RoboTwin.
- In five real-robot iterations without human intervention, stack-cups improved from 40% to 90% and insert-wallet from 25% to 80%.
- Under the same online data budget, Q-Planning was the only tested method to improve stably from failures without training an auxiliary actor.
Sources (1)
Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
Public signals
Hugging Face upvotes 2
TL;DR - Q-Planning enables large behavior-cloned robot policies to improve from successful and failed deployment rollouts by training a small off-policy Q-function while keeping the original policy frozen. It delivers stable gains in simulation and on contact-rich real-robot tasks without further human demonstrations.
- The Q-function guides inference using a single-step, Q-weighted average over actions sampled from the behavior-cloned policy.
- After ten self-improvement iterations, success rose from 93% to 99% on LIBERO-10 and from 83.8% to 91.4% on bimanual RoboTwin.
- In five real-robot iterations without human intervention, stack-cups improved from 40% to 90% and insert-wallet from 25% to 80%.
- Under the same online data budget, Q-Planning was the only tested method to improve stably from failures without training an auxiliary actor.