In-Context Robot Learning with VLM Agents

TL;DR - GPT-Policy is a framework that uses vision-language model agents to learn robot tasks from deployment-time context without gradient updates. Real-robot experiments suggest that human videos improve task completion, while aligned action references provide additional gains for contact-sensitive tasks.

• A context compiler preserves task-relevant visual transitions from demonstrations, examples, and interaction feedback.
• A VLM proposes robot-tool actions, while a constrained controller verifies and executes them and reports outcomes.
• The study evaluates task success, efficiency, model differences, and the effects of removing context components.
• Human demonstrations can help even without robot action labels, indicating a path toward more adaptable general-purpose robotic policies.