In-Context Robot Learning with VLM Agents
Ranking
Overall
85
Content
95
Popularity
62
Observed public metrics from 1 member.
Merged summary
TL;DR - GPT-Policy is a framework that uses vision-language model agents to learn robot tasks from deployment-time context without gradient updates. Real-robot experiments suggest that human videos improve task completion, while aligned action references provide additional gains for contact-sensitive tasks.
- A context compiler preserves task-relevant visual transitions from demonstrations, examples, and interaction feedback.
- A VLM proposes robot-tool actions, while a constrained controller verifies and executes them and reports outcomes.
- The study evaluates task success, efficiency, model differences, and the effects of removing context components.
- Human demonstrations can help even without robot action labels, indicating a path toward more adaptable general-purpose robotic policies.
Sources (1)
In-Context Robot Learning with VLM Agents
Public signals
Hugging Face upvotes 22
TL;DR - GPT-Policy is a framework that uses vision-language model agents to learn robot tasks from deployment-time context without gradient updates. Real-robot experiments suggest that human videos improve task completion, while aligned action references provide additional gains for contact-sensitive tasks.
- A context compiler preserves task-relevant visual transitions from demonstrations, examples, and interaction feedback.
- A VLM proposes robot-tool actions, while a constrained controller verifies and executes them and reports outcomes.
- The study evaluates task success, efficiency, model differences, and the effects of removing context components.
- Human demonstrations can help even without robot action labels, indicating a path toward more adaptable general-purpose robotic policies.