李飞飞第一篇「触觉」论文：机器人会盲摸麻将了
Merged summary
TL;DR - T-Rex is a vision-language-action model that gives robotic hands a separate high-frequency tactile pathway, achieving 65% average success across 12 real-world dexterous tasks versus 35% for the strongest vision-only baseline. It shows tactile feedback is essential for precise, contact-rich manipulation.
- A Mixture-of-Transformer-Experts architecture separates visual-language understanding, low-frequency action planning, and high-frequency tactile refinement.
- Cascaded Flow Matching lets the action expert establish a trajectory before the tactile expert adjusts it using real-time force feedback.
- A VQ-VAE compresses 16-frame force histories into discrete codes representing temporal contact patterns such as pressing, sliding, and insertion.
- Three-stage training combines 22,889 hours of human video, 100 hours of teleoperation across 207 objects and 22 action primitives, and roughly 100 task-specific demonstrations.
Sources (1)
李飞飞第一篇「触觉」论文：机器人会盲摸麻将了
TL;DR - T-Rex is a vision-language-action model that gives robotic hands a separate high-frequency tactile pathway, achieving 65% average success across 12 real-world dexterous tasks versus 35% for the strongest vision-only baseline. It shows tactile feedback is essential for precise, contact-rich manipulation.
- A Mixture-of-Transformer-Experts architecture separates visual-language understanding, low-frequency action planning, and high-frequency tactile refinement.
- Cascaded Flow Matching lets the action expert establish a trajectory before the tactile expert adjusts it using real-time force feedback.
- A VQ-VAE compresses 16-frame force histories into discrete codes representing temporal contact patterns such as pressing, sliding, and insertion.
- Three-stage training combines 22,889 hours of human video, 100 hours of teleoperation across 207 objects and 22 action primitives, and roughly 100 task-specific demonstrations.