Towards Surgical World-Action Modeling: A Preliminary Joint Visual-Trajectory Forecasting for Surgical Motion Planning

TL;DR - This paper introduces a preliminary surgical world-action model that jointly predicts future operative video frames and instrument trajectories. Joint forecasting could improve motion planning by capturing both tool movement and its visual consequences.

• Historical video frames and tool trajectories are encoded jointly, processed by a temporal-spatial encoder, and decoded through separate visual and trajectory heads.
• A chunked autoregressive rollout predicts 15 future steps and consistently outperforms direct one-shot prediction across evaluated horizons.
• For the first segment, chunking raises PSNR from 18.86 to 23.11 dB and reduces average displacement error from 45.77 to 22.22 pixels.
• Longer forecasts still suffer from progressive visual degradation and accumulated trajectory errors.