Marionette: Predicting World States, Rendering Geometry, Painting Appearance

TL;DR - Marionette is an interactive game world model that predicts explicit 3D character states, renders geometric controls, and uses video diffusion for photorealistic appearance. This separation enables direct control and rule-based correction of long-horizon errors without retraining the observation model.

• Predicts a 276-dimensional state containing articulated skeletons, metric root trajectories, and rotations.
• A zero-parameter renderer computes geometry and occlusion before control-conditioned video synthesis.
• State-level terrain and separation rules reduced ground penetration by 66% and prevented character drift.
• Structured state routing showed no detected fidelity loss, scoring 831 FVD versus 799 with recorded poses.