Can Video World Models Track Unobserved World States?

TL;DR - This paper tests whether video world models can track hidden world states using an action-conditioned Shell Game. Several architectures produce plausible videos but fail beyond the five-swap training horizon, while mechanisms with mutable internal state extrapolate successfully.

• Bidirectional and autoregressive Transformers, Mamba, and constrained linear attention fall toward chance on longer swap sequences despite maintaining visual quality.
• Extra denoising steps do not improve tracking because pixel-based diffusion targets provide no supervision for unobserved state.
• Standard Transformers must repeatedly reconstruct hidden arrangements from history because their append-only KV caches cannot revise state in place.
• Linear attention with negative transition eigenvalues and nonlinear fast-weight TTT succeed by carrying and updating state across chunks.