Can Video World Models Track Unobserved World States?
Ranking
Overall
88
Content
100
Popularity
59
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper tests whether video world models can track hidden world states using an action-conditioned Shell Game. Several architectures produce plausible videos but fail beyond the five-swap training horizon, while mechanisms with mutable internal state extrapolate successfully.
- Bidirectional and autoregressive Transformers, Mamba, and constrained linear attention fall toward chance on longer swap sequences despite maintaining visual quality.
- Extra denoising steps do not improve tracking because pixel-based diffusion targets provide no supervision for unobserved state.
- Standard Transformers must repeatedly reconstruct hidden arrangements from history because their append-only KV caches cannot revise state in place.
- Linear attention with negative transition eigenvalues and nonlinear fast-weight TTT succeed by carrying and updating state across chunks.
Sources (1)
Can Video World Models Track Unobserved World States?
Public signals
Hugging Face upvotes 1
TL;DR - This paper tests whether video world models can track hidden world states using an action-conditioned Shell Game. Several architectures produce plausible videos but fail beyond the five-swap training horizon, while mechanisms with mutable internal state extrapolate successfully.
- Bidirectional and autoregressive Transformers, Mamba, and constrained linear attention fall toward chance on longer swap sequences despite maintaining visual quality.
- Extra denoising steps do not improve tracking because pixel-based diffusion targets provide no supervision for unobserved state.
- Standard Transformers must repeatedly reconstruct hidden arrangements from history because their append-only KV caches cannot revise state in place.
- Linear attention with negative transition eigenvalues and nonlinear fast-weight TTT succeed by carrying and updating state across chunks.