VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders

TL;DR — VideoRAE is a representation autoencoder that turns frozen Video Foundation Model features into compact, reconstruction-capable video latents for generative modeling, showing that video-understanding representations can double as generation-friendly latent spaces.

• Instead of pixel-optimized 3D-VAEs, it reuses multi-scale hierarchical features from a frozen VFM encoder (e.g., V-JEPA 2, VideoMAEv2), compressed via a lightweight 1D self-attention projector.
• Supports both continuous latents for Diffusion Transformers and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization.
• A local-and-global alignment objective with the frozen VFM teacher preserves semantics and removes the need for KL regularization.
• Reports state-of-the-art UCF-101 class-to-video gFVDs of 40 (AR) and 93 (DiT), ~5x faster convergence than baselines, and faster convergence than LTX-VAE in a 2B-scale text-to-video study.