Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

TL;DR - This paper studies image tokenizers as the “visual language” of unified autoregressive multimodal models using task-specific training losses. It finds that tokenizer quality cannot be judged by reconstruction alone and that image-to-text loss is a comparatively consistent predictor of downstream generation and visual-understanding performance.

• Text, image, text-to-image, and image-to-text losses scale differently and can rank tokenizers differently.
• Across tokenizers, text-to-image loss is confounded by differing image-token spaces, while image-to-text loss uses a shared text vocabulary and is more comparable.
• Better image reconstruction does not necessarily produce lower task losses or stronger downstream results.
• Tokenizer design—including discriminator use, semantic supervision, and vocabulary size—affects joint image-text learning and can influence text modeling.