Virtual Encoders in Multimodal Transformers

TL;DR - Multimodal transformers without dedicated perceptual encoders can develop “Virtual Encoders,” performing encoder-like processing within their early-to-middle layers. This suggests perception and language processing may emerge as functional regimes rather than align with separate architectural modules.

• The study examines models receiving lightly projected patches, audio frames, or discrete visual tokens instead of continuous encoder-derived features.
• Linear probing and representation-similarity analyses identify internal states resembling task-usable perceptual representations.
• Causal analyses indicate that the shared transformer internalizes perceptual encoding before downstream language processing.
• The findings challenge the assumption that architectural module boundaries must define the boundary between perception and language.