Progressive Multimodal Alignment for Continual Instruction Tuning

TL;DR - Progressive Multimodal Alignment (PMA) addresses projector-level forgetting during continual instruction tuning of multimodal LLMs. It selectively adds projector experts while retaining the pretrained projector as a stable alignment anchor.

• Detects multimodal distribution shifts using a lightweight representation descriptor.
• Expands projector experts only when needed, enabling sub-linear parameter growth.
• Routes multimodal features across experts to balance adaptation with retention.
• Improves prior methods across two MCIT benchmarks and multiple MLLM backbones.