Salient Knowledge Pathways: Sparse Cross-Modal Routing for Efficient Knowledge-Intensive Multimodal Question Answering

TL;DR - SKIP is a sparse inference architecture for knowledge-intensive multimodal question answering that dynamically limits visual processing, retrieval, and cross-modal fusion. It matches or surpasses dense baselines while using 3.4–6.8× fewer FLOPs and 2.7× less latency.

• Combines question-guided visual token pruning, region-conditional retrieval, and sparse cross-attention.
• Adapts compute budgets to predicted question difficulty and speculatively verifies retrieved knowledge.
• Derives an information-bottleneck bound suggesting optimal visual sparsity scales as (O(1/\sqrt{N})).
• Evaluated across five benchmarks, including OK-VQA, InfoSeek, and Encyclopedic-VQA.