Salient Knowledge Pathways: Sparse Cross-Modal Routing for Efficient Knowledge-Intensive Multimodal Question Answering
Ranking
Overall
77
Content
90
Popularity
45
Observed public metrics from 1 member.
Merged summary
TL;DR - SKIP is a sparse inference architecture for knowledge-intensive multimodal question answering that dynamically limits visual processing, retrieval, and cross-modal fusion. It matches or surpasses dense baselines while using 3.4–6.8× fewer FLOPs and 2.7× less latency.
- Combines question-guided visual token pruning, region-conditional retrieval, and sparse cross-attention.
- Adapts compute budgets to predicted question difficulty and speculatively verifies retrieved knowledge.
- Derives an information-bottleneck bound suggesting optimal visual sparsity scales as (O(1/\sqrt{N})).
- Evaluated across five benchmarks, including OK-VQA, InfoSeek, and Encyclopedic-VQA.
Sources (1)
Salient Knowledge Pathways: Sparse Cross-Modal Routing for Efficient Knowledge-Intensive Multimodal Question Answering
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - SKIP is a sparse inference architecture for knowledge-intensive multimodal question answering that dynamically limits visual processing, retrieval, and cross-modal fusion. It matches or surpasses dense baselines while using 3.4–6.8× fewer FLOPs and 2.7× less latency.
- Combines question-guided visual token pruning, region-conditional retrieval, and sparse cross-attention.
- Adapts compute budgets to predicted question difficulty and speculatively verifies retrieved knowledge.
- Derives an information-bottleneck bound suggesting optimal visual sparsity scales as (O(1/\sqrt{N})).
- Evaluated across five benchmarks, including OK-VQA, InfoSeek, and Encyclopedic-VQA.