Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation

TL;DR - Prompted Information Bottlenecks (PIB) improves parameter-efficient adaptation of frozen vision foundation models by explicitly regulating how task-relevant information is preserved and compressed across layers. It reports strong accuracy and robustness across 34 datasets while tuning only 0.35% of parameters on average.

• Applies information-bottleneck principles to Visual Prompt Tuning’s layer-wise representations.
• Retains useful local evidence in early layers while progressively suppressing nuisance factors and redundancy.
• Reaches 92.1% on FGVC, 93.01% on HTA, and 77.33% on VTAB-1k.
• Reduces shortcut reliance and improves performance under distribution shift and fine-grained recognition.