别再乱调图像塔了！浙大 IJCAI 论文揭露 VLM 非对称性真相，给 CLIP 微调「踩刹车」

TL;DR - An IJCAI paper proposes an adaptive asymmetric adapter for few-shot CLIP tuning that freely adapts the text encoder while suppressing image-encoder updates on uncertain inputs. This preserves pretrained visual generalization under distribution shift while retaining in-domain adaptation.

• Experiments across 11 datasets find text-side tuning generally more beneficial, while aggressive image-side tuning harms out-of-distribution performance.
• A confidence-driven regularizer pushes image-adapter corrections toward zero for low-confidence samples, effectively reverting to pretrained CLIP visual features.
• The adapter uses a shared down-projection, multiple routed up-projection experts, and load balancing to prevent expert collapse.
• It reportedly outperforms 11 baselines across base-to-novel, cross-dataset, and domain-generalization evaluations, with lightweight ViT and ResNet-50 support.