PRiSM: Prototype Regularization for Few-Shot VLMs

TL;DR - PRiSM is a plug-and-play prototype regularization method for training-free few-shot adaptation of vision-language models. It improves robustness when labeled examples are class-imbalanced or span many classes.

• Introduces a Dirichlet-sampled benchmark varying class balance and effective class count.
• Finds substantial degradation in existing methods under realistic imbalance, sometimes worsening with more labeled samples.
• Optimizes prototypes using inter-class separation, support-feature alignment, and baseline-fidelity terms.
• Uses an efficient block Majorize-Minimize optimizer with Lipschitz bounds derived via the Gershgorin circle theorem.