When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning

TL;DR - Fine-tuning MedSAM improves in-domain and nearby-domain segmentation but can hurt performance under large distribution shifts. The study links these failures to decoder representation drift and identifies adaptation and prompt-training strategies that better preserve robustness.

• Full fine-tuning offered the best overall tradeoff across in-distribution, close-OOD, and far-OOD datasets.
• Encoder-only LoRA was the strongest parameter-efficient method, outperforming standard LoRA and visual prompt tuning on far-OOD data.
• Centered Kernel Alignment analysis associated far-OOD degradation with decoder drift; encoder similarity alone did not explain robustness.
• Training with random 0–100-pixel prompt jitter improved performance and resilience to noisy prompts.