When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning
Ranking
Overall
83
Content
90
Popularity
65
Observed public metrics from 1 member.
Merged summary
TL;DR - Fine-tuning MedSAM improves in-domain and nearby-domain segmentation but can hurt performance under large distribution shifts. The study links these failures to decoder representation drift and identifies adaptation and prompt-training strategies that better preserve robustness.
- Full fine-tuning offered the best overall tradeoff across in-distribution, close-OOD, and far-OOD datasets.
- Encoder-only LoRA was the strongest parameter-efficient method, outperforming standard LoRA and visual prompt tuning on far-OOD data.
- Centered Kernel Alignment analysis associated far-OOD degradation with decoder drift; encoder similarity alone did not explain robustness.
- Training with random 0–100-pixel prompt jitter improved performance and resilience to noisy prompts.
Sources (1)
When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning
Public signals
Semantic Scholar citations 1 · Semantic Scholar influential citations 1
TL;DR - Fine-tuning MedSAM improves in-domain and nearby-domain segmentation but can hurt performance under large distribution shifts. The study links these failures to decoder representation drift and identifies adaptation and prompt-training strategies that better preserve robustness.
- Full fine-tuning offered the best overall tradeoff across in-distribution, close-OOD, and far-OOD datasets.
- Encoder-only LoRA was the strongest parameter-efficient method, outperforming standard LoRA and visual prompt tuning on far-OOD data.
- Centered Kernel Alignment analysis associated far-OOD degradation with decoder drift; encoder similarity alone did not explain robustness.
- Training with random 0–100-pixel prompt jitter improved performance and resilience to noisy prompts.