Should This Case Be Adapted? Prediction Fragmentation Controls Test-Time Adaptation

TL;DR - This paper uses prediction fragmentation—the disagreement geometry between original and adapted segmentation masks—to decide whether each medical-imaging case should undergo test-time adaptation. The approach substantially reduces harmful adaptations without requiring labels or additional backward passes at decision time.

• Fixed-step adaptation worsened 58.7% of cases on cross-vendor cardiac MRI despite producing no statistically significant mean Dice change.
• Prediction fragmentation correlates with harmful accepted area across three benchmarks (Spearman ρ 0.50–0.60) at one-quarter the latency of a gradient-norm signal.
• On the selected cardiac benchmark, the router cut harmful accepted area from 0.129 to 0.013 at matched Dice and reduced harmed cases from 58.7% to 20.0%.
• The routing template transferred across segmentation architectures and domains, though on prostate data it reduced harmful edits at the cost of accuracy.