UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists

TL;DR - UpgradeBench evaluates how organizations should migrate fine-tuned LLM specialists across successive base-model releases. Its decision policy nearly matches full retraining while avoiding behavioral regressions and using roughly one-third of the compute and labeling cost.

• The benchmark covers four consecutive Qwen releases, six tasks, two model sizes, and OLMo checkpoints with known training lineage.
• Adapter portability decays with continued-pretraining distance: OLMo retention fell from 0.88–0.99 after 46B additional tokens to zero after 2.9T tokens.
• Teacher relabeling can recover specialists on a new base model without fresh gold annotations, although it does not guarantee compute savings.
• A lightweight CKA probe using 256 prompts predicts adapter portability across versions with Spearman correlation 0.74 over eight model pairs.