UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists
Ranking
Overall
78
Content
90
Popularity
N/A
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - UpgradeBench evaluates how organizations should migrate fine-tuned LLM specialists across successive base-model releases. Its decision policy nearly matches full retraining while avoiding behavioral regressions and using roughly one-third of the compute and labeling cost.
- The benchmark covers four consecutive Qwen releases, six tasks, two model sizes, and OLMo checkpoints with known training lineage.
- Adapter portability decays with continued-pretraining distance: OLMo retention fell from 0.88–0.99 after 46B additional tokens to zero after 2.9T tokens.
- Teacher relabeling can recover specialists on a new base model without fresh gold annotations, although it does not guarantee compute savings.
- A lightweight CKA probe using 256 prompts predicts adapter portability across versions with Spearman correlation 0.74 over eight model pairs.
Sources (1)
UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists
Public signals
N/A
TL;DR - UpgradeBench evaluates how organizations should migrate fine-tuned LLM specialists across successive base-model releases. Its decision policy nearly matches full retraining while avoiding behavioral regressions and using roughly one-third of the compute and labeling cost.
- The benchmark covers four consecutive Qwen releases, six tasks, two model sizes, and OLMo checkpoints with known training lineage.
- Adapter portability decays with continued-pretraining distance: OLMo retention fell from 0.88–0.99 after 46B additional tokens to zero after 2.9T tokens.
- Teacher relabeling can recover specialists on a new base model without fresh gold annotations, although it does not guarantee compute savings.
- A lightweight CKA probe using 256 prompts predicts adapter portability across versions with Spearman correlation 0.74 over eight model pairs.