🛰️ Daily AI Frontier
‹ back to 2026-09-09

Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

Research LLMs & Foundation Models

Ranking

Overall 78
Content 95
Popularity 37

Observed public metrics from 1 member.

Merged summary

TL;DR - A 30B mixture-of-experts study finds that checkpoints with better pretraining loss or benchmark scores do not necessarily produce the best models after supervised fine-tuning and the full downstream training stack. Robustness to local weight perturbations—termed higher solution density—better characterizes checkpoints that retain strong downstream performance.

  • Checkpoint rankings can change across different stages of model training.
  • Strong pretraining metrics alone may be insufficient for selecting downstream initialization checkpoints.
  • Better final checkpoints maintain downstream performance under local perturbations to their weights.
  • Solution density may offer an additional signal for checkpoint selection.

Sources (1)

Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

arXiv cs.AI Sohir Maskey, Philipp Scholl, Jonas Knupp, Pit Neitemeier, Sascha Wirges 2026-09-08 arXiv:2609.08966
Public signals Semantic Scholar citations 0 · Semantic Scholar influential citations 0
Providers: Hugging Face · N/A OpenAlex · N/A Publisher · N/A Semantic Scholar · Citations 0 · Influential citations 0 X · N/A Fetched 2026-09-21 14:23:42.134152 UTC

TL;DR - A 30B mixture-of-experts study finds that checkpoints with better pretraining loss or benchmark scores do not necessarily produce the best models after supervised fine-tuning and the full downstream training stack. Robustness to local weight perturbations—termed higher solution density—better characterizes checkpoints that retain strong downstream performance.

  • Checkpoint rankings can change across different stages of model training.
  • Strong pretraining metrics alone may be insufficient for selecting downstream initialization checkpoints.
  • Better final checkpoints maintain downstream performance under local perturbations to their weights.
  • Solution density may offer an additional signal for checkpoint selection.
item →