Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model
Ranking
Overall
77
Content
95
Popularity
34
Observed public metrics from 1 member.
Merged summary
TL;DR - Mixture of Training (MoT) splits Transformer pretraining into independent layer-block runs within a frozen scaffold, then recombines them into a larger model. A 1.3B-parameter experiment demonstrates feasibility and baseline perplexity parity, though practical compute gains depend on scaffold reuse.
- Trains contiguous layer blocks independently using a frozen pretrained aligner scaffold.
- Reassembled blocks form a usable model, optionally followed by brief end-to-end adaptation.
- The parity schedule matches the reported perplexity of monolithic training but processes more aggregate tokens.
- MoT is presented as a small-scale framework for reusable training units, not a general replacement for monolithic pretraining.
Sources (1)
Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - Mixture of Training (MoT) splits Transformer pretraining into independent layer-block runs within a frozen scaffold, then recombines them into a larger model. A 1.3B-parameter experiment demonstrates feasibility and baseline perplexity parity, though practical compute gains depend on scaffold reuse.
- Trains contiguous layer blocks independently using a frozen pretrained aligner scaffold.
- Reassembled blocks form a usable model, optionally followed by brief end-to-end adaptation.
- The parity schedule matches the reported perplexity of monolithic training but processes more aggregate tokens.
- MoT is presented as a small-scale framework for reusable training units, not a general replacement for monolithic pretraining.