Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model

TL;DR - Mixture of Training (MoT) splits Transformer pretraining into independent layer-block runs within a frozen scaffold, then recombines them into a larger model. A 1.3B-parameter experiment demonstrates feasibility and baseline perplexity parity, though practical compute gains depend on scaffold reuse.

• Trains contiguous layer blocks independently using a frozen pretrained aligner scaffold.
• Reassembled blocks form a usable model, optionally followed by brief end-to-end adaptation.
• The parity schedule matches the reported perplexity of monolithic training but processes more aggregate tokens.
• MoT is presented as a small-scale framework for reusable training units, not a general replacement for monolithic pretraining.