Matryoshka Language Model Suites

TL;DR - An arXiv preprint proposing "Matryoshka" language model suites, where progressively larger sub-models are nested inside a single architecture trained end-to-end, so one training run and one set of weights yields an entire model family. It matters because it cuts suite-level training compute and parameter count while making speculative decoding structurally natural — the draft model literally lives inside the verifier.

• Nested sub-models are trained jointly end-to-end, reducing the suite's total parameter count versus training and serving each size independently.
• The shared architecture enables low-cost distillation from the largest sub-model to all smaller ones at every training step.
• Validated on a 500M / 1.5B / 3B suite: on par with independently trained baselines on benchmarks plus validation and out-of-domain perplexity, using 36% less training compute.
• Speculative decoding throughput improves 14–26% since the draft is contained in the verifier; the paper also ablates key architectural choices as design guidance.