‹ back to 2026-07-16

In-Place Tokenizer Expansion for Pre-trained LLMs

arXiv cs.CL Efficiency & Systems Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak, Aditya Tadimeti, Tim Seyde, Maxime Labonne, Alexander Amini, Mathias Lechner 2026-07-16

TL;DR - A method for expanding a pre-trained LLM's tokenizer in place so later-added languages tokenize more efficiently, cutting on-device decode latency without retraining from scratch. It matters because compact/on-device models ship small vocabularies that fragment non-core languages, raising cost and latency for those users.

Continues the existing BPE merges on a multilingual corpus so most source tokens carry over unchanged and every new token decomposes exactly into source tokens; new embedding rows are initialized as the mean of their source sub-token embeddings.
A two-stage adaptation (embedding-only training, then full-model continued pre-training) recovers source-checkpoint quality.
Applied to a continued-pretrained LFM2-8B-A1B (8B MoE) to produce LFM2.5-8B-A1B with a 128K tokenizer; encodes Hindi/Vietnamese in ~2.4×/2.6× fewer tokens (up to 4.0× on Thai).
Estimated 2.2–3.7× per-character decode speedup on reference devices; model weights and expanded tokenizer released, along with reported negative findings.

view merged work →