In-Place Tokenizer Expansion for Pre-trained LLMs
Merged summary
TL;DR - A method for expanding a pre-trained LLM's tokenizer in place so later-added languages tokenize more efficiently, cutting on-device decode latency without retraining from scratch. It matters because compact/on-device models ship small vocabularies that fragment non-core languages, raising cost and latency for those users.
- Continues the existing BPE merges on a multilingual corpus so most source tokens carry over unchanged and every new token decomposes exactly into source tokens; new embedding rows are initialized as the mean of their source sub-token embeddings.
- A two-stage adaptation (embedding-only training, then full-model continued pre-training) recovers source-checkpoint quality.
- Applied to a continued-pretrained LFM2-8B-A1B (8B MoE) to produce LFM2.5-8B-A1B with a 128K tokenizer; encodes Hindi/Vietnamese in ~2.4×/2.6× fewer tokens (up to 4.0× on Thai).
- Estimated 2.2–3.7× per-character decode speedup on reference devices; model weights and expanded tokenizer released, along with reported negative findings.
Sources (1)
In-Place Tokenizer Expansion for Pre-trained LLMs
TL;DR - A method for expanding a pre-trained LLM's tokenizer in place so later-added languages tokenize more efficiently, cutting on-device decode latency without retraining from scratch. It matters because compact/on-device models ship small vocabularies that fragment non-core languages, raising cost and latency for those users.
- Continues the existing BPE merges on a multilingual corpus so most source tokens carry over unchanged and every new token decomposes exactly into source tokens; new embedding rows are initialized as the mean of their source sub-token embeddings.
- A two-stage adaptation (embedding-only training, then full-model continued pre-training) recovers source-checkpoint quality.
- Applied to a continued-pretrained LFM2-8B-A1B (8B MoE) to produce LFM2.5-8B-A1B with a 128K tokenizer; encodes Hindi/Vietnamese in ~2.4×/2.6× fewer tokens (up to 4.0× on Thai).
- Estimated 2.2–3.7× per-character decode speedup on reference devices; model weights and expanded tokenizer released, along with reported negative findings.