Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Merged summary
TL;DR - Byte-Prefix Marginalization enables on-policy distillation between language models with different tokenizers by mapping teacher probabilities into a shared byte space. It preserves probability mass and improves mathematical and programming performance over existing cross-tokenizer methods.
- Maps each teacher token’s probability to the longest matching student-token byte prefix.
- Aggregates shared mappings and retains unmatched probability in an explicit residual category.
- Exactly recovers teacher byte-prefix marginals at over 99% of training positions; otherwise, it uses a mass-preserving lower bound.
- Improves six-benchmark avg@8 by 3.7–6.6 points over the strongest baselines across three teacher-model families.
Sources (1)
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
TL;DR - Byte-Prefix Marginalization enables on-policy distillation between language models with different tokenizers by mapping teacher probabilities into a shared byte space. It preserves probability mass and improves mathematical and programming performance over existing cross-tokenizer methods.
- Maps each teacher token’s probability to the longest matching student-token byte prefix.
- Aggregates shared mappings and retains unmatched probability in an explicit residual category.
- Exactly recovers teacher byte-prefix marginals at over 99% of training positions; otherwise, it uses a mass-preserving lower bound.
- Improves six-benchmark avg@8 by 3.7–6.6 points over the strongest baselines across three teacher-model families.