Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization

TL;DR - Byte-Prefix Marginalization enables on-policy distillation between language models with different tokenizers by mapping teacher probabilities into a shared byte space. It preserves probability mass and improves mathematical and programming performance over existing cross-tokenizer methods.

• Maps each teacher token’s probability to the longest matching student-token byte prefix.
• Aggregates shared mappings and retains unmatched probability in an explicit residual category.
• Exactly recovers teacher byte-prefix marginals at over 99% of training positions; otherwise, it uses a mass-preserving lower bound.
• Improves six-benchmark avg@8 by 3.7–6.6 points over the strongest baselines across three teacher-model families.