Meta新研究：字节模型蒸馏后，天花板破了

TL;DR - Meta FAIR and University of Washington researchers propose distilling token-level teacher distributions into byte-level student models. Their scaling-law projections suggest an End-Of-Token byte model could eventually exceed conventional token distillation by 4 percentage points in average downstream accuracy while reducing data and storage needs.

• End-Of-Token adds an explicit marker after each token, preserving token-boundary probability mass when converting teacher predictions into byte-level targets.
• With Llama 3-8B as teacher, projected asymptotic accuracy was 52.4% for End-Of-Token, versus 50.5% for Marginalize-It and 48.4% for token distillation.
• Byte-level students have roughly 256 basic output choices rather than Llama 3’s 128,256-token vocabulary, enabling storage of complete teacher distributions instead of top-k truncation.
• The projected gains require more compute: End-Of-Token adds about 30.94% training cost over ordinary byte modeling for the same text, and equal-inference-cost comparisons remain unfinished.