Meta新研究：字节模型蒸馏后，天花板破了
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - Meta FAIR and University of Washington researchers propose distilling token-level teacher distributions into byte-level student models. Their scaling-law projections suggest an End-Of-Token byte model could eventually exceed conventional token distillation by 4 percentage points in average downstream accuracy while reducing data and storage needs.
- End-Of-Token adds an explicit marker after each token, preserving token-boundary probability mass when converting teacher predictions into byte-level targets.
- With Llama 3-8B as teacher, projected asymptotic accuracy was 52.4% for End-Of-Token, versus 50.5% for Marginalize-It and 48.4% for token distillation.
- Byte-level students have roughly 256 basic output choices rather than Llama 3’s 128,256-token vocabulary, enabling storage of complete teacher distributions instead of top-k truncation.
- The projected gains require more compute: End-Of-Token adds about 30.94% training cost over ordinary byte modeling for the same text, and equal-inference-cost comparisons remain unfinished.
Sources (1)
Meta新研究：字节模型蒸馏后，天花板破了
TL;DR - Meta FAIR and University of Washington researchers propose distilling token-level teacher distributions into byte-level student models. Their scaling-law projections suggest an End-Of-Token byte model could eventually exceed conventional token distillation by 4 percentage points in average downstream accuracy while reducing data and storage needs.
- End-Of-Token adds an explicit marker after each token, preserving token-boundary probability mass when converting teacher predictions into byte-level targets.
- With Llama 3-8B as teacher, projected asymptotic accuracy was 52.4% for End-Of-Token, versus 50.5% for Marginalize-It and 48.4% for token distillation.
- Byte-level students have roughly 256 basic output choices rather than Llama 3’s 128,256-token vocabulary, enabling storage of complete teacher distributions instead of top-k truncation.
- The projected gains require more compute: End-Of-Token adds about 30.94% training cost over ordinary byte modeling for the same text, and equal-inference-cost comparisons remain unfinished.