Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content
Merged summary
TL;DR - This study compares token, byte, and pixel text encodings under controlled linguistic content and downstream capacity. It finds no universally superior encoding; effectiveness depends on the task, languages, capacity, and compute budget.
- Tests verified parallel sentences across 13 languages and five scripts using a shared, variable-width bottleneck.
- Pixels best preserve surface form, while bytes best retain cross-lingual alignment, particularly for same-script languages.
- Tokens perform best for topic classification.
- Sequence length alone does not explain utility: longer representations may preserve highly compressible information, while shorter ones can lose meaning.
Sources (1)
Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content
TL;DR - This study compares token, byte, and pixel text encodings under controlled linguistic content and downstream capacity. It finds no universally superior encoding; effectiveness depends on the task, languages, capacity, and compute budget.
- Tests verified parallel sentences across 13 languages and five scripts using a shared, variable-width bottleneck.
- Pixels best preserve surface form, while bytes best retain cross-lingual alignment, particularly for same-script languages.
- Tokens perform best for topic classification.
- Sequence length alone does not explain utility: longer representations may preserve highly compressible information, while shorter ones can lose meaning.