Objective vs. Search: Decomposing What Makes a Good Tokeniser

TL;DR - This paper disentangles tokeniser optimization objectives from search procedures using a controlled 2×2 comparison. Bottom-up search generally produces language models with better compression efficiency, while neither design choice consistently improves grammatical performance.

• Introduces BottomUpLL and TopDownComp to complement BPE and UnigramLM across objective and search dimensions.
• Tests tokenisers across model sizes, vocabulary sizes, and English-only versus multilingual domains.
• Bottom-up tokenisers consistently achieve lower bits-per-byte in most settings, indicating search procedure matters more than objective.
• BLiMP results show no consistent relationship between tokeniser design and grammatical performance.