Language Identification via Compositional Data Analysis: A Linear-Time Classifier Based on Log-Ratio Geometry
Merged summary
TL;DR - A method for language identification that treats character/bigram frequencies as compositional data, using the centered log-ratio (CLR) transformation for a deterministic, linear-time classifier. It matters as an interpretable, low-resource alternative to neural language ID models.
- Models unigram and bigram frequency distributions as compositional vectors on the simplex, mapped bijectively via CLR onto the zero-sum subspace where Euclidean distances equal Aitchison distances.
- Combines CLR-transformed unigram and bigram features with Laplace smoothing to handle sparsity; evaluated on six languages.
- Reports robust accuracy across text lengths, with stronger performance on longer sequences.
- Positioned as deterministic and computationally efficient, favoring interpretability and low resource consumption over neural approaches.
Sources (1)
Language Identification via Compositional Data Analysis: A Linear-Time Classifier Based on Log-Ratio Geometry
TL;DR - A method for language identification that treats character/bigram frequencies as compositional data, using the centered log-ratio (CLR) transformation for a deterministic, linear-time classifier. It matters as an interpretable, low-resource alternative to neural language ID models.
- Models unigram and bigram frequency distributions as compositional vectors on the simplex, mapped bijectively via CLR onto the zero-sum subspace where Euclidean distances equal Aitchison distances.
- Combines CLR-transformed unigram and bigram features with Laplace smoothing to handle sparsity; evaluated on six languages.
- Reports robust accuracy across text lengths, with stronger performance on longer sequences.
- Positioned as deterministic and computationally efficient, favoring interpretability and low resource consumption over neural approaches.