Log-Depth Recurrent Language Modeling
Ranking
Overall
75
Content
95
Popularity
27
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper introduces an autoregressive language-model architecture based on balanced-tree recursive operators, achieving logarithmic computational depth and linear runtime. Early experiments show robust length extrapolation and performance approaching ALiBi-based Transformers.
- Computes representations for all prefixes in logarithmic depth, enabling more parallelism than conventional recurrent models.
- Requires linear runtime in the number of input tokens, avoiding Transformers’ quadratic scaling.
- Extends balanced-tree sequence-encoding operators to autoregressive prediction.
- Offers a potential alternative to Transformers, though the experiments provide only an initial characterization.
Sources (1)
Log-Depth Recurrent Language Modeling
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper introduces an autoregressive language-model architecture based on balanced-tree recursive operators, achieving logarithmic computational depth and linear runtime. Early experiments show robust length extrapolation and performance approaching ALiBi-based Transformers.
- Computes representations for all prefixes in logarithmic depth, enabling more parallelism than conventional recurrent models.
- Requires linear runtime in the number of input tokens, avoiding Transformers’ quadratic scaling.
- Extends balanced-tree sequence-encoding operators to autoregressive prediction.
- Offers a potential alternative to Transformers, though the experiments provide only an initial characterization.