Log-Depth Recurrent Language Modeling

TL;DR - This paper introduces an autoregressive language-model architecture based on balanced-tree recursive operators, achieving logarithmic computational depth and linear runtime. Early experiments show robust length extrapolation and performance approaching ALiBi-based Transformers.

• Computes representations for all prefixes in logarithmic depth, enabling more parallelism than conventional recurrent models.
• Requires linear runtime in the number of input tokens, avoiding Transformers’ quadratic scaling.
• Extends balanced-tree sequence-encoding operators to autoregressive prediction.
• Offers a potential alternative to Transformers, though the experiments provide only an initial characterization.