Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

TL;DR - SiPE is a method that learns a lightweight syntactic prior from dependency parses during pretraining and injects it into Transformer positional embeddings, improving both syntactic generalization and general language understanding without touching self-attention. It matters because prior syntax-injection methods typically degrade perplexity, while SiPE improves it.

• Works across all three dominant PE families (absolute, relative, rotary) for both encoders and decoders, leaving self-attention and the rest of the architecture unchanged.
• Optimal injection point is architecture-dependent: autoregressive decoders with relative PE benefit most from multiplicative coupling with the relative-position term of the attention score, while encoders do best with the prior added directly to input embeddings.
• Reported gains: up to +10.3% on SyntaxGym, −9.0% perplexity vs. a base model without syntactic supervision, and up to +8.2% on GLUE.
• Conditions on a single parse at inference rather than marginalizing over many parses or discarding syntax at runtime, claiming a new Pareto frontier between syntactic supervision and inference cost.