‹ back to 2026-08-10

Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLU

arXiv cs.LG LLMs & Foundation Models Yuting Ge, Pengju Yang, Mingkai Nie 2026-08-07

TL;DR - An arXiv preprint testing whether SwiGLU's unbounded positive tail is actually required in decoder-only LLM feed-forward networks, using a closed-tail gating alternative called MemGLU. At small pretraining scales it isn't: MemGLU matches SwiGLU within ~0.1% validation NLL, suggesting activation-function design has more slack than commonly assumed.

MemGLU is introduced as a closed-tail comparator derived from a memristive branch geometry, contrasting with SwiGLU's open (unbounded) positive tail.
Evaluation used paired pretraining runs at 9M and 30M parameters with three seeds; MemGLU stayed within roughly 0.1% of SwiGLU in validation negative log-likelihood.
Trained SwiGLU checkpoints degrade under positive-tail suppression, and mechanism diagnostics show the two models use their gates differently despite comparable loss — implying models adapt to whatever gate geometry is present during pretraining.
Claims are explicitly scoped to "the tested scales" (9M/30M); no evidence is offered for larger models or downstream task performance.

view merged work →