DeltaMerge-LowRes: Composing Language and Task Deltas for Low-Resource Adaptation

TL;DR — DeltaMerge-LowRes trains separate language and task weight deltas independently, then merges them in weight space at inference to adapt a multilingual encoder to new language+task pairs, avoiding expensive joint fine-tuning in low-resource settings.

• Learns a language delta (Δ_L) from unlabeled monolingual text and a task delta (Δ_T) from labeled English data, composing them via four rules: additive, activation-guided, sparsity-aware, and a novel cross-axis TIES.
• Cross-axis TIES adapts TIES-Merging (trimming, sign election, merging) to language vs. task axes; it wins summarization on 3/4 African languages by +4 to +7 chrF (18.59 vs. 13.80 task-only) and improves QA F1 by +2.32, EM by +2.91.
• Sparsity-aware merging cuts classification calibration error (ECE) by 36% at parity macro-F1; evaluation spans 158 cells (4 task families × 4 languages) with 10,000-sample paired bootstrap per cell.
• The choice of composition rule materially affects what the merged model preserves, suppresses, and calibrates; authors release JSON traces and a claim ledger.