‹ back to 2026-08-05

SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

arXiv cs.CL LLMs & Foundation Models Kejian Zhu, Zhuoran Jin, Shangqing Tu, Hongbang Yuan, Yushi Bai, Kang Liu, Juanzi Li, Jun Zhao 2026-08-04

TL;DR - This paper argues that reinforcement learning handles multi-task LLM reasoning more effectively than supervised fine-tuning by producing sparse, nearly orthogonal task updates. It introduces Parallel-RL to improve training efficiency and flexibility.

Multi-stage SFT exhibits severe gradient conflicts across diverse tasks, while RL supports stable coexistence.
SFT interference scales with absolute gradient magnitude; RL interference is bounded by gradient variance.
Advantage normalization and on-policy optimization help keep RL gradient variance small and task directions near-orthogonal.
Parallel-RL decouples multi-task training based on these findings.

view merged work →