TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training

TL;DR - TrainSDC is a low-overhead framework for protecting LLM training from silent data corruption. It targets distinct fault-propagation patterns in Transformer forward and backward passes, keeping training close to fault-free behavior with 1.65%–6.76% runtime overhead.

• Forward-pass faults are location-dependent, with Q/K-path corruption causing persistent training deviations.
• Backward-pass vulnerability depends mainly on gradient exponent distributions rather than fault location.
• TrainSDC combines Q/K-path recomputation, residual-gain monitoring, and exponent-aware gradient scaling.
• Tests on Llama 3.2-1B and Qwen3-0.6B show resilience under sparse and dense fault injection.