TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training
Ranking
Overall
81
Content
100
Popularity
37
Observed public metrics from 1 member.
Merged summary
TL;DR - TrainSDC is a low-overhead framework for protecting LLM training from silent data corruption. It targets distinct fault-propagation patterns in Transformer forward and backward passes, keeping training close to fault-free behavior with 1.65%–6.76% runtime overhead.
- Forward-pass faults are location-dependent, with Q/K-path corruption causing persistent training deviations.
- Backward-pass vulnerability depends mainly on gradient exponent distributions rather than fault location.
- TrainSDC combines Q/K-path recomputation, residual-gain monitoring, and exponent-aware gradient scaling.
- Tests on Llama 3.2-1B and Qwen3-0.6B show resilience under sparse and dense fault injection.
Sources (1)
TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - TrainSDC is a low-overhead framework for protecting LLM training from silent data corruption. It targets distinct fault-propagation patterns in Transformer forward and backward passes, keeping training close to fault-free behavior with 1.65%–6.76% runtime overhead.
- Forward-pass faults are location-dependent, with Q/K-path corruption causing persistent training deviations.
- Backward-pass vulnerability depends mainly on gradient exponent distributions rather than fault location.
- TrainSDC combines Q/K-path recomputation, residual-gain monitoring, and exponent-aware gradient scaling.
- Tests on Llama 3.2-1B and Qwen3-0.6B show resilience under sparse and dense fault injection.