SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training

TL;DR - SCOUT is a runtime failure-localization framework for LLM pre-training that pinpoints the originating rank behind job-wide stalls, stragglers, and numerical errors by finding outliers via strict-majority consensus among equivalent replicas. It matters because synchronization masks rank-local faults, and existing in-process monitors, post-mortem logs, and offline health checks all lose the evidence needed to attribute them.

• Core primitive is a Consensus Collective Communication (C3) abstraction: replica progress, timing, and numerical evidence are aligned, and ranks whose compact signatures disagree with peers are flagged as outliers.
• An out-of-band CPU observer stays responsive after the trainer blocks or terminates, addressing the case where in-process monitors go silent during hangs.
• In-situ replay reproduces recurring stragglers and silent data corruption alongside the live job, preserving model state, kernels, allocations, communication path, and thermal/memory pressure; collective fingerprints surface rank-local protocol divergence.
• Clean replay coverage certifies checkpoint numerical integrity so recovery avoids SDC-corrupted state; integrates with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without training-loop or framework-source changes, and is open source.