Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference

TL;DR - A large-scale study finds that properly optimized layer dropout can reduce both LLM training compute and inference costs without sacrificing accuracy. Across more than 2,400 experiments, it saved up to 25% of training FLOPs and enabled up to 1.5× faster inference.

• Effective layer dropout requires optimizing its layer distribution, schedule, and optimizer hyperparameters.
• At equal training FLOPs, layer dropout produced lower loss; at equal training steps, it achieved similar or lower validation loss with less compute.
• Dropout-trained models supported early exit, intermediate-layer skipping, and self-speculative decoding with negligible accuracy loss.
• Results covered 271M–8.2B-parameter models and datasets up to 160B tokens, trained on Cerebras CS-3 systems.