LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training

TL;DR - LazyTrain optimizes checkpointing, activation placement, recomputation, and data transfers for LLM training on memory-constrained hardware. It improves throughput and enables larger batches by reducing exposed communication and optimizer-state overhead.

• Uses mixed-integer scheduling to coordinate CPU, GPU, and NVMe resources.
• Combines 8-bit optimizer states with fast gradient clipping in a Hybrid 8-bit operator.
• Improves sustained TFLOPS by approximately 1.24× over matched H800 baselines.
• Trains Qwen3.6-27B at 1,361 tokens/s with batch size 72 and 68.84 GB peak GPU memory.