UE5M3 FP4 Block Scaling for Stable Language Model Pretraining

TL;DR - This paper presents an FP4 language-model pretraining recipe that pairs E2M1 values with wide-range UE5M3 block scales, simplifying quantization while improving reported loss and benchmark aggregates. It demonstrates the approach on an 8B Nemotron-H model trained for nearly 190 billion tokens.

• UE5M3 block scales enable periodic tensor scaling without randomized Hadamard transforms.
• The recipe selectively applies stochastic rounding to backward gradients and uses FP4 for all eligible internal linear layers.
• A block-16 configuration achieved lower final-window training loss and quantized-inference validation loss than NVIDIA Transformer Engine’s NVFP4 recipe.
• Removing RHT and the BF16 final-block exemption increased measured model-body token throughput by 21.2% in a native NVFP4 execution ablation.