UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
Ranking
Overall
81
Content
100
Popularity
37
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper presents an FP4 language-model pretraining recipe that pairs E2M1 values with wide-range UE5M3 block scales, simplifying quantization while improving reported loss and benchmark aggregates. It demonstrates the approach on an 8B Nemotron-H model trained for nearly 190 billion tokens.
- UE5M3 block scales enable periodic tensor scaling without randomized Hadamard transforms.
- The recipe selectively applies stochastic rounding to backward gradients and uses FP4 for all eligible internal linear layers.
- A block-16 configuration achieved lower final-window training loss and quantized-inference validation loss than NVIDIA Transformer Engine’s NVFP4 recipe.
- Removing RHT and the BF16 final-block exemption increased measured model-body token throughput by 21.2% in a native NVFP4 execution ablation.
Sources (1)
UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper presents an FP4 language-model pretraining recipe that pairs E2M1 values with wide-range UE5M3 block scales, simplifying quantization while improving reported loss and benchmark aggregates. It demonstrates the approach on an 8B Nemotron-H model trained for nearly 190 billion tokens.
- UE5M3 block scales enable periodic tensor scaling without randomized Hadamard transforms.
- The recipe selectively applies stochastic rounding to backward gradients and uses FP4 for all eligible internal linear layers.
- A block-16 configuration achieved lower final-window training loss and quantized-inference validation loss than NVIDIA Transformer Engine’s NVFP4 recipe.
- Removing RHT and the BF16 final-block exemption increased measured model-body token throughput by 21.2% in a native NVFP4 execution ablation.