QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides

TL;DR - QUADS is a method that makes the NVFP4 4-bit format usable for RL rollout in Mixture-of-Experts LLMs, avoiding the training collapse that naive NVFP4 suffers while beating FP8 on throughput. It matters because rollout generation is a key RL bottleneck and NVFP4 promises higher throughput than FP8.

• Diagnoses FP4 RL instability: naive NVFP4 rollout with BF16 training collapses after ~150 steps with growing rollout-trainer log-prob gaps; error analysis pins the cause on activation error (amplified by the coarse E2M1 grid) rather than weight error.
• Trainer-side fix — Asymmetric Quantization-Aware Training: fake-quantizes weights but keeps activations unquantized for better rollout-trainer alignment.
• Rollout-side fix — Residual Activation Compensation: corrects high-error activation channels while preserving native W4A4 FP4 GEMMs.
• Reported results: BF16-level accuracy, +21.49 average pass@1 over naive NVFP4 RL, and ~16% higher rollout throughput than FP8.