Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening

TL;DR - A field report on full fine-tuning a 32.76B-parameter Qwen3-32B model across 16 NVIDIA B300 GPUs (two nodes, FSDP/ZeRO-3), offering operational triage tooling, calibrated scaling numbers, and negative results rather than new algorithms. It matters as one of the first published practitioner accounts on B300 hardware, with transferable debugging practice for large distributed training jobs.

• Watch power, not utilization: a B300-calibrated power-draw table distinguishes compute / communication / data-starvation / checkpoint-or-deadlock / idle states, because GPU utilization% falsely reads 100% during an NCCL hang.
• Negative results dispel folklore: a controlled A/B found per-step NFS reads matched a pretokenized local cache (~53k tok/s) since the corpus fits in page cache and the job is compute-bound; an earlier "throughput collapse" was re-diagnosed as NFS/CPU contention, not a storage-medium limit.
• Failure case and hardening: an epoch-end NCCL deadlock caused by per-rank token-packing imbalance was addressed with a 2.7-second pre-run invariant gate plus an external watcher, converting multi-hour silent failures into instant rejections; the authors position this against PyTorch's documented Join / equalize-to-minimum practice.
• Reference data: 4/8/16-GPU strong-scaling and GPU-hour measurements on B300 are reported as absolute values, showing near-linear scaling as expected in this regime.