16张B200才能跑的Kimi K3，8张AMD就装下了

TL;DR - Wafer AI deployed the 2.8T-parameter Kimi K3 on a single 8-GPU AMD MI355X node, versus 16 NVIDIA B200s across two servers, claiming ~3.8x per-node throughput and better cost efficiency — a sign AMD's larger HBM capacity is becoming a real systems advantage for trillion-parameter open models.

• Memory, not compute, is the binding constraint: K3 weights alone exceed 1.5 TB, so 192 GB B200 cards (≈1.5 TB/node) require cross-node deployment over ~195 Gb/s RoCE v2, while 288 GB MI355X (≈2.3 TB/node) keeps the model on one node.
• Reported numbers (1024-in/400-out): MI355X 952 tok/s total, 118 tok/s per user; 16x B200 498 tok/s total (~249 tok/s/node), 90 tok/s per user; 8x B300 1568 tok/s, 172 tok/s per user. At assumed $2.5/$4.25/$6 per GPU-hour, MI355X yields ~48 tok/s per dollar vs ~7 (B200) and ~33 (B300).
• ROCm needed minimal work: a missing top_k_renorm_prob was patched in plain PyTorch (no custom kernel), after which speculative decoding via an external block-diffusion draft model gave ~2.2x single-stream, ~1.7x at medium concurrency, ~18% peak throughput.
• TTFT fix was shape-related, not capability-related: 12 attention heads per GPU under 8-way TP missed AITER's MLA prefill kernel shape constraints (multiples of 4/8/16), forcing a slow Triton fallback; zero-padding to 16 heads raised prefill to ~13k tok/s (from ~4–7k), cutting a 172k-token cold prefill from ~51s toward B300's ~23s.