16张B200才能跑的Kimi K3，8张AMD就装下了
Ranking
Overall
71
Content
80
Popularity
N/A
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - Wafer AI deployed the 2.8T-parameter Kimi K3 on a single 8-GPU AMD MI355X node, versus 16 NVIDIA B200s across two servers, claiming ~3.8x per-node throughput and better cost efficiency — a sign AMD's larger HBM capacity is becoming a real systems advantage for trillion-parameter open models.
- Memory, not compute, is the binding constraint: K3 weights alone exceed 1.5 TB, so 192 GB B200 cards (≈1.5 TB/node) require cross-node deployment over ~195 Gb/s RoCE v2, while 288 GB MI355X (≈2.3 TB/node) keeps the model on one node.
- Reported numbers (1024-in/400-out): MI355X 952 tok/s total, 118 tok/s per user; 16x B200 498 tok/s total (~249 tok/s/node), 90 tok/s per user; 8x B300 1568 tok/s, 172 tok/s per user. At assumed $2.5/$4.25/$6 per GPU-hour, MI355X yields ~48 tok/s per dollar vs ~7 (B200) and ~33 (B300).
- ROCm needed minimal work: a missing
top_k_renorm_probwas patched in plain PyTorch (no custom kernel), after which speculative decoding via an external block-diffusion draft model gave ~2.2x single-stream, ~1.7x at medium concurrency, ~18% peak throughput. - TTFT fix was shape-related, not capability-related: 12 attention heads per GPU under 8-way TP missed AITER's MLA prefill kernel shape constraints (multiples of 4/8/16), forcing a slow Triton fallback; zero-padding to 16 heads raised prefill to ~13k tok/s (from ~4–7k), cutting a 172k-token cold prefill from ~51s toward B300's ~23s.
Sources (1)
16张B200才能跑的Kimi K3，8张AMD就装下了
Public signals
N/A
TL;DR - Wafer AI deployed the 2.8T-parameter Kimi K3 on a single 8-GPU AMD MI355X node, versus 16 NVIDIA B200s across two servers, claiming ~3.8x per-node throughput and better cost efficiency — a sign AMD's larger HBM capacity is becoming a real systems advantage for trillion-parameter open models.
- Memory, not compute, is the binding constraint: K3 weights alone exceed 1.5 TB, so 192 GB B200 cards (≈1.5 TB/node) require cross-node deployment over ~195 Gb/s RoCE v2, while 288 GB MI355X (≈2.3 TB/node) keeps the model on one node.
- Reported numbers (1024-in/400-out): MI355X 952 tok/s total, 118 tok/s per user; 16x B200 498 tok/s total (~249 tok/s/node), 90 tok/s per user; 8x B300 1568 tok/s, 172 tok/s per user. At assumed $2.5/$4.25/$6 per GPU-hour, MI355X yields ~48 tok/s per dollar vs ~7 (B200) and ~33 (B300).
- ROCm needed minimal work: a missing
top_k_renorm_probwas patched in plain PyTorch (no custom kernel), after which speculative decoding via an external block-diffusion draft model gave ~2.2x single-stream, ~1.7x at medium concurrency, ~18% peak throughput. - TTFT fix was shape-related, not capability-related: 12 attention heads per GPU under 8-way TP missed AITER's MLA prefill kernel shape constraints (multiples of 4/8/16), forcing a slow Triton fallback; zero-padding to 16 heads raised prefill to ~13k tok/s (from ~4–7k), cutting a 172k-token cold prefill from ~51s toward B300's ~23s.