AI本地部署不如官方版的元凶找到了：734个依赖包，每一个都可能坑
Ranking
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - Tests of Qwen3.6-27B show that inference-stack choices—not just model weights and hardware—can materially change token outputs and tool-call reliability, especially over long contexts. Attention kernels, KV-cache precision, weight quantization, and tensor parallelism can all introduce numerical drift that accumulates into consequential errors.
- Switching among FlashAttention 2, Flash Inference, and Triton Attention produced Top-1 token flips despite identical prompts, weights, GPU, and KV-cache settings.
- INT4 KV-cache quantization caused rapidly increasing divergence and unrecoverable tool-call failures; INT8 sometimes recovered, while BF16 remained stable.
- A community W8A16 INT8 quantization outperformed official FP8 and NVFP4 variants in Top-1 consistency; NVFP4 approached a 50% flip rate near 88K context in this setup.
- Tensor-parallel configurations also changed outcomes, likely due to numerical differences in NCCL reductions, underscoring the need to report complete runtime and evaluation environments.
Sources (1)
AI本地部署不如官方版的元凶找到了：734个依赖包，每一个都可能坑
TL;DR - Tests of Qwen3.6-27B show that inference-stack choices—not just model weights and hardware—can materially change token outputs and tool-call reliability, especially over long contexts. Attention kernels, KV-cache precision, weight quantization, and tensor parallelism can all introduce numerical drift that accumulates into consequential errors.
- Switching among FlashAttention 2, Flash Inference, and Triton Attention produced Top-1 token flips despite identical prompts, weights, GPU, and KV-cache settings.
- INT4 KV-cache quantization caused rapidly increasing divergence and unrecoverable tool-call failures; INT8 sometimes recovered, while BF16 remained stable.
- A community W8A16 INT8 quantization outperformed official FP8 and NVFP4 variants in Top-1 consistency; NVFP4 approached a 50% flip rate near 88K context in this setup.
- Tensor-parallel configurations also changed outcomes, likely due to numerical differences in NCCL reductions, underscoring the need to report complete runtime and evaluation environments.