AI가 움직이고 미래가 업데이트됩니다.Qwen3.8-27B-int4-AutoRound
INT4 (W4A16) AutoRound quantization of Qwen/Qwen3.8-27B — **18 GB**, with the **MTP speculative-decoding head quantized and working**. The MTP head is quantized along with the model, so `--speculative-config` in vLLM works out of the box and roughly **doubles decode throughput** (measured +98 % on temp-0 prose vs the same GPU without MTP; draft acceptance 41–67 % depending on content). At the time of upload, the available 4-bit options either dropped the MTP tensors (no speculative decoding) or kept hundreds of tensors in bf16 (28 GB — no longer fits a 24–32 GB GPU next to a useful KV cache). This one follows the recipe that served Qwen3.6-27B in production for months:
번역 준비 중입니다. 원문을 표시합니다.
INT4 (W4A16) AutoRound quantization of Qwen/Qwen3.8-27B — **18 GB**, with the **MTP speculative-decoding head quantized and working**. The MTP head is quantized along with the model, so `--speculative-config` in vLLM works out of the box and roughly **doubles decode throughput** (measured +98 % on temp-0 prose vs the same GPU without MTP; draft acceptance 41–67 % depending on content). At the time of upload, the available 4-bit options either dropped the MTP tensors (no speculative decoding) or kept hundreds of tensors in bf16 (28 GB — no longer fits a 24–32 GB GPU next to a useful KV cache). This one follows the recipe that served Qwen3.6-27B in production for months: