AIが動く。未来が更新される。ThinkingCap-Qwen3.8-27B-abliterated-NVFP4A16
A 4-bit weight-only quantization (NVFP4 weights, bf16 activations) of ThinkingCap-Qwen3.8-27B-abliterated, the uncensored variant of bottlecapai/ThinkingCap-Qwen3.8-27B. 20.6 GB, down from 55.6 GB in bf16, at KL 0.0155 against the bf16 model. The weights fit a 24 GB GPU with a little room for KV cache; 32 GB and up is comfortable. vLLM serves it on Ampere and newer (sm80+) with a weight-only kernel: Marlin FP4 on most GPUs (Ampere, Ada, Hopper, RTX 50, DGX Spark) and a dedicated BF16×FP4 kernel on B200/B300. Activations stay bf16, so no GPU computes in FP4 here. **Which build?** The W4A16 build (19.5 GB, KL 0.0260) runs on the same GPUs (vLLM: Marlin, Machete on Hopper) and is 1.1 GB smaller; this build has the lower KL. Speed was not measured for either. With memory to spare, the FP8-BLOCK build (31.3 GB, KL 0.0046) has the lowest KL of the published builds; on Ada GPUs use FP8-DYNAMIC (31.3 GB, KL 0.0068) instead.
翻訳準備中のため原文を表示しています
A 4-bit weight-only quantization (NVFP4 weights, bf16 activations) of ThinkingCap-Qwen3.8-27B-abliterated, the uncensored variant of bottlecapai/ThinkingCap-Qwen3.8-27B. 20.6 GB, down from 55.6 GB in bf16, at KL 0.0155 against the bf16 model. The weights fit a 24 GB GPU with a little room for KV cache; 32 GB and up is comfortable. vLLM serves it on Ampere and newer (sm80+) with a weight-only kernel: Marlin FP4 on most GPUs (Ampere, Ada, Hopper, RTX 50, DGX Spark) and a dedicated BF16×FP4 kernel on B200/B300. Activations stay bf16, so no GPU computes in FP4 here. **Which build?** The W4A16 build (19.5 GB, KL 0.0260) runs on the same GPUs (vLLM: Marlin, Machete on Hopper) and is 1.1 GB smaller; this build has the lower KL. Speed was not measured for either. With memory to spare, the FP8-BLOCK build (31.3 GB, KL 0.0046) has the lowest KL of the published builds; on Ada GPUs use FP8-DYNAMIC (31.3 GB, KL 0.0068) instead.