GLM-5.3-Flash-W4A16-MTP

模型介绍

INT4 weight-only quantization of zai-org/GLM-5.3-Flash with the BF16 MTP draft head kept for speculative decoding. Only the 36,288 routed-expert GEMMs are INT4 (GPTQ, symmetric, group-size 128); attention, router, shared experts, embeddings, the vision tower and the MTP head stay in BF16. Not a new model — all capability comes from the base model. > **Two things that bite.** If your `config.json` predates 2026-09-08, re-download it — older copies fail in vLLM with `KeyError: 'layers.0.mlp.gate_up_proj.weight'` (weights are unchanged). And always pass `--max-num-seqs ≤ 512` — the vLLM default of 1024 exceeds this hybrid linear-attention model's 512 Mamba/KDA-state cache blocks. **H100 / H200, TP=4 (pinned image)** — the Quick start command is the benchmarked recipe. `num_speculative_tokens: 2` is the sweet spot on Hopper (52–55% acceptance; N=5 collapses acceptance to ~30%). Keep prefix caching off — it measured −2…−5% on H200. On 8× H200, two independent TP=4 replicas behind a load balancer beat one TP=8 endpoint by +28–38% aggregate at c128–c512; TP=8 wins single-stream and holds one 7.8M-token pool. On 2× H200 (89.5 GiB weights per GPU) add `--tensor-parallel-size 2 --max-num-seqs 128 --max-cudagraph-capture-size 128` for 262K, or `--max-model-len 1048576 --max-num-seqs 16 --gpu-memory-utilization 0.95 --max-cudagraph-capture-size 64 --max-num-batched-tokens 4096` for 1M.

主要特点

特点摘要待依据来源补齐，不凭模型名称推断能力。

局限与注意事项

暂无足够来源支持具体缺点结论。请重点验证实际任务表现、时延、数据政策、许可与版本生命周期。

来源报价与费用分析

同一模型在不同渠道、地区和配置下可能采用不同价格。下表独立保留各渠道报价，不将渠道价格视为厂商官方统一价格。

公开评测与运行配置

暂无已采集的可核验公开评分。模型未上榜不代表能力较低。

常见问题

资料来源与可追溯记录

许可证：mit。模型介绍与第三方评测均以原来源的具体版本和方法为准。