Llama-3.2-1B-Instruct-quantized.w8a8

Model overview

Quantized version of Llama-3.2-1B-Instruct. It achieves scores within 5% of the scores of the unquantized model for MMLU, ARC-Challenge, GSM-8k, Hellaswag, Winogrande and TruthfulQA. This model was obtained by quantizing the weights of Llama-3.2-1B-Instruct to INT8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%. Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, where a fixed linear scaling factor is applied between INT8 and floating point representations for each output channel dimension. Activations are quantized with a symmetric dynamic per-token scheme, computing a linear scaling factor at runtime for each token between INT8 and floating point representations. The SmoothQuant algorithm is used to alleviate outliers in the activations, whereas rhe GPTQ algorithm is applied for quantization. Both algorithms are implemented in the llm-compressor library. GPTQ used a 1% damping factor and 512 sequences sequences taken from Neural Magic's LLM compression calibration dataset.

Key features

特点摘要待依据来源补齐，不凭模型名称推断能力。

Limitations & considerations

Sources are insufficient to establish specific limitations. Verify task performance, latency, data policies, licensing and version lifecycle.

Source prices & cost analysis

Prices vary by channel, region and configuration. Channel quotes are listed independently and are not treated as a single official provider price.

Public benchmarks & run configuration

暂无已采集的可核验公开评分。模型未上榜不代表能力较低。

Frequently asked questions

Sources & traceability

许可证：llama3.2。模型介绍与第三方评测均以原来源的具体版本和方法为准。