phi-4-FP8-dynamic

This model was obtained by quantizing activation and weights of phi-4 to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%. Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme. The llm-compressor library is used for quantization. The model was evaluated on the OpenLLM leaderboard tasks (version 1) with the lm-evaluation-harness and the vLLM engine, using the following command:

译文尚未就绪，显示来源原文