AI is happening. The future is updating.DeepSeek-Coder-V2-Instruct-FP8
It achieves an average score of 88.98 on the HumanEval+ benchmark, whereas the unquantized model achieves 87.63. This model was obtained by quantizing the weights and activations of DeepSeek-Coder-V2-Instruct to FP8 data type, ready for inference with vLLM >= 0.5.2. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. In particular, this model can now be loaded and evaluated with only 4xH100 GPUs, as opposed to 8. Only the weights and activations of the linear operators within transformers blocks are quantized. Symmetric per-tensor quantization is applied, in which a single linear scaling maps the FP8 representations of the quantized weights and activations. AutoFP8 is used for quantization with 512 sequences of UltraChat.
It achieves an average score of 88.98 on the HumanEval+ benchmark, whereas the unquantized model achieves 87.63. This model was obtained by quantizing the weights and activations of DeepSeek-Coder-V2-Instruct to FP8 data type, ready for inference with vLLM >= 0.5.2. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. In particular, this model can now be loaded and evaluated with only 4xH100 GPUs, as opposed to 8. Only the weights and activations of the linear operators within transformers blocks are quantized. Symmetric per-tensor quantization is applied, in which a single linear scaling maps the FP8 representations of the quantized weights and activations. AutoFP8 is used for quantization with 512 sequences of UltraChat.