AI is happening. The future is updating.diffusiongemma-26B-A4B-it-FP8-dynamic
This model is an FP8 quantized version of google/diffusiongemma-26B-A4B-it. The model has both weights and activations quantized to FP8 using vllm/llm-compressor and in the compressed-tensors format. It was evaluated on several tasks to assess its quality in comparison to the unquantized model using vLLM. The following metrics were generated when serving the quantized model with vLLM on a single B200 GPU.
This model is an FP8 quantized version of google/diffusiongemma-26B-A4B-it. The model has both weights and activations quantized to FP8 using vllm/llm-compressor and in the compressed-tensors format. It was evaluated on several tasks to assess its quality in comparison to the unquantized model using vLLM. The following metrics were generated when serving the quantized model with vLLM on a single B200 GPU.