AI is happening. The future is updating.gemma-4-E4B-it-W4A16-AutoRound-GPTQ
This repository hosts **W4A16 INT4-quantized** versions of `google/gemma-4-E4B-it`, a multimodal mixture-of-experts model supporting text, vision, and audio inputs. Two quantized variants are available: > **Note on MTP / Speculative Decoding:** > If you want to use the official speculative decoding assistant model (`google/gemma-4-E4B-it-assistant`) for MTP support, it is recommended to use the `vllm/vllm-openai:gemma4-0505-cu129` Docker image, which includes newer Gemma 4 support and decoding patches. > > Due to INT4 quantization, the assistant acceptance rate may be lower compared to the original unquantized `google/gemma-4-E4B-it` model. > **Note:** Only the language model (LM) layers are quantized to INT4. The vision tower, audio tower, and multimodal projectors are kept at full precision (BF16) to preserve multimodal quality.
This repository hosts **W4A16 INT4-quantized** versions of `google/gemma-4-E4B-it`, a multimodal mixture-of-experts model supporting text, vision, and audio inputs. Two quantized variants are available: > **Note on MTP / Speculative Decoding:** > If you want to use the official speculative decoding assistant model (`google/gemma-4-E4B-it-assistant`) for MTP support, it is recommended to use the `vllm/vllm-openai:gemma4-0505-cu129` Docker image, which includes newer Gemma 4 support and decoding patches. > > Due to INT4 quantization, the assistant acceptance rate may be lower compared to the original unquantized `google/gemma-4-E4B-it` model. > **Note:** Only the language model (LM) layers are quantized to INT4. The vision tower, audio tower, and multimodal projectors are kept at full precision (BF16) to preserve multimodal quality.