AI is happening. The future is updating.Qwen2.5-VL-7B-Instruct-quantized.w8a8
This model was obtained by quantizing the weights of Qwen/Qwen2.5-VL-7B-Instruct to INT8 data type, ready for inference with vLLM >= 0.5.2. This model was created with llm-compressor by running the code snippet below as part a multimodal announcement blog. The model was evaluated using mistral-evals for vision-related tasks and using lm_evaluation_harness for select text-based benchmarks. The evaluations were conducted using the following commands:
This model was obtained by quantizing the weights of Qwen/Qwen2.5-VL-7B-Instruct to INT8 data type, ready for inference with vLLM >= 0.5.2. This model was created with llm-compressor by running the code snippet below as part a multimodal announcement blog. The model was evaluated using mistral-evals for vision-related tasks and using lm_evaluation_harness for select text-based benchmarks. The evaluations were conducted using the following commands: