Search FoxAI
AI Models 84762
- TDD↗
RED-AIGC
- Llama-3.2-3B-Instruct-FP8↗
RedHatAI
Quantized version of Llama-3.2-3B-Instruct. It achieves scores within 0.7% of the scores of the unquantized model for MMLU, ARC-Challenge, GSM-8k, Hellaswag, Winogrande and TruthfulQA. This model was obtained by quantizing the weights of Llama-3.2-3B-Instruct to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%. Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, where a fixed linear scaling factor is applied between FP8 and floating point representations for each output channel dimension. Activations are quantized with a symmetric per-tensor scheme, where a fixed linear scaling factor is applied between FP8 and floating point representations for the entire activation tensor. Weights are quantized by rounding to nearest FP8 representation. The llm-compressor library was applied to quantize the model, usin 512 sequences sequences taken from Neural Magic's LLM compression calibration dataset.
- NousResearch_-_Hermes-3-Llama-3.1-70B-gguf↗
RichardErkhov
- recogna-nlp_-_phibode-3-mini-4k-ultraalpaca-gguf↗
RichardErkhov
- 38-2012-12↗
rose33300
- mxbai-edge-colbert-v0-17m-onnx-int8↗
ryandono
- metal-acestep1.5-v1↗
ryanontheinside
- rock-acestep1.5-xl-v1↗
ryanontheinside
- xgen-7b-8k-inst↗
Salesforce
- Palmyra-Med-70B-32K-GGUF↗
second-state
