AI is happening. The future is updating.LLaMA-MoE-v1-3_0B-2_16
β€οΈ This repo contains the model `LLaMA-MoE-v1-3.0B (2/16)`, which activates 2 out of 16 experts (3.0B parameters). This model is NOT fine-tuned by instruction pairs, so it may not be good enough to act like a chatbot. π’ LLaMA-MoE is a series of Mixture-of-Expert (MoE) models based on LLaMA-2. You can find the code for training this model at this repo. π This series of models are obtained by partitioning original LLaMA FFNs into experts and further continual pre-training. The total model size is only 6.7B parameters, which is very convenient for deployment and research usage. More details could be found at our technical report.
β€οΈ This repo contains the model `LLaMA-MoE-v1-3.0B (2/16)`, which activates 2 out of 16 experts (3.0B parameters). This model is NOT fine-tuned by instruction pairs, so it may not be good enough to act like a chatbot. π’ LLaMA-MoE is a series of Mixture-of-Expert (MoE) models based on LLaMA-2. You can find the code for training this model at this repo. π This series of models are obtained by partitioning original LLaMA FFNs into experts and further continual pre-training. The total model size is only 6.7B parameters, which is very convenient for deployment and research usage. More details could be found at our technical report.