Qwen3-1.6B-A0.9B

Model overview

The checkpoint is stored as a single `model.safetensors` file with individual expert weights matching the original Qwen3 structure. Each layer has 16 experts with separate `gate_proj`, `up_proj`, and `down_proj` weights per expert. The model was fine-tuned on a toy copypasta dataset and achieves: - **Perplexity**: 1.0 (on validation text) - **Generation**: Successfully generates coherent continuations 1. **Configuration**: Created with 10 layers and 16 experts (8 activated per token) to achieve ~1.6B total parameters with 50% MoE activation 2. **Initialization**: Randomly initialized weights using transformers `init_weights()` 3. **Fine-tuning**: Trained on famous internet copypastas until perplexity < 3.0 4. **Checkpoint Conversion**: Converted from batched expert format to individual expert format to match original Qwen3 checkpoint structure 5. **Validation**: Confirmed perplexity ~1.0 and successful text generation

Key features

特点摘要待依据来源补齐，不凭模型名称推断能力。

Limitations & considerations

Sources are insufficient to establish specific limitations. Verify task performance, latency, data policies, licensing and version lifecycle.

Source prices & cost analysis

Prices vary by channel, region and configuration. Channel quotes are listed independently and are not treated as a single official provider price.

Public benchmarks & run configuration

暂无已采集的可核验公开评分。模型未上榜不代表能力较低。

Frequently asked questions

Sources & traceability

许可证：mit。模型介绍与第三方评测均以原来源的具体版本和方法为准。