AI is happening. The future is updating.chinese-babylm-2026-v3
A compact DeBERTaV2 masked language model pretrained from scratch on Chinese child-directed and child-produced text. Designed for data-constrained language modeling with a pure-Chinese WordPiece tokenizer. Key findings: - Tokenizer granularity is the primary factor for HANZI track performance (8K vocab >> 32K vocab) - DeBERTaV2 achieves the highest NLU scores (zhoblimp: 69.89) - Radical embedding hurts performance with character-level tokenization
A compact DeBERTaV2 masked language model pretrained from scratch on Chinese child-directed and child-produced text. Designed for data-constrained language modeling with a pure-Chinese WordPiece tokenizer. Key findings: - Tokenizer granularity is the primary factor for HANZI track performance (8K vocab >> 32K vocab) - DeBERTaV2 achieves the highest NLU scores (zhoblimp: 69.89) - Radical embedding hurts performance with character-level tokenization