AIが動く。未来が更新される。modernbert-ja-130m
ModernBERT is a new variant of the BERT model that combines local and global attention, allowing it to handle long sequences while maintaining high computational efficiency. It also incorporates modern architectural improvements, such as RoPE. Our ModernBERT-Ja-130M is trained on a high-quality corpus of Japanese and English text comprising **4.39T tokens**, featuring a vocabulary size of 102,400 and a sequence length of **8,192** tokens. Additionally, if your GPUs support Flash Attention 2, we recommend using our models with Flash Attention 2.
翻訳準備中のため原文を表示しています
ModernBERT is a new variant of the BERT model that combines local and global attention, allowing it to handle long sequences while maintaining high computational efficiency. It also incorporates modern architectural improvements, such as RoPE. Our ModernBERT-Ja-130M is trained on a high-quality corpus of Japanese and English text comprising **4.39T tokens**, featuring a vocabulary size of 102,400 and a sequence length of **8,192** tokens. Additionally, if your GPUs support Flash Attention 2, we recommend using our models with Flash Attention 2.