AI가 움직이고 미래가 업데이트됩니다.modernbert-ja-130m
ModernBERT is a new variant of the BERT model that combines local and global attention, allowing it to handle long sequences while maintaining high computational efficiency. It also incorporates modern architectural improvements, such as RoPE. Our ModernBERT-Ja-130M is trained on a high-quality corpus of Japanese and English text comprising **4.39T tokens**, featuring a vocabulary size of 102,400 and a sequence length of **8,192** tokens. Additionally, if your GPUs support Flash Attention 2, we recommend using our models with Flash Attention 2.
번역 준비 중입니다. 원문을 표시합니다.
ModernBERT is a new variant of the BERT model that combines local and global attention, allowing it to handle long sequences while maintaining high computational efficiency. It also incorporates modern architectural improvements, such as RoPE. Our ModernBERT-Ja-130M is trained on a high-quality corpus of Japanese and English text comprising **4.39T tokens**, featuring a vocabulary size of 102,400 and a sequence length of **8,192** tokens. Additionally, if your GPUs support Flash Attention 2, we recommend using our models with Flash Attention 2.