AI is happening. The future is updating.ModernGBERT_134M
ModernGBERT 134M is a German ModernBERT language model with 134 million parameters and a native context length of up to 8,192 tokens. This model follows the same BERT-style architecture and training procedure as the ModernBERT codebase. ModernGBERT 134M has been pre-trained on 470 billion tokens from the German portion of RedPajama V2, a subset of the same training dataset used for our LLäMmlein decoder family. You can use ModernGBERT with the `transformers` library from version v4.48.0 onwards. (Optional: install `flash-attn` to achieve highest efficiency.) Since ModernGBERT 134M is a Masked Language Model (MLM), you can load it via `AutoModelForMaskedLM`. For downstream tasks such as classification, retrieval, or QA, fine-tune the model by following standard BERT fine-tuning recipes.
ModernGBERT 134M is a German ModernBERT language model with 134 million parameters and a native context length of up to 8,192 tokens. This model follows the same BERT-style architecture and training procedure as the ModernBERT codebase. ModernGBERT 134M has been pre-trained on 470 billion tokens from the German portion of RedPajama V2, a subset of the same training dataset used for our LLäMmlein decoder family. You can use ModernGBERT with the `transformers` library from version v4.48.0 onwards. (Optional: install `flash-attn` to achieve highest efficiency.) Since ModernGBERT 134M is a Masked Language Model (MLM), you can load it via `AutoModelForMaskedLM`. For downstream tasks such as classification, retrieval, or QA, fine-tune the model by following standard BERT fine-tuning recipes.