AI가 움직이고 미래가 업데이트됩니다.limite-1b-violetto
Limite 1B - Violetto is Paradigma’s first model, designed for high-throughput solutions of difficult mathematical problems. Limite is a 1-billion parameter dense autoregressive transformer, trained from scratch on a mixture of less than 300B highly curated tokens, capable of handling sequences up to 131k tokens long. Apart from data curation, we achieve such high sample efficiency by equipping Limite with an architecture strongly inspired by recent advancements made by the community on pre-training speedrun competitions. The base model itself scores well on math benchmarks with few-shot prompting.
번역 준비 중입니다. 원문을 표시합니다.
Limite 1B - Violetto is Paradigma’s first model, designed for high-throughput solutions of difficult mathematical problems. Limite is a 1-billion parameter dense autoregressive transformer, trained from scratch on a mixture of less than 300B highly curated tokens, capable of handling sequences up to 131k tokens long. Apart from data curation, we achieve such high sample efficiency by equipping Limite with an architecture strongly inspired by recent advancements made by the community on pre-training speedrun competitions. The base model itself scores well on math benchmarks with few-shot prompting.