AI is happening. The future is updating.kobigbird-bert-base
BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. BigBird relies on **block sparse attention** instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. *NOTE:* Use `BertTokenizer` instead of BigBirdTokenizer. (`AutoTokenizer` will load `BertTokenizer`)
BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. BigBird relies on **block sparse attention** instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. *NOTE:* Use `BertTokenizer` instead of BigBirdTokenizer. (`AutoTokenizer` will load `BertTokenizer`)