bigbird-roberta-large

BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle. It is a pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. Disclaimer: The team releasing BigBird did not write a model card for this model so this model card has been written by the Hugging Face team.