AI is happening. The future is updating.tinyroberta-squad2
This is the *distilled* version of the deepset/roberta-base-squad2 model. This model has a comparable prediction quality and runs at twice the speed of the base model. **Language model:** tinyroberta-squad2 **Language:** English **Downstream-task:** Extractive QA **Training data:** SQuAD 2.0 **Eval data:** SQuAD 2.0 **Code:** See an example extractive QA pipeline built with Haystack **Infrastructure**: 4x Tesla v100 This model was distilled using the TinyBERT approach described in this paper and implemented in haystack. Firstly, we have performed intermediate layer distillation with roberta-base as the teacher which resulted in deepset/tinyroberta-6l-768d. Secondly, we have performed task-specific distillation with deepset/roberta-base-squad2 as the teacher for further intermediate layer distillation on an augmented version of SQuADv2 and then with deepset/roberta-large-squad2 as the teacher for prediction layer distillation.
This is the *distilled* version of the deepset/roberta-base-squad2 model. This model has a comparable prediction quality and runs at twice the speed of the base model. **Language model:** tinyroberta-squad2 **Language:** English **Downstream-task:** Extractive QA **Training data:** SQuAD 2.0 **Eval data:** SQuAD 2.0 **Code:** See an example extractive QA pipeline built with Haystack **Infrastructure**: 4x Tesla v100 This model was distilled using the TinyBERT approach described in this paper and implemented in haystack. Firstly, we have performed intermediate layer distillation with roberta-base as the teacher which resulted in deepset/tinyroberta-6l-768d. Secondly, we have performed task-specific distillation with deepset/roberta-base-squad2 as the teacher for further intermediate layer distillation on an augmented version of SQuADv2 and then with deepset/roberta-large-squad2 as the teacher for prediction layer distillation.