RoBERTa: A Robustly Optimized BERT Pretraining Approach

Abstract

This paper introduces and evaluates the idea described in “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, and reports empirical results that helped shape subsequent work in nlp, transformers.