ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

Abstract

This paper introduces and evaluates the idea described in “ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators”, and reports empirical results that helped shape subsequent work in nlp, pretraining.