BERTology Meets Biology: Interpreting Attention in Protein Language Models

Abstract

This paper introduces and evaluates the idea described in “BERTology Meets Biology: Interpreting Attention in Protein Language Models”, and reports empirical results that helped shape subsequent work in interpretability, transformers.