XLNet: Generalized Autoregressive Pretraining for Language Understanding | AISC
41For slides and more information on the paper, visit https://aisc.ai.science/events/2019-08-06
Discussion lead: Alec Robinson
Motivation:
With the capability of modeling bidirectional contexts, denoising autoencoding
based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions
and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we
propose XLNet, a generalized autoregressive pretraining method that (1) enables
learning bidirectional contexts by maximizing the expected likelihood over all
permutations of the factorization order and (2) overcomes the limitations of BERT
thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas
from Transformer-XL, the state-of-the-art autoregressive model, into pretraining.
Empirically, XLNet outperforms BERT on 20 tasks, often by a large margin, and
achieves state-of-the-art results on 18 tasks including question answering, natural
language inference, sentiment analysis, and document ranking.