Ernie 2.0: A Continual Pre-Training Framework for Language Understanding | AISC

Channel:
Canada LLMs Explained - Aggregate Intellect - AI.SCIENCE
Subscribers:
22,300
Published on ● Video Link: https://www.youtube.com/watch?v=8K1IX7VJ5Fc


Duration: 1:40:01
1,721 views
53

For slides and more information on the paper, visit https://aisc.ai.science/events/3919-09-03

Discussion lead: Royal Sequeira


Motivation:
Recently, pre-trained models have achieved state-of-the-art results in various language understanding
tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural
language processing. Current pre-training procedures usually focus on training the model with several
simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there
exists other valuable lexical, syntactic and semantic information in training corpora, such as named
entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical,
syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant
multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms BERT and XLNet
on 16 tasks including English tasks on GLUE benchmarks and several common tasks in Chinese. The
source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.



Other Videos By LLMs Explained - Aggregate Intellect - AI.SCIENCE


2019-09-25MLOps: Overview of Machine Learning Operations on the Cloud | AISC
2019-09-24Lookahead Optimizer: k steps forward, 1 step back
2019-09-24Similarity of neural network representations revisited
2019-09-23Detecting Customer Complaint Escalation w/ Recurrent Neural Networks & Manually-Engineered Features
2019-09-23Graph Normalizing Flows
2019-09-23CNN Architectures for Large-Scale Audio Classification | AISC
2019-09-222019 AI Squared Forum Paper Track | AISC
2019-09-16Making of a conversational agent platform | AISC
2019-09-09A Survey of Singular Learning | AISC
2019-09-04Overview of Reinforcement Learning | AISC
2019-09-03Ernie 2.0: A Continual Pre-Training Framework for Language Understanding | AISC
2019-08-28Consistency by Agreement in Zero-shot Neural Machine Translation | AISC
2019-08-26TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing | AISC
2019-08-21Science of science: Identifying Fundamental Drivers of Science | AISC
2019-08-19AI Product Stream Meet and Greet | AISC
2019-08-12[Original ResNet paper] Deep Residual Learning for Image Recognition | AISC
2019-08-11[GAT] Graph Attention Networks | AISC Foundational
2019-08-06XLNet: Generalized Autoregressive Pretraining for Language Understanding | AISC
2019-07-31Overview of Generative Adversarial Networks | AISC
2019-07-29Multi-Armed Bandit Strategies for Non-Stationary Reward Distributions and Delayed Feedback Processes
2019-07-22AISC Abstract Night