Understanding AI #3 - What does Pre-training mean in Generative Pre-Training Transformer (GPT)?

Channel:

The Flourishing

Subscribers:

2,350

Published on March 25, 2023 3:00:02 AM ● Video Link: https://www.youtube.com/watch?v=5UpZTGo8Bng

Duration: 7:22

46 views

3

Welcome to my new series called Understaing AI where I research and talk about various topics within Artificial Intelligence.

In this episode I talk about the term pre-training in GPT models and models generally.

Here are some links to relevant documents for further study
Semi-supervised Sequence Learning, A. M. Dai, Q. V. Le (2015) - https://arxiv.org/abs/1511.01432

Improving Language Understanding by Generative Pre-Training. Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. Preprint (2018). - https://openai.com/research/language-unsupervised
Direct Link to pdf of above paper - https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf