Turing-NLG, DeepSpeed and the ZeRO optimizer

Channel:
Switzerland Yannic Kilcher
Subscribers:
284,000
Published on ● Video Link: https://www.youtube.com/watch?v=tC01FRB0M7w


Duration: 21:18
10,268 views
298

Microsoft has trained a 17-billion parameter language model that achieves state-of-the-art perplexity. This video takes a look at the ZeRO optimizer that enabled this breakthrough. ZeRO allows you to do model- and data-parallelism without having huge cuts in training speed.

https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/
https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/
https://github.com/microsoft/DeepSpeed
https://arxiv.org/abs/1910.02054

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher



Other Videos By Yannic Kilcher


2020-04-03Dream to Control: Learning Behaviors by Latent Imagination
2020-04-02Can we Contain Covid-19 without Locking-down the Economy?
2020-04-01State-of-Art-Reviewing: A Radical Proposal to Improve Scientific Publication
2020-03-31Agent57: Outperforming the Atari Human Benchmark
2020-03-30Axial Attention & MetNet: A Neural Weather Model for Precipitation Forecasting
2020-03-25[Rant] coronavirus
2020-03-23Online Education - How I Make My Videos
2020-02-24Deep Learning for Symbolic Mathematics
2020-02-21NeurIPS 2020 Changes to Paper Submission Process
2020-02-12Growing Neural Cellular Automata
2020-02-11Turing-NLG, DeepSpeed and the ZeRO optimizer
2020-01-27[Interview] Mark Ledwich - Algorithmic Extremism: Examining YouTube's Rabbit Hole of Radicalization
2020-01-22Reformer: The Efficient Transformer
2020-01-10Go-Explore: a New Approach for Hard-Exploration Problems
2019-12-11NeurIPS 19 Poster Session
2019-12-10Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions
2019-12-08NeurIPS 2019
2019-11-21MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
2019-11-07A neurally plausible model learns successor representations in partially observable environments
2019-11-03SinGAN: Learning a Generative Model from a Single Natural Image
2019-11-02AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning


Tags:
deep learning
machine learning
nlp
natural language processing
machine translation
arxiv
attention mechanism
attention
transformer
seq2seq
bert
long sequence
memory
gpt-2
Megatron
Microsoft
distributed
parallelism