Addendum for Supermasks in Superposition: A Closer Look (Paper Explained)

Channel:
Switzerland Yannic Kilcher
Subscribers:
284,000
Published on ● Video Link: https://www.youtube.com/watch?v=Jqvb7jp4Nm8


Duration: 48:52
2,694 views
87

I take a closer look at "Supermasks in Superposition" after I've already done a video on it. Specifically, I look at: 1. The intuition and theoretical justification behind the G objective, 2. Whether Supermasks and Superposition can be viewed as two distinct ideas and 3. The Paper's Broader Impact Statement.

OUTLINE:
0:00 - Intro & Overview
2:00 - SupSup Recap
4:00 - In-Depth Analysis of the G Objective
20:30 - Superposition without Supermasks
25:40 - Broader Impact Statement
36:40 - Conclusion
37:20 - Live Coding

Part 1 on SupSup: https://youtu.be/3jT1qJ8ETzk
My Code: https://colab.research.google.com/drive/1bEcppdN6qZRpEFplIiv41ZI3vDwDjcvC?usp=sharing
Paper: https://arxiv.org/abs/2006.14769

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher



Other Videos By Yannic Kilcher


2020-07-26[Classic] Playing Atari with Deep Reinforcement Learning (Paper Explained)
2020-07-23[Classic] ImageNet Classification with Deep Convolutional Neural Networks (Paper Explained)
2020-07-21Neural Architecture Search without Training (Paper Explained)
2020-07-19[Classic] Generative Adversarial Networks (Paper Explained)
2020-07-16[Classic] Word2Vec: Distributed Representations of Words and Phrases and their Compositionality
2020-07-14[Classic] Deep Residual Learning for Image Recognition (Paper Explained)
2020-07-12I'M TAKING A BREAK... (Channel Update July 2020)
2020-07-11Deep Ensembles: A Loss Landscape Perspective (Paper Explained)
2020-07-10Gradient Origin Networks (Paper Explained w/ Live Coding)
2020-07-09NVAE: A Deep Hierarchical Variational Autoencoder (Paper Explained)
2020-07-08Addendum for Supermasks in Superposition: A Closer Look (Paper Explained)
2020-07-07SupSup: Supermasks in Superposition (Paper Explained)
2020-07-06[Live Machine Learning Research] Plain Self-Ensembles (I actually DISCOVER SOMETHING) - Part 1
2020-07-05SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization (Paper Explained)
2020-07-04Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (Paper Explained)
2020-07-03On the Measure of Intelligence by François Chollet - Part 4: The ARC Challenge (Paper Explained)
2020-07-02BERTology Meets Biology: Interpreting Attention in Protein Language Models (Paper Explained)
2020-07-01GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (Paper Explained)
2020-06-30Object-Centric Learning with Slot Attention (Paper Explained)
2020-06-29Set Distribution Networks: a Generative Model for Sets of Images (Paper Explained)
2020-06-28Context R-CNN: Long Term Temporal Context for Per-Camera Object Detection (Paper Explained)


Tags:
deep learning
machine learning
arxiv
explained
neural networks
ai
artificial intelligence
paper
supsup
supermasks
lottery ticket
lottery ticket hypothesis
gradient
entropy
surplus
superfluous neurons
lifelong learning
multitask learning
catastrophic forgetting
continuous learning
binary mask
random network
optimization
hopfield network
gradient descent
superposition