[Live Machine Learning Research] Plain Self-Ensembles (I actually DISCOVER SOMETHING) - Part 1

Channel:
Switzerland Yannic Kilcher
Subscribers:
284,000
Published on ● Video Link: https://www.youtube.com/watch?v=z_3Qv4In2ac


Duration: 1:18:43
12,086 views
508

I share my progress of implementing a research idea from scratch. I attempt to build an ensemble model out of students of label-free self-distillation without any additional data or augmentation. Turns out, it actually works, and interestingly, the more students I employ, the better the accuracy. This leads to the hypothesis that the ensemble effect is not a process of extracting more information from labels.

OUTLINE:
0:00 - Introduction
2:10 - Research Idea
4:15 - Adjusting the Codebase
25:00 - Teacher and Student Models
52:30 - Shipping to the Server
1:03:40 - Results
1:14:50 - Conclusion

Code: https://github.com/yk/PyTorch_CIFAR10

References:
My Video on SimCLRv2: https://youtu.be/2lkUNDZld-4
Born-Again Neural Networks: https://arxiv.org/abs/1805.04770
Deep Ensembles: A Loss Landscape Perspective: https://arxiv.org/abs/1912.02757

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher



Other Videos By Yannic Kilcher


2020-07-21Neural Architecture Search without Training (Paper Explained)
2020-07-19[Classic] Generative Adversarial Networks (Paper Explained)
2020-07-16[Classic] Word2Vec: Distributed Representations of Words and Phrases and their Compositionality
2020-07-14[Classic] Deep Residual Learning for Image Recognition (Paper Explained)
2020-07-12I'M TAKING A BREAK... (Channel Update July 2020)
2020-07-11Deep Ensembles: A Loss Landscape Perspective (Paper Explained)
2020-07-10Gradient Origin Networks (Paper Explained w/ Live Coding)
2020-07-09NVAE: A Deep Hierarchical Variational Autoencoder (Paper Explained)
2020-07-08Addendum for Supermasks in Superposition: A Closer Look (Paper Explained)
2020-07-07SupSup: Supermasks in Superposition (Paper Explained)
2020-07-06[Live Machine Learning Research] Plain Self-Ensembles (I actually DISCOVER SOMETHING) - Part 1
2020-07-05SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization (Paper Explained)
2020-07-04Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (Paper Explained)
2020-07-03On the Measure of Intelligence by François Chollet - Part 4: The ARC Challenge (Paper Explained)
2020-07-02BERTology Meets Biology: Interpreting Attention in Protein Language Models (Paper Explained)
2020-07-01GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (Paper Explained)
2020-06-30Object-Centric Learning with Slot Attention (Paper Explained)
2020-06-29Set Distribution Networks: a Generative Model for Sets of Images (Paper Explained)
2020-06-28Context R-CNN: Long Term Temporal Context for Per-Camera Object Detection (Paper Explained)
2020-06-27Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures (Paper Explained)
2020-06-26On the Measure of Intelligence by François Chollet - Part 3: The Math (Paper Explained)


Tags:
deep learning
machine learning
arxiv
explained
neural networks
ai
artificial intelligence
paper
ensemble
pytorch
lightning
cifar10
github
vim
code
cuda
gpu
research
ml
ml research
how to
implement
live coding
python
self
distillation
born again
deep ensembles
cnn
resnet
vgg
torchvision
imagenet