Do ImageNet Classifiers Generalize to ImageNet? (Paper Explained)

Channel:
Switzerland Yannic Kilcher
Subscribers:
284,000
Published on ● Video Link: https://www.youtube.com/watch?v=fvctpYph8Pc


Duration: 25:37
20,052 views
381

Has the world overfitted to ImageNet? What if we collect another dataset in exactly the same fashion? This paper gives a surprising answer!

Paper: https://arxiv.org/abs/1902.10811
Data: https://github.com/modestyachts/ImageNetV2

Abstract:
We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.

Authors: Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher



Other Videos By Yannic Kilcher


2020-05-07WHO ARE YOU? 10k Subscribers Special (w/ Channel Analytics)
2020-05-06Reinforcement Learning with Augmented Data (Paper Explained)
2020-05-05TAPAS: Weakly Supervised Table Parsing via Pre-training (Paper Explained)
2020-05-04Chip Placement with Deep Reinforcement Learning (Paper Explained)
2020-05-03I talk to the new Facebook Blender Chatbot
2020-05-02Jukebox: A Generative Model for Music (Paper Explained)
2020-05-01[ML Coding Tips] Separate Computation & Plotting using locals
2020-04-30The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies (Paper Explained)
2020-04-29Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask (Paper Explained)
2020-04-28[Rant] Online Conferences
2020-04-27Do ImageNet Classifiers Generalize to ImageNet? (Paper Explained)
2020-04-26[Drama] Schmidhuber: Critique of Honda Prize for Dr. Hinton
2020-04-25How much memory does Longformer use?
2020-04-24Supervised Contrastive Learning
2020-04-23Thinking While Moving: Deep Reinforcement Learning with Concurrent Control
2020-04-22[Rant] The Male Only History of Deep Learning
2020-04-21Gradient Surgery for Multi-Task Learning
2020-04-20Longformer: The Long-Document Transformer
2020-04-20Backpropagation and the brain
2020-04-18Shortcut Learning in Deep Neural Networks
2020-04-17Feature Visualization & The OpenAI microscope


Tags:
deep learning
machine learning
imagenet
cifar10
cifar10.1
generalization
overfitting
mturk
arxiv
vision
models
research
hardness
accuracy
classifier
resnet