Deep Ensembles: A Loss Landscape Perspective (Paper Explained)
848#ai #research #optimization
Deep Ensembles work surprisingly well for improving the generalization capabilities of deep neural networks. Surprisingly, they outperform Bayesian Networks, which are - in theory - doing the same thing. This paper investigates how Deep Ensembles are especially suited to capturing the non-convex loss landscape of neural networks.
OUTLINE:
0:00 - Intro & Overview
2:05 - Deep Ensembles
4:15 - The Solution Space of Deep Networks
7:30 - Bayesian Models
9:00 - The Ensemble Effect
10:25 - Experiment Setup
11:30 - Solution Equality While Training
19:40 - Tracking Multiple Trajectories
21:20 - Similarity of Independent Solutions
24:10 - Comparison to Baselines
30:10 - Weight Space Cross-Sections
35:55 - Diversity vs Accuracy
41:00 - Comparing Ensembling Methods
44:55 - Conclusion & Comments
Paper: https://arxiv.org/abs/1912.02757
Abstract:
Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable variational Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space. Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods. Finally, we evaluate the relative effects of ensembling, subspace based methods and ensembles of subspace based methods, and the experimental results validate our hypothesis.
Authors: Stanislav Fort, Huiyi Hu, Balaji Lakshminarayanan
Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher