Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates | TDLS

Channel:
Canada LLMs Explained - Aggregate Intellect - AI.SCIENCE
Subscribers:
22,300
Published on ● Video Link: https://www.youtube.com/watch?v=nBE_ClJzYEM


Duration: 1:24:13
991 views
17

Toronto Deep Learning Series, 27 August 2018

For slides and more information, visit https://tdls.a-i.science/events/2018-08-27/

Paper Review: https://arxiv.org/abs/1611.01505

Speaker: https://www.linkedin.com/in/dave-macdonald-9934154/
Organizer: https://www.linkedin.com/in/amirfz/

Host: Statflo

Paper abstract:
Adaptive gradient methods for stochastic optimization adjust the learning rate for each parameter locally. However, there is also a global learning rate which must be tuned in order to get the best performance. In this paper, we present a new algorithm that adapts the learning rate locally for each parameter separately, and also globally for all parameters together. Specifically, we modify Adam, a popular method for training deep learning models, with a coefficient that captures properties of the objective function. Empirically, we show that our method, which we call Eve, outperforms Adam and other popular methods in training deep neural networks, like convolutional neural networks for image classification, and recurrent neural networks for language tasks.



Other Videos By LLMs Explained - Aggregate Intellect - AI.SCIENCE


2018-11-15(Original Paper) Latent Dirichlet Allocation (algorithm) | AISC Foundational
2018-10-31[Transformer] Attention Is All You Need | AISC Foundational
2018-10-25[Original attention] Neural Machine Translation by Jointly Learning to Align and Translate | AISC
2018-10-16[StackGAN++] Realistic Image Synthesis with Stacked Generative Adversarial Networks | AISC
2018-10-11Bayesian Deep Learning on a Quantum Computer | TDLS Author Speaking
2018-10-02Prediction of Cardiac arrest from physiological signals in the pediatric ICU | TDLS Author Speaking
2018-09-24Junction Tree Variational Autoencoder for Molecular Graph Generation | TDLS
2018-09-19Reconstructing quantum states with generative models | TDLS Author Speaking
2018-09-13All-optical machine learning using diffractive deep neural networks | TDLS
2018-09-05Recurrent Models of Visual Attention | TDLS
2018-08-28Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates | TDLS
2018-08-20TDLS: Large-Scale Unsupervised Deep Representation Learning for Brain Structure
2018-08-14Principles of Riemannian Geometry in Neural Networks | TDLS
2018-08-07Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond | TDLS
2018-07-30Program Language Translation Using a Grammar-Driven Tree-to-Tree Model | TDLS
2018-07-23Explainable Neural Networks based on Additive Index Models | TDLS
2018-07-18TMLS2018 - Machine Learning in Production, Panel Discussion
2018-07-16Flexible Neural Representation for Physics Prediction | AISC Trending Paper
2018-07-10Connectionist Temporal Classification, Labelling Unsegmented Sequence Data with RNN | TDLS
2018-06-25Learning to Represent Programs with Graphs | TDLS
2018-06-19Quantum generative adversarial networks | TDLS Author Speaking


Tags:
deep learning
adam optimizer
eve optimizer
sgd