A Survey of Singular Learning | AISC

Channel:
Canada LLMs Explained - Aggregate Intellect - AI.SCIENCE
Subscribers:
22,300
Published on ● Video Link: https://www.youtube.com/watch?v=ADaxmxQ4hm4


Duration: 1:47:02
598 views
19

For slides and more information on the paper, visit https://aisc.ai.science/events/2019-09-09

Discussion lead: Mehdi Garrousian


Motivation:
Singular Learning

This session is a survey of results from the works of Sumio Watanabe [1] on using resolution of singularity techniques from nonlinear algebra to improve learning and model selection when the Fisher information matrix of the learning machine is singular. This happens to be almost always the case!

The notion of singularity in mathematics refers to the points on an algebraic manifold where the tangent space is ill-behaved. We shall see that singularities make the learning process more challenging by substantially worsening the bias-variance tradeoff and lacking the desired convergence properties regardless of the number of training examples.

The Fisher information matrix is the Hessian of the KL-distance (loss function) at the true parameter. We follow [2] to take a closer look at how singularities are manifest in practice by examining the spectrum of the eigenvalues of the loss function for some typical neural network examples.

[1] Almost All Learning Machines are Singular, Sumio Watanabe
[2] Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond, Levent Sagun, Leon Bottou, Yann LeCun



Other Videos By LLMs Explained - Aggregate Intellect - AI.SCIENCE


2019-09-26EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
2019-09-26[Gated-SCNN] Gated Shape CNNs for Semantic Segmentation
2019-09-25MLOps: Overview of Machine Learning Operations on the Cloud | AISC
2019-09-24Lookahead Optimizer: k steps forward, 1 step back
2019-09-24Similarity of neural network representations revisited
2019-09-23Detecting Customer Complaint Escalation w/ Recurrent Neural Networks & Manually-Engineered Features
2019-09-23Graph Normalizing Flows
2019-09-23CNN Architectures for Large-Scale Audio Classification | AISC
2019-09-222019 AI Squared Forum Paper Track | AISC
2019-09-16Making of a conversational agent platform | AISC
2019-09-09A Survey of Singular Learning | AISC
2019-09-04Overview of Reinforcement Learning | AISC
2019-09-03Ernie 2.0: A Continual Pre-Training Framework for Language Understanding | AISC
2019-08-28Consistency by Agreement in Zero-shot Neural Machine Translation | AISC
2019-08-26TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing | AISC
2019-08-21Science of science: Identifying Fundamental Drivers of Science | AISC
2019-08-19AI Product Stream Meet and Greet | AISC
2019-08-12[Original ResNet paper] Deep Residual Learning for Image Recognition | AISC
2019-08-11[GAT] Graph Attention Networks | AISC Foundational
2019-08-06XLNet: Generalized Autoregressive Pretraining for Language Understanding | AISC
2019-07-31Overview of Generative Adversarial Networks | AISC