When is Scale Enough?

Channel:
United States Simons Institute for the Theory of Computing
Subscribers:
68,700
Published on ● Video Link: https://www.youtube.com/watch?v=Jpu3kQv39L4


Duration: 1:12:25
1,438 views
38

Ethan Dyer (Google Research, Blueshift Team)
https://simons.berkeley.edu/talks/tutorial-emergent-behaviors-deep-learning
Deep Learning Theory Workshop and Summer School

Abstract: Deep learning continues its march of performance progress as models and datasets are scaled up. This talk will discuss work investigating performance predictability with model, dataset, and compute scale for deep learning in general and large language models in particular. I will review scaling in linear models -- a simple analytic system exhibiting many of the phenomena characteristic of realistic networks. I will also discuss empirical work attempting to investigate what types of problems can practically be solved by scale alone and what types cannot.



Other Videos By Simons Institute for the Theory of Computing


2022-08-24Competitive Analysis of Online Algorithms (Part 1)
2022-08-24Online Reinforcement Learning and Regret
2022-08-24Foundations of RL
2022-08-24Foundations of RL
2022-08-23Fundamentals of Markov Decision Processes
2022-08-23Bridging Stochastic and Adversarial Bandits
2022-08-23Adversarial Bandits: Theory and Algorithms
2022-08-23Stochastic Bandits: Foundations and Current Perspectives
2022-08-22Stochastic Bandits: Foundations and Current Perspectives
2022-08-06Deep Learning in Structural Biology and Protein Design: How, Where, and Why
2022-08-05When is Scale Enough?
2022-08-05Universality of Approximate Message Passing on Semi-random Matrices
2022-08-05A Theoretical Framework of Convolutional Kernels on Image Datasets
2022-08-04Tutorial: Methods from Statistical Physics III
2022-08-04A New Perspective on High-Dimensional Causal Inference
2022-08-04Distribution Shift as Underspecification, and What We Might Do About It
2022-08-04Tutorial: Methods from Statistical Physics II
2022-08-03Tutorial: Methods from Statistical Physics I
2022-08-03Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting
2022-08-03The Devil is in the Tails and Other Stories of Interpolation
2022-08-03Tutorial: Implicit Bias II


Tags:
Simons Institute
theoretical computer science
UC Berkeley
Computer Science
Theory of Computation
Theory of Computing
Deep Learning Theory Workshop and Summer School
Ethan Dyer