Learning Part-Whole Structure by Chunking - More Efficient than Deep Learning!!!
13I met Shuchen at NeurIPS 2022, and she is doing really interesting work on chunking at her lab, Max Planck Institute for Biological Cybernetics! Happy to invite her to give a talk about her work at our discussion session:
The world is complex - how do we learn a compact representation of the world? Shuchen will introduce the Hierarchical Chunking Model (HCM), a cognitive-inspired algorithm that learns nested hierarchies of chunks from sequences. Hierarchical chunking may be the key to creating part-whole structures that we use to perceive the world and applies to data domains, including text, image, audio, and other temporal sequences.
We had a great discussion, and ultimately, the consensus is that chunking may be a phenomenon that is online (that is, learned through experience), and we can use chunking to get the representation needed for semantic meaning later. This is much like Byte Pair Encoding generating tokens, which are then projected in an embedding space in the Transformer Architecture.
Paper: https://openreview.net/pdf?id=LceHl9wKmoQ
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
0:00 Introduction
4:11 Chunking is a widespread learning phenomenon
21:36 Hierarchical Generative Model
24:35 How to generate from hierarchical structure
25:28 Hierarchical Chunking Model
37:16 Link to Digital Signal Processing
44:14 Concerns with Exponential Increase in Chunk Length (no concern - only finitely many chunks possible)
55:07 HCM is better than RNN when sequence contains hierarchy
1:07:32 Forgetting is important - Pre-learned chunks can lead to worse performance if environment is different
1:11:26 HCM is similar to human chunk learning
1:15:15 Learning part-whole relationships
1:27:33 Summary
1:29:20 Start of discussion
1:37:35 Link to Hinton’s Part Whole Hierarchy and Link to LeCun’s Self-Supervised Learning
1:40:16 Link to Byte-Pair Encoding
1:43:38 Chunking as representation for more efficient deep learning models
1:51:24 What is the most basic unit for chunking?
2:02:10 Detailed discussion on linkage to Byte-Pair Encoding
2:14:13 Online Adaptive HCM!!
2:32:15 Concluding Remarks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator.
Discord: https://discord.gg/fXCZCPYs
LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/
Online AI blog: https://delvingintotech.wordpress.com/.
Twitter: https://twitter.com/johntanchongmin
Try out my games here: https://simmer.io/@chongmin