Memory: How is it encoded, retrieved and how it can be used for learning systems
9How should memory be encoded and retrieved for learning systems? I provide some of my recent thoughts on this matter, and also go through some of the proposed ways to extend Transformer context length.
There are various new and interesting ways to store memory, most notably, using text itself. It comes with drawbacks of verbose and lengthy storage, but it can be useful for human interpretation.
While I am still thinking of the final answer for how memory should be stored, I am increasingly inclined to think it must consist of an initial reference point, and a movement. Such a construct can be generalizable across reference frames, and can be used hierarchically as well.
Related Past Videos:
A New Framework of Memory for Learning: https://www.youtube.com/watch?v=q9uMEAcB3lM
Learning, Fast & Slow: https://www.youtube.com/watch?v=Hr9zW7Usb7I
Generative Agents: https://www.youtube.com/watch?v=_pkktFIcZRo
OpenAI Vector Embeddings: https://www.youtube.com/watch?v=lIoLCip0HwM
ARC Challenge with GPT4: https://www.youtube.com/watch?v=vt2yG1da8Fg
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Reference Materials:
Slides: https://github.com/tanchongmin/TensorFlow-Implementations/blob/main/Paper_Reviews/Memory%20Systems.pdf
Eric Kandel - In Search of Memory: http://evolbiol.ru/docs/docs/large_files/kandel.pdf
Memformer: https://aclanthology.org/2022.findings-aacl.29/
Recurrent Memory Transformer: https://arxiv.org/abs/2207.06881
Scaling Transformers to 1M tokens and beyond with RMT: https://arxiv.org/abs/2304.11062
Memorizing Transformers: https://arxiv.org/abs/2203.08913
Generative Agents: https://arxiv.org/abs/2304.03442
Baby AGI: https://github.com/yoheinakajima/babyagi
Recitation-Augmented Language Models: https://arxiv.org/abs/2210.01296
Lab42 ARC Essay Challenge: https://lab42.global/essay-arc/
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
0:00 Introduction
1:33 Narrow view of memory (synapse level)
3:03 Memory as a network
6:58 Vector Representation of Memory
11:46 How to generalize from memory
17:37 Abstraction Space for Memory
26:42 Biological and non-biological memory
31:45 Learning, Fast and Slow
34:47 Memory in vector embeddings
35:20 Memformer
43:06 Recurrent Memory Transformer
48:07 Memorizing Transformer
54:26 Hierarchical Memory Referencing in Embeddings
59:42 Text-based Memory
1:09:55 Baby AGI
1:14:48 Recitation-Augmented Generation
1:20:44 Hierarchical Memory Referencing in Text (ARC Challenge Essay)
1:23:42 Discussion
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator.
Discord: https://discord.gg/bzp87AHJy5
LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/
Online AI blog: https://delvingintotech.wordpress.com/
Twitter: https://twitter.com/johntanchongmin
Try out my games here: https://simmer.io/@chongmin