Everything about LLM Agents - Chain of Thought, Reflection, Tool Use, Memory, Multi-Agent Framework

Channel:
Singapore John Tan Chong Min
Subscribers:
5,330
Published on ● Video Link: https://www.youtube.com/watch?v=Ll7lRBaP378


Duration: 1:55:31
1,115 views
42

How do LLM Agents work?

How does a language model understand the world, and know how to use tools/plugins/APIs?

How can we use LLMs as a System for more complicated tasks?

If you seek to find out the answers to these, this session is for you!

https://www.youtube.com/watch?v=Ll7lRBaP378

~~~~~~~~~~~~~~~~

Slides: https://github.com/tanchongmin/TensorFlow-Implementations/blob/main/Paper_Reviews/Introduction%20to%20Agents.pdf

My own referenced research:
Learning, Fast and Slow: https://www.youtube.com/watch?v=Hr9zW7Usb7I
LLMs as a System for the ARC Challenge: https://www.youtube.com/watch?v=plVRxP8hQHY

My own referenced framework:
StrictJSON: https://www.youtube.com/watch?v=A6sIh-lmApk

Reference Papers:
Planning:
ReAct: https://arxiv.org/abs/2210.03629
Reflexion: https://arxiv.org/abs/2303.11366
SayCan: https://say-can.github.io/

Tool Usage:
Visual ChatGPT: https://arxiv.org/abs/2303.04671
HuggingGPT: https://arxiv.org/abs/2303.17580
Voyager: https://arxiv.org/abs/2305.16291
Ghost in the MineCraft: https://arxiv.org/abs/2305.17144

Memory:
Retrieval Augmented Generation: https://proceedings.neurips.cc/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf
Recitation Augmented Generation (change the retrieved memory according to hints): https://arxiv.org/abs/2210.01296
Knowledge Graph as JSON - Generative Agents: Interactive Simulacra: https://arxiv.org/abs/2304.03442
Pyschology - Eyewitness Testimony (Loftus et al, 1975) - How memory retrieval is influenced by wording: https://link.springer.com/content/pdf/10.3758/BF03336715.pdf

Multi-agent:
AutoGPT: https://github.com/Significant-Gravitas/AutoGPT
BabyAGI: https://github.com/yoheinakajima/babyagi
Camel - Society of Minds: https://arxiv.org/abs/2303.17760
ChatDev - Sequential Product Development using Camel: https://arxiv.org/abs/2307.07924

My relevant videos on LLMs:
How ChatGPT works: https://www.youtube.com/watch?v=wA8rjKueB3Q
SayCan: https://www.youtube.com/watch?v=iS3ikfSsp6Y
OpenAI Vector Embeddings: https://www.youtube.com/watch?v=lIoLCip0HwM
Generative Agents: Interactive Simulacra: https://www.youtube.com/watch?v=_pkktFIcZRo
Voyager: https://www.youtube.com/watch?v=Y-pgbjTlYgk
Ghost in the MineCraft: https://www.youtube.com/watch?v=_VXOczXIkks
LLMs and Knowledge Graphs: https://www.youtube.com/watch?v=1RZ5yIyz31c
LLM Agents as a System to solve a 2D Escape Room: https://www.youtube.com/watch?v=OQeBg7-pME4

~~~~~~~~~~~~~~~~

0:00 Introduction
0:38 Story of an Agent
30:40 What are agents?
33:52 Chain of Thought to various levels of Abstractions
39:36 Incorporating World Feedback - ReAct and Reflexion
46:36 Voyager - Iterative Prompting with World Feedback
50:36 Tool Usage
1:03:30 Tool Learning and Composing
1:07:52 Memory
1:26:11 Multi-agent systems
1:38:04 Challenges of Implementing Agents
1:48:30 Discussion

~~~~~~~~~~~~~~~~

AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator.

Discord: https://discord.gg/bzp87AHJy5
LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/
Online AI blog: https://delvingintotech.wordpress.com/
Twitter: https://twitter.com/johntanchongmin
Try out my games here: https://simmer.io/@chongmin



Other Videos By John Tan Chong Min


2023-11-07Learning, Fast and Slow: My Landmark Idea for fast, adaptable agents (ICDL 2023 Best Paper Finalist)
2023-11-06A roadmap for AI: Past, Present and Future (Part 2): Fixed vs Flexible, Memory Soup vs Hierarchy
2023-11-03AI & Education: Education when AI tools are smarter than us - Discussion with Kuang Wen (Part 2)
2023-11-03AI & Education: RAG Question-Answer, Test Question Generator, Autograder by Kuang Wen! (Part 1)
2023-10-31A Roadmap for AI: Past, Present and Future (Part 1)
2023-10-28Tutorial #10: StrictJSON v2 (StrictText): Handle any output - quotation marks or backslash!
2023-10-24ChatDev: Can LLM Agents really replace a software company?
2023-10-17LLMs and Robotics: An Overview by Daniel Tan!
2023-10-17LLM Q&A #1: Prompting vs Fine-Tuning, More vs Fewer Sources for RAG, Prompting vs LLMs as a System
2023-10-10LLMs as a System of Multiple Expert Agents to solve the ARC Challenge (Detailed Walkthrough)
2023-09-26Everything about LLM Agents - Chain of Thought, Reflection, Tool Use, Memory, Multi-Agent Framework
2023-09-19Moving Beyond Probabilities: Memory as World Modelling
2023-09-05Symbolic Regression: Doing What LLMs cannot - Deriving Arbitrary Mathematical Relations!
2023-08-29LLM Agents as a System (Prelim Findings Sharing): An Attempt to solve a 2-player 2D Escape Room!
2023-08-23LLM as Pattern Machines(Part 2) - Goal Directed Decision Transformers, 10-Year Plan for Intelligence
2023-08-18Tutorial #9: Evolution Game v2: ChatGPT (Text) and Dall-E (Image) API Integration!
2023-08-17Tutorial #8: Create a Web Scraper using ChatGPT and Selenium!
2023-08-17Tutorial #7: Create a Chatbot with Gradio and ChatGPT!
2023-08-15LLMs as General Pattern Machines: Use Arbitrary Tokens to Pattern Match?
2023-08-08Tutorial #6: LangChain & StrictJSON Implementation of Knowledge Graph Question Answer with LLMs
2023-08-08Large Language Models and Knowledge Graphs: Merging Flexibility and Structure