Creating JARVIS: ChatGPT + APIs - HuggingGPT, Memory-Augmented Context, Meta GPT structures

Channel:
Singapore John Tan Chong Min
Subscribers:
5,330
Published on ● Video Link: https://www.youtube.com/watch?v=wLjJ34ygZVc


Duration: 1:35:17
1,847 views
40

This is Part 2 of the LLMs with Tools/Plugins/API discussion session. In this one crazy week of AI, we already have TaskMatrix.AI, which can link millions of APIs with one GPT model, and HuggingGPT, an interface of GPT with multiple HuggingFace models. HuggingGPT is particularly interesting, as it has a memory-augmented retrieval of various input/output examples to form a broad plan. TaskMatrix.AI has a mechanism that enables the API to learn from feedback.

We then see how GPTs can be linked to a greater ecosystem in a hierarchical fashion, such as Prompt Manager and GPT, and recurrent fashion, such as Socratic Models.

I believe with an increase in the APIs used, we will soon reach the token limit of in-context prompting, and increasing the context length will be of paramount importance. We discuss Memformer, Memorizing Transformer, and a task input/output memory storage and retrieval system, and whether they can solve the problem.

LLMs with tools will be the next step forward. Whether it can achieve AGI remains to be seen. However, I do believe that multiple models of GPT in a multi-agent format might be able to achieve more than the sum of its parts, and can become a very powerful system. Watch this space.

~~~~~~~~~~~~~~~~

Slides: https://github.com/tanchongmin/TensorFlow-Implementations/blob/main/Paper_Reviews/LLMs%20as%20API%20Interface.pdf

Related videos:
Part 1 of LLM and Tools (Toolformer, Visual ChatGPT, Wolfram Alpha Plugin): https://www.youtube.com/watch?v=J1Xj0xXmtHU
How ChatGPT works: https://www.youtube.com/watch?v=wA8rjKueB3Q
Learning, Fast and Slow (Adaptive learning): https://www.youtube.com/watch?v=Hr9zW7Usb7I

Reference Materials:
TaskMatrix.AI: https://github.com/microsoft/visual-chatgpt/tree/main/TaskMatrix.AI
HuggingGPT (aka JARVIS): https://github.com/microsoft/JARVIS
Lottery Ticket Hypothesis: https://arxiv.org/abs/1803.03635
Neural Darwinism: https://en.wikipedia.org/wiki/Neural_Darwinism
Reflexion: https://arxiv.org/abs/2303.11366
Socratic Models: https://arxiv.org/abs/2204.00598
Memformer: https://aclanthology.org/2022.findings-aacl.29/
Memorizing Transformers: https://arxiv.org/abs/2203.08913
Task-driven Autonomous Agent Utilizing GPT-4, Pinecone, and LangChain for Diverse Applications: https://yoheinakajima.com/task-driven-autonomous-agent-utilizing-gpt-4-pinecone-and-langchain-for-diverse-applications/

~~~~~~~~~~~~~~~~~~

0:00 Introduction
0:22 TaskMatrix.AI
10:02 HuggingGPT (aka JARVIS)
24:57 Prompts for HuggingGPT
31:18 HuggingGPT Input to Output Flowchart
41:56 Emergence via LLMs connected hierarchically or recurrently
1:03:12 Memory
1:11:51 Discussion
1:34:16 Conclusion

~~~~~~~~~~~~~~~~~~~

AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator.

Discord: https://discord.gg/fXCZCPYs
LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/
Online AI blog: https://delvingintotech.wordpress.com/.
Twitter: https://twitter.com/johntanchongmin
Try out my games here: https://simmer.io/@chongmin



Other Videos By John Tan Chong Min


2023-05-09Memory: How is it encoded, retrieved and how it can be used for learning systems
2023-05-02I created a Law Court Simulator with GPT4!
2023-05-02I created a Law Court Simulator with ChatGPT!
2023-04-25Creating a ChatGPT Harry Potter Text-based RPG game!
2023-04-25Learn from just Memory Storage and Retrieval: Generative Agents Interacting in Simulation!
2023-04-18The future is neuro-symbolic: Expressiveness of ChatGPT and generalizability of symbols (SymbolicAI)
2023-04-17Can GPT4 solve the Abstraction and Reasoning Corpus (ARC) Challenge Zero-Shot?
2023-04-12GPT4: Zero-shot Classification without any examples + Fine-tune with reflection
2023-04-11OpenAI Vector Embeddings - Talk to any book or document; Retrieval-Augmented Generation!
2023-04-11Tutorial #2: OpenAI Vector Embeddings and Pinecone for Retrieval-Augmented Generation
2023-04-04Creating JARVIS: ChatGPT + APIs - HuggingGPT, Memory-Augmented Context, Meta GPT structures
2023-04-02Is GPT4 capable of self-improving? Are we heading for AGI or AI doom?
2023-03-28How Visual ChatGPT works + Toolformer/Wolfram Alpha. LLMs with Tools/APIs/Plugins is the way ahead!
2023-03-21Tokenize any input, even continuous vectors! - Residual Vector Quantization - VALL-E (Part 2)
2023-03-07Using Transformers to mimic anyone's voice! - VALL-E (Part 1)
2023-02-28Learning Part-Whole Structure by Chunking - More Efficient than Deep Learning!!!
2023-02-21High-level planning with large language models - SayCan
2023-02-13Learning, Fast and Slow: Towards Fast and Adaptable Agents in Changing Environments
2023-02-07Using Logic Gates as Neurons - Deep Differentiable Logic Gate Networks!
2023-01-31Learn from External Memory, not just Weights: Large-Scale Retrieval for Reinforcement Learning
2023-01-17How ChatGPT works - From Transformers to Reinforcement Learning with Human Feedback (RLHF)