How Visual ChatGPT works + Toolformer/Wolfram Alpha. LLMs with Tools/APIs/Plugins is the way ahead!

Channel:
Singapore John Tan Chong Min
Subscribers:
5,330
Published on ● Video Link: https://www.youtube.com/watch?v=J1Xj0xXmtHU


Game:
Wolfram (2012)
Category:
Let's Play
Duration: 1:54:53
1,842 views
40

ChatGPT has been giving us very human-like responses, and has proven to be a user friendly interface to talk to. What if we can bring it to the next level by integrating it with plugins/tools/APIs? These tools can enable ChatGPT to communicate with applications, retrieve data from the web, buy your favourite meal from your favourite store, play chess using AlphaZero, send an email to your boss and many more.

We will go through how tools can be integrated in a few-shot manner using the Toolformer Input-Output example approach, in a zero-shot manner using the Visual ChatGPT/OpenAI plugin description approach. We also illustrate some failure cases (based on my own experimentation) of failing to call the right tool, or calling the tool with the wrong input. These failure cases reflect some of the weaknesses of ChatGPT to understand complex rules, such as finding words with no vowels, or comparing numbers against a certain threshold.

Overall, ChatGPT with plugins/tools/APIs is very promising, and the tools help to mitigate some of the flaws of ChatGPT. If infused with my "Learning, Fast & Slow"-style architecture, it can even be adaptive to the environment! There is a lot of promise for this approach, and I believe we are not too far from creating Jarvis from Iron Man.

~~~~~

Slides: https://github.com/tanchongmin/TensorFlow-Implementations/blob/main/Paper_Reviews/LLMs%20as%20API%20Interface.pdf

Related videos:
Part 2: https://www.youtube.com/watch?v=wLjJ34ygZVc
How ChatGPT works: https://www.youtube.com/watch?v=wA8rjKueB3Q
Learning, Fast and Slow (Adaptive learning): https://www.youtube.com/watch?v=Hr9zW7Usb7I

References:
Toolformer Paper: https://arxiv.org/abs/2302.04761
Visual ChatGPT Code + Paper: https://github.com/microsoft/visual-chatgpt
Wolfram Alpha Plugin announcement from Stephen Wolfram: https://writings.stephenwolfram.com/2023/03/chatgpt-gets-its-wolfram-superpowers/
Emergence properties of LLM paper: https://arxiv.org/abs/2206.07682
ReAct paper: https://arxiv.org/abs/2210.03629
LangChain documentation: https://python.langchain.com/en/latest/
OpenAI API Key: https://platform.openai.com/account/api-keys
OpenAI Plugin Documentation: https://platform.openai.com/docs/plugins/introduction

~~~~~

0:00 Introduction
6:38 LLMs vs Tools
11:15 What’s possible
35:03 What APIs can be interfaced
39:49 Neuro-symbolic AI
41:50 Make LLMs adaptive - Learning, Fast and Slow
44:30 Toolformer
52:38 Visual ChatGPT
58:53 ReAct for Chain of Thought prompting
1:04:53 Overall workflow of API calling
1:20:53 Code walkthrough
1:29:00 Live Demo of Visual ChatGPT
1:43:50 Limitations of LLMs + Tools
1:48:50 Failure cases for calling tools
1:53:13 Conclusion

~~~~~~

AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator.

Discord: https://discord.gg/fXCZCPYs
LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/
Online AI blog: https://delvingintotech.wordpress.com/.
Twitter: https://twitter.com/johntanchongmin
Try out my games here: https://simmer.io/@chongmin



Other Videos By John Tan Chong Min


2023-05-02I created a Law Court Simulator with ChatGPT!
2023-04-25Creating a ChatGPT Harry Potter Text-based RPG game!
2023-04-25Learn from just Memory Storage and Retrieval: Generative Agents Interacting in Simulation!
2023-04-18The future is neuro-symbolic: Expressiveness of ChatGPT and generalizability of symbols (SymbolicAI)
2023-04-17Can GPT4 solve the Abstraction and Reasoning Corpus (ARC) Challenge Zero-Shot?
2023-04-12GPT4: Zero-shot Classification without any examples + Fine-tune with reflection
2023-04-11OpenAI Vector Embeddings - Talk to any book or document; Retrieval-Augmented Generation!
2023-04-11Tutorial #2: OpenAI Vector Embeddings and Pinecone for Retrieval-Augmented Generation
2023-04-04Creating JARVIS: ChatGPT + APIs - HuggingGPT, Memory-Augmented Context, Meta GPT structures
2023-04-02Is GPT4 capable of self-improving? Are we heading for AGI or AI doom?
2023-03-28How Visual ChatGPT works + Toolformer/Wolfram Alpha. LLMs with Tools/APIs/Plugins is the way ahead!
2023-03-21Tokenize any input, even continuous vectors! - Residual Vector Quantization - VALL-E (Part 2)
2023-03-07Using Transformers to mimic anyone's voice! - VALL-E (Part 1)
2023-02-28Learning Part-Whole Structure by Chunking - More Efficient than Deep Learning!!!
2023-02-21High-level planning with large language models - SayCan
2023-02-13Learning, Fast and Slow: Towards Fast and Adaptable Agents in Changing Environments
2023-02-07Using Logic Gates as Neurons - Deep Differentiable Logic Gate Networks!
2023-01-31Learn from External Memory, not just Weights: Large-Scale Retrieval for Reinforcement Learning
2023-01-17How ChatGPT works - From Transformers to Reinforcement Learning with Human Feedback (RLHF)
2023-01-09HyperTree Proof Search - Automated Theorem Proving with AlphaZero and Transformers!
2022-12-23CodinGame Fall Challenge 2022: A First Look (managed to get to Silver!)


Other Statistics

Wolfram Statistics For John Tan Chong Min

There are 1,842 views in 1 video for Wolfram. About an hours worth of Wolfram videos were uploaded to his channel, less than 0.61% of the total video content that John Tan Chong Min has uploaded to YouTube.