CRADLE (Part 2): An AI that can play Red Dead Dedemption 2. Reflection, Memory, Task-based Planning
10CRADLE - An AI that can play Red Dead Redemption 2
Following the days of MineCraft agents like Voyager, Ghost in the Minecraft, JARVIS-1, we have the latest attempt to crack an AAA game, Red Dead Redemption 2, with AI.
It uses GPT-4V to decipher the images of the game, coupled with augmentations like VideoSubFinder to get the subtitles of conversation, GroundingDino to get bounding boxes for objects.
It truly is trying to do something like multiple abstraction spaces for image/video domain, an idea which I truly like.
That, and coupled with procedural memory of skills (via code) and episodic memory of current and past experiences in both long form and summarised form.
It does not do everything perfectly, but it is a great first step at achieving Artificial General Intelligence.
I posit that if we can tackle the image domain well, we would be more than 50% there. Currently, our image processing tools leave much to be desired.
~~~
Part 1 here: https://www.youtube.com/watch?v=MDGsGnvWfKg
Main resources:
CRADLE github: https://github.com/BAAI-Agents/Cradle
CRADLE video: https://www.youtube.com/watch?v=Cx-D708BedY
My slides on CRADLE: https://github.com/tanchongmin/TensorFlow-Implementations/blob/main/Paper_Reviews/CRADLE.pdf
Past Agentic Frameworks (my videos):
Voyager: https://www.youtube.com/watch?v=Y-pgbjTlYgk
Ghost in the MineCraft: https://www.youtube.com/watch?v=_VXOczXIkks
JARVIS-1: https://www.youtube.com/watch?v=JUAec-dAt5c
LLMs as a System to solve the ARC Challenge (mine): https://www.youtube.com/watch?v=sTvonsD5His
Referenced resources for Task-based planning:
TaskGen (my Agentic framework): https://www.youtube.com/watch?v=O_XyTT7QGH4
Chain of Thought (CoT) prompting: https://arxiv.org/abs/2201.11903
Referenced resources for Image Processing:
VideoSubFinder: https://sourceforge.net/projects/videosubfinder/
Grounding DINO: https://arxiv.org/abs/2303.05499
Multi-template Matching (MTM): https://pyimagesearch.com/2021/03/29/multi-template-matching-with-opencv/
~~~
0:00 Introduction
16:07 CRADLE workflow
31:45 Reasoning and Planning Overview
33:37 Reasoning and Planning Module
37:02 Task Inference
49:08 Priors for Action Space
53:35 Learning Skills from in-game prompts
59:49 Action Planning
1:07:20 Memory
1:23:03 Why is it so hard?
1:28:47 Is RDR2 hard for decision making?
1:34:27 Overall Thoughts
1:38:13 Discussion
~~~
AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator.
Discord: https://discord.gg/bzp87AHJy5
LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/
Online AI blog: https://delvingintotech.wordpress.com/
Twitter: https://twitter.com/johntanchongmin
Try out my games here: https://simmer.io/@chongmin
Other Videos By John Tan Chong Min
Other Statistics
Cradle Statistics For John Tan Chong Min
There are 260 views in 1 video for Cradle. About 2 hours worth of Cradle videos were uploaded to his channel, less than 0.64% of the total video content that John Tan Chong Min has uploaded to YouTube.