LLMs as a System of Multiple Expert Agents to solve the ARC Challenge (Detailed Walkthrough)
31Can GPT4 solve the ARC Challenge?
I have been working on this problem for half a year, and I present the LLM Agents as a System approach to solve it!
We utilise various different views to represent the problem - pixel, object, grid, use common primitive functions and conditional functions to ground the action space, incorporate environment feedback like in Voyager.
The result - 50 solves out of 111 training set problems (which can fit in 3000 tokens context length, catering for environment feedback)!
~~~~~~~~~~~~~~~~
Paper: https://arxiv.org/abs/2310.05146
Slides: https://github.com/tanchongmin/TensorFlow-Implementations/blob/main/Paper_Reviews/LLMs%20as%20a%20System%20for%20the%20ARC%20Challenge.pdf
Github: https://github.com/tanchongmin/ARC-Challenge
Earlier ARC Videos:
LLMs as a System to Solve ARC (Part 1): https://www.youtube.com/watch?v=plVRxP8hQHY
Initial Experimentation using GPT4: https://www.youtube.com/watch?v=vt2yG1da8Fg
Language-complete ARC (LARC): https://arxiv.org/pdf/2106.07824.pdf
Michael Hodel's Domain-Specific Language: https://github.com/michaelhodel/arc-dsl
Voyager: https://arxiv.org/abs/2305.16291
LLMs and Object View: https://arxiv.org/abs/2305.18354
ARCathon: https://lab42.global/arcathon/
On The Measure of Intelligence: https://arxiv.org/abs/1911.01547
~~~~~~~~~~~~~~~~
0:00 Introduction
1:50 Methods to solve ARC
4:32 Transformers
19:28 Language ARC
20:40 Seven Steps to Greatness (Solving ARC)
21:10 Step 1: LARC
22:28 Step 2: Chain of Thought
26:53 Step 3: Code Generation
28:49 Step 4: Multiple Abstraction Spaces
40:55 Step 5: Grounding in Action Space
46:41 Step 6: Using Conditionals
50:26 Step 7: Environment Feedback
51:48 Using Memory for Learning
1:03:00 Results
1:05:25 Example Flow using problem d0f5fe59
1:09:21 Visual Results of Solves and Non-Solves
1:17:30 Potential Improvements
1:18:15 Discussion
~~~~~~~~~~~~~~~~
AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator.
Discord: https://discord.gg/bzp87AHJy5
LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/
Online AI blog: https://delvingintotech.wordpress.com/
Twitter: https://twitter.com/johntanchongmin
Try out my games here: https://simmer.io/@chongmin