LLMs as a system to solve the Abstraction and Reasoning Corpus (ARC) Challenge!
12Following my earlier experiments on the zero-shot learning capabilities of GPT4 to solve the ARC challenge, I refine it into a code-grounded mechanism with conditional code generation and LLMs in a system in order to solve more ARC tasks. The system involves grounding GPT in successful examples previously, and having broad generic high-level functions which imbue human priors for GPT to call. We also provide an abstracted input view in the form of objects, sections (work in progress), so that GPT can better associate the input to output.
The grounding of GPT4 is very important, and here, we ground it in a broad to specific way, from the broad generation of overall pattern to the specific steps of the Python code. We also provide some "hints" or chain of thought reasoning process to ask GPT4 to highlight differences between input and output in pixel, section, object view, mimicing a mixture of experts discussing with each other to get the right approach. This helps to filter the solution space greatly.
Furthermore, we incorporate a Voyager / Ghost in the Minecraft like approach and do recursive code generation / action function generation via feedback from the environment (whether training set is solved and what is the generated output of the code versus the actual output). This enables GPT4 to better reflect on its output and hopefully solve the problem iteratively.
There is still a long way to go, but I am certain this is a step in the right direction.
~~~~~~~~~~~~~
Previous video on ARC: https://www.youtube.com/watch?v=vt2yG1da8Fg
Slides: https://github.com/tanchongmin/TensorFlow-Implementations/blob/main/Paper_Reviews/Solving%20the%20ARC%20Challenge.pdf
Latest Jupyter Notebook (so you can try yourself): https://github.com/tanchongmin/ARC-Challenge
Quick Summary of the Presentation (very brief, for details, please refer to the video until I write the details up on another arxiv paper): https://github.com/tanchongmin/TensorFlow-Implementations/blob/main/Paper_Reviews/LLMs%20as%20a%20System%20to%20solve%20the%20ARC%20Challenge.pdf
Paper (Initial views, not the most recent one): https://arxiv.org/abs/2306.03553
Language-complete ARC (LARC): https://arxiv.org/pdf/2106.07824.pdf
Michael Hodel's Domain-Specific Language: https://github.com/michaelhodel/arc-dsl
LLMs and Object View: https://arxiv.org/abs/2305.18354
ARCathon: https://lab42.global/arcathon/
ARC Playground: https://arc-editor.lab42.global/playground
On The Measure of Intelligence: https://arxiv.org/abs/1911.01547
~~~~~~~~~~~~~
0:00 Introduction
3:30 Traditional Methods and why they fail
5:21 Transformers as a new way ahead
13:05 Language ARC
15:26 Initial Ideas
30:55 Limitations of purely text-based instructions
37:41 Solution: Limited Context Length
39:56 Solution: Object View
45:13 Solution: Use Code as Fact Checking Tool
51:39 Imbuing Flexibility: Conditional Code
59:54 Conditional Code is more efficient than sequence of Unary Functions
1:05:53 Refined Broad to Specific Prompt
1:16:25 Voyager-like Recursive Environment Loop
1:18:08 Overall LLM as a System approach to solve ARC
1:20:38 Live Demo
1:36:12 Reflections on the demo
1:39:09 Further Ideas for Improvement
1:44:43 Discussion and Conclusion
~~~~~~~~~~~~~~
AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator.
Discord: https://discord.gg/bzp87AHJy5
LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/
Online AI blog: https://delvingintotech.wordpress.com/
Twitter: https://twitter.com/johntanchongmin
Try out my games here: https://simmer.io/@chongmin