LLM Agents as a System (Prelim Findings Sharing): An Attempt to solve a 2-player 2D Escape Room!
9We started building this system as part of an LLM Agent Hackathon, and the idea is to have collaborative actions taken by LLM Agents to solve an overall task!
Richard shares his conversational framework where the agents converse with one another and with a coach to give feedback to the agents.
I share about my framework which uses a planner to plan macro-level actions for the two agents, and the two agents will then use these macro-level actions as their goal and plan micro-level actions. In order to save inference costs, the micro-level plan is in the form of a list of actions, which will be taken one at a time until completion before querying the planner again for a new goal.
We share tips about what works and what does not, as well as some suggested reason behind them. In short, navigation is not native to LLMs and we need to ground it in the environment space like move (1, 0) rather than just text - the main thing for LLM Agents is that there must be semantic grounding in the action space. The order of prompting also matters and the LLM Agent can be prompted to give more specific outputs if we prompt it in the right sequence for it to make decisions in a step by step (broad to specific) manner.
As improvement, we can consider giving the environment feedback to the agents for planning (like Voyager for Minecraft), and we can also have a better way of feeding context into the LLM, perhaps using context-dependent knowledge graphs.
~~~~~~~~~~~~~~~~~~~~~
Richard Github Repo: https://github.com/richard087/chatterbug
John Github Repo: https://github.com/tanchongmin/strictjson/tree/main/EscapeRoom
Voyager: https://www.youtube.com/watch?v=Y-pgbjTlYgk
Ghost in the MineCraft: https://www.youtube.com/watch?v=_VXOczXIkks
~~~~~~~~~~~~~~~~~~~~~
0:00 Introduction
1:06 Richard’s Work on Conversational Agents with a Coach
45:19 Step on two squares 2D Escape Room
1:13:36 Pick-Place 2D Escape Room
~~~~~~~~~~~~~~~~~~~~~
AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator.
Discord: https://discord.gg/bzp87AHJy5
LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/
Online AI blog: https://delvingintotech.wordpress.com/
Twitter: https://twitter.com/johntanchongmin
Try out my games here: https://simmer.io/@chongmin