flowchart LR
agent(Agent)
env((Environment))
env-- sense -->agent
agent-- reason -->agent
agent-- act -->env
10 Agents
Agents are our last stop in the road towards giving language models more autonomy. So far, we’ve been able to connect LLMs with other systems, extending their knowledge and enhancing their capacity to solve complicated problems by leveraging different tools. But all tasks the tasks we can solve so far are, essentially, single-shot: requiring little to no planning once the task is well understood and the right tools are selected. Regardless of how complex the tasks we’ve seen, the number of steps they require is fixed from the beginning.
To handle the most complex tasks, we must give LLMs the ability to not only pick and use arbitrary tools, but to actively reason about the task while performing it, incorporate new insights, and change their minds during the solution process. LLMs must become active agents rather than passive tools. This is what the agent paradigm is supposed to unlock. Agents are not a new thing in AI, or computer science in general, though. In fact, it is one of the most important software design paradigms since primorial ages—back when Turing was thinking about German codes.
In computational terms, a agent is just a software component that acts autonomously, interacting with an (computational) environment and constantly making choices as to further some goals. The difference between traditional software and agent-based software is that, in traditional software, all changes to a system are in response to some user input: that is, the user is the active element. In agent-based software, the software itself is active, in the sense that it is always making adjustments and producing effects on the computational system, regardless of whether there is a user at all.
As an example, consider the difference between a passive AI assistant, like Alexa or Siri, and an active one. In the former case, the assistant is essentially dormant until you make a request. Then, it turns on, parses your query, and takes any actions needed to fullfil your request, before going back to sleep. All interactions are started bu you, the user.
In contrast, an active AI assistant would be always processing stuff, reading the news, scanning your emails, searching in Google, etc. It could forward you any interesting links it finds online and update your calendar as new tasks arrive at your email. But, more importantly, it could be working on a specific task for as long as necessary, without continued input from you.
Now, you can stretch the definition so that any piece of software looks like agent-based. Your operating system is always on, doing stuff, ready to ping you when there’s a problem. It has a goal—keeping the system running—and it can interact with an environment—hardware drivers, peripherals, hard drives, etc.—and exert actions to make changes to the system. But this is seldom a useful way to think about your operating system, mostly because its goals are pretty straightforward, and the actions it can perform as very restricted.
Things turn interesting when you give a software agent long-term goals that have no immediately obvious steps. The agent must then observe the environment, gather information, maybe ask you for clarifications, and make a plan to further those goals. Moreover, as it discovers new information, it may refine those goals, split them into lower-level subgoals, and generally navigate around a complex decision-making process that wasn’t explicitly programmed, or even obvious to an external observer from the beginning.
Adding language models to the mix unlocks exciting new possibilities, as now your agent can not only communicate with you in natural language, but also talk to other agents easily—no need to define formal communication protocols or APIs—, and even reason and argue with itself in natural language.
Anatomy of an LLM-powered agent
An LLM agent is, ultimately, just a plain-old AI agent with some linguistic capabilities. So let’s review the general definition of agent in the AI field, and explore some of the basic agent architectures that have been invented.
In its simplest conception, an agent is a software component embedded within a larger computational system we call the environment. This environment can be anything from a computer simulation, to the whole Internet, including other agents (computational or otherwise). THe agent always has some goals—which can as abstract and ambiguous as you want—that it must further by performing some actions. For this purpose, the agent has some sensors by which it can obtain information from the environment, and some actuators that allow it to exert changes in the environment. Finally, the agent has some internal reasoning process to decide which actions to perform given the current environment state (see Figure 10.1).
This picture is complete from the outside, but if we look inside the agent, we can further refine it. Here is where literature is—quite literally—overflown with variants of “standard” agent architectures. So I won’t attempt to cover the majority of variants or even be too formal. Rather, I’ll explain what are the main ingredients in most used agent architectures, using a casual nomenclature.
The most basic agent that does some form of reasoning needs to do two separate things: keep track of the state of the world (which may include it’s own state) and then reason about that world to decide which actions to take. This reasoning can be cyclical, updating its own state with new information inferred from previous state. We can thus come up with a sensible basic agent architecture (see Figure 10.2).
flowchart TB
subgraph Agent
sensors[Sensors]
state(Internal\nRepresentation\nof World State)
reason[Reasoning\nModule]
actuators[Actuators]
end
subgraph Environment
env((World State))
end
env-- perceive -->sensors
sensors -- update -->state
state -- inform -->reason
reason -- update -->state
reason -- choose -->actuators
actuators -- execute -->env
classDef white fill:#fff
class Agent white
class Environment white
This architecture is sufficiently detailed for us to discuss different variants, while being sufficiently abstract to allow many different use cases and implementations. For example, we haven’t explicitely defined what goes in the internal representation of the world: it could be anything from a structured set of values to a list of natural language facts to a full-blown database. We also haven’t specified what sensors or actuators actually are, which depend heavily on the concrete use case.
Adding natural language
So far we haven’t used the fact our agent is LLM-powered, and for good reason. The basic agent architecture we just defined is agnostic to implementation details. Adding an LLM to the agent architecture just means some, or all of the modules are implemented using language models: that is, with prompts, RAG, function calling, and everything else we’ve seen so far.
First, perception may be LLM-based: your agent can receive information from the environment in the form of natural language documents. Likewise, the actuators can be LLM-based, e.g., via function calling. But more importantly, the reasoning process and the internal representation may involve language models as well. The internal representation can be a straightforward collection natural language claims about the environment, which the agent updates and refines via reasoning. And the reasoning itself can be guided with some clever prompting and thus occur entirely in a linguistic framework. Finally, the environment itself can contain other language models, or otherwise be implemented also as a language-based simulation with clever prompts.
Use cases for LLM agents
The most obvious use case for an LLM-based agent is some sort of always-on assistant; your personal Jarvis, if you will. Unfortunately, most AI assistants today are more tools than agents. They turn on after a user command, do their thing, and go back to iddle. All interactions between you and Siri, Cortana, or Google Assistant, are started by you.
Moving towards more agentic behavior, what we should expect is an AI assistant that has long-term goals—e.g., keeping you informed, or keeping your calendar well organized—and is always consuming information and taking actions to further those goals. So here are a few possible, very broad use cases for such a type of agent.
The planning agent
This is one of the most useful and yet unexplored use cases. An agent that is connected to all your input sources: email, work and family chat groups, etc., collecting all information about upcoming events and tasks. Its goal would be to help you maintain a healthy work-life balance by keeping your calendar up to date. Potentially, the agent could reply back to a meeting email and ask for a reschedule if it knows you’ll be busy that day. Now imagine everyone has such an agent, and they all negotiate with themselves and find the optimal schedule for the group meeting.
The research agent
This is an agent whose goal is to answer some complicated, nuanced research question. It can be something related to your work, like mapping a business opportunity, or something more akin to a typical scientific research question. The agent would search online for relevant sources, extract insights, update its internal representation and generate new follow-up questions, building a web of interconnected pieces of information for a given domain.
The coding agent
This is the software engineering holy grail: an agent that continuosly scans and improves an existing codebase, perhaps adding unit tests, comments in code, or refactoring existing functions. Additionally, this agent could pick low-complexity issues (like obvious bug fixes) and attempt to solve them automatically, even crafting a pull request when all test cases pass. Furthermore, it could revise existing pull requests and make suggestions for improvements or general comments before a human revisor takes over the final merge.
The gossip agent
And this is more tongue-in-cheek but, imagine an agent that is monitoring all your social accounts: Facebook (who has that?), Twitter/X, LinkedIn, your friends’ WhatsApp group chats, etc. Everytime some event happens—e.g., someone announces they got a job, or are traveling somewhere, etc.—the agent updates it’s internal representation of your social network. And periodically, or whenever something important changes, it will send you updates on who’s doing what or going where.
Beyond the single agent paradigm
All of the above are reasonably straightforward applications of the agent paradigm using LLMs. While the devil is in the details, and implementation is often much harder than one first thought, I can definitely see all of the above examples working in the near future, at least within reasonable constraints. However, single-agent architectures are just scratching the surface of what this paradigm can bring. The real deal is in multi-agent environments.
In a multi-agent setup, different agents coexist, collaborating and/or competing to achieve common or distinct goals. The power of mulit-agent architectures is that a complex problem may be very hard to model with a single agent, but feasible when subdivided into distinct goals and skills. Take, for example, the coding agent scenario mentioned before. Now, instead of a single agent taking all these tasks one at a time, imagine multiple agents working in parallel, some optimized to fix bugs, others to comment code, and others yet to revise, add tests, make suggestions, etc.
And this is more than just N different agents working in parallel on a codebase. These agents may have to coordinate to solve a big task. For example, maybe one “software architect” agent will take a complex issue and make plan, dividing it into subtasks for different features, and then other “implementation” agents will work out the simpler tasks and do the actual coding.
Furthermore, you can have gatekeeper agents that revise code and emit some sort of qualitative evaluation. They can even ask questions, and the implementors must convince them of their solution. A lot of both old and recent research suggests than, when combined in such a manner, multiple agents can solve problems that none of them was able to tackle on its own.
Why would this work better than just a super coder LLM? You can consider it an advanced form of ensembling mixed with self-reflection, which are in turn advanced forms of chain-of-thought. By combining multiple LLMs with different tasks (i.e., different prompts) in iterated interactions, we give them more chances to revise and regenerate, which translates in more computational power. At the same time, we’re leveraging the ensemble effect of pooling different solutions to the same problem which tends to produce better solutions on average as mistakes are smoothed out.
Symbolic learning in LLM agents
Putting a bunch of LLM-powered agents to work together is neat, but we can go further. The next step is make our agents actually learn from their interactions with the environment.
In the classic agent architecture we’ve discussed so far, there is no explicit learning happening inside the agent. To some extent, you can think of updating the internal representation as some sort of learning about the environment, but there is no learning about the strategies the agent can use. The reasoning module, however is implemented, has no way (at least so far) to update itself so that, given the same environment, it could perform better in future ocassions instead of repeating the same mistakes.
The standard solution to this problem is reinforcement learning. Assume some optimizable inference architecture inside the agent’s reasoning module (e.g., a neural network) and use feedback from the environment to update its parameters. But what if we could achieve something similar using a purely linguistic approach?
One way to achieve this is to give the agent the ability to reason about its own performance, and generate hypotheses (which are nothing more than natural language claims) that describe what it has learned. These hypotheses can be stored in a private knowledge base (separate from the environment description) and retrieved during reasoning.
In the simplest case, this is a long-term memory of what the agent has tried in different situations, and what has worked. In future situations, a RAG approach could provide examples of things that have worked before. However, this can be refined further, by progressively asking the LLM to produce high-level hypotheses that abstract and summarize similar situations, eventually building operative knowledge, more general than any specific past situation.
What’s incredible is this approach has been shown to improve reasoning and problem-solving skills in domains as complicated as medical question answering. In a recent paper, researchers put agents in a simulated hospital environment, and generated thousands of cases of patients with different conditions. The whole simulation is run by linguistic agents, from patients to nurses to doctors, who interact with each other and decide on a treatment. External rules (from a medical knowledge base) are used to simulate if the treatment is effective or not. And after each case is completed, the doctor agents write a detailed report of the case, which is stored in their internal knowledge base.
After ten thousand or so iterations, the doctor agents had built a case database that contained a distilled knowledge about every possible rundown of diseases, symptoms, treatments, etc., they had seen. This is equivalent to what medical residents see in two years of training. And incredibly, this database, when provided to a GPT-4 single-shot question answering bot, improved its results in a standard medical question benchmarks by almost 10%.
This is something akin to reinforcement learning, but it is parameter-free and 100% transparent and explainable. Instead of updating some opaque weights in some neural networks, your LLM-powered agent is learning like humans do: by analysing its actions, taking notes, and constructing hypotheses. While extremely experimental, this is one of the most exciting use cases for LLM agents I’ve seen recently.
Caveats and limitations
As in everything related to LLMs, you must beware of hallucinations and biases. Ensembles in general, and agents in particular are more robust to random errors because of their distributed nature. This means that unlikely hallucinations that could be deal-breakers in single-LLM scenarios might get smoothed away when you pool together multiple LLMs taking turns to make choices.
However, biases are not random but systematic errors. Using the same model to power all your agents means they are all susceptible to making the same judgement mistakes over and over. In general, there is no algorithmic solution to reducing biases that doesn’t attack the original problem: biased data.
Another major caveat of LLM agents is that, compared to a single zero-shot or a simple chain-of-thought prompt, they are significantly more costly. Your agent is always running, which means you’re constantly calling the LLM API, even if there is nothing new to be done. This underscores once again the need for in-house, open-source language models that can be scaled to these heavy-duty scenarios in a cost-effective manner.
And finally, agents in AI have a long history of being abused as an overengineered solution to problems that could otherwise be solved with more direct methods. Do your own experiments and test whether simple prompts, or other, more straightforward augmentation techniques are sufficient for your use case before going the length of building an entire simulated environment.
Conclusions
The agent model is one of the most exciting paradigms in artificial intelligence, with a long and rich history of architectures and approaches. It is also one of the hardest to put into practice, at least until now, because it always required far more intelligence that was possible with previous approaches. But with LLMs, agents are seeing a rebirth as one of the most promising approaches to building reliable and truly intelligent AI solutions.
I am, however, fairly biased. This is one of my dearest research topics and I may well be over-excited about it. So take everything I said here with a grain of salt. Modern agent architectures are still brittle and hard to deploy in large-enough scale for their distributed strenth to be noticeable. Still, they hold great promise, and are one of the few paths forward that may take us all the way to general AI.
In Part 3 we will build some agent-based simulations, especially towards the final chapters. Due to their complexity and cost, we won’t spend as much time building agents and simulations as I would like, but you’ll get a taste of how powerful this paradigm can be in ?sec-dnd and ?sec-storyteller.