Lots of people are starting to explore what can be done with agent-based AI systems. It feels like there’s huge potential in them. But getting an agentic system to work reliably enough to use in practice often turns out to be very hard.
Here are a few techniques that can help with reliability.
Technique 1: Optimise The Agent’s Tools
The performance of agents depends heavily on the tools they have available. Give them better tools (or improve the ones they’ve got) and, naturally, they’re likely to perform better.
For example:
- If your agent is doing online research, instead of giving it access to a generic search result API, give it access to one designed specifically for LLMs.
- If your agent only needs certain fields returned by an API, filter the other fields out.
Technique 2: Add a Retry Mechanism
Let’s suppose that, 20% of the time, your agent produces a result that can, in an automated way, be identified as poor. If latency isn’t a concern, you can check each result and ask the agent to try again if necessary. (You may need to give the agent some information about its previous attempts so that it can avoid repeating itself.) You can quickly significantly increase the reliability of the overall system.
Technique 3: Generate Extra Responses and Pick the Best
If your system involves generating a set of N items but some of those items can be weak, consider generating slightly more than N items, automatically scoring them, then picking and using the highest-scoring N.
Technique 4: Eliminate the Agent
AI agents are appealing precisely because they can figure things out in an iterative, exploratory way. But the flipside of that open-endedness is that they can be very unreliable. You probably don’t want to hear this, but your best bet for now may actually be to eliminate your agent if possible.
How?
It depends on your situation but, if you’re asking an agent to complete a particular sort of task, try thinking through the steps of how you’d complete that task yourself. See if you can find a fixed sequence of steps that will do the job in the vast majority of cases and that you can automate (using LLMs if appropriate).
Technique 5: Avoid the Agent in Certain (Important) Cases
Sometimes eliminating your agent entirely isn’t possible (but it’s worth thinking hard about it.) Even in these situations, you may have some important cases that can be handled using a fixed sequence of steps that is more reliable than the agent would be. You may be able to classify incoming requests and route them to either your (relatively reliable) fixed sequence of steps or to your (relatively unreliable) agent. The overall reliability of your system can be improved as a result.
Final Words
Agent-based AI systems have a lot of promise but can be very unreliable and pose challenges that can be unfamiliar if you’re used to more traditional, deterministic software development. Dealing with that unreliability often becomes a key challenge and a focus of your engineering.
I hope the techniques I’ve outlined here give you some useful ideas and help you a little on your journey.
Let me know what other techniques you’re finding useful when developing AI-based software systems.
Leave a Reply