AI agents and tool use, explained

What is an AI agent?

An AI agent is a language model set up to take actions in a loop, rather than just answer a question once. Instead of emitting one block of text, it observes the state of a task and decides on a step. It acts, then reads the result back in before deciding the next step.

The actions might be calling an API, running a search, or editing a file in a codebase. What makes the system an agent, rather than a chatbot, is that its own outputs change the world it is reasoning about. It keeps going until the task is done or it gives up.

What is tool calling?

Tool calling is a model's ability to invoke an external function — a calculator, a search engine, a calendar — mid-generation and use the result to keep writing. The Toolformer paper (Schick et al., 2023) showed a 6.7-billion-parameter GPT-J model could teach itself this. Given a handful of examples per API, it sampled candidate calls into its own training text, executed them, and kept only the ones that measurably lowered its loss on the following tokens.

Finetuned on that self-built dataset, Toolformer gained a calculator, a question-answering system, a Wikipedia search tool, a machine translator and a calendar. It improved sharply on factual lookup and arithmetic tasks. It beat the much larger GPT-3 (175 billion parameters) on several of them, without hurting its ordinary language-modelling perplexity. Schick et al. also found the ability to use tools well did not appear in their GPT-2-family models until around 775 million parameters.

How does reasoning-and-acting work?

ReAct, introduced by Yao et al. (2022), interleaves written reasoning steps — called thoughts — with actions, inside the same prompt. A thought doesn't touch the environment; it just restates or updates the model's plan. Then an action does something, like searching Wikipedia, and the result comes back as an observation for the next thought.

On the question-answering benchmark HotpotQA and the fact-checking benchmark Fever, ReAct came close to or beat chain-of-thought prompting while interacting with a simple Wikipedia API. A human review found it hallucinated facts far less often than chain-of-thought: 6% of successes versus 14%. On the interactive benchmarks ALFWorld and WebShop, ReAct beat imitation- and reinforcement-learning baselines trained on tens of thousands of examples. It won by 34 and 10 absolute points of success rate respectively, using just one or two prompt examples.

Yao et al. also found that acting without any reasoning — a baseline called Act — consistently did worse. It lost track of plans and produced hallucinated actions in text-based games. Combining ReAct with chain-of-thought self-consistency, switching between the two depending on confidence, did better than either alone on both HotpotQA and Fever.

What is a coding agent, and how good are they?

A coding agent is a model that reads a real codebase and a bug report, then edits files to fix it, rather than writing an isolated function from scratch. SWE-bench, built by Jimenez et al. (2023), tests this directly. It uses 2,294 tasks drawn from real GitHub issues and their merged pull requests, across 12 popular Python repositories, each graded by running the repository's own tests.

Results were stark. Claude 2, the best model in the 2023 paper, resolved only 1.96% of issues with a standard BM25 retriever supplying context. That rose to 4.8% when given an "oracle" retriever that was handed the exact files the real fix touched. GPT-4 scored lower still, at 1.31% with BM25. Fine-tuned open models, SWE-Llama 7B and 13B built on CodeLlama, were competitive with Claude 2 in some settings despite being far smaller.

Jimenez et al. found performance fell as context length grew: models struggled to locate the relevant lines among thousands of files and hundreds of thousands of lines of code. Model-written patches were also consistently shorter and simpler than the real fix, averaging under half as many edited lines. They rarely touched more than one file.

Where do agents fail?

The clearest failure mode across all three papers is getting lost in long context. SWE-bench's Claude 2 results dropped as the amount of retrieved code grew, even when a larger context window should have given the model more of the right files to work with.

ReAct's own error analysis on HotpotQA found a distinct loop failure. The model sometimes repeated the same thought and action over and over, unable to reason its way to a new step. Yao et al. attributed this partly to greedy decoding. A separate 23% of its failures came from the Wikipedia search simply returning nothing useful, derailing the reasoning that followed.

Toolformer's limits were narrower but telling: it could call a tool, but it couldn't yet reformulate a bad query or browse past the first search result, so weak retrieval just produced weak answers. And removing all reasoning from an agent, as in ReAct's Act-only baseline, reliably made it worse at tracking subgoals and more prone to hallucinated actions.

Why does this matter for a build decision now?

These three papers mark a rough timeline. ReAct (October 2022) and Toolformer (February 2023) established that a model can reason and call tools inside one loop. SWE-bench (October 2023) then asked whether that loop could do a working engineer's job, and found that even Claude 2 cleared barely 2% of real issues unaided.

None of these results describe current frontier systems. They describe the specific models tested at the time, on PaLM-540B, GPT-J-6.7B and Claude 2 respectively. Anyone evaluating a present-day coding or tool-using agent should ask for a benchmark like SWE-bench run on the current model. Don't infer success from an architecture description.