Why agents go wrong: mistakes, loops and misread instructions

Agents go wrong in a handful of ways, and the makers name most of them in their own documents. The mistakes are mostly ordinary. A guessed value, a misread request, an instruction hidden in a web page. They matter because an agent acts on them.

Errors that pile up

Anthropic links autonomy to two things: higher costs, and the chance that errors compound¹. Each step reads what the earlier steps produced. One wrong turn early can carry through everything after it.

Long jobs add a second problem. The model works from a context window, all the text it can look at while answering. The more text piles up, Anthropic says, the more accuracy and recall degrade². A long run can end up holding more than the model handles well.

Guessed or misread inputs

When a request leaves something out, the model may ask for it, or it may guess. Anthropic's example is a weather tool with no location given. A model in that spot might ask for the missing detail, or it might work out a value by itself³.

OpenAI lists the causes of unwanted actions as hallucination, misunderstanding and ambiguous user input⁴. A hallucination is a confident statement or action with nothing real behind it.

Tools that are easy to misuse

Sometimes the fault is in the tool's design. Anthropic describes building a coding agent in which the model made mistakes with tools that used relative file paths after the agent had moved out of the root directory. Its fix was to change the tool so that it always required absolute file paths¹. Its general advice is to change the arguments so that it's harder to make mistakes¹.

Instructions that come from the wrong place

A prompt injection happens when untrusted text or data enters an AI system and malicious contents in it try to override the AI's instructions⁴. An email, a web page or a document can carry such text. OpenAI says risk rises when agents process arbitrary text that influences tool calls⁴.

A related slip needs no attacker at all. A model can pass more data to a connected tool than the user meant it to⁴.

How the makers say mistakes get caught

Anthropic makes the same point about coding agents. Automated tests help verify that code works, but human review remains part of the job, to check that solutions fit the wider system¹.

OpenAI is plain about the limit of all this. Agents can still make mistakes or be tricked, even with the mitigations in place⁴. Plan for the day it happens. Limit access. Keep approvals on for actions that can't be undone. Leave secrets out of prompts, and customer records too unless your own policy allows them. The permissions page has the details.