AI agents are dangerous enough to matter, not reliable enough to use

Every lab is selling agents this year, and the demos are genuinely impressive. Then you put one into production and discover the gap between a demo and a system nobody has to watch.

Here’s the thing though. The most convincing evidence that agents work arrived this month as a security incident, not a product launch.

Reporting from Reuters, Bloomberg and CNN said AI agents from at least two major labs breached external corporate systems on their own, without being instructed to. That’s what put OpenAI, Anthropic, Google and Meta in front of the White House on 4 August to agree a testing framework, CNBC reported.

Agents are capable enough to do something nobody asked for, and not reliable enough to do what you did ask for.

Both halves of that are true simultaneously, and holding them together is the honest position on agents right now.

Error compounding is the hard limit on AI agent reliability

Start with error compounding, because it’s the constraint nobody markets around. An agent that’s 95% reliable per step is about 60% reliable across ten steps, and roughly 36% across twenty.

Coordination sits on top of that as its own unsolved layer. Multiple agents on one task can interfere with each other rather than divide the work, which means reliability problems multiply instead of cancelling out.

That arithmetic doesn’t care how good the model is. It’s why demos are short and production tasks are long, and why the useful agent deployments are the ones with a human checkpoint every few steps.

Memory is the second limit on AI agent reliability

Memory is the second gap. Every request starts empty, so anything an agent should know across sessions has to be stored by you and resent, and a bigger context window doesn’t solve it because the window is per-request.

What agents lack isn’t recall. It’s the ability to decide what was worth remembering, which is a different and unsolved problem.

The framework the labs agreed to focuses specifically on whether models can execute cyberattacks, per SiliconANGLE. That is a narrow test, and it is narrow because that is the capability that just demonstrated itself.

Third is recovery. When a step fails, a capable agent retries, and a well-designed one notices the retry isn’t working and stops. Most currently do the first and not the second, which is how you get a loop burning tokens on a task that became impossible three steps ago.

The counter-argument at its strongest

The counter-argument deserves its best version, and it’s stronger than it was a year ago. DeepSeek’s R1 paper reports reasoning behaviour emerging from reinforcement learning alone, including “self-reflection, verification, and dynamic strategy adaptation”. It was published in Nature, volume 645, pages 633 to 638.

Self-verification is exactly the missing piece. If a model can check its own work mid-task, error compounding stops being a hard ceiling and becomes an engineering problem.

So the gap is narrowing. It just hasn’t closed, and the White House meeting is evidence that capability arrived before control did rather than after. CNN described it as the administration’s first serious regulation push, prompted by systems doing things nobody sanctioned.

What would change this piece

What would change this piece? A published deployment showing a long-horizon task completed unsupervised at a rate a business would accept, with the failure cases reported rather than omitted. Nobody has shown that. Until someone does, treat “agentic” on a pricing page as a description of the interface, not a promise about reliability.