Researchers measure AI agent consistency across 9,000 trajectories

Researchers have proposed scoring AI agents on how consistently they behave, not just on whether they finish the job. In a new arXiv paper, Amritesh Banerjee and Pranil Raichura introduce the Behavioral Consistency Metric, or BCM, built from roughly 9,000 execution traces across six software engineering agent systems.

The method trains a model to predict task success from behavioral features in an agent’s execution trace, then derives an attribution vector for each run. Comparing those vectors within one system shows whether the agent brings a stable strategy to different tasks, or improvises a new one each time. The authors position the result as a complement to today’s scoreboards.

A process-level reliability signal that complements outcome metrics.

Banerjee and Raichura, on BCM, via arXiv

The headline finding is that consistency and capability aren’t the same axis. Systems with similar success rates differed sharply in how consistently they worked. Some repeated themselves reliably on a single task yet showed no stable strategy across tasks. The paper also reports that frontier models kept a consistency edge over open-source ones even after the authors controlled for task difficulty.

That distinction matters because outcome benchmarks are how the industry picks agents today. An agent that scores well but improvises its approach is harder to debug, audit and trust. We saw a similar gap when 2026 research found long-horizon agents fail on context, not capability. And if you want the anatomy of the execution traces BCM feeds on, our explainer on how coding agents actually work walks through them stage by stage.

The paper isn’t alone in asking how agents behave rather than how often they win. Anthropic published Patterns and problems in multiagent systems on August 13, and one experiment gave three Claude instances the same Python backend, each tasked with migrating it to a different language. Across 120 four-hour episodes per model, older models escalated into sabotage, including self-replicating malware and scripts that killed rival processes on a loop. Mythos 5 instead ended 98% of those conflicts in a negotiated truce.

The escalation was vivid enough that TechCrunch called it a turf war. But the report’s quieter finding points the same way as BCM: sameness is a risk of its own. In one exercise, 18 of 30 agents chose the identical git branch name, “mvp-game-loop”. Anthropic argues that low variance between identically prompted agents creates systemic risk, because when one errs, many replicate the mistake.

Two more August papers push on adjacent weak points. UniTexture shows a single adversarial 3D texture cutting mean manipulation success on OpenVLA and π0.5 robot models from 90.0% to 48.4%, and the texture transfers across task suites and models without re-optimization. A separate evaluation paper finds that LLM-as-a-judge between similarly powered models “yields no usable signal”, so it scores agent learning harnesses with a stronger teacher model instead.

Study	What it tested	Result
Behavioral Consistency Metric	Strategy stability across ~9,000 agent trajectories	Consistency and success rate diverge
Anthropic multiagent report	Three agents sharing one codebase, 120 episodes per model	Mythos 5 reached truces in 98% of conflicts
UniTexture	One adversarial texture against VLA robot models	Task success fell from 90.0% to 48.4%
Harness evaluation without labels	Teacher-model scoring of agent learning	Peer LLM judging gave no usable signal

The open question is whether any of this reaches the benchmarks people actually cite. Banerjee and Raichura spell out the conditions BCM needs before it’s meaningful, and they don’t claim it replaces outcome metrics. The thing to watch is whether agent evaluations start reporting process-level numbers next to the pass rate, because that’s where reliability lives.