IBM Research lifts AI agent consistency on AppWorld from 53% to 69%
IBM Research has published a method for measuring and reducing how often an AI agent solves a task on one run and fails the identical task on the next. On the AppWorld benchmark, a ReAct agent running GPT-4.1 passed all five repetitions of a task only 53.0% of the time, even though its average per-run pass rate was 77.4%. Adding what the team calls consistency guidelines lifted that all-five figure to 69.0%.
The distance between those two numbers is the whole story. The team names it the consistency gap, defined as Mean@k minus Pass^k, and puts the baseline at 24.4 percentage points. After the intervention it’s 12.0pp, and average accuracy rose rather than fell.
Large language model (LLM)-powered agents can be accurate on average yet unreliable in production, a discrepancy that has been observed but remains largely unaddressed.
Duesterwald, Elder, Ngweta, Ubaru and Zimon, Closing the Consistency Gap, via arXiv
How the diagnosis works
The method ships inside ALTK-Evolve, IBM’s system for turning an agent’s own past trajectories into reusable guidelines. The new component is a Consistency Analyzer. Given one recorded trajectory, it replays each decision step through controlled resampling, drawing k completions at once with k=5 by default, and scores how much the output actually varies. That’s one extra model call per decision step, run offline against the already-recorded context, with no logits and no second end-to-end rollout of the task.
Every flagged step then becomes a candidate guideline in the format ALTK-Evolve already stores and retrieves. The published examples aren’t task trivia. One tells the agent to count checkbox-style markers with a line-anchored regex rather than a plain substring count, because note titles often repeat the marker symbol in a legend line. The analyzer targets instability rather than failure, so it catches steps the agent happened to get right this time.
| Measure on AppWorld test_normal | Baseline | With consistency guidelines |
|---|---|---|
| Mean@5, average per-run pass rate | 77.4% | 81.0% |
| Pass^5, passes on all five runs | 53.0% | 69.0% |
| Consistency gap | 24.4pp | 12.0pp |
So why does an agent at temperature 0.0 flip at all? The team’s answer is the shape of the token distribution, not the sampling on top of it. A sharp distribution puts most of its mass on one token and shrugs off small numerical noise. A flat one spreads comparable mass across near-tied tokens, so GPU floating-point non-associativity and request batching can reorder the winner, and a trajectory chains dozens of those decisions.
An agent can be capable and inconsistent at the same time.
IBM Research, on Hugging Face
Where the gains land
The improvement concentrates in the middle and hard tiers. Medium tasks gained 22.9 points of Pass^5, a 44% relative increase, and hard tasks gained 14.3 points, a 45% relative increase. Easy tasks gained 12.2 points, having had the least room to move. Mean@5 held or improved at every difficulty level, which was a stated requirement rather than a bonus.
Guidelines mined from one task also carried over. Applied to a different but related task in the same AppWorld scenario, they still lifted Pass^5 by 13.0 points, roughly 3 points below the same-task result. On the weaker gpt-oss-120b, same-task Pass^5 rose from 10.1% to 16.1%, while the similar-task number gained 8.7 points and so exceeded the same-task figure.
The evidence is still narrow, though. Every number here comes from AppWorld test_normal, which is 168 tasks, and the headline results rest on a single model. AppWorld itself is a 750-task suite spanning 9 simulated apps and 457 APIs, so test_normal is one slice of it. Academic work has attacked the same question from other angles, including a metric built from 9,000 recorded trajectories.
What’s worth watching is whether Pass^k starts appearing next to Mean@k on public leaderboards, because the diagnosis no longer needs a grader or a live replay. The open-source repo now carries the Consistency Analyzer and the guideline generation, and the technical report went up on arXiv on 8 September 2026. That puts it alongside recent work on portable memory for coding agents, because the pitch is the same one: stop the agent relearning what it already knew.
Get the daily rundown
One email each weekday with the AI news that matters, every claim linked to its primary source.
Free, one email each weekday, unsubscribe in one click. We never sell or share your address.
