Apple generates 337 AI agent tests from MCP specs with no live tool access

Apple researchers have built a pipeline that writes evaluation tests for tool-using AI agents from nothing but the tool’s own specification. The system, called Agent Seer, takes a single Model Context Protocol (MCP) spec and produces graded scenarios, fake tool outputs and multi-turn dialogues. It needs no examples, no live tool access and no domain-specific tuning, according to the paper posted to Apple’s machine learning research site in August 2026.

The insight behind it is simple. An MCP tool already ships with a name, a natural-language description and a typed parameter schema, which the MCP specification defines as a JSON Schema for the expected inputs. The authors, Harish Karumuri, Mahesh Vemula and David Lopes Pegna, argue that’s enough semantic information for a language model to infer plausible workflows and fill in realistic parameter values. The bottleneck shifts from human curation to structured extraction.

That matters because hand-written agent benchmarks don’t scale and go stale as APIs change. It’s the same measurement problem we covered when the Stanford AI Index found benchmarks saturating within months. The paper, submitted to arXiv on June 24, 2026, calls this the cold-start evaluation problem: producing realistic test data for a tool suite that has none.

How Agent Seer scored on seven MCP servers

The team ran the pipeline on seven open-source MCP specifications drawn from the official MCP reference server repository and the MCP server registry. Gemini 2.5 Flash Lite generated 337 scenarios, and Gemini 2.5 Flash judged them at temperature zero. Mean tool-calling correctness came in at 0.911 on a zero to one scale, with 31.7% of records scoring a perfect mark and only 2.3% scoring below 0.5.

MCP server	Tools	Mean params per tool	Tool-calling score	Coherence	Tool coverage
Illustrator	64	3.6	0.898	0.855	56%
Selenium	56	1.8	0.935	0.850	100%
Redis	47	2.1	0.966	0.902	100%
Git	33	11.2	0.857	0.757	100%
Elasticsearch	20	1.8	0.930	0.902	100%
Slack	16	2.2	0.886	0.938	100%
Filesystem	14	1.8	0.876	0.825	100%

The table shows the paper’s headline finding, which is that tool count barely matters: Selenium’s 56 tools scored higher than Filesystem’s 14. What drags quality down is parameter complexity. Git averages 11.2 parameters per tool, three times the next highest, with 95% of them optional, and it finished last on both tool calling and coherence. Across the seven servers, parameters per tool correlated with tool-calling quality at r = -0.60, while tool count correlated at a modest r = +0.40.

Where the generated tests go wrong

The dominant failure isn’t picking the wrong tool but filling in the wrong value, and that shows up in the argument scores. Only 42% of records scored perfectly on arguments, and 57% scored partial marks. Value accuracy accounted for 223 failing records, far ahead of relevancy at 44 and format at 35. In Redis, the pipeline kept calling the set tool with the right key and value but skipped the optional expiration field when a scenario implied time-bounded storage.

Git exposed a second problem, though a rarer one. In three scenarios the generator emitted real Git commands that weren’t in the MCP spec at all: fetch, revert and filter-repo. The authors call that a pretraining-knowledge leak past the spec. It’s rare, at 3 of 893 tool invocations, but it’s the kind of error a simple name-match metric would never catch, which is why the paper argues for scoring arguments below the function-call level.

Coverage, meanwhile, held up on the smaller servers. Every tool in Redis, Selenium, Git, Elasticsearch, Slack and Filesystem appeared in at least one generated scenario. Illustrator, the only spec with more than 56 tools, reached 56%, which the authors flag as the one sign of a coverage ceiling.

The catch: an LLM grading an LLM

The obvious objection is circularity. Both the scenarios and the scores come from language models, so tests that look good to a model could score well regardless of actual quality. The authors address it two ways. The judge is a stronger model than the generator, and they re-scored the full corpus with Alibaba’s Qwen3.5-122B-A10B-FP8, which agreed with Gemini on tool calling at a paired r of 0.79 and preserved the ranking of the seven servers.

The most significant limitation is reliance on LLM-generated ground truth, which introduces systematic biases from the generating model.

Harish Karumuri, Mahesh Vemula and David Lopes Pegna, Apple, in the paper’s limitations section

Even so, the Qwen judge was stricter on coherence, shifting the mean down by about 0.16, and the two judges disagreed on which server had the worst conversational flow. Multi-turn expansion only succeeded on 54 of the 337 scenarios, and the authors list a human evaluation study as future work. Those are the gaps a sceptical reader should hold onto, the same ones that make us wary of leaderboard scores in general.

What’s worth watching is whether the approach spreads beyond MCP. The paper says extending it to OpenAPI, gRPC and function-calling schemas is structurally straightforward, since the same name-plus-typed-parameter scaffolding is present. If that holds, any team shipping an internal API could generate its own agent test suite before a single real call is made.