Apple tests 256 model pairs: 92.9% at best, 9.4% open-weight average

Apple researchers put a headline number on how badly language models garble structure when they talk to each other, and the number everyone quotes is the flattering one. The round-trip study, posted by four researchers at Apple, reports a best pairing of 92.9%. That’s the single strongest cell in a 16 by 16 grid of 256 generator and extractor pairings. Across the 12 open-weight models in the same matrix, the average round trip lands at 9.4%.

The protocol is deliberately narrow so the scoring can be exact. One model, the generator, turns a procedurally built arithmetic expression into a short word problem. A separate model, the extractor, has to recover the original expression from that prose alone, and SymPy checks symbolic equivalence. A rule-based guard rejects any word problem that leaks mathematical notation, which forces the structure through language rather than through symbols.

Apple ran 16 models in both roles on 2,450 expressions, with operator counts from 2 to 8 and tree depths from 2 to 6. Twelve were open-weight, from 0.6B to 32B across five families, and four were frontier models reached by API: Claude Haiku 4.5, GPT-5, Gemini-3-Flash and Gemini-3.1-Pro. Every model ran with greedy decoding and thinking disabled, which the paper says isolates serialization from test-time reasoning.

Panel	Round-trip accuracy
12 open-weight models, all pairings	9.4%
Open-weight models at 8B and above	17.0%
4 frontier models, all pairings	74.7%
GPT-5 reading its own word problems	91.3%
Gemini-3.1-Pro reading GPT-5	92.9%

Which model writes and which model reads changes everything

That 92.9% is itself an argument against using one model for both jobs. GPT-5 is the best extractor in the panel but only the second-best generator, while Gemini-3.1-Pro leads generation and comes second at extraction. Pairing them in that order beats Gemini-3.1-Pro reading its own output by 4.2 points. Swapping the two roles for a single pair, Gemma-3-27B with Gemini-3.1-Pro, moves accuracy by 60.4 points.

The failures sit on the writing end. Using agreement across the extractor panel, Apple attributes at least 73.6% of failed round trips to the generator, meaning no extractor in the panel recovered the target. Wrong operator count accounts for 71.8% of all errors. Tree shape predicts difficulty better than model family does, and right-branching expressions are harder than left-branching ones of the same size.

The fix only clears the frontier under matched conditions

The paper’s most quotable claim is that roughly 3,600 fine-tuning messages lift every open-weight model above untrained Gemini-3.1-Pro. The catch is in the training data. Those examples use the same four arithmetic operators and the same tree topology as the evaluation, with only the variable letters and constants held back. Apple is explicit about what that means.

This is best read as an upper bound on what the channel admits under matched semantics, not a generalization claim.

Suau, Ferrando, Zappella and Bengio, Apple

The harder test tells a different story. A second regime trained on an assembly domain with a novel operator vocabulary, mixed in arity and built on coloured-object nouns that share nothing with arithmetic, lifted generation quality for all ten open-weight models by 3.3 to 16.6 points. Even so, the strongest of them, Phi-4 at 32.7%, finished 12.4 points below untrained Gemini-3.1-Pro at 45.1%. The skill transfers, just not far enough to close the gap.

Apple’s recommendation follows from a figure it calls the NL tax. When the same Qwen3 models at 14B to 32B read deterministic S-expression or JSON-AST encodings instead of prose, extraction reaches 85% to 88% against 51% to 70% from the best prose generators, an 18 to 20 point penalty for the detour through language. So the paper points teams at structured intermediates whenever the consumer can parse them, naming Anthropic’s Model Context Protocol and Google’s Agent2Agent protocol as schemas that already exist. We’ve covered Apple generating agent tests from MCP specifications before.

The closest prior work, round-trip correctness for code models at ICML 2024, kept the same model on both ends on purpose. Apple’s paper says that choice sidestepped a “communication chasm” between distinct models, and this study makes the chasm the object of measurement. The result lands near our earlier reporting that self-reflection beat a multi-agent pipeline in a Spanish school study.

Worth holding onto: these are scores with reasoning modes switched off, in English, with one untuned prompt pair, which the authors describe as a reproducible lower bound. Extraction tracks existing benchmarks closely, correlating at 0.98 with MMLU, but generation correlates only moderately at 0.47 to 0.89. That gap is the claim to test next, and the paper says the code gets released on acceptance, so nobody outside Apple’s machine learning group can rerun the matrix yet.