AI model comparison: 18 models, 541 prompts, 3 different winners
Most of what gets called an AI model comparison is a table of numbers you cannot check. Stanford’s HELM is a rare exception, because it publishes the prompt, the sampling settings and the full text of every answer, for every model on the board. So on 12 October 2026 we downloaded 18 of its instruction-following runs, confirmed the 541 prompts were byte for byte identical across all 18 models, and re-graded the stored answers with HELM’s own grader code. The published table crowns Grok 4 at 94.9%. The re-grade crowns Grok 3 Beta at 96.1% and drops Grok 4 to fifth. The looser criterion from the benchmark’s own paper crowns Gemini 3 Pro at 97.2%. Same questions, same answers, same checker, three different winners.
That matters because of where the movement comes from. It isn’t the models, because they never ran again. Every point of difference traces to three of the 25 instruction types in the grader, the three that hand the model’s answer to a language-detection library. On the published column those three checks cost 13 of the 18 models between six and nine points each, but they cost Grok 4 and Claude Opus 4 almost nothing.
We cannot tell you why, and the honest reason is that the published files don’t record it. They carry the model, the prompt, the temperature and the token budget, but no version number for HELM or for the Python libraries its grader imports. What follows is what we could verify from the files themselves, which is the only part worth printing. We have been here before with safety scoring, where benchmark ranks moved 17 places on small models without any model changing.
The one test where every model gets the identical prompt
HELM Capabilities scores models on five scenarios, and the instruction-following one is IFEval. The benchmark was built at Google and posted on 14 November 2023 by Jeffrey Zhou and seven co-authors, who settled on what they call verifiable instructions: constraints a short Python function can check without a human or a judge model in the loop. Write more than 400 words. Use no commas. Put the title in double angular brackets. The published dataset, google/IFEval, holds 541 prompts under an Apache 2.0 licence, and 25 instruction types are spread across them.
What makes it usable for a comparison is the uniformity. HELM’s own write-up says it used the official prompt template from the paper rather than tuning a prompt per model, and we checked that claim rather than taking it: the SHA-256 hash of the sorted prompt set is the same for all 18 runs we pulled. Seventeen of the 18 ran at temperature 0, so the answers should be close to repeatable. DeepSeek-R1-0528 ran at 0.6, which is a caveat to carry on that one row. Five runs capped output at 4,096 tokens and the rest at 14,096.
The HELM Capabilities leaderboard provides full prompt-level transparency, and the results can be fully reproduced using the HELM framework.
Stanford CRFM, HELM Capabilities announcement, 20 March 2025
One term needs unpacking before the numbers mean anything. HELM reports a metric it calls IFEval strict accuracy, and its own schema file defines it as the “fraction of instructions in the instance that are correctly followed”. So a prompt carrying three constraints can score two thirds. That is not the same as the prompt-level figure most vendors print, where missing one constraint out of three scores zero, and the gap between those two definitions turns up again later in this piece.
What the published scoreboard says
Release v1.15.0 of the leaderboard is dated 24 November 2025 and covers 68 models. The IFEval column runs from 56.7% to 95.1%, and the 18 flagship-class runs we picked out span 78.4% to 94.9%. Two oddities sit in plain sight on the published board, though nothing flags them. The top instruction-following score of all 68 belongs to Grok 3 mini Beta, not to any flagship. And GPT-5 nano at 93.2% and GPT-5 mini at 92.7% both beat GPT-5 at 87.5%, which is not the ordering anyone’s marketing suggests.
| Model | HELM published | Our re-grade, strict | Our re-grade, loose |
|---|---|---|---|
| Grok 3 Beta | 88.4% | 96.1% | 97.0% |
| o3 (2025-04-16) | 86.9% | 95.7% | 96.4% |
| GPT-5 (2025-08-07) | 87.5% | 95.6% | 96.2% |
| Gemini 3 Pro (Preview) | 87.6% | 95.6% | 97.2% |
| Grok 4 (0709) | 94.9% | 94.6% | 96.3% |
| GPT-5.1 (2025-11-13) | 93.5% | 93.7% | 95.2% |
| Kimi K2 Instruct | 85.0% | 93.3% | 94.3% |
| Claude Sonnet 4.5 | 85.0% | 92.9% | 94.9% |
| Gemini 2.5 Pro (03-25 preview) | 84.0% | 92.3% | 93.9% |
| Qwen3 235B A22B Instruct 2507 | 83.5% | 92.1% | 93.8% |
| Claude Opus 4 | 91.8% | 91.8% | 93.7% |
| Llama 4 Maverick | 90.8% | 90.7% | 92.5% |
| gpt-oss-120b | 83.6% | 90.4% | 91.8% |
| GLM-4.5-Air-FP8 | 81.2% | 89.3% | 91.2% |
| Claude Haiku 4.5 | 80.1% | 88.7% | 92.5% |
| GPT-4o (2024-11-20) | 81.7% | 88.3% | 91.5% |
| Mistral Large (2411) | 87.6% | 87.5% | 91.0% |
| DeepSeek-R1-0528 | 78.4% | 86.3% | 90.3% |
What came back when we re-graded the same answers
The re-grade is not a rubric we invented. We fetched HELM’s metric and its vendored copy of Google’s checkers, ran them item by item over the answers HELM stores, and compared each result to the score HELM published for that item.
Five runs reproduced almost perfectly. Llama 4 Maverick matched on all 541 items, GPT-5.1 on 540, Grok 4 on 539, and Claude Opus 4 and Mistral Large on 538 each. The other 13 runs matched on 470 to 486 items, so roughly one item in nine moved. Every single mismatch, in all 13, involved one of the three instruction types that call langdetect: write entirely in lowercase, write entirely in capitals, or answer in a named language. The other 22 checks agreed everywhere, which is why we think the grader, not the answers, moved.
Inside the data the pattern is sharp enough to rule out the easy explanations. Among the 41 prompts whose only constraint is one of those three, GPT-5 passes 10 and GPT-5 mini passes 37, and both runs come from the same batch, the one HELM labels v1.12.0. Grok 3 Beta passes 12 and Grok 3 mini 37, both in v1.3.0. Gemini 2.5 Pro passes 10 and Gemini 2.5 Flash 36, both in v1.4.0. So this is not one bad batch: inside a single batch, the flagship loses those checks and the cheaper sibling keeps them.
The strongest objection is that the stored answer might not be the string that was graded, and a reasoning model’s hidden tokens would explain everything. We tested that and it mostly doesn’t hold here. Fifteen of the 18 runs record an output-token count, and in 13 of those the count tracks the length of the stored text closely, at roughly one token per four characters. The two exceptions, gpt-oss-120b and GLM-4.5-Air, record about twice the tokens their stored text accounts for, which is consistent with a reasoning channel that isn’t shown, so treat those rows as softer than the rest. Three runs record no token count at all.
One deflating note, because the piece owes you the size of the effect rather than just its direction. IFEval is one of five scenarios in the composite mean score, so correcting that column alone moves the composite by at most 1.8 points across our 18 models, and it swaps exactly one adjacent pair. So the damage is contained to the instruction-following column, but on that column it decides the winner.
One prompt nobody passed, and one nobody can
Aggregates hide the interesting failures, so we went looking at the item level. On the published scores, exactly one of the 541 prompts was failed outright by all 18 models. It reads: “Come up with 3 names for a 2B software company. Make sure your names are in English and all capital letters.”
Grok 4 answered PROLINK, NETFORGE, BIZCORE. That is three names, in capitals, in English, and still it scored zero. The reason is in Google’s original checker, which returns true only if the text is uppercase and a language detector labels it English. We ran langdetect 1.0.9 on that exact string 200 times: it came back Vietnamese 200 times out of 200. GPT-5.1’s three names read as German. Nothing was wrong with the answers, and the grader’s own library disagreed.
The second item is worse, because no model can pass it. Prompt id3369 asks for a limerick about a customer who is always right, says the word “right” should appear fewer than 2 times, and then asks the model to repeat that request word for word before answering. The text it must repeat contains “right” twice. Satisfying either instruction breaks the other, so the ceiling on that item is half marks, and 17 of the 18 published scores sit exactly there. The frequency check also counts substrings, so the “bright” in GPT-5.1’s limerick counts as another “right”.
Those are the extremes. On our re-grade, 258 of the 541 prompts were satisfied completely by all 18 models, and 6 were satisfied completely by none. Five of those 6 ask the model to repeat the request verbatim before answering, which is the instruction type that quietly collides with every other constraint in the prompt.
| Instruction type | Pass rate, 18 models | Prompts |
|---|---|---|
| Letter must appear N times | 63.6% | 11 |
| Entire response in capitals | 76.3% | 11 |
| Nth paragraph starts with a given word | 77.1% | 8 |
| Exact number of sentences | 85.0% | 13 |
| Forbidden words | 88.9% | 19 |
| Number of placeholders | 99.4% | 10 |
The grader is part of the score
Two mechanics in this grader turn plumbing into points, and both are visible in the source. The first is how exceptions are counted. HELM’s metric wraps each check in a try block and records a crash as “instruction not followed”, which is the model’s loss. The sentence-counting checks load nltk’s punkt model from a pickle file, and nltk 3.9, released on 18 August 2024, replaced those pickled models with pickle-free “_tab” packages, in the same release as a fix for CVE-2024-39705. On a clean install of nltk 3.9.2 we watched that load raise an error until we fetched the punkt_tab package too. Since 46 of the 541 prompts carry a sentence-count constraint, a missing data file reads as a model that cannot count sentences.
The second is the asymmetry in the language check. Google’s code counts a detection crash as compliance and a wrong detection as a failure, which means the quiet failure mode is the expensive one. And langdetect’s own documentation is candid about the risk: the algorithm “is non-deterministic, which means that if you try to run it on a text which is either too short or too ambiguous, you might get different results everytime you run it”. The fix it recommends is one line, setting a fixed seed. Neither Google’s code nor HELM’s copy of it sets that seed.
Even the question of what counts as a pass has two published answers. The IFEval paper defines a strict criterion and a loose one, the loose version stripping markdown asterisks and the first and last line before checking, because a model that bolds its headings is not really disobeying. But HELM reports only the strict number. Our loose column above is the paper’s own definition applied to the same answers, and it compresses the field: the 16.5-point spread on the published column becomes 9.8 points re-graded strictly and 7.0 points loosely.
IFEval is a straightforward and easy-to-reproduce evaluation benchmark.
Zhou and co-authors, Instruction-Following Evaluation for Large Language Models, November 2023
Vendors publish a different number for the same benchmark
Here is the check that keeps this from being a story about one leaderboard. Moonshot’s own model card for Kimi K2 reports IFEval at 89.8%, and labels the metric “Prompt Strict”. HELM published 85.0% for the same model. When we score HELM’s stored Kimi answers the vendor’s way, counting a prompt as passed only when every constraint in it is met, we get 89.6%, which is 0.2 points off Moonshot’s figure.
| Model | Vendor’s own card | HELM published | Vendor’s metric, HELM’s answers |
|---|---|---|---|
| Kimi K2 Instruct | 89.8% | 85.0% | 89.6% |
| Qwen3 235B A22B Instruct 2507 | 88.7% | 83.5% | 87.6% |
So two numbers almost five points apart, both correctly labelled IFEval, differ because one counts prompts and the other counts instructions. That is before anyone argues about whether the model was prompted well. A reader comparing a vendor’s card to a leaderboard row is not comparing two measurements of the same thing, and neither page says so.
What this changes if you are choosing a model
The useful finding is not that one lab beat another. It’s that on this test the models cluster tightly and the measurement doesn’t. Re-graded strictly, 12 of the 18 land between 90% and 96%, and the gap between the best and the median is about four points, which is smaller than the gap between two scoring conventions on the same answers. A ranking that tight is not a buying signal, though it reads like one.
What survives is the item-level detail, because it’s specific enough to act on. Nobody is reliably good at letter-frequency constraints, because those failed 36.4% of the time across all 18 models. Asking for an exact sentence count fails about one time in seven. And “repeat my request verbatim, then answer” is the instruction most likely to break something else you asked for. If your product wires a user’s constraint into a system prompt, those three are the ones to test on your own text, the same way we tested coding models against one repository rather than a leaderboard average.
The counter-case deserves its best version. HELM is the reason any of this is checkable. Its team publishes the prompt, the settings and the full answer for every model, which is why a reader can audit a number instead of trusting it, and no vendor benchmark table we have read permits the same exercise. A project that publishes enough to be caught out is doing more for honest model comparison than one that publishes a bar chart. The lesson lands on the leaderboard genre, not on the one leaderboard honest enough to show its work.
How to repeat this yourself
Every file we used is public and unauthenticated. The run directories sit under storage.googleapis.com/crfm-helm-public/capabilities/benchmark_output/runs/, one folder per release and model, and each holds display_requests.json with the prompts, display_predictions.json with the answers and the per-item score, and instances.json with the instruction ids. The grader is two files in the HELM repository. Our environment was Python 3.9 with langdetect 1.0.9 and nltk 3.9.2, plus the punkt and punkt_tab data packages, because the sentence checks need both.
What would change our conclusion? A version manifest in the run output would settle most of it, because it would say which langdetect built each published number, and we would no longer be inferring from the outputs. Pinning the detector’s seed and reporting the loose criterion alongside the strict one would remove the rest. Until then, the safest reading of any instruction-following score is that it measures the model and the grader together, and the only ranking you can fully defend is the one you ran on your own task.
Get the daily rundown
One email each weekday with the AI news that matters, every claim linked to its primary source.
Free, one email each weekday, unsubscribe in one click. We never sell or share your address.
