Geneva benchmark: Claude Opus 4.7 scores below 4.6 on legal text

A new benchmark for international legal text puts Anthropic’s Claude Opus 4.6 at the top and its newer sibling, Opus 4.7, in fourth place. The abstract of the paper describing it names the winner and leaves that comparison out. In the body, the University of Geneva authors note that the confidence intervals overlap, so they report the gap as “the absence of an improvement rather than a regression.”

The dataset is called IntLawNER, and it reached arXiv on 18 September 2026. It carries 2,987 gold-annotated sentences and 8,094 entity spans taken from International Court of Justice decisions, UN Security Council resolutions from 1946 to 2026, and European Court of Human Rights judgments. Seven entity types sort those references by legal status rather than by wording.

Named entity recognition tags the spans in a text that name a thing, so here a treaty, a provision, a court instrument or a case citation. Genis Skura, Roland Bouffanais and Didier Wernli argue that international law had no token-level resource of this kind. The existing legal sets cover domestic systems, so German court decisions, Indian judgments, Greek legislation, Turkish legal texts and English contracts. That gap matters because the same sentence often cites a national criminal code and a convention side by side, and spaCy gives both the same LAW label.

So the team benchmarked nine large language models, two fine-tuned transformers, a regex baseline and one zero-shot span model. The test split holds 605 sentences and 1,660 spans. Scoring is strict entity-level F1, which means a span counts only when the label and both boundaries match. The dataset card on Hugging Face shows the same split we read in the paper, 2.11k train rows, 272 validation and 605 test, under a CC BY 4.0 licence.

System	Micro-F1	Macro-F1
Claude Opus 4.6, 5-shot	0.873	0.806
Claude Sonnet 4.6, 5-shot	0.871	0.797
GPT-5.5, 5-shot	0.861	0.777
XLM-RoBERTa-large, fine-tuned	0.849	0.736
Claude Opus 4.7, 5-shot	0.842	0.751
Legal-BERT, fine-tuned	0.820	0.627
GLiNER, zero-shot	0.243	0.167

Opus 4.7 also trails Opus 4.6 without examples, at 0.767 against 0.793 micro-F1. That’s the finding the authors draw their general lesson from, and a reader tracking Anthropic’s Opus releases wouldn’t get it from a leaderboard. Sonnet 4.6 is the other surprise, rising from near-worst zero-shot to second place with five examples, a gain of 14.8 points.

General capability improvements do not automatically transfer to domain-specific entity recognition.

Skura, Bouffanais and Wernli, IntLawNER

The fine-tuned models had training data, and they still didn’t clear the prompted ones. XLM-RoBERTa-large averaged 0.849 micro-F1 across five seeds and Legal-BERT 0.820, which inverts the encoder-versus-LLM result we covered in Valencia. Both slide on the rare labels, because there are so few examples of them: Legal-BERT scores 0.338 on court instruments, which have 58 training spans, and 0.078 on soft law.

Worse off is GLiNER large v2.1, the zero-shot span model, at 0.243 micro-F1 and 0.000 on soft law. The paper’s explanation is structural rather than a matter of model quality. GLiNER’s label conditioning can’t see the annotation glossary, and its fixed span window truncates the composite ICJ citations that routinely run past 30 tokens.

The methodology carries the caveat most likely to travel beyond this dataset. A hybrid pipeline cut 468,000 source sentences to about 3,000 candidates, and Claude Opus 4.6 agents then ran roughly 800 calls over 250 iterations to label them. Human reviewers in Label Studio accepted 89.6% of those spans unchanged.

Cohen’s kappa on the boundary-matched spans came out at 0.964, which would pass any quality threshold. Score the full spans, including missing entities and boundary fixes, and agreement drops to a macro-F1 of 0.753. That gap is what their recommendation rests on.

So the authors recommend per-label confusion matrices as a minimum standard for any NER dataset labelled by a model. Their own limits section is blunt about the rest: 2,987 sentences is modest, court instruments have 83 spans in total and 18 in the test split, and no independent international-law expert has reviewed the schema. Customary law is out of scope, because it reads as doctrine rather than as named entities.

Whether the schema survives that expert review is the thing to watch, and the code and pipeline are public for anyone who wants to rebuild it at a larger n. The label inventory’s example of a provision span is “Article 2, paragraph 4”, the prohibition on the use of force that Mykhailo Soldatenko examined in Lawfare on 23 September through the Cuban missile crisis. Extraction tools reach that clause from the other end, so they count where it’s cited instead of arguing what it constrains.