Stanford wins Databricks’ Grounded Reasoning Cup at 63.3% accuracy

Stanford won the inaugural Grounded Reasoning Cup, a live AI agent competition hosted by Databricks, with 63.3% accuracy on a benchmark no team had seen before competition day. The win wasn’t comfortable. Stanford trailed UMass Amherst from halftime on and pulled ahead in the final 56 seconds, taking the title by 1.75 points.

The format explains why the result matters more than a typical leaderboard update. Eleven academic teams from the U.S. and Canada spent two months building agents against OfficeQA, Databricks’ grounded-reasoning benchmark, each paired with a mentor lab from OpenAI, Anthropic, or Google DeepMind and restricted to that partner’s model family. On June 17 they applied those agents in real time to OfficeQA Pro V2, a freshly released test set, across six rounds of 15 questions with 15 minutes per round. The event carried $120,000 in Databricks credits for the top teams.

That design targets a question benchmarks keep dodging: do gains on a public test generalize to similar tasks the developer hasn’t seen? It’s the measurement problem behind benchmarks saturating within months of release. Here, the average team scored roughly 41%, the top three cleared 50%, and Stanford’s 63.3% beat the average team by about 22 points. Databricks says that’s also around 35 points above the average frontier agent’s offline baseline.

Place	Team	Models	Result
1st	Stanford	Claude Opus 4.8, via Claude Code	63.3% accuracy, 57 correct of 88 attempted
2nd	UMass Amherst	Claude Opus 4.8 Fast	36 speed bonuses, 4-minute average correct submission
3rd	Yale	Gemini 3.1 Pro and Gemini 3.5 Flash	49 of 90 questions correct

What separated the winners is the part worth studying. Stanford’s edge came from preparation rather than raw model power. During development the team traced wrong answers back to the agent’s exact misstep, then converted those failure patterns into reusable skills for its Claude Code agent running Claude Opus 4.8: table localization, answer formatting, clarifying common financial wordings. By competition day the playbook held about 100 skills, plus logic for when to search parsed text and when to fall back to source PDFs.

Even so, the team had to gamble mid-match. A second Claude Code agent verified answers in the first three rounds but earned Stanford only two speed bonuses, so the team switched it off to cut latency and collected bonuses on 14 questions. In the final round Stanford turned the verifier back on, and it caught an error that a resubmission fixed, sealing the win.

UMass bet on speed instead. Running Claude Opus 4.8 Fast with three agents in parallel and a final Opus call to pick the best answer, the team averaged four minutes per correct submission against a field average of eight minutes 30 seconds. The result: 36 speed bonuses worth 0.25 points each, more than double Stanford’s 16, and a 10.25-point lead at halftime that shrank to 3.75 entering the last round. Yale took third with a four-arm harness spanning ReAct agents and a planner-verifier pipeline on Gemini 3.1 Pro and Gemini 3.5 Flash, answering 49 of 90 questions correctly.

The competition made one thing clear: grounded reasoning over enterprise-style document corpora has improved since we released the OfficeQA benchmark 7 months ago, but it is still far from solved.

Databricks, Grounded Reasoning Cup recap

The headroom is real. Even with every team’s attempts pooled, 18.8% of questions went unsolved. The underlying OfficeQA Pro paper shows why: its corpus spans 89,000 pages of U.S. Treasury Bulletins with over 26 million numerical values, and frontier agents given direct corpus access still averaged only 34.1% on its 133 questions.

The larger lesson is that the surrounding system mattered more than model choice, echoing research on why long-horizon agents fail on context rather than capability. Independent work points the same way: the FinSkillBench paper, which tested agents on 2,603 investment-management task episodes, found curated skill packages lifted mean scores from 0.366 to 0.528 across nine models, while self-generated skills added little. What to watch next is whether these competition-honed techniques close that unsolved 18.8%, or whether it takes another benchmark to find out.