Max Planck study: Claude searches on 83% of prompts, GPT-5.3 on 14%

Researchers at the Max Planck Institute for Software Systems replayed the same 1,000 user prompts through four frontier chatbot APIs and counted how often each one reached for the web. Claude Sonnet 4.6 called web search on 825 of them. GPT-5.3-chat called it on 140.

That 825-to-140 gap sits at the centre of a paper posted to arXiv on 16 September, written with co-authors at Saarland University, Ruhr University Bochum, Seoul National University and Microsoft. The team also studied 171,264 real conversations donated by 613 users of ChatGPT, Claude, Grok and DeepSeek. They describe it as the first study of agentic search across multiple commercial platforms using observable search traces.

So the gap isn’t purely a model property, which the authors tested directly. They lifted the published system prompt from one OpenAI model and handed it to another, which cut o4-mini’s search calls from 634 prompts to 478. Even so, o4-mini kept searching far more often than GPT-5.3-chat did under any harness. Instructions move the dial, and the underlying model still decides.

Model (API)	Prompts that triggered search, of 1,000	Factuality with search	Factuality, search disabled
GPT-5.3-chat	140	3.50	2.81
Claude Sonnet 4.6	825	3.29	3.41
Grok-4.3	766	3.81	3.30
DeepSeek-v4-flash	584	3.13	2.50

Searching more didn’t mean answering better. Claude scored 3.29 on factuality when it searched and 3.41 on the same prompts with search switched off, which the paper calls a mixed effect at best. Every one of the four models scored highest on the prompts where it chose not to search at all. That reads as good judgement about when internal knowledge is enough, not as evidence that retrieval is doing heavy lifting.

while Web search is useful, it is still not sufficiently effective at plugging the gaps in models’ parametric knowledge

Amani et al., Characterizing Web Search by Conversational LLM Agents

What comes back differs as much as when it’s asked for. In the live traces ChatGPT’s search returned about 14 URLs per query and roughly 43 per prompt, while DeepSeek’s returned about 5 and 14. Reddit and YouTube dominate the results served to ChatGPT and Grok, and the paper reports they’re completely absent from Claude’s. Results from xiaohongshu.com show up prominently in DeepSeek search and nowhere else.

But attribution is where it gets uncomfortable. Citation rates across the donated traces ran from 1.8% on Grok to 34.1% on DeepSeek. Across ChatGPT, Claude and Grok, between 14% and 53% of claims turned out to be supported by pages the agent had retrieved but never cited, and at least 15% to 20% stayed ungrounded entirely. In ChatGPT those ungrounded claims scored 2.84 on factuality, against 3.47 for claims backed by the citation next to them.

That pattern matches what a separate team found with SearchAtlas, accepted to Findings of EMNLP 2026, which maps search trajectories into graphs and flags unverified parametric knowledge entering the answer. For anyone leaning on these tools for sourcing, our guide to using AI for research without inventing your sources covers the checks that catch it. The retrieval-versus-memory tradeoff is the same one behind long context versus RAG.

Vendors are already selling the fix, because the variation is exactly what a platform layer claims to absorb. Databricks argued on 17 September that bundled harness search is a default nobody chose, and pitched naming a partner provider on the web_search builtin instead.

That post cites Nimble’s testing as lifting benchmark accuracy from 46% to 71%. But Nimble’s own write-up puts baseline web search at 22%, so the 46% figure is generic deep research rather than the built-in tool. Both figures come from the vendor’s own evaluation.

The Max Planck team released its analysis code and mock data on GitHub, though the donated conversations stay private under the consent terms. Worth watching is whether the platforms start reporting citation rates themselves, because right now a 1.8% rate and a 34.1% rate look identical to the person reading the answer.