How to use AI for research without fake citations

Can you use AI for research without it inventing sources? Yes, but not by trusting the output and not by avoiding the tools. It takes a workflow, and the workflow has to survive deadline pressure or you won’t follow it.

Start with why the failure happens, because the fix follows directly from the cause.

Why models invent citations

A model predicts likely text. A citation has a recognisable shape: author names, a plausible title, a year, a journal, a DOI-looking string.

Producing something with that shape is easy. Producing one that exists requires the specific reference to have been memorised during training and retrieved correctly, which is a much harder ask.

The model is not lying to you. It is generating a plausible citation, and plausibility is the only thing it was ever optimising for.

Preference tuning makes this worse rather than better. OpenAI’s InstructGPT work showed a 1.3B tuned model being preferred over 175B GPT-3, and what preference rewards is confident, well-formed answers, not hedged ones.

So the fabricated reference arrives with exactly the same confidence as a real one, and there’s nothing in the output to distinguish them.

Where AI for research genuinely helps

Task	Safe?	Why
Summarising a document you supply	Yes	The source is in front of it and you can check
Explaining a concept you’ll verify	Yes	Errors are detectable once you know what to look up
Suggesting search terms	Yes	Output is a starting point, not a finding
Extracting figures from a paper you pasted	Mostly	Check every number against the page
Recalling a citation from memory	No	This is where fabrication lives
Stating a statistic without a source	No	Confident and unverifiable is the worst combination

The pattern is consistent. Give it a document and ask about the document, and it performs well. Ask it to recall the world, and you’re relying on memory it has no way to flag as unreliable.

The three-step workflow

This survives deadline pressure because each step is fast, which is the only reason people keep doing it.

One: separate the claim from the source. When the model states a fact, note it as unverified rather than writing it into your draft. Anything you type into a document acquires false authority the moment it looks finished.

Two: open the source yourself. Not search for it, not check the title looks right. Open the page and find the sentence containing the figure.

Three: copy the wording, not the paraphrase. If the source says “up to 7x increases in pre-training speed”, write that rather than “seven times faster”, because the qualifier usually matters and paraphrase quietly strips it.

Retrieval changes the odds

Tools that search and then answer are structurally safer than tools answering from memory, and it’s worth understanding why.

The original RAG paper described combining “pre-trained parametric and non-parametric memory”, and the practical effect is that answers come from documents that demonstrably exist.

It doesn’t eliminate the problem, though. Retrieval matches on similarity rather than correctness, so a system can cite a real source that doesn’t support the claim, which our piece on how RAG actually works covers.

That failure is harder to catch than invention, because the citation checks out until you read it.

There’s a second-order problem arriving as well. Villalobos and colleagues projected that models “will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032“, and the gap is being filled with generated text.

Which means a web page confirming a statistic may itself be model output rather than an independent check. Verification now has to reach the primary source, not the first page that agrees with the claim.

That’s a real change from a few years ago. Corroboration used to be cheap because independent sources were genuinely independent, and searching for agreement no longer establishes that.

Prompting that reduces fabrication

Two instructions do most of the work here.

Give the model a licensed exit: “If the answer isn’t in the material provided, reply exactly NOT FOUND.” That single line removes a large share of invented answers, because you’ve made not knowing a valid low-cost response.

And ask for reasoning before the conclusion rather than after, since Google Research found in Chain-of-Thought Prompting that working through steps improves accuracy substantially. Both are covered further in our prompting guide.

Long documents have a positional trap

If you paste a large paper and ask questions, where the answer sits inside it affects whether you get it.

Stanford’s Lost in the Middle found that “performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts”.

So a summary of a fifty-page report can silently miss the middle twenty pages. Ask about specific sections rather than requesting one summary of everything.

The habit that actually holds

Every rule above collapses under deadline unless one thing is true: you never let an unverified number into a draft.

Keep them in a separate list until each is checked. It sounds fussy and it’s the only version of this that works, because a figure sitting in your document at 11pm will get published.

If you cannot verify something, write around the gap or say the data isn’t available. An honest absence beats a plausible fabrication, and readers can tell the difference more often than writers expect.

What would change this

Models that express calibrated uncertainty would help enormously, and nothing in current preference training rewards that.

Verifiable rewards point somewhere better. DeepSeek’s R1 work reported “self-reflection, verification” emerging where answers could be checked mechanically, and a model that checks its own citations against a real index would close most of this gap.

Until then the burden sits with you, and the workflow above is the cheapest way to carry it.

One last framing that helps. Treat the model as a very fast research assistant who has read enormously and remembers imperfectly, and who has never been told that admitting a gap is acceptable.

You’d check that person’s references before publishing under your own name. The same standard applies here, and it’s the only one that holds up.