UT Dallas and NIST detect LLM hallucinations with 2 answers, not 10

Researchers at the University of Texas at Dallas and the National Institute of Standards and Technology say they can spot a hallucinating language model using two generated answers rather than the ten that leading methods sample. Their paper on diversion decoding, first submitted on 11 July 2026 and revised on 19 August, reports an AUROC of 78.49% on Llama 2 13B. The strongest baseline they tested, semantic entropy, scored 72.1% on the same setup.

The mechanism is adversarial rather than statistical. The team prompts the model, keeps its greedy answer, then prompts it again while rejecting any token that would rebuild that answer. So their own worked example asks for the capital of France, gets “Paris”, then blocks “Paris” and takes the next most likely token. How stubbornly the model fights its way back becomes the uncertainty signal, and a gradient boosting classifier turns those features into a hallucination score.

That matters because the paper’s stated case against the existing options is cost, not accuracy. The paper measures an expansion ratio, meaning generated tokens divided by tokens in the first answer. Sampling methods that draw ten responses sit at 10. Diversion decoding sits at 3.6, a median taken across 4,900 examples.

Method	AUROC, Llama 2 7B	AUROC, Llama 2 13B	Expansion ratio
Predictive entropy	67.0%	70.1%	1
Normalized predictive entropy	67.8%	63.8%	1
Lexical similarity	64.4%	61.7%	10
Semantic entropy	71.9%	72.1%	10
Diversion decoding	74.66%	78.49%	3.6

Existing approaches are often probabilistic and computationally expensive, limiting their practical applicability.

Abdeen et al., arXiv

Still, the evaluation is narrower than the headline number suggests. Everything runs on TriviaQA, with 1,300 questions for training and 3,600 for testing, on 4-bit Llama 2 models and a single NVIDIA H100. The method also needs output token probabilities, so it’s suited to open weights rather than a closed API. Push the training ratio to 90% and the paper reports roughly 82% for the 7B model and around 78% for the 13B, which is the smaller model overtaking the larger one.

The gap between those two figures is the part worth sitting with. A detector that improves on one backbone and stalls on another isn’t measuring hallucination in general, it’s measuring one model’s behaviour under pressure. That’s a recurring problem in this area, and it’s part of why the word hallucination itself makes these errors harder to fix.

The field is splitting by modality

Three more detection papers on arXiv show how far apart this work has drifted, and each one measures something different. ReWEIGH, posted on 19 August, is a training free decoding intervention for vision language models that cuts hallucinated object mentions by up to 21.3% on four 7B backbones, at an average added latency of 1.33% per token. DeMTS treats the denoising trajectories of diffusion language models as multivariate time series, tested on two backbones and three benchmarks.

The fourth isn’t a method at all, but a dataset. HalluTruthQA-4K holds 4,000 expert curated Arabic question answering instances across Islamic knowledge, history, science and geography. Of those, 1,643 are hallucinated and 2,357 are not, with 1,843 annotated erroneous spans. It’s the official dataset for Track 2 of the HalluScoring 2026 shared task.

Put together, they show a field that hasn’t agreed what it’s measuring. One group scores trivia answers from an autoregressive model, another counts objects a vision model invented, a third reads denoising steps, and a fourth is still building the ground truth. Their numbers don’t line up, so none of them checks another, though that’s a familiar problem if you’ve watched safety benchmark rankings shift on small models.

What would move this along is somebody who runs diversion decoding against a backbone the authors didn’t pick, on a dataset that isn’t TriviaQA. The efficiency claim is the interesting one, because a 3.6 expansion ratio is a different budget from a 10. Whether the accuracy survives contact with a different model is still open.