IIT Delhi: safety prompts make VLMs refuse answers they can still see

Two researchers at IIT Delhi have published evidence that a safety instruction makes vision-language models refuse questions they can still see the answer to. The paper went up on arXiv on 19 August 2026. It measures the distance between what a model perceives and what it’s willing to say, and that gap turns out to be controllable.

Mehak Gupta and Tanmoy Chakraborty tested three aligned models: Qwen3-VL-8B-Instruct, LLaVA-v1.6-Vicuna-7B and Phi-3-Vision-128K-Instruct. Each of them saw 2,000 image-question pairs randomly sampled from each of VQA-v2, GQA and POPE. The image, the question, the model weights and the decoding settings all stayed fixed. Only the instruction changed, adding a line telling the model to abstain when an answer isn’t explicitly observable from the image.

That single line moved a lot of behaviour. In the paper’s paired comparison, Qwen3-VL answered 63% of VQA-v2 pairs correctly under the safety-constrained setup and abstained on 32%. On POPE, the benchmark built to probe object hallucination, it abstained on 56% of them.

Qwen3-VL-8B, safety-constrained	Correct	Abstain	Incorrect
VQA-v2, original	0.63	0.32	0.05
VQA-v2, refusal suppressed	0.81	0.09	0.10
GQA, original	0.53	0.28	0.19
GQA, refusal suppressed	0.63	0.05	0.32
POPE, original	0.41	0.56	0.03
POPE, refusal suppressed	0.68	0.01	0.33

The obvious reading is that the safety prompt breaks perception. The authors tested that by blurring the image with a Gaussian filter, then comparing the logit of the first generated token with and without the corruption, layer by layer. Abstained generations held their sensitivity to the image right through decoding. That rules out a collapse in visual grounding, which is the part that translates pixels into tokens in the first place.

Together, these findings reveal a previously underexplored failure mode in aligned VLMs: safety alignment can override grounded visual expression even when perceptual evidence remains internally preserved.

Mehak Gupta and Tanmoy Chakraborty, IIT Delhi, via arXiv

So the team went looking for where the refusal lives, using the tools of mechanistic interpretability. They took the difference in mean hidden states between abstained and answered samples at each decoding layer, which yields a refusal direction, then subtracted it from activations at inference time with a strength of one. No weights were retrained, so any behaviour change traces to the activations themselves. Suppressing that direction cut Qwen3-VL’s POPE abstention from 0.56 to 0.01 and lifted correct answers from 0.41 to 0.68.

The catch is what filled the space the refusals left. Incorrect answers on that same POPE run rose from 0.03 to 0.33, and on GQA they went from 0.19 to 0.32. The paper describes those increases as modest relative to the reduction in abstention. But on a benchmark built to catch hallucination, a move from 0.03 to 0.33 in wrong answers is a figure a sceptical reader can weigh differently.

The finding doesn’t sit alone. TRAPSBench, posted six days earlier, reports the mirror image: across 16 VLMs from five families, linear probes decode answerability from hidden states at up to 0.91 AUROC, yet the best score on its calibration metric was 0.292. Those models know when they should abstain and don’t. Gupta and Chakraborty’s models know the answer and won’t give it, and both papers land on expression rather than perception as the bottleneck.

That same direction is what attackers reach for. Abliteration strips refusal by projecting weight matrices orthogonal to it, and a preprint on refusal aliases treats the ease of extracting the direction as the vulnerability itself. Arditi and colleagues showed in 2024 that refusal sits in a one-dimensional subspace across 13 open chat models. Steering it is the same knob either way, which is why another 2026 preprint clamps the direction back on to restore refusals in Yoruba, Igbo, Igala and Hausa.

What’s worth watching is whether the effect reproduces beyond three models and three benchmarks, because the scope here is narrow. This is a v1 preprint with no peer review, the refusal directions were estimated once on Qwen3-VL and VQA-v2 before being applied elsewhere, and the intervention strength was fixed at one across every architecture. If the abstention rates hold on newer systems, the gap between seeing and saying stops being a curiosity and becomes an evaluation problem.