How multimodal AI models actually process images

A multimodal model reads an image and answers questions about it, which sounds like it must be seeing. It isn’t, and knowing what it does instead tells you exactly where it will fail you.

The mechanism is simpler than the capability suggests, and unusually predictive of the failure modes.

How multimodal AI models turn everything into tokens

A language model processes a sequence of vectors. It has no opinion about where those vectors came from.

So the trick for images is to convert them into vectors that sit in the same space as text, and then feed both to the same network. Audio works the same way.

The model is not looking at your picture. It is reading a translation of your picture into the language it already spoke.

That translation is where the quality is won or lost, and it’s also where information gets discarded before the model ever reasons about anything.

How image and text got into one space

The foundational move was contrastive training on captioned images.

OpenAI’s CLIP trained on 400 million image-text pairs to place matching pairs near each other, and matched a fully supervised ResNet-50 on ImageNet “without needing to use any of the 1.28 million training examples it was trained on”.

Connecting that to a language model came next. DeepMind’s Flamingo bridged a frozen vision encoder to a frozen language model and reported state-of-the-art few-shot results across a range of vision-language tasks.

The efficient version of that bridge is what most systems now use. Li and colleagues’ BLIP-2 trained only a small querying transformer between the two frozen components, reporting it outperformed Flamingo80B on zero-shot visual question answering with “54x fewer trainable parameters”.

The encoder underneath

The component doing the translation is almost always a vision transformer, which cuts the image into fixed patches and treats each as a token.

Dosovitskiy and colleagues introduced the approach in An Image is Worth 16×16 Words, and the title is the mechanism. Patch size sets the finest detail the model can distinguish, permanently.

Teaching a model to follow instructions about images came later and cheaply. Visual Instruction Tuning generated multimodal instruction data using a language model and used it to train an open assistant, which is how most open vision models are built now.

What the architecture predicts about failures

Task	Reliable?	Why the architecture says so
Describing a scene	Yes	Exactly what caption training rewards
Reading large clear text	Mostly	Survives the downsampling
Reading small or dense text	No	Detail is lost before the model sees it
Counting objects	No	Captions rarely give exact counts
Precise spatial relations	Unreliable	Encoders bind position loosely
Fine detail in a corner	No	Compressed into a global representation

The pattern is that global understanding is strong and local precision is weak. An image is compressed into a few hundred vectors, and whatever didn’t survive that compression is unavailable no matter how you phrase the question.

Which gives you a practical rule. Crop before you ask. A model given the relevant region at full size answers questions it gets wrong on the whole page.

Native multimodality changed the ceiling

Early systems bolted a vision encoder onto a finished language model. Recent ones train on multiple modalities from the start, and the difference shows.

Google’s Gemini report described models “built from the ground up to be multimodal” across image, audio, video and text, rather than assembling separately trained components.

Joint training means the representation isn’t optimised for captions alone, so the model retains information that a caption-trained encoder would have thrown away as irrelevant.

That’s why text-in-image handling improved sharply between generations while counting barely moved. One is a representation problem and the other is an algorithmic one.

Video is mostly an economics problem

Video is handled as a sequence of frames, which makes the cost arithmetic brutal.

A minute at even one frame per second is sixty images, each consuming hundreds of tokens. That’s a large context spent before any question is asked, and our piece on what an AI feature costs works through why that bill grows faster than people expect.

Sampling fewer frames is the obvious response and it loses anything that happened between them. Systems trade coverage against cost, and neither choice is free.

Long inputs bring their own problem. Stanford’s Lost in the Middle found accuracy “significantly degrades when models must access relevant information in the middle of long contexts”, and a hundred frames is a long context by any measure.

Using multimodal AI models well

Ask specific questions rather than requesting a description. What is the total on this invoice beats describe this document, because the first names what to look for and the second invites the model to summarise the average of everything.

Give the image at the highest resolution the system accepts, and crop rather than shrink when you have to choose.

Verify anything numerical. A model reading a figure off a chart is doing the task it’s weakest at, and it will supply a confident number regardless, which our guide to verifying AI output covers.

For documents specifically, dedicated OCR still beats a general model on dense text. Use the multimodal model to interpret what OCR extracted rather than to do the extraction.

The security consequence people miss

If an image becomes tokens in the same stream as your instructions, then text inside an image is an input the model may follow.

That’s prompt injection with a new delivery route. A screenshot uploaded by a user can carry instructions your system never intended to accept, and it looks like a picture to everyone reviewing it.

Anything building agents on multimodal input needs to treat image content as untrusted, the same way it would treat a web page, which our guide to building agents covers on the permissions side.

Where this is going

The compression bottleneck is the thing to watch. Approaches that let a model request a closer look at a region, rather than working from one fixed encoding, would fix most of the failures in that table at once.

Some of those internal representations have now been isolated and named, which our piece on mechanistic interpretability covers, including features that fire on the same concept in text and in images.

The generation side inherits the same constraints in reverse, and our guide to why image prompts fail shows the counting and binding problems from the other direction.

Until then, the useful mental model is a system that understands scenes well and reads details badly. Design around that and these models are genuinely useful; assume they see what you see and they will disappoint you in ways that are hard to debug.