How AI image generation works, and why prompts fail

Why does an AI image generator ignore half of what you asked for? Usually because the request contained something the system has no mechanism for, and knowing which half is the whole skill.

This covers how these models work, what that means for prompts, and where they fail no matter how you phrase it.

AI image generation is denoising, not drawing

The model starts with pure noise and removes a little of it at each step, guided by your text, until an image appears.

Ho and colleagues set out the approach in Denoising Diffusion Probabilistic Models, training a network to reverse a process that gradually adds noise to real images. Run the reversal from noise alone and you get something new.

Nothing is composed. The image emerges everywhere at once, which is exactly why counting and text placement go wrong.

Doing this on full-resolution pixels was too slow to be practical. Rombach and colleagues moved it into a compressed latent space in the Latent Diffusion paper, which is what made consumer image generation possible at all.

How your words reach the image

Your prompt is turned into a vector and used to steer the denoising. It never becomes instructions in any literal sense.

The link between text and pictures comes from contrastive training. OpenAI’s CLIP learned from 400 million image-text pairs and matched a fully supervised ResNet-50 on ImageNet “without needing to use any of the 1.28 million training examples it was trained on”.

That’s the foundation, and it also explains the limits. The model learned what captions tend to accompany what images, so anything captions rarely describe is something it cannot reliably produce.

Guidance strength controls how hard the text pulls. Ho and Salimans introduced classifier-free guidance as a way to trade diversity for prompt adherence, and the slider labelled CFG in most tools is exactly this.

What prompts can and cannot control

You ask for	Reliable?	Why
Subject and setting	Yes	Heavily represented in captions
Style, medium, lighting	Yes	Captions describe these constantly
Camera angle, composition	Mostly	Common photographic vocabulary
Exact object counts	No	Nothing in the process counts anything
Specific text in the image	Improving	Letters are shapes to a denoiser
Precise spatial relations	Unreliable	Encoders bind attributes loosely

The counting failure is structural rather than a bug awaiting a fix. There’s no stage at which the system tallies objects, so “five birds” is a vibe rather than a specification.

Attribute binding is the same story. Ramesh and colleagues noted in the unCLIP work behind DALL-E 2 that models built on CLIP embeddings struggle to bind attributes to the right objects, which is why a red cube next to a blue sphere so often arrives with the colours swapped.

Splitting the request across two images and combining them by hand is frequently faster than fighting this, and it’s what most people doing production work end up doing.

Text rendering improved substantially once text encoders got stronger. Google’s Imagen found that “scaling the size of the frozen text encoder” improved sample fidelity more than scaling the image model, which is why recent systems handle short words far better than early ones.

Writing prompts that work

Describe the finished picture rather than issuing instructions. “A copper kettle on a wooden table, morning light from a side window” beats “make me an image of a kettle and put good lighting on it”.

Front-load what matters. Attention weakens across a long prompt, so the tenth clause carries less force than the first, and a paragraph of qualifiers dilutes everything in it.

Negation mostly doesn’t work in the prompt itself. Asking for no people frequently produces people, because the encoder registers the concept without the operator. Use the negative prompt field if your tool has one, which applies guidance in the opposite direction.

And fix the seed while you experiment. Changing one word with the seed held constant tells you what that word did; changing prompt and seed together tells you nothing.

The settings that actually matter

Guidance scale is the one to learn first. Low values produce loose, varied images that drift from the prompt; high values follow the text closely and start to look oversaturated and stiff.

Step count has sharply diminishing returns. Beyond a certain point you’re paying for compute that changes almost nothing, and the exact threshold depends on the sampler.

Aspect ratio matters more than people expect, because models are strongest at the shapes they were trained on. Extreme ratios often produce duplicated subjects and stretched anatomy.

Editing beats regenerating

When an image is almost right, most people rewrite the prompt and roll again. That throws away everything that worked.

Inpainting regenerates a masked region while keeping the rest, which is the correct tool for a bad hand or a wrong object. Image-to-image starts from an existing picture and applies a partial denoise, letting you keep composition while changing style.

Getting to a usable image is normally three or four passes of this kind, and treating generation as a single shot is what makes the tools feel unreliable.

The part you cannot ignore

Provenance is now a live obligation rather than an ethical nicety, and the rules arrived this month.

The EU AI Act’s transparency provisions require generated content to be marked, and within days of the deadline Anthropic began watermarking output worldwide. Image tools are squarely inside that scope, and our explainer on what the Act requires covers who it reaches.

Training data is the other exposure. Rulings so far have turned on how data was obtained and whether output competes with its source, so check what your provider says about its corpus and what indemnity it offers.

Style prompts naming a living artist sit in the most contested part of that argument. Whatever your view of it, it’s the request most likely to cause you a problem commercially.

Where AI image generation is heading

The gap between text encoders and image quality keeps closing, and Imagen’s finding suggests the remaining wins are in language understanding rather than in the denoiser.

Generation and understanding are converging, and the reading half works differently. Our explainer on how multimodal models process an image covers why they describe scenes well and read fine detail badly.

Expect counting and spatial reasoning to improve when generation is paired with a model that can plan a layout first. Until then, prompt for the picture rather than the specification, and edit rather than reroll.