Mechanistic interpretability explained: reading a model from the inside

We build language models by specifying a training process, not a design. Nobody writes down what the finished network does, which means nobody starts out knowing.

Mechanistic interpretability is the attempt to fix that by reverse-engineering the computation inside a trained model into something a person can read.

A different question from explainability

The older field of explainable AI asks which inputs mattered to an output. That’s useful and it stays outside the model.

Mechanistic interpretability asks what algorithm the weights are implementing. The goal is a description you could reimplement by hand and get the same behaviour.

The ambition is not a heatmap showing which words mattered. It is a circuit diagram.

Anthropic’s Mathematical Framework for Transformer Circuits established much of the vocabulary, treating attention heads as composable operations rather than an undifferentiated blob of parameters.

Some circuits have actually been found

This is the part that makes the field credible rather than aspirational, because specific mechanisms have been isolated and tested.

Induction heads are the best-known example. Anthropic’s work on in-context learning and induction heads identified attention heads that find an earlier occurrence of the current token and copy what followed it, and argued they drive much of a model’s ability to learn from examples in the prompt.

That’s a real explanation of a capability that had previously been described as emergent, which is a word that mostly means we don’t know.

Whole circuits have been mapped too. Interpretability in the Wild traced how GPT-2 small completes sentences requiring an indirect object, identifying the specific heads involved and confirming the account by ablating them.

Why it is so hard: superposition

The obvious approach is to look at individual neurons and ask what each responds to. That fails, and the reason is structural.

Anthropic’s Toy Models of Superposition showed that networks pack more features than they have dimensions, storing them in overlapping directions rather than one per neuron.

So a single neuron fires for several unrelated concepts, which is why neuron-level inspection produces confusing results and always did.

The breakthrough was to stop looking at neurons. Towards Monosemanticity used a sparse autoencoder to decompose activations into features that each correspond to one interpretable thing, which turned an intractable problem into a merely difficult one.

It scales, and the features are steerable

A fair objection to early work was that it studied small models, and toy findings often don’t survive contact with production systems.

Anthropic’s Scaling Monosemanticity extracted millions of interpretable features from Claude 3 Sonnet, a production model, including abstract ones covering code errors, deception and bias.

Crucially, amplifying or suppressing a feature changed behaviour in the way the feature’s description predicted. That’s the test that separates a plausible story from an actual mechanism.

Method	What it gives you	Limit
Attention visualisation	Where the model looked	Looking is not reasoning
Probing	Whether information is present	Present does not mean used
Ablation	Whether a component is necessary	Says nothing about what it does
Sparse autoencoders	Interpretable features you can steer	Expensive, incomplete coverage

What a feature looks like

The word feature does a lot of work in this field, and it’s worth making concrete.

A feature is a direction in the model’s activation space that fires on one recognisable thing. Some are narrow, like a specific programming construct. Others are startlingly abstract, firing on a concept whether it appears in English, in code, or in an image.

That cross-modal behaviour is the strongest evidence these are real internal units rather than surface pattern matching, since nothing in the training data labels the connection.

Why anyone should care

The safety argument is the one usually made, and it’s stronger than it sounds.

Behavioural testing can only tell you a model behaved well on the cases you tried. A mechanism that produces deceptive output would be invisible to testing and visible to interpretability, at least in principle.

That matters more as models get agency. The events that prompted this month’s voluntary testing framework involved agents acting without instruction, and the government bodies doing that testing have only behavioural tools to work with.

There’s a practical case too. Understanding why a model fails at something tells you whether prompting can fix it or whether the capability isn’t there, which is a more useful answer than another benchmark score.

What it cannot do yet

Honesty about the limits matters here, because the field attracts more excitement than its current results support.

Coverage is partial. Millions of features sounds enormous and is a fraction of what a frontier model represents, so an audit finding nothing worrying has checked a small part of the system.

Cost is significant. Training sparse autoencoders on a large model is a serious compute expense, which limits how routinely this can be done.

And findings don’t transfer. Features found in one model are specific to that model, so every new release starts the work again, which is a poor fit for a release cadence measured in months.

The sceptical view

One serious criticism is that we may be imposing legibility rather than discovering it.

Sparse autoencoders are trained to produce interpretable features, so finding interpretable features is partly what they were built to do. Whether those features are the model’s own units of computation is a genuinely open question.

The counter is steering. If amplifying a feature changes behaviour exactly as its label predicts, the feature is doing causal work rather than being a convenient description, and that evidence is hard to dismiss.

A second criticism is timing. Capability is advancing faster than interpretability, so the models being deployed are always harder to understand than the ones already analysed.

What would count as progress

An interpretability finding that predicted a failure before behavioural testing caught it would change how seriously this is taken, and nothing public has cleared that bar.

Watch for automated feature labelling and for methods that transfer between models, since both attack the cost problem that currently keeps this out of routine deployment.

And watch whether regulators start asking for it. The EU AI Act’s documentation duties stop well short of requiring mechanistic understanding, and our explainer on what the Act requires covers where that line currently sits.