Mixture-of-experts explained: why parameter counts stopped mattering

Why does a model with a trillion parameters cost roughly the same to run as one with a few billion? Because it isn’t using all of them on any given word, and that trick is called mixture-of-experts.

It’s the reason parameter counts stopped being a useful way to compare models, and the reason nearly every major lab now builds this way.

Dense models use every parameter, every time

In a standard transformer, every token that passes through touches every weight in the network. A 70-billion-parameter model performs computation proportional to all 70 billion, for every single token, whether the task is hard or trivial.

That’s wasteful in a specific way. Most of what a model knows is irrelevant to any particular sentence, and yet the arithmetic runs anyway.

The idea is older than the current wave

This is not a 2024 invention. In January 2017, a team including Noam Shazeer, Quoc Le, Geoffrey Hinton and Jeff Dean published Outrageously Large Neural Networks, describing a layer of “up to thousands of feed-forward sub-networks” with “a trainable gating network” choosing which to use per example.

They claimed “greater than 1000x improvements in model capacity with only minor losses in computational efficiency”, using up to 137 billion parameters between stacked LSTM layers. That was before transformers dominated, and the core mechanism has barely changed since.

What changed was that the engineering caught up, which is the part the later work is really about.

Sparse models pick a few experts per token

A mixture-of-experts layer replaces one large feed-forward block with many smaller ones, called experts, plus a small router that decides which experts each token should visit.

Google Research set out the cleanest version of this in Switch Transformers, where the model “selects different parameters for each incoming example”. Only the chosen experts do any work, so the compute per token stays flat while the total parameter count grows enormously.

Total parameters describe how much a model knows. Active parameters describe what it costs to ask.

The Switch paper reported “up to 7x increases in pre-training speed with the same computational resources” against a comparable dense baseline, and a “4x speedup over the T5-XXL model”. Those gains came from the same compute budget, spent differently.

What an expert actually is

The word invites a misconception worth clearing up. An expert is not a specialist in medicine or French or code. It is a feed-forward sub-network, and what it ends up handling is whatever the router learned to send it.

To see why that matters, it helps to know where these layers sit. The transformer, introduced in Attention Is All You Need, alternates attention blocks with feed-forward blocks, “dispensing with recurrence and convolutions entirely”.

Mixture-of-experts replaces the feed-forward half. Attention stays dense, because every token genuinely does need to look at every other token, so the saving comes from only one side of the architecture.

Researchers who have inspected trained routers usually find divisions that look arbitrary rather than semantic: punctuation here, numerals there, and a lot of overlap. The partition is real, it just doesn’t map onto categories a person would have chosen.

Active versus total parameters

This is the distinction that makes model announcements confusing, so it’s worth pinning down.

Figure	What it tells you
Total parameters	Memory needed to hold the model. Every expert must be loaded.
Active parameters	Compute and latency per token. Only the routed experts run.
Number of experts	How finely the knowledge is divided
Experts per token	How many run at once, usually one or two

So a headline parameter count tells you about the hardware you need to hold the thing, not about the speed you’ll get. Two models with identical totals can differ several-fold in serving cost depending on how many experts fire.

Why the economics pushed everyone here

Training happens once. Serving happens on every request, forever, and that asymmetry decides architecture.

The context is that training compute has been growing at roughly 5x per year since 2020, per Epoch AI. When each generation costs an order of magnitude more, any technique that buys capability without buying proportional compute gets adopted quickly.

DeepMind’s compute-optimal training work sharpened the same pressure from the training side, finding that “for every doubling of model size the number of training tokens should also be doubled”. Sparsity is one answer to a budget that has to stretch across both.

If you can add capability by adding experts without adding compute per token, you get a better model at roughly unchanged marginal cost. Given how hard frontier training budgets have become to justify, that is a very attractive trade.

It shows up directly in what models cost. The spread between $6 and $30 per million output tokens across models at similar capability is partly an architecture story.

The parts that are genuinely hard

Sparsity is not free, and the Switch authors were direct about what stood in the way: “complexity, communication costs and training instability”.

Routing has to be learned alongside everything else, and it can collapse. If the router sends most tokens to a handful of favourites, the other experts never train and you have paid for parameters that do nothing.

Communication is the other tax. Experts are spread across devices, so every token routed to a distant expert means moving data between chips, which is exactly the operation that hardware is worst at.

Labs counter this with auxiliary losses that penalise the router for over-using any single expert, effectively forcing balance. That works, and it also means some of the routing decisions are made for load-balancing reasons rather than because they produce the best answer.

Which is a genuine cost worth being clear about. A dense model routes nothing and therefore never trades quality for utilisation.

Switch Transformers also reported enabling training of large sparse models in bfloat16 for the first time, which mattered because numerical instability had been a practical blocker rather than a theoretical one.

What it means if you are choosing a model

Three things follow, and they’re all practical.

Self-hosting a sparse model needs memory for the full parameter count even though compute stays low, so a model advertised as efficient can still be the one that will not fit on your card. That’s a common surprise when weighing self-hosting against an API.

Meta’s Llama 3 flagship is a useful contrast: “a dense Transformer with 405B parameters”. Dense means every one of those 405 billion runs on every token, which is simpler to serve and considerably more expensive per request than a sparse model of similar total size.

Comparing a sparse model to a dense one on parameters alone is meaningless. Compare on measured latency and price per token, since those reflect what actually runs.

Reasoning models complicate this further, since they spend extra compute at inference time on top of whatever the architecture costs. Our piece on when that extra spend is worth it covers the arithmetic.

And expect uneven performance across domains. Knowledge is partitioned between experts, so a sparse model can be excellent on subjects with well-trained experts and noticeably weaker on the ones the router rarely picked.

The counter-case for staying dense

Sparsity has won on economics, but the argument for dense models is stronger than its current unpopularity suggests.

Dense models are predictable. Every token gets identical treatment, so quality doesn’t vary with which experts happened to be selected, and there is no routing failure mode to debug when output degrades on a particular domain.

They’re also far easier to fine-tune. Adapting a sparse model means reasoning about which experts your data touches, and a narrow fine-tuning set can leave most of the network untouched while overfitting the few experts it reaches.

And the memory requirement is not a detail. Holding every expert in VRAM to use a fraction of them per token is a poor deal on a single machine, which is why the models people actually run locally skew dense.

So the honest framing is that sparsity is an answer to a serving-cost problem at scale. If you are not serving at scale, it may be solving a problem you do not have.

What would change this picture

Two developments would matter. If memory bandwidth stops being the binding constraint on serving, the advantage of sparsity narrows, because the whole point is trading memory for compute.

And if routing quality plateaus while dense architectures keep improving, the added complexity stops paying for itself. Neither has happened, which is why the technique spread from a Google research paper to nearly every serious model in about four years.

The published evidence is also thinner than it looks. Frontier labs rarely disclose expert counts or routing details now, so most claims about how a given commercial model is built are inference rather than fact, which is worth remembering when reading any architecture comparison.