Signum
Learn

Mixture of experts explained: why a trillion-parameter model can run like a much smaller one

By Elliot Marsh, Editor · · 4 min read

When Mistral released the weights of Mistral Large 4 in October, the headline figure was about one trillion parameters. The more useful number was further down: about 52 billion active parameters per step. That gap is the signature of a mixture-of-experts model, the design behind many of today's largest open and closed models.

The short version

A mixture-of-experts (MoE) model splits part of its network into many smaller sub-networks called experts. For each token it processes, a small router picks a few experts to use and skips the rest. So the model has a huge number of parameters in total, but only a fraction of them do work for any given token.

Dense models first

In a conventional dense Transformer, every parameter is used for every token. A 70-billion-parameter dense model does roughly 70 billion parameters' worth of computation per token. Making it smarter by making it bigger makes every token proportionally more expensive.

What MoE changes

Transformer layers contain a feed-forward block, a large part of the model's parameters. In an MoE model, that block is replaced by many parallel experts plus a router:

  1. A token arrives at the layer.
  2. The router, a small learned network, scores every expert for this token.
  3. The token is sent to the top few experts, often one or two (or a handful of smaller ones in newer designs).
  4. Their outputs are combined, weighted by the router's scores.

Different tokens can go to different experts, and the same token can go to different experts in different layers.

So the model has the capacity of a very large network (its total parameters) and the computation per token of a much smaller one (its active parameters).

A short history

The idea dates back to the early 1990s, but the modern version comes from a 2017 Google paper, Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al.), which used thousands of experts in a language model. Google's Switch Transformers paper (Fedus, Zoph and Shazeer, 2021) simplified routing to a single expert per token and scaled to over a trillion parameters. Mistral's Mixtral 8x7B, published in 2024, brought the approach to widely used open-weight models: eight experts per layer, two chosen per token.

What "experts" really are

The name is misleading. Experts do not neatly specialise in subjects such as "biology" or "French". Analyses of trained MoE models, including the Mixtral paper, have found routing tends to follow lower-level patterns such as syntax and token types more than topics. Think of experts as parallel capacity the model learns to use, not a panel of specialists.

Total versus active parameters

When you see both numbers, they tell you different things:

  • Active parameters roughly determine compute cost and speed per token. A model with 52 billion active parameters does work closer to a 52-billion dense model than a trillion-parameter one.
  • Total parameters determine memory. All the experts must be loaded, because any token might need any of them. That is why Mistral Large 4, despite modest active parameters, needs a rack-scale setup: reports cite eight Nvidia B300 GPUs for the standard version.

That is the central trade-off: MoE saves compute but not memory.

Why labs use MoE

  • Better quality for the compute. For a fixed training and serving budget, MoE models generally reach better results than dense models with the same active size.
  • Faster responses. Fewer active parameters per token means quicker generation once the model is loaded.
  • Cheaper serving at scale. For providers handling many users at once, the compute savings outweigh the memory cost.

The downsides

  • Memory hungry. Hard to run on a single consumer GPU or laptop, even when active parameters look small.
  • Harder to train. Routers can collapse onto a few favourite experts, leaving others unused. Training adds "load-balancing" terms to spread tokens out.
  • Communication overhead. When experts sit on different chips, tokens have to be shuffled between them, which can bottleneck performance.
  • Fine-tuning is fiddlier than with dense models of similar active size.

How to read model announcements

When a release says "1T parameters", check:

  1. Is it MoE? If so, how many parameters are active per token?
  2. What hardware does it need? That follows total size and precision (quantisation reduces it).
  3. What do independent benchmarks say? Parameter counts, total or active, are not a measure of quality on their own.

The bottom line

Mixture of experts is how labs build models with enormous capacity without paying the full compute cost on every token. It makes models faster and cheaper to serve at scale, but no easier to fit in memory. When you see a giant parameter count, look for the active figure next to it.

Sources