Mixture-of-experts

Hook

A bigger feedforward layer usually means a slower one — every token has to pass through every parameter. What if a model could have dozens of feedforward blocks, each specialized, and every token only ever paid the cost of the two or so that actually matter for it?

Intuition

expert 0

2.00

expert 1 (not run)

0.00

expert 2

1.00

expert 3 (not run)

0.00

token 2 routes to experts {2, 0} — output = 1.269

Four small experts, one router. Pick a token and watch which two experts light up — the other two contribute nothing to the output and never even get evaluated.

Formalize

A mixture-of-experts layer replaces one feedforward block with $N$ of them plus a router. The router scores every expert, keeps only the top $K$ , renormalizes their scores to sum to 1, and combines just those:

y = \sum_{i \in \text{top-}K(\text{router}(x))} \frac{p_i}{\sum_{j \in \text{top-}K} p_j} \cdot \text{Expert}_i(x)

$y$ — the mixture-of-experts layer's output for this input.
$x$ — the input token or vector being routed.
$K$ — the number of experts kept per token (top- $K$ routing).
$\text{router}(x)$ — the router's raw score for each expert, given input $x$ .
$p_i$ — the router's (softmax) score for expert $i$ , before renormalizing over just the selected top- $K$ .
$i, j$ — indices ranging over experts, restricted to the top- $K$ selected ones.
$\text{Expert}_i(x)$ — the output of the $i$ -th expert sub-network applied to $x$ .

Unselected experts aren't down-weighted — they're never run
Every expert outside the top $K$ contributes exactly zero — not a small weight, but literally never computed.
Total capacity and per-token compute decouple
A model can have far more total parameters than a dense one while running only a fraction of them per token.

Play

expert 0

2.00

expert 1 (not run)

0.00

expert 2

1.00

expert 3 (not run)

0.00

token=2 → experts {2, 0}

expert 0 (not run)

0.00

expert 1

7.00

expert 2 (not run)

0.00

expert 3

0.00

token=-2 → experts {3, 1} — no overlap with token=2's experts

Token $x=2$ and token $x=-2$ activate completely disjoint pairs of experts — zero overlap. Each token gets routed to whichever specialists the router thinks fit it best, and the rest of the model, for that token, might as well not exist.

Worked example

Four experts, router weights $[0.5, -0.5, 1.0, -1.0]$ , top-2 routing:

Token x=2 routes to experts 2 and 0
Router logits: $[1, -1, 2, -2]$ . Softmax, subtracting the max logit ( $2$ ) for stability:
- $e^{1-2}\approx0.368$
- $e^{-1-2}\approx0.050$
- $e^{2-2}=1$
- $e^{-2-2}\approx0.018$
- Sum $\approx0.368+0.050+1+0.018=1.436$
- Full probabilities:
  - expert 0 $\approx0.368/1.436\approx0.256$
  - expert 1 $\approx0.050/1.436\approx0.035$
  - expert 2 $\approx1/1.436\approx0.696$
  - expert 3 $\approx0.018/1.436\approx0.013$
Top-2 are expert 2 ( $\approx0.696$ ) and expert 0 ( $\approx0.256$ ), summing to $\approx0.952$ . Renormalizing:
- $0.696/0.952\approx0.731$ for expert 2
- $0.256/0.952\approx0.269$ for expert 0
Experts 1 and 3 are never evaluated.
The combined output blends only the selected two
Expert 2's output is $1$ , expert 0's is $2$ . Combined: $0.731(1) + 0.269(2) \approx 1.269$ — a weighted blend of exactly two specialists, not all four.
A different token activates a disjoint pair entirely
Token $x=-2$ flips every router logit's sign, routing instead to experts 3 and 1 — the exact two experts that sat idle for $x=2$ . Same router, same experts, completely different active subset.

Checkpoint

Find the token, among the three candidates, that routes to expert 1.

Pick a token to try it

Summary

y = \sum_{i \in \text{top-}K} \tilde p_i \cdot \text{Expert}_i(x) \qquad\text{— total capacity can grow, per-token compute doesn't have to}

Mixture-of-experts decouples a model's total parameter count from its per-token compute cost: adding more experts adds capacity, but as long as $K$ stays fixed, every token still only touches $K$ of them. That's the entire mechanism behind how modern large language models scale to hundreds of billions of parameters while keeping inference cost close to a much smaller dense model's. The next chapter asks the question this raises directly: given a compute budget, how should it actually be spent — more parameters, or more data?