Mixture of Experts.
One model. Selective routing.

“Expert” means a trained sub-network, not a little specialist agent.

A Mixture of Experts (MoE) is a neural-network design that combines a set of expert sub-networks with a gating network or router. In many Transformer language models, an MoE layer uses feed-forward networks as its experts. They are trained as parts of the larger model, not launched as separate chatbots that each receive the whole question.

In a sparse MoE, the router uses a token’s current representation to choose one or more experts. Those experts process that token, and the model combines their outputs before continuing. Another token in the same prompt may be routed differently.

The name can suggest clean human-readable specialties, such as a “math expert” and a “poetry expert.” The architecture does not promise that kind of division. Experts learn useful internal patterns through training; interpreting those patterns takes separate evidence.

A sparse model does not run every expert for every token.

This is a form of conditional computation: the model has a larger pool of parameters available, while each token uses only a selected portion of the expert computation. The original sparsely-gated MoE paper describes a trainable gating network choosing a sparse combination of experts for each input.

Architectures make different routing choices. The Switch Transformer uses a simplified top-one route: each token goes to one expert in an MoE layer. Mixtral 8x7B chooses two of eight feed-forward experts for each token at each MoE layer. “MoE” describes a family of designs, not one universal routing algorithm.

Active parameters tell you about a token’s path, not the whole machine.

For an MoE model, the total parameter count includes the expert weights that belong to the model. The active-parameter count estimates the parameters used to process one token along its selected route, including the relevant shared parts. The two numbers describe different things.

Mixtral’s 8x7B report gives a clear example: about 47 billion parameters in total, with about 13 billion active for each token during inference. Different tokens can select different pairs of experts. Those numbers describe that specific architecture and should not be applied to other MoE models.

Actual memory and latency depend on precision, cache, batching, hardware, how experts are distributed, and the serving software. For a device-level comparison, see how small models fit a deployment budget ↗ and what happens during AI inference ↗.

Sparse computation can shift the bottleneck.

MoE can add model capacity while limiting the expert computation applied to each token. But choosing and dispatching experts adds work. Distributed models may need communication between devices to reach the selected experts; poor routing balance can leave some experts overloaded while others do little work.

Training also has to keep routing workable. Switch Transformer research discusses balancing expert use, routing instability, and communication cost. In practice, a model with fewer active parameters is not automatically faster or cheaper for every device, batch size, or request pattern. Measure the deployed runtime.

Four quick distinctions.

MoE is one architectural way to balance model capacity and computation. If you are comparing model families, also see how large language models generate text ↗ and how quantization changes model weights ↗.