Mixture-of-experts (MoE) model

A mixture-of-experts (MoE) model is an AI model built from many smaller parts, the experts, of which only a few do the work for each word. It needs about as much computation per word as a much smaller model, so it runs faster and cheaper than an ordinary model of its size. It still needs memory for all its parts. DeepSeek-V3 and Mistral Small 4 are built this way.

How does it work?

Each layer of a transformer has an attention part and a feed-forward network. An MoE layer replaces that one feed-forward network with a set of experts, each a smaller feed-forward network, plus a router: a small network that scores the experts for every token and sends the token to the best few. Their outputs are combined, weighted by the router's scores. In Mixtral 8x7B (Mistral AI, January 2024), each layer has 8 experts and the router picks 2 per token; the choice can differ from token to token.

Two numbers describe such a model. The total parameters all have to be in memory; the active parameters per token set the computation. Mixtral has 47 billion parameters in total and uses 13 billion per token, and it matched or beat Llama 2 70B and GPT-3.5 on the benchmarks its makers tested. Model names often carry both numbers: Gemma 4 26B A4B (Google, April 2026) has 26 billion parameters in total and 4 billion active.

Where does the idea come from?

Shazeer et al. (2017) built MoE layers with up to thousands of experts and reported more than 1,000 times the model capacity at a minor loss of computational efficiency. Switch Transformers (Fedus et al., 2021) sent each token to a single expert, trained models of up to a trillion parameters and reached up to 7 times faster pre-training with the same computing resources. Newer models use many small experts: DeepSeekMoE (2024) split experts into finer ones and added shared experts that every token uses. DeepSeek-V3 (December 2024) has 671 billion parameters with 37 billion active per token, and MiniMax-M2 (2025) has 256 experts per layer, of which 8 are active.

Where does it go wrong?

• Memory stays large. Serving Mixtral takes memory for all 47 billion parameters, although each token uses 13 billion. Hugging Face's explainer (December 2023) sums it up: MoE models need a lot of GPU memory because all experts are loaded.
• The router can favour a few experts. If most tokens go to a few popular experts, training becomes inefficient, so MoE training usually adds an extra loss that rewards an even spread. DeepSeek-V3 balances the load without such a loss, and MiniMax-M2 does it with bias terms in the router.
• The experts are not specialists in the everyday sense. When Mistral AI checked which experts handled arXiv papers, biology abstracts and philosophy texts, it found no obvious pattern by topic; the routing followed syntax more than subject.
• Routing costs time. The Mixtral paper notes overhead from routing and memory loads, and calls MoE layers more suitable for batched workloads.

Example

Suppose a company wants to run Mistral Small 4 (March 2026) on its own server. The model has 128 experts, uses 4 of them per token and has 119 billion parameters, of which 6 billion are active (8 billion including the embedding and output layers). Per token it computes like a small model, but all 119 billion parameters must be loaded: about 238 GB at 16 bits, or about 60 GB with 4-bit quantization. An ordinary dense model with 8 billion parameters would need about 16 GB at 16 bits.