Explainer · AI Architecture · How It Works

What Is Mixture-of-Experts? The AI Architecture That Lets a 1.6-Trillion-Parameter Model Cost Less to Run Than You Would Expect

Almost every major AI model released in the past two years uses Mixture-of-Experts architecture. The headlines mention the trillion-parameter count. What they rarely explain is why that number is misleading — and what is actually happening when an AI processes your request.

The Problem MoE Solves

For the first decade of modern AI development, making a model more capable meant making it larger. More parameters — the numbers stored in the model’s layers that determine how it responds to input — meant better performance. GPT-3, released by OpenAI in 2020, had 175 billion parameters. More capable models needed more parameters, more compute to train them, more compute to run them, and more electricity to power the hardware. The relationship between capability and cost was roughly linear: double the capability, roughly double everything else.

This created a practical ceiling. Training costs for frontier models reached hundreds of millions of dollars. Running them required specialised hardware. The energy consumption became a genuine concern. Something had to change about the fundamental architecture, not just the hardware.

Mixture-of-Experts (MoE) is the answer the research community settled on. It breaks the linear relationship between parameter count and computation cost by ensuring that not all parameters are used for every request. A model can have 1.6 trillion parameters while only activating 48 billion of them for a given input — keeping the model’s breadth of knowledge without paying the full computational cost every time.

A visual walkthrough of MoE architecture by Maarten Grootendorst, covering routing, sparse activation, and how MoE integrates into Transformer models. November 2024. Not a Wangdoo production.

The Hospital Analogy

The clearest way to understand MoE is to think of a large hospital.

When you call with a medical question, the hospital does not wake up every specialist on staff to answer you. A triage nurse first decides who you should speak to — a cardiologist if you describe chest symptoms, a dermatologist if you describe a rash, a neurologist if you describe headaches. Each specialist has deep knowledge in their area. The triage nurse does not need to know everything each specialist knows. Their job is routing — matching your specific question to the right expert.

A Mixture-of-Experts AI model works the same way. Inside the model, there are many specialised sub-networks called “experts.” Each expert is a small neural network that has, during training, developed stronger handling for certain types of input — certain languages, certain kinds of reasoning, certain technical domains. Alongside these experts sits a separate small network called the “router” or “gating network.” Its only job is to look at the input and decide which experts should handle it.

For each token — each word or word-fragment in your input — the router selects a subset of the available experts. Those experts process the token. The others sit idle. The model’s total parameter count reflects all experts combined. Its operational compute cost reflects only the experts that were active.

How the Router Actually Works

The routing decision happens at a layer-by-layer level inside the model. At each relevant layer of the Transformer architecture, instead of passing the token through a single large feed-forward network (as in a dense model), the token goes to the router first. The router produces a score for each available expert — essentially a probability of how relevant each expert is for this specific token. The top-scoring experts are selected. How many are selected depends on the model’s design; common choices are top-1 (one expert per token) or top-2 (two experts per token, with their outputs weighted by the router’s scores).

This routing happens millions of times during a single conversation — for every token, at every relevant layer, the model is making a fast routing decision. The routers are learned during training: they get better at assigning tokens to the right experts as the model trains on data. The experts themselves also specialise through training — not explicitly assigned roles, but developing them organically through the patterns in the training data they handle most often.

Where MoE Came From

The Mixture-of-Experts concept is not new. It was first formalised in a 1991 paper by Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton — “Adaptive Mixtures of Local Experts.” The original idea: train multiple expert networks, each specialising in different parts of the input space, with a gating network that learns to route inputs to the most appropriate expert. The architecture was sound but the compute needed to make it work at scale did not exist.

The crucial step toward modern MoE came in 2017, when Shazeer et al. at Google published “Outrageously Large Neural Networks” — introducing sparse gating, which meant only the top-k experts were activated per input rather than all of them. This reduced the compute overhead dramatically and made MoE practical at meaningful scale. Noam Shazeer, a co-author of that 2017 MoE paper, was also a co-author of the 2017 Transformer paper “Attention Is All You Need” — the same person helped create both the architecture and the efficiency layer now used to scale it. In June 2026, Shazeer announced he was leaving Google — where he had been co-leading the Gemini model family — to join OpenAI, confirmed on X and reported by Yahoo Finance and TechTimes.

The modern era of MoE in large language models began with Google’s GShard in 2020 — a 600-billion-parameter model that demonstrated sparse MoE architecture could work at unprecedented scale. The Switch Transformer in 2021, also from Google, extended this to 1.6 trillion parameters and simplified the routing to a single expert per token, addressing training instability issues. The GLaM model in 2022 showed that a 1.2-trillion-parameter MoE model could consume roughly one-third of the energy used to train GPT-3 while achieving better performance on 29 NLP evaluation tasks — the first clear demonstration that the efficiency gains were real and quantifiable.

Which Models Use MoE Today

MoE architecture is now standard practice among the most capable AI models. The confirmed and unconfirmed picture looks like this:

Model	Total parameters	Active per token	Confirmed?
Mistral Mixtral 8x7B	46.7B	~12.9B (2 of 8 experts)	Yes — Mistral published the architecture
DeepSeek-V3	671B	37B	Yes — confirmed verbatim in DeepSeek technical report (arXiv:2412.19437). Now superseded by V4
DeepSeek V4-Pro	1.6T	49B	Yes — confirmed from DeepSeek’s own Hugging Face model card and API docs. Released April 24 2026, MIT licence
Meituan LongCat-2.0	1.6T	~48B (range 33B–56B)	Yes — confirmed in Meituan’s own GitHub and model card
Google Gemini 1.5 and later	Not disclosed	Not disclosed	Widely reported; Google has not published architecture specifics
GPT-4	Estimated ~1.8T across 16 experts	Not disclosed	Not confirmed by OpenAI — widely believed based on leaked reports

Why This Changes the Pricing Conversation

Understanding MoE makes the AI pricing landscape significantly more legible. When comparing model costs, the total parameter count is the wrong number to focus on. The active parameter count — and the efficiency of the routing mechanism — are what actually determine how much it costs to run a query.

Meituan’s Wangdoo’s coverage of AI infrastructure developments. Meituan’s LongCat-2.0 costs $0.75 per million input tokens at standard pricing. GPT-5.5 costs $5 per million input tokens. LongCat-2.0 has 1.6 trillion total parameters; GPT-5.5 is a dense model whose parameter count OpenAI has not disclosed. The pricing difference is not purely a function of corporate strategy — it reflects the fundamental compute difference between activating ~48 billion parameters per token versus whatever GPT-5.5’s dense architecture requires. DeepSeek’s aggressive pricing is partly the same story: MoE architecture reduces the operational compute cost per query, which feeds through to lower pricing.

This also explains why “1.6 trillion parameters” in a headline is not as alarming as it sounds. The model does not cost 1.6 trillion parameters’ worth of compute to answer your question. It costs roughly 48 billion parameters’ worth — comparable to running a mid-tier dense model. The 1.6 trillion total parameters represent the model’s potential knowledge, held in reserve and drawn on selectively.

The Trade-Off MoE Creates

MoE is not a free efficiency gain. It creates a specific hardware trade-off that matters for anyone running AI infrastructure.

To run a MoE model, all of the model’s parameters — including the inactive experts — must be loaded into GPU or accelerator memory. The active parameter count determines the compute cost per token. The total parameter count determines the memory requirement. A 1.6-trillion-parameter MoE model might cost as much per query as a 48-billion-parameter dense model, but it requires the memory footprint of a 1.6-trillion-parameter model to run at all.

For large-scale inference providers — the companies running these models as APIs — this memory overhead is manageable across many servers. For someone attempting to self-host a large MoE model, it is the primary practical constraint. The 1.6-trillion-parameter total is the figure that determines how much hardware you need. The 48-billion-parameter active count is the figure that determines how fast each query runs once you have that hardware.

My Take — Mr Wangdoo

The reason this architecture matters beyond the technical details is that it changes what is possible at a given price point. Dense models hit a wall where making them more capable makes them meaningfully more expensive to run. MoE breaks that wall — you can increase total knowledge (more experts, more total parameters) without proportionally increasing the cost of a single query. That is why the dominant AI labs converged on this approach.

The GPT-4 situation is worth sitting with. OpenAI has a model that is almost certainly MoE — the parameter estimates and performance characteristics both point that way — and they have never said so publicly. That silence is itself a signal: the architecture has become too competitively significant to disclose voluntarily. When a company builds something genuinely important, the architectural details become proprietary information rather than research contributions to share. MoE went from a 1991 academic paper to a trade secret in about thirty years. The research community open-sourced enough of the approach (GShard, Switch Transformer) that everyone could use it. The specific implementations became the competitive moat.

Frequently Asked Questions