Sparse Mixture of Experts Explained for 2026 Models

The Parameter Count on the Box Stopped Meaning Anything

In 2023 you could read a model's size and roughly predict what hardware it needed. A 70B model wanted about 40GB of VRAM in 4-bit and would run, slowly, on two consumer cards. That heuristic is now broken. Models advertising 2.4 trillion parameters exist, and so do 480B models that serve faster than dense 70B ones.

The reason is sparse mixture of experts, or MoE. Almost every frontier open-weight release in 2026 uses it, and understanding the architecture is the difference between reading a model card accurately and badly over- or under-buying hardware.

This guide covers what the numbers mean, how routing works, and what it changes for people running models locally. If you're choosing quantisation formats instead, our LLM quantisation guide covers that, and our comparison of local LLM servers covers the software layer.

• Total parameters set your memory bill; active parameters set your speed — they are different numbers and both belong on the model card
• A sparse MoE model routes each token to a few experts out of many, so most of the network sits idle on any given token
• MoE is memory-hungry and compute-light — the opposite bottleneck profile to a dense model of the same speed
• For self-hosters, unified-memory machines punch far above their price on MoE workloads because capacity matters more than raw bandwidth

Quick Picks

• Best mental model: a hospital with many specialists where the triage nurse sends you to two of them, not to all of them
• Number to check first on any model card: active parameters per token — if it is missing, you cannot estimate serving cost
• Best hardware shape for MoE at home: high unified memory (a 96GB or 128GB machine) rather than a single fast GPU with 24GB
• Best value approach for most people: run a small dense model locally and call a large MoE by API — the crossover rarely favours self-hosting the giants
• Biggest trap: assuming a 4-bit quantisation of a 400B MoE will fit because "it is only 100 billion active" — it will not

What Are Active Parameters?

Every model card for a MoE release lists two numbers, and they answer two different questions.

Total parameters is how much model exists. All of it has to be somewhere your inference engine can reach — VRAM, system RAM, or a fast SSD it can page from. This is your capacity requirement.

Active parameters is how much of the model participates in producing one token. This is roughly what determines how much arithmetic happens per token, and therefore how fast generation runs.

In a dense model these two numbers are identical. In a sparse MoE they diverge dramatically. Qwen3.8-Max, announced this week, carries around 2.4 trillion total parameters and activates roughly 95 billion per token. It has the knowledge capacity implied by the larger figure and something closer to the per-token compute cost implied by the smaller one.

That is the entire value proposition. You are buying capacity at memory prices instead of at compute prices.

How Does Routing Actually Work?

Inside a transformer, each layer has an attention block and a feed-forward block. In a MoE model, the feed-forward block is replaced by a set of parallel feed-forward networks — the experts — plus a small router network.

For every token, at every MoE layer, the router scores the experts and picks the top few, typically two to eight out of anywhere from eight to hundreds. Only those experts run. Their outputs are combined, weighted by the router's confidence, and passed on.

Three consequences follow, and they explain most of the practical behaviour:

• Routing is per-token and per-layer, not per-request. A single sentence will touch a large fraction of the experts across its length, which is why you cannot simply keep "the experts you need" in memory and evict the rest
• The router is trained, not rule-based. Experts do not correspond to human-legible topics. An expert is not "the Python expert" — it is whatever specialisation minimised loss during training
• Load balancing is a training problem. Left alone, routers collapse onto a few favourite experts, so training includes auxiliary losses to spread traffic. This is why expert counts and top-k values are architecture decisions rather than tunable dials at inference time

Why MoE Changes What Hardware You Want

Here is the part that trips people up. A dense model and a MoE model with the same active-parameter count have similar compute needs but wildly different memory needs — and consumer hardware is optimised for the opposite balance.

A high-end consumer GPU offers enormous memory bandwidth and compute against a small pool of VRAM. That is an excellent match for a dense model that fits, and a poor match for a MoE model that does not. If your MoE weights spill to system RAM across a PCIe link, the effective speed collapses regardless of how few parameters are technically active, because the router keeps asking for experts that live on the wrong side of the bus.

This is why unified-memory machines have become disproportionately interesting for local AI. A workstation with 128GB of unified memory has mediocre peak throughput compared to a discrete GPU, but it can hold a large MoE model entirely resident. On MoE workloads it frequently wins, and our mini computer and local AI hardware coverage has tracked several such machines through 2026.

The rules of thumb worth internalising:

• Estimate memory from total parameters, at roughly 0.5GB per billion parameters in 4-bit, plus context overhead
• Estimate tokens-per-second from active parameters, then discount heavily if anything is spilling off-device
• Prefer capacity over bandwidth when choosing between two machines at the same price for MoE work
• Watch the batch-size behaviour — MoE serving efficiency improves markedly with concurrency, which is why hosted APIs price these models so aggressively and why a single-user local instance sees worse economics than the provider does

Is a Small Dense Model Better Than a Big MoE?

For a lot of local use, yes — and this is the most useful thing to say plainly.

A 27B or 32B dense model in 4-bit fits comfortably on a single 24GB card, runs fast, and handles summarisation, extraction, classification, and routine code assistance well. A 400B-plus MoE will be smarter on hard reasoning and long-horizon agentic work, but you will spend an order of magnitude more on hardware to find that out.

The pattern that works for most people: run the small dense model locally for volume and privacy-sensitive work, and call a hosted MoE for the hard 5%. The economics of that split are better than either extreme, and it is why labs now ship a frontier MoE and a small dense sibling together — Qwen shipped a 27B checkpoint alongside its 2.4T flagship for precisely this reason.

What Should You Check Before Downloading a MoE?

A short checklist that saves an afternoon:

• Active parameters per token — if the card does not state it, treat serving cost as unknown
• Expert count and top-k — these determine how much of the model a typical request touches
• Whether the quantisation you want actually exists for that architecture; MoE support in quantisation toolchains lags dense support by weeks
• Context window versus your real workload — a million-token window costs memory you may never use
• Licence terms for commercial use, which vary far more across open-weight MoE releases than the phrase "open weights" implies

The Direction of Travel

Sparse MoE won because it decouples the thing that makes models smart (capacity) from the thing that makes them expensive to run (per-token compute). That decoupling is why frontier-quality inference has fallen in price faster than most people expected through 2026, and it is why the headline parameter count on a model card has quietly stopped being a specification and started being a marketing figure.

Read the second number. It is the one that determines what you will actually pay.

Sources: MarkTechPost on Qwen3.8-Max architecture — August 3, 2026; TNGlobal on Qwen3.8-Max parameters and context — August 4, 2026; Alizila — August 3, 2026.