
MoE Expert Streaming: A 180B Model on a 32GB Mac in Python
Hugging Face's MoE expert streaming runs a 180B model on a 32GB Mac at about 14 tokens a second, within 6% of a dedicated engine, using plain Transformers.
Running a frontier-scale open model at home usually means a rack of GPUs or a very large unified-memory machine. A new Hugging Face engineering post shows another route. Engineer Tarek Ziadé got a 180-billion-parameter mixture-of-experts model running on a MacBook with 32GB of memory through the standard Transformers library, and pushed it to within about 6% of a specialized inference engine on token generation. The code is open, and the technique, called MoE expert streaming, is under review for Transformers itself.
- Target model: Qwen3.8-Flash-Next, 48 layers of 512 experts with 10 active per token, 121GB of experts in FP8.
- Memory footprint: experts quantized to 3 bits (49GB on disk) with a 12GiB device cache; about 18GiB in use after loading and a 25.4GiB peak on a 6,000-token prompt.
- Final speed: 13.9 tokens per second decode, 228 tokens per second prefill, against 14.8 and 220 for the Strata engine on the same M5 MacBook.
- Status: an open pull request (huggingface/transformers#50717) plus a demo Space; the API is still being discussed.
How Does MoE Expert Streaming Work?
A mixture-of-experts model only uses a handful of its expert sub-networks for each token, and those experts make up more than 90% of the weights. So the idea is simple: keep the dense layers in memory, leave the experts on the SSD, and read in only the ones the router actually picks. A bounded least-recently-used cache holds the hot experts on the device.
The clever part is how the cache is shaped. Rather than looping over experts one by one in Python, which ran at a third of normal speed in early tests, the cache is a single preallocated "slot pool" shared by every layer. Remapping the router's expert IDs to slots makes the pool look like an ordinary expert bank, so Transformers' existing grouped matrix-multiply kernel runs on it unchanged. Eight I/O threads fetch misses, and the next two layers' routers run early to guess which experts to prefetch. If you are new to the architecture, our sparse mixture-of-experts explainer covers the basics.
What Speed Can You Expect on a 32GB Mac?
The first working version managed about 10 tokens per second. Ziadé's team then closed most of the gap to Strata with targeted fixes: committing GPU work early on Apple's MPS backend for a 10% gain, runtime-compiled Metal kernels for model parts with no fast path on macOS, larger prefill chunks, and reading the next layer's experts while the current one computes. Prefill jumped from 68 to about 228 tokens per second.
The author is candid that roughly 14 tokens per second is still too slow for his own coding-agent workflow, which runs on a dual-GPU desktop. The remaining gap comes down to host round trips of about 10 milliseconds per token, which would need deeper control over GPU synchronization than PyTorch currently exposes.
Why This Matters for Local LLM Users
The comparison table in the post is the most useful takeaway for anyone running models at home. On a 24GB A10G, llama.cpp is hard to beat when experts fit in system RAM, while vLLM's offloading suits big batched workloads better than a single user. Streaming from disk fills the remaining gap: models whose experts do not fit in RAM at all. That makes the approach a promising fit for laptops and compact machines, and our guide on how much RAM a local LLM needs shows why that ceiling matters.
Everything in the project is Python plus Metal kernels compiled at runtime, now published on the Hugging Face Hub. Ziadé also notes that the code was written with AI assistance while he chose what to try and read the profiles. For more research like this, see our AI coverage.
Sources: Hugging Face — November 18, 2026; GitHub pull request — November 2026.
More AI Stories

OpenAI Chip Design Results: Up to 14% PPA Gains on AMD GPU
OpenAI says its models delivered up to 14% better power, performance and area on AMD's flagship GPU design, and AMD has sent some changes to the fab.

Gemini API Tool Retrieval: Load Tool Schemas Only on Demand
Gemini API Tool Retrieval lets agents keep hundreds of tools on standby and load schemas only when needed, cutting input tokens at no separate charge.

Qwen 4 Max Explained: 2.4T Parameters, $1.50 API Pricing
Alibaba's Qwen 4 Max pairs 2.4 trillion parameters with a 1M-token context at $1.50 per million input tokens. Here is what developers get today.
