ESP32-S3 Runs a 28.9M-Parameter LLM at 9 Tokens/s

Nine Tokens a Second, No Cloud, No Linux

Developer Slava S. has a 28.9-million-parameter language model generating text at roughly 9 tokens per second on an ESP32-S3 development board that costs about $8. Not streaming from an API, not offloading to a phone — the whole inference loop runs on a microcontroller with 512KB of SRAM and 8MB of PSRAM, driving a small I2C display. CNX Software wrote it up on August 3, 2026.

• 28.9M-parameter model trained on TinyStories, quantized to 4-bit and stored as a 14.9MB file on 16MB of flash
• Roughly 9 tokens per second on an ESP32-S3 board costing about $8, with no network connection required
• Google's Per-Layer Embeddings technique keeps a 25M-parameter embedding table in slow flash, pulling only ~450 bytes per token into SRAM
• Built on Karpathy's llama2.c, with source published at github.com/slvDev/esp32-ai

How Do You Fit a Language Model in 512KB of SRAM?

You don't — and that's the trick. The interesting engineering here is not the model, it's the memory choreography around it.

The ESP32-S3 has three tiers of storage with wildly different speeds: 512KB of fast internal SRAM, 8MB of external PSRAM, and 16MB of flash that is slow but plentiful. The naive approach loads weights into RAM and immediately runs out of room. This project instead splits the model by access pattern. The transformer layers — the "thinking" core touched on every single token — live in SRAM. The output head and working buffers sit in PSRAM. And the embedding table, which is by far the largest single structure at around 25M parameters, stays in flash.

That last part only works because of Per-Layer Embeddings, a technique Google published for its on-device model work. Rather than needing the whole table resident, the model pulls roughly 450 bytes of embedding data per token directly from flash. A lookup that touches half a kilobyte is cheap even on slow storage; a lookup that touches 25 million parameters is not.

What Can a 28.9M-Parameter Model Actually Do?

Write short stories, and that is genuinely the whole scope. The model was trained on TinyStories, a dataset of simple children's narratives built specifically to study how small a language model can get while still producing coherent English. It is not going to answer questions, follow instructions, or call tools.

Being precise about that matters, because the achievement here is a systems achievement rather than a capability one. Nobody is replacing a hosted API with this. What has changed is the ceiling on what a $8 microcontroller can hold — roughly 100x more parameters than comparable ESP32 demos managed in 2023. The same memory-tiering approach applies directly to the small task-specific models that are actually useful at this scale: intent classification, command parsing, keyword spotting, structured text generation for a device that has no business talking to a server.

Readers following our mini computer coverage will recognize this as the far end of a trend we've been tracking, from Moonshine putting offline voice AI on a Pico 2 to NightRun booting a local LLM on a Pi 5 with no operating system. Each of these strips away another layer that everyone assumed was load-bearing.

Why the ESP32-S3 and Not a Pi?

Cost, power, and the fact that it is already there. A Raspberry Pi 5 running a small model is an unremarkable result — it has gigabytes of RAM and a real operating system. An ESP32-S3 has neither, sells for single-digit dollars, and sits inside an enormous number of already-deployed sensors, displays, and controllers.

The S3 specifically brings vector instructions and PSRAM support, which is why it and not the older ESP32 variants is the target. It's the same reason the chip keeps showing up in ambitious builds like the ESP32 web browser that loads real pages in text mode — there is more headroom in it than the price suggests.

Try It Yourself

The code is on GitHub under slvDev/esp32-ai, built on top of Andrej Karpathy's llama2.c, which remains the clearest single reference for how transformer inference actually works when you strip out the framework. The bill of materials is one ESP32-S3 board with PSRAM and a small display.

For anyone building embedded products, the useful takeaway is not "microcontrollers can run LLMs now." It is that the memory hierarchy on these chips is far more exploitable than most firmware treats it as, and the per-layer embedding trick generalizes well beyond language models.

Sources: CNX Software — August 3, 2026; slvDev/esp32-ai on GitHub — August 2026; llama2.c by Andrej Karpathy — reference implementation.