28.9M Parameter LLM Runs on $8 Microcontroller

An experimental project demonstrates that a 28.9 million parameter language model can now run entirely inside an ESP32-S3 microcontroller costing about $8. The model needs no internet connection; it generates text word by word on a connected small screen at roughly 9 tokens per second. The approach uses an architectural trick called Per-Layer Embeddings borrowed from Google's Gemma model research, allowing a model a hundred times larger than what was previously possible on such a tiny chip.

Leap from Mini Models to Larger Ones

Previously, language models that could run on microcontrollers like the ESP32-S3 were only around 260,000 parameters. The reason is simple: the chip has just 512KB of fast SRAM usable for storing the entire model during inference. A 260,000-parameter model was already near that limit. Now, this model is 100 times larger, with 28.9 million total parameters, though only 3.9 million of them are active in fast memory during inference. The remaining 25 million parameters sit in a lookup table stored on 16MB of flash memory.

How Does This Memory Trick Work?

The key is separating the "thinking part" from the "knowledge part." In many language models, most parameters reside in the embedding layers, which act like a giant dictionary. The model doesn't recompute this dictionary each time; it simply looks up entries as needed. With Google's Per-Layer Embeddings architecture, originally designed for the Gemma 3n and 4 models, this large parameter table can sit in slow storage like flash. When processing one token, the model only reads about 450 bytes from that table, not all 25 million entries. Meanwhile, the core computation part stays in fast SRAM, and the output head plus working memory goes into 8MB of PSRAM. A similar efficiency approach is seen in Gemini 3.6 Flash, which Google designed for speed, low cost, and safety.

Performance on Paper and in the Real World

The model is quantized to 4-bit, bringing the total size down to just 14.9MB. In tests, it reaches 9.5 tokens per second overall, with 9.7 tokens per second of pure compute time. The entire process runs inside the ESP32-S3 chip; no data is sent to any external server. Full hardware specs: the microcontroller has 512KB SRAM, 8MB PSRAM, and 16MB flash, forming the physical foundation for this experiment.

What Can This Model Actually Do?

The model was trained on the TinyStories dataset, a collection of simple synthetic short stories from Microsoft Research (arXiv:2305.07759). As a result, it can write coherent short stories, albeit in a very limited style. This capability grew out of the llama2.c project popularized by Andrej Karpathy, which showed that small language models can be trained and run in pure C code.

It will not answer questions, follow instructions, write code, or know facts.

These limitations are not due to the memory trick, but because the reasoning part of the model is still small. So don't expect to use it as a personal assistant. Its main strength lies in demonstrating the architecture, paving the way for large models to operate on extremely constrained hardware.

Who Is This Project Useful For?

Developers working on IoT devices, educational toys, or simple voice assistant prototypes will benefit from this approach. Imagine a doll that can tell its own stories, or an information panel displaying short narratives without needing an internet connection. However, for anyone needing a language model for complex tasks like programming, data analysis, or factual conversation, this model isn't the answer. It's also not suited for applications requiring very low latency, since 9 tokens per second is still slow for rapid real-time interaction.

Transparency: Code, Data, and Bug History Publicly Available

The project is completely open. Every stage, from firmware, assembly and flashing instructions, to training code, ablation, and quantization, is documented in a public repository. A full report on the method, on-chip measurements, and findings is recorded in the RESULTS.md file. The developer intentionally included the messy development history, including a bug in the initial parameter count that made the numbers look inflated, and the correction that followed. The commit history transparently shows the numbers shifting, offering a valuable lesson in honest research.

What's Still Unclear and Needs Testing?

Several questions remain unanswered. First, what's the actual power consumption of the ESP32-S3 when running this model, and is it low enough for battery-powered devices? Second, can this architecture scale to models with broader capabilities, for instance by enlarging the reasoning part or training on more diverse datasets? Third, long-term stability and reliability haven't been tested. All of these are areas for further exploration, not just a rehashing of existing tricks.

This project adds to the evidence that running AI directly on small devices is becoming more feasible. The on-device AI trend is also embraced by major companies, like when Google introduced Gemini Intelligence at Galaxy Unpacked to run on smartphones. While an $8 microcontroller is far more limited, this experiment reminds us that architectural innovation can break barriers once thought impossible.