Building an On-Device LLM with Real-Time Input Using ESP32-S3 and 16MB Flash

When you set out to run a 28.9 million parameter language model on an $8 ESP32-S3 board, you quickly hit a wall. A model that runs smoothly in demo videos immediately freezes the moment it meets your custom hardware peripherals. With only 512KB of SRAM and having to reflash a 15MB partition every time you change a single sentence, development quickly becomes frustrating. To break through these constraints and build a device that actually accepts keypad inputs and operates, you have to design your memory caching and I/O buffers differently from the ground up.

Injecting Prompts via Serial Without Re-flashing Every Time

Flashing the board all over again just to change a single test sentence is a waste of time. By tying together asynchronous UART interrupts and FreeRTOS semaphores, you don't need to reboot the board at all. The model instantly understands sentences as you type them through the serial monitor or a keypad.

Assign GPIO pins 16/17 to the UART_NUM_2 port, set the hardware FIFO threshold to 120 bytes, and open the receive interrupt.
Create a 2048-byte ring buffer in the internal SRAM. This prevents data overflow crashes while inference operations are in progress.
Attach a streaming parser that detects newline characters, and directly convert the input text into an array of token IDs using a 43,056-byte BTK1 encoder library.
Set up an xPromptSemaphore binary semaphore. Once the receive task finishes writing tokens and calls xSemaphoreGive, the waiting inference task acquires the memory lock via xSemaphoreTake and starts computation.

Having this setup cuts down the time spent on prompt modification and testing by over 80%.

`
+------------------+ +-------------------+ +--------------------+
| External UART | ---> | HW FIFO Buffer | ---> | SW Ring Buffer |
| (Keypad/Monitor) | | (120 Bytes) | | (2048 Bytes) |
+------------------+ +-------------------+ +--------------------+
|
v
+------------------+ +-------------------+ +--------------------+
| LLM Forward Pass | <--- | xPromptSemaphore | <--- | BTK1 Tokenizer |
| (Inference Task) | | (Binary Lock) | | (43,056 B Library) |
+------------------+ +-------------------+ +--------------------+

Achieving 14 Tokens Speed with Per-Layer Embeddings and SRAM Caching

By using the Per-Layer Embeddings (PLE) architecture proposed by the Google Gemma 3n researchers, you can split and load a 4-bit quantized 14.9MB model. The embedding table, amounting to 25 million parameters, is placed in the 16MB SPI flash, while the Output Head weights (3.1 million parameters) and KV cache are placed in the 8MB PSRAM. Only the most frequently used 559K parameter Dense Compute Core and Hot Activation buffer are packed into the 512KB SRAM.

Pushing the speed above 14 tokens per second is trickier than it sounds.

Slap the __attribute__((noinline)) attribute directly onto the matvec_i8_range computation function. If the compiler automatically handles inlining, I-RAM cache misses occur, and computation time actually tanks from 94.9ms to 155.2ms.
Split the ple_model_proj operation and the qkv operation across the Dual Xtensa LX7 cores of the ESP32-S3. You have to run both cores simultaneously to get the proper speed.
Widen the prefetch buffer that preloads early layer embedding data into remaining SRAM space to clear the flash memory bottleneck.

A plain C port yields a desperate speed of 0.57 tokens per second. However, once you finish INT8 staging and SRAM prefetching, you exceed 14.0 tokens per second.

Optimization Stage	Latency per Token	Token Generation Speed	Key Technologies Applied
Pure C Port (Baseline)	1,757.2 ms	0.57 tok/s	Single core, FP32 operations
PSRAM Head & Scalar Optimization	193.9 ms	4.61 tok/s	Output Head placement in PSRAM
Dual-Core FP32 Applied	139.4 ms	6.22 tok/s	Dual-core layer-split computation
INT8 Staging + SRAM Optimization	94.9 ms	9.88 tok/s	INT8 quantization, noinline applied
SRAM Prefetch Buffer Expansion	~71.4 ms	14.0+ tok/s	Early layer SRAM prefetch

Controlling Current at 210mA with Light Sleep

Running dual cores continuously at a 240MHz clock spikes current consumption up to 210mA (777mW). Aside from the board getting hot, the battery melts away in no time. You need to use the esp_pm_configure command to lower the clock to 40MHz and enable automatic light sleep when there is no inference.

In the esp_pm_config_esp32s3_t struct, set max_freq_mhz to 240, min_freq_mhz to 40, and turn light_sleep_enable to true.
Execute esp_sleep_pd_config(ESP_PD_DOMAIN_VDDSDIO, ESP_PD_OPTION_ON). Even when entering sleep state, power to the SPI flash and RAM must be maintained so data isn't lost.
Use uart_set_wakeup_threshold and esp_sleep_enable_uart_wakeup to set a recovery interrupt so the CPU wakes up from sleep the moment a serial signal comes in.

Continuously running computations on a 3.7V 1000mAh LiPo battery lasts only 4.7 hours. However, in light sleep standby mode, the current drops sharply to 0.24mA (0.88mW). In real-world usage environments mixed with standby time, the average current is maintained around 15 to 30mA, extending battery life by more than 3x.

Building an On-Device LLM with Real-Time Input Using ESP32-S3 and 16MB Flash

Injecting Prompts via Serial Without Re-flashing Every Time

Assign GPIO pins 16/17 to the UART_NUM_2 port, set the hardware FIFO threshold to 120 bytes, and open the receive interrupt.

Create a 2048-byte ring buffer in the internal SRAM. This prevents data overflow crashes while inference operations are in progress.

Attach a streaming parser that detects newline characters, and directly convert the input text into an array of token IDs using a 43,056-byte BTK1 encoder library.

Set up an xPromptSemaphore binary semaphore. Once the receive task finishes writing tokens and calls xSemaphoreGive, the waiting inference task acquires the memory lock via xSemaphoreTake and starts computation.

Having this setup cuts down the time spent on prompt modification and testing by over 80%.

Achieving 14 Tokens Speed with Per-Layer Embeddings and SRAM Caching

Pushing the speed above 14 tokens per second is trickier than it sounds.

Slap the __attribute__((noinline)) attribute directly onto the matvec_i8_range computation function. If the compiler automatically handles inlining, I-RAM cache misses occur, and computation time actually tanks from 94.9ms to 155.2ms.

Split the ple_model_proj operation and the qkv operation across the Dual Xtensa LX7 cores of the ESP32-S3. You have to run both cores simultaneously to get the proper speed.

Widen the prefetch buffer that preloads early layer embedding data into remaining SRAM space to clear the flash memory bottleneck.

A plain C port yields a desperate speed of 0.57 tokens per second. However, once you finish INT8 staging and SRAM prefetching, you exceed 14.0 tokens per second.

Optimization Stage

Latency per Token

Token Generation Speed

Key Technologies Applied

Pure C Port (Baseline)

1,757.2 ms

0.57 tok/s

Single core, FP32 operations

PSRAM Head & Scalar Optimization

193.9 ms

4.61 tok/s

Output Head placement in PSRAM

Dual-Core FP32 Applied

139.4 ms

6.22 tok/s

Dual-core layer-split computation

INT8 Staging + SRAM Optimization

94.9 ms

9.88 tok/s

INT8 quantization, noinline applied

SRAM Prefetch Buffer Expansion

~71.4 ms

14.0+ tok/s

Early layer SRAM prefetch

Controlling Current at 210mA with Light Sleep

In the esp_pm_config_esp32s3_t struct, set max_freq_mhz to 240, min_freq_mhz to 40, and turn light_sleep_enable to true.

Execute esp_sleep_pd_config(ESP_PD_DOMAIN_VDDSDIO, ESP_PD_OPTION_ON). Even when entering sleep state, power to the SPI flash and RAM must be maintained so data isn't lost.

Use uart_set_wakeup_threshold and esp_sleep_enable_uart_wakeup to set a recovery interrupt so the CPU wakes up from sleep the moment a serial signal comes in.

Building an On-Device LLM with Real-Time Input Using ESP32-S3 and 16MB Flash

Related Video

This $8 Microcontroller Is Running a Language Model Locally!

Building an On-Device LLM with Real-Time Input Using ESP32-S3 and 16MB Flash

Injecting Prompts via Serial Without Re-flashing Every Time

Achieving 14 Tokens Speed with Per-Layer Embeddings and SRAM Caching

Controlling Current at 210mA with Light Sleep

More from the community

사내 시스템에 llm api 붙일 때 마주하는 현실적인 한계와 대응법

레거시 백엔드에 GPT-6 Astra 붙일 때 예산 승인과 보안 통과를 먼저 끝내는 법이 있습니다

에이전트끼리 대화하다 6천만 원 청구서가 나오는 이유

사내 RAG 벡터 검색에 Okta 권한 필터를 직접 거는 방법

브라우저 에이전트에게 내 구글 계정을 통째로 넘기면 안 되는 이유

Apple Won the AI Race

Comments (0)

Building an On-Device LLM with Real-Time Input Using ESP32-S3 and 16MB Flash

Injecting Prompts via Serial Without Re-flashing Every Time

Achieving 14 Tokens Speed with Per-Layer Embeddings and SRAM Caching

Controlling Current at 210mA with Light Sleep