I Got Real Speech Recognition Running on a $10 Chip (Whistle)

Highlights

• Whistle is a 16.9 megabyte speech-to-text model released by Cactus Compute on October 2nd.

• The model runs completely offline on an ESP32-S3 microcontroller costing 10 dollars with no Wi-Fi or Bluetooth connection.

• Whistle supports English, German, French, Spanish, Italian, Dutch, and Polish while automatically detecting the spoken language.

• A trimmed four-layer version of the decoder fits onto the microcontroller flash storage after stripping out extra layers and taking up about 9 megabytes.

• Short spoken phrases achieve an 82 to 93 percent recognition rate, whereas single standalone words work only 44 percent of the time.

Timeline

1. 00:00 - 03:06

   Overview of Whistle and Flash Memory Constraints

   Demonstrations show the microcontroller responding to spoken voice commands in multiple languages by changing built-in LED colors. Cactus Compute developed Whistle as a small speech-to-text model that supports word timestamps and direct audio embeddings without writing transcripts. Unlike pre-built engine packages for phones and laptops, running this model on an ESP32 chip requires custom implementation.

   • An ESP32-S3 microcontroller running offline processes offline voice commands via a tiny microphone and button.
   • Whistle is almost nine times smaller than OpenAI's WhisperBase model while beating it on LibriSpeech benchmarks.
   • The complete model file size is 16.9 megabytes, which exceeds the 16 megabytes of flash storage available on the board.
2. 03:06 - 06:49

   Trimming the Model and Audio Processing Pipeline

   A custom tool strips unnecessary layers from the decoder to make the file fit inside the flash storage partition alongside firmware. The encoder processes 80 millisecond frames using eight layers where every frame references every other frame in the clip. The decoder writes out text sequentially using cross-attention to check against encoder outputs and n-gram memory tables.

   • The model decoder contains eight layers that can be cut down because each truncated version functions independently.
   • N-gram lookup tables consume 5.3 megabytes, representing nearly a third of the total file size.
   • Audio samples are recorded at 16,000 samples per second and chopped into 10 millisecond slices before convolution layers group them into 80 millisecond frames.
3. 06:49 - 10:31

   Hardware Setup and Processing Optimization

   Holding the button activates white recording mode, pulses during processing, and executes the requested command. Heavy matrix multiplications are split across both chip cores and moved to 8-bit integers on the vector unit. Testing reveals that full phrases yield up to 93 percent accuracy compared to 44 percent for isolated single-word commands.

   • The hardware setup connects an INMP441 digital microphone and a push button to an ESP32-S3 board using eight wires.
   • Processing time dropped from nearly 24 seconds down to about 10 seconds per command through core load balancing and vector math optimization.
   • Short phrases achieve significantly higher recognition accuracy than single words because the model relies on conversational sentence context.
4. 10:31 - 12:21

   Transcription Accuracy and Supplemental Performance Tricks

   Custom performance enhancements allow the trimmed microcontroller model to surpass full-size model accuracy benchmarks in specific test categories. Whistle is distributed under the Apache 2.0 license on Hugging Face for public developer tinkering.

   • English color commands achieve a 78 percent success rate that remains stable even with background microphone noise present.
   • Keyword biasing nudges the model toward expected color words whenever confidence drops.
   • A retry rule automatically re-evaluates failed attempts in alternative languages to verify high-confidence outputs.