esp32-p4 speech recognition: A 65.7M Transducer, No Accelerator

esp32-p4 speech recognition just crossed a line that matters for anyone building voice into agent devices. A new report describes running Kroko, a 65.7-million-parameter streaming transducer, faster than real time on the ESP32-P4, a microcontroller with two 400 MHz RISC-V cores and no neural accelerator. The system transcribes the board's microphone live, finalizes a word about two seconds after it is spoken, and does it offline, with no cloud round trip. Speech recognition on microcontrollers has historically meant fixed command lists; Espressif's own ESP-SR recognizes up to 200 commands on the ESP32-S3. Writing down arbitrary speech is a different problem, and this post works through the report's numbers to explain how it was done, what the accuracy costs are, and where it fits alongside the Oído project on the ESP32-S3. Source

What the report demonstrates

The paper, published on arXiv in October 2026 by Billal Boumedine of USTHB, ports Kroko, a streaming Zipformer2 transducer for English trained by Banafo and released as int8 ONNX weights for sherpa-onnx, to the ESP32-P4 without any retraining. The target chip has two 400 MHz RISC-V cores with a 128-bit SIMD extension called PIE, 32 MB of PSRAM and 16 MB of flash, and no neural accelerator; the board adds an ES8311 codec with a microphone and a screen that shows the live transcript. Each 1.28-second chunk of audio is processed in 0.90 seconds using both cores, beam search included, which is 0.70 times real time, and the whole pipeline, model, engine, firmware and quantized weights, is published on GitHub with a browser-based installer. Source

The compression method that makes it fit

The released int8 weights are 71 MB, far beyond what fits comfortably on the board. The report's answer is a training-free quantization pipeline applied to the encoder's 290 matrices. Weights are stored at 2 or 4 bits per weight in Hadamard-rotated groups of 128 with Lloyd-Max levels and one norm per group; the bit width of each matrix comes from its measured sensitivity; and each matrix is first corrected toward the original model's output, then rounded with GPTQ, then tuned with a search over the group norms. That pipeline brings the model to a 26.65 MB file for about half a point of word error rate on LibriSpeech test-clean, and an ablation shows GPTQ doing most of the repair work. The report is equally honest about the failed branch: an attempt to reach 16 MB, motivated by sharing the PSRAM budget with a second model, stalled at 19.63 MB because the remaining rounding nearly doubled the error rate to 8.83%. That kind of negative result is rare in release notes and genuinely useful for anyone planning a similar port. Source

The engineering underneath the speed claim

The speed numbers come from disciplined measurement, not a single lucky build. The report logs the port's speed history from 51.3 seconds down to 0.62 seconds of encoder time per chunk, with each design step re-measured in one build, including what did not work. The activations run as int16 between matrix products and int8 into them, with hand-written PIE kernels for the chip's 128-bit SIMD extension, and the Hadamard rotation is done with an unnormalized transform that needs only additions and is exact, so it costs almost nothing in integer arithmetic. The memory system turns out to be the real adversary: decoding fake codes instead of streaming them from PSRAM saved 39 ms of the 79 ms decoding budget per chunk in an earlier build, which tells you the external memory stream, not the math, was the bottleneck at that stage. Even the screen is accounted for, costing 10.4 ms of encoder time per chunk. For embedded engineers, this measurement discipline is arguably the most transferable part of the report: the bottleneck on a microcontroller is rarely where desktop intuition says it is. Source

The accuracy and speed numbers, with caveats

On LibriSpeech, the port scores 4.34% WER on test-clean and 10.26% on test-other, against 3.94% and 9.17% for the released int8 model run through sherpa-onnx; on real speech from FLEURS and Common Voice the gap grows to 2.3 and 1.1 points. A GPU simulator showed the integer arithmetic itself costs nothing and the whole gap comes from the quantized weights. The caveats deserve the same attention as the numbers. Accuracy is measured with a PC build of the engine that computes the same bits as the chip, not on the device itself, so on-device acoustics with a real microphone are a separate question. The model is English-only. The report explicitly says energy was not measured, so battery implications are unknown. And the 51.3 s to 0.62 s encoder speed history is a single developer's optimization log, instructive but specific to this chip and kernel work. None of this diminishes the result; it defines what still needs testing before a product decision. Source

Where it sits against Oído and ESP-SR

The comparison landscape matters. Espressif's ESP-SR MultiNet recognizes a fixed list of up to 200 commands on the ESP32-S3, which covers wake words and command control but not dictation. Oído, the closest prior work, runs a 13-million-parameter Conformer-CTC model on one 240 MHz Xtensa core of the ESP32-S3 with 8 MB of PSRAM, transcribing at 1.97 times the audio duration, slower than real time, with reported WERs of 3.70% and 8.23% measured on a PC build. The new port runs a model five times larger, faster than real time and in streaming mode, on the P4's two RISC-V cores. The report is careful here too: the chips differ, Oído's model is five times smaller and fits a leaner chip, and its streaming variant lands 1.8 to 2.2 WER points above the new port, so the honest summary is not "better" but "a different point on the size, speed and accuracy surface." For device builders, the practical takeaway is that open-vocabulary transcription now has two working open-source reference points on Espressif silicon, at different price and power points. Source

What this changes for agent devices, in our assessment

This section is our read, separate from the sources. Every voice-driven agent device today makes the same compromise: stream audio to the cloud for transcription, or accept a fixed command list on device. A faster-than-real-time streaming transducer on a $5-class chip removes that fork for English dictation-class workloads, and it pairs naturally with the patterns we have covered before: our ESP32 AI voice agent post assumed cloud transcription, our ESP32 wake word detection guide covered the keyword-spotting layer that would still sit in front of a model like this, and our ESP32 MCP server post covered the tool-calling side that a local transcript would feed. Privacy-sensitive products, offline-first devices and cost-sensitive fleets all benefit directly. Our caution is the report's own: nobody has measured energy yet, and battery-powered products live or die on that number, so treat this as a proven inference result awaiting a power budget, not a finished product design. The fact that the engine, firmware and weights are all published means your team can measure it yourself this week. Source