Have a project in mind? Let's talk.
The ESP32 AI voice agent has quietly become the default way to put a large language model into a physical product. A five-dollar microcontroller wakes on a spoken phrase, streams compressed audio to a cloud model, and plays the answer back through a speaker, all in about a second. The pattern powers everything from hobbyist desk robots to shipping consumer devices, and the open-source XiaoZhi project alone now supports 138 board directories and 171 release variants across six Espressif chip families. Source
This post breaks down the architecture that makes it work, layer by layer, using primary sources: Espressif's own audio framework documentation, the XiaoZhi protocol specification, and OpenAI's realtime voice API guide. If you are scoping a voice-enabled product, this is the map.
The ESP32-S3, the current sweet spot for voice agents, is a dual-core Xtensa LX7 running at 240 MHz with 512 KB of internal SRAM, octal PSRAM support, 2.4 GHz Wi-Fi, and Bluetooth 5 LE. The detail that matters for voice: it includes vector instructions that accelerate neural network computing and signal processing, exposed through Espressif's ESP-DSP and ESP-NN libraries. Source
Those vector instructions are what let a cheap MCU run a real audio front-end locally. Wake word detection, echo cancellation, and noise suppression all execute on-chip, continuously, without touching the network. The cloud only gets involved once the user is actually talking. That split (always-on local listening, on-demand cloud intelligence) is the foundation of the whole design. ESP-SR audio front-end documentation
Every ESP32 voice agent, whether a weekend build or a commercial product, ends up with the same three layers.
Edge audio front-end. Captures mic input, cleans it up, detects the wake word, and decides when speech starts and stops. Runs entirely on the ESP32.
Streaming transport. Moves compressed audio between the device and a server with low latency and session semantics. Usually Opus frames over WebSocket, sometimes MQTT plus UDP.
Cloud brain. Turns speech into text, reasons with an LLM, and turns the reply back into speech. Either a classic ASR, LLM, TTS pipeline or a realtime end-to-end voice model.
The XiaoZhi project is the reference implementation worth studying: it implements both transports, both cloud patterns, device-side and cloud-side MCP for tool control, offline wake with ESP-SR, and speaker recognition, all in ESP-IDF. Source
Espressif's ESP-SR Audio Front-end (AFE) framework is the piece most designs build on. It chains together the algorithms a voice agent needs before any audio leaves the device. ESP-SR audio front-end documentation
The documented building blocks:
AEC (Acoustic Echo Cancellation): up to two-mic processing, removes the device's own speaker output from the mic signal. This is what makes full-duplex possible, so the user can barge in while the agent is talking.
NS (Noise Suppression): single-channel suppression of non-human noise, especially stationary noise like fans.
BSS and MISO: dual-channel source separation and channel selection for noisy two-mic scenes.
VAD (Voice Activity Detection): real-time per-frame voice state, used to segment speech and decide when an utterance ends.
AGC (Automatic Gain Control): normalizes quiet and loud talkers.
WakeNet: a neural-network wake word engine designed for low-power embedded MCUs.
WakeNet deserves emphasis: wake word detection runs offline on the device, and XiaoZhi builds on it for customizable wake words. The practical implication is privacy and cost. Raw audio does not stream to the cloud around the clock; the device listens locally and only opens a session on wake or button press. ESP-SR audio front-end documentation Source
Hardware note: full-duplex interaction depends on AEC-capable hardware, meaning boards with the mic and speaker wiring (and typically two mics) that AEC needs. XiaoZhi's documentation calls this out directly, and its supported-board list (ESP32-S3-BOX-3, M5Stack AtomS3R plus Echo Base, and many more) is a good shortlist of proven audio designs. Source
The transport is where voice agents differ from ordinary IoT telemetry. You are streaming bidirectional audio inside a session, and latency compounds at every hop.
XiaoZhi's WebSocket protocol document describes the canonical flow. The device boots, initializes its audio codec and network, and waits. When a voice session starts (wake word or button), it calls OpenAudioChannel, reads the server WebSocket URL from settings, and opens the connection. Audio then flows as Opus-encoded frames in both directions until the session closes. XiaoZhi WebSocket protocol
Two design points from the protocol worth copying. First, the session is opened on demand, not held open: the always-on cost stays on the cheap local wake engine. Second, the protocol is documented as an interface with a server you can implement yourself, which keeps the device firmware decoupled from any particular cloud brain. XiaoZhi WebSocket protocol
XiaoZhi also ships an alternative MQTT plus UDP transport: MQTT for control messages, UDP for the audio stream itself. That pattern trades WebSocket's simplicity for lower overhead on lossy links, which matters on 4G modules the project supports (ML307, EC801E, NT26 Cat.1). Source
Why Opus: it is the standard low-latency speech codec, and both XiaoZhi's device firmware and the realtime voice APIs have standardized around compressed or compact audio rather than raw WAV. On the cloud side, OpenAI's realtime voice-over-WebSocket guide accepts mono 16-bit PCM at 16 or 24 kHz (or G.711 at 8 kHz) as a continuous base64 stream, which shows how little ceremony the transport actually needs. OpenAI realtime voice WebSockets guide Source
There are two working patterns for the server side, and XiaoZhi supports both. Source
The pipeline pattern chains three services: streaming ASR turns speech into text, an LLM generates the reply (XiaoZhi demos use Qwen and DeepSeek), and TTS synthesizes the response. Each stage is independently swappable, debuggable, and billable, which is why most production systems start here. Tool use fits naturally: XiaoZhi wires device control (speaker, LED, servo, GPIO) through device-side MCP and cloud capabilities through cloud-side MCP. Source
The realtime pattern replaces the chain with a single end-to-end voice model over a persistent socket. OpenAI's realtime guide shows the shape: open a WebSocket session, set the audio format once at startup, then stream input audio and receive output audio, with chunk boundaries arbitrary as long as the stream stays ordered. Fewer moving parts, lower latency, but less control over each stage and a different cost profile. OpenAI realtime voice WebSockets guide
A pragmatic read, and this is our synthesis rather than any vendor's guidance: start with the pipeline pattern for control and observability, and move to realtime when the product's feel demands sub-second turn-taking and your AEC hardware can support full duplex.
Users judge a voice agent on one number: the gap between the end of their sentence and the start of the answer. This section is our engineering heuristic, not a vendor spec. Budget roughly like this on a healthy Wi-Fi link: wake or end-of-speech detection around 100 to 300 ms (VAD plus endpointing), uplink and ASR finalization a few hundred ms, LLM first token a few hundred ms, TTS first audio a few hundred ms. Each layer you remove or merge (realtime models, speculative endpointing, streaming TTS) buys back fractions of a second. If your agent feels slow, measure each hop before optimizing anything; the bottleneck is rarely where intuition says.
A voice agent is a microphone with credentials. Three controls belong in the design from day one, and we have written about each in depth.
Encrypt the transport. The audio socket carries voice data and session tokens; treat TLS as non-negotiable, exactly as we covered for ESP32 MQTT over TLS. The same certificate discipline applies to WebSocket.
Plan for updates. A shipping voice device will need model endpoints rotated, wake words tuned, and firmware patched for years. Secure OTA is the difference between a product and a recall; see our guide to ESP32 OTA update security.
Keep wake detection local. The ESP-SR design pattern (on-device WakeNet, session opened only on wake) is not just a battery decision; it is the strongest privacy property the architecture offers. Preserve it when you customize. ESP-SR audio front-end documentation
For the broader device-security picture, our piece on Meta's Muse Gadgets SDK and ESP32 agent device security covers how on-device models are changing this space.
An ESP32 AI voice agent is a device built on an Espressif ESP32-class microcontroller that wakes on a spoken word, streams speech to a cloud LLM, and speaks the answer back. The heavy lifting splits into three layers: an on-chip audio front-end (wake word, echo cancellation, VAD), a streaming transport (Opus audio over WebSocket or MQTT plus UDP), and a cloud brain (ASR, LLM, TTS, or a realtime end-to-end voice model). XiaoZhi ESP32 project
The ESP32-S3 is the common choice: dual-core 240 MHz Xtensa LX7, vector instructions for neural network and signal processing acceleration, octal PSRAM support, Wi-Fi, and BLE. XiaoZhi also supports ESP32, C3, C5, C6, and P4 targets, but S3 boards dominate its 138-board support list. Espressif ESP32-S3 XiaoZhi ESP32 project
Not the frontier models. The ESP32 runs the audio front-end locally (wake word, noise suppression, VAD, echo cancellation via ESP-SR), while the LLM runs in the cloud. On-device models are getting smaller, but the working pattern today is local listening plus cloud reasoning. ESP-SR audio front-end documentation
Both work, and XiaoZhi implements both. WebSocket is simpler and carries the Opus audio and control messages on one connection. MQTT plus UDP splits control from audio, which can perform better on lossy links such as 4G. Pick WebSocket unless you have a measured reason to split. XiaoZhi WebSocket protocol XiaoZhi ESP32 project
It can be, if the design takes security seriously from the start: TLS on the audio socket, signed OTA updates, local-only wake detection, and least-privilege API credentials on the server side. None of these come for free in a hobby build; all of them are standard practice in production firmware.
Ready to take the first step towards unlocking opportunities, realizing goals, and embracing innovation? We're here and eager to connect.