How to stream microphone audio over WebSocket
> TL;DR — The realtime API wants PCM16LE, 16kHz, mono, in ~100ms frames. Your browser hands you float32 at 44.1/48kHz stereo. The integration that breaks is almost never the WebSocket — it's the audio pipeline in between: wrong sample rate, resampling in the wrong place, frames that are too big, or ignoring backpressure. Set the sample rate at the AudioContext, convert to PCM16LE in an AudioWorklet, frame at 100ms, send binary, and respect lanson.throttled.
Problem
You have a working WebSocket to the realtime endpoint. You start sending mic audio. One of three things goes wrong:
All three are pipeline problems, not API problems. Get the pipeline right and the WebSocket is boring, which is exactly what you want.
Architecture
The pipeline has five stages. Each has one job and one common failure mode:
mic ──► AudioContext(16kHz) ──► AudioWorklet ──► 100ms frames ──► WS (binary)
set rate HERE float→PCM16LE buffer to respect
(not resample later) + downmix mono 1600 samples backpressure
1. Capture at 16kHz. Create the AudioContext with sampleRate: 16000. This is the single most important line. Resampling after capture (in the worklet) is a common source of drift and clicks; setting the rate at the context level lets the browser do it cleanly.
2. Convert to PCM16LE. The worklet receives float32 in [-1, 1]; convert to 16-bit signed integers. Downmix to mono if the source is stereo.
3. Frame at 100ms. At 16kHz, 100ms = 1600 samples = 3200 bytes. This is the recommended sweet spot — 50ms is lower latency but higher CPU, 200ms+ adds noticeable latency. Hard cap is 1 MiB per frame (exceeding it closes the socket with 1009).
4. Send binary, not base64. Base64 text frames add ~33% overhead — fine for debugging, wasteful in production. Send raw ArrayBuffer frames.
5. Respect backpressure. When the upstream can't keep up, the server drops frames and sends lanson.throttled (soft, 1 MiB in-flight) or closes with 1013 (hard, 8 MiB). Pause sending when throttled; don't fight it.
Code
1. The AudioWorklet (convert + frame)
// audio-processor.js — loaded via ctx.audioWorklet.addModule()
class PCMProcessor extends AudioWorkletProcessor {
constructor() {
super();
this.pending = [];
this.frameSize = 1600; // 100ms @ 16kHz
}
process(inputs) {
const input = inputs[0];
if (!input || input.length === 0) return true;
const channel = input[0]; // mono (AudioContext downmixes if source is stereo)
for (let i = 0; i < channel.length; i++) this.pending.push(channel[i]);
// Emit complete 100ms frames as PCM16LE
while (this.pending.length >= this.frameSize) {
const frame = this.pending.splice(0, this.frameSize);
const pcm16 = new Int16Array(frame.length);
for (let j = 0; j < frame.length; j++) {
const s = Math.max(-1, Math.min(1, frame[j]));
pcm16[j] = s < 0 ? s 0x8000 : s 0x7fff;
}
this.port.postMessage(pcm16.buffer, [pcm16.buffer]);
}
return true;
}
}
registerProcessor("pcm-processor", PCMProcessor);
2. Main thread (capture + send)
const ctx = new AudioContext({ sampleRate: 16000 }); // ← the important line
const stream = await navigator.mediaDevices.getUserMedia({ audio: { channelCount: 1 } });
// Browser auth: session token from your backend (never the secret key)
const { token } = await fetch("/your-backend/session-token", { method: "POST" }).then(r => r.json());
const ws = new WebSocket(wss://audio.lansonai.com/v1/audio/transcriptions/stream?access_token=${token});
let throttled = false;
await ctx.audioWorklet.addModule("/audio-processor.js");
const source = ctx.createMediaStreamSource(stream);
const node = new AudioWorkletNode(ctx, "pcm-processor");
// Keep the graph active without playing audio back
const mute = ctx.createGain();
mute.gain.value = 0;
source.connect(node);
node.connect(mute);
mute.connect(ctx.destination);
node.port.onmessage = (e) => {
if (throttled) return; // backpressure: pause, don't queue
if (ws.readyState === WebSocket.OPEN) ws.send(e.data); // binary PCM16LE frame
};
ws.onmessage = (e) => {
const ev = JSON.parse(e.data);
if (ev.type === "lanson.throttled") { throttled = true; / resume after a beat / }
if (ev.type === "conversation.item.input_audio_transcription.completed") {
throttled = false;
console.log([${ev.utterance_index}] ${ev.text});
}
};
> Server-side (Node) is simpler: use the Authorization: Bearer sk-... header and send binary PCM frames directly — no AudioContext, no worklet. Just keep sending to avoid the idle timeout.
3. Reconnect by close code
ws.onclose = (e) => {
const { code } = e;
if (code === 1000) return; // normal close — don't reconnect
if (code === 4408 || code === 1008) { // idle / duration limit — reconnect now
reconnect();
} else if (code === 1013) { // upstream / backpressure — back off
reconnectWithBackoff();
} else {
reconnectWithBackoff();
}
// After reconnect: re-send session.update to restore configuration
};
Result
Why it matters
1. Set the sample rate at the AudioContext, not in the worklet. This is the highest-leverage line in the whole pipeline. Resampling float32→16kHz by hand in the worklet is where drift, clicks, and "it works on my machine" bugs come from. Let the browser's context do the rate conversion; your worklet only does float→int and framing.
2. Frame size is a tradeoff you should choose deliberately, not by accident. 100ms is the sweet spot, but the real lesson is that frame size couples latency to CPU. Smaller frames = lower latency + more work; bigger frames = the opposite. Pick based on your device, and know the 1 MiB hard cap exists.
3. Backpressure is a feature, not an error. Most tutorials treat a dropped frame or a 1013 close as a bug. It's the server telling you it can't drain as fast as you're producing. The correct response is to pause and resume, not to buffer everything and crash the socket. Treat your audio pipeline as a producer that must respect the consumer's drain rate — the same principle that applies to any streaming system.
4. Every stage should start at the earliest signal. This lesson runs in both directions. On our server side, simply moving VAD detection from "wait for a full buffer" to "first-byte arrival" cut ~80ms. Your client side follows the same rule: don't hold audio longer than you must. The 100ms frame exists precisely so the server gets signal early enough to work with — the pipeline is a relay, and every handoff that waits is latency someone will feel.