[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"article-doc:cookbook\u002Fstream-mic-audio-over-websocket":3},"---\ntitle: \"How to stream microphone audio over WebSocket\"\ndescription: \"Your browser gives you float32 at 48kHz stereo; the realtime API wants PCM16LE at 16kHz mono in 100ms frames. The full capture → convert → frame → send pipeline, plus backpressure and reconnect.\"\nseries: \"A\"\ndifficulty: \"intermediate\"\nreading_time: \"9 min\"\nstatus: \"draft\"\nlang: \"en\"\nrelated_docs:\n  - \"\u002Frealtime\u002Faudio-input\"\n  - \"\u002Frealtime\u002Fconnection-lifecycle\"\n  - \"\u002Frealtime\u002Fquickstart\"\n  - \"\u002Fguides\u002Freconnection\"\nrelated_cookbook:\n  - \"choose-realtime-vs-batch\"\n  - \"stable-captions-no-jitter\"\nsource_urls:\n  - \"https:\u002F\u002Fdocs.lansonai.com\u002Frealtime\u002Faudio-input\"\n  - \"https:\u002F\u002Fdocs.lansonai.com\u002Frealtime\u002Fconnection-lifecycle\"\nclaim_sources:\n  - claim: \"Realtime API requires PCM16LE, 16000 Hz, mono.\"\n    source: \"lanson-audio-docs\u002Fcontent\u002F2.realtime\u002F4.audio-input.md (Format requirements)\"\n    type: \"fact\"\n  - claim: \"100ms frames (3200 bytes) is the recommended balance; max frame is 1 MiB (exceed → close 1009).\"\n    source: \"lanson-audio-docs\u002Fcontent\u002F2.realtime\u002F4.audio-input.md (Frame size and latency \u002F Max frame size)\"\n    type: \"fact\"\n  - claim: \"Base64 text frames have ~33% overhead; binary frames are recommended for production.\"\n    source: \"lanson-audio-docs\u002Fcontent\u002F2.realtime\u002F4.audio-input.md (Transport methods)\"\n    type: \"fact\"\n  - claim: \"Backpressure: soft 1 MiB in-flight drops frames + lanson.throttled; hard 8 MiB closes (1013).\"\n    source: \"lanson-audio-docs\u002Fcontent\u002F2.realtime\u002F4.audio-input.md (Backpressure)\"\n    type: \"fact\"\n  - claim: \"Close codes: 4408 idle_timeout, 1008 duration limit, 1009 frame_too_large, 1013 upstream\u002Fbackpressure.\"\n    source: \"lanson-audio-docs\u002Fcontent\u002F2.realtime\u002F3.connection-lifecycle.md (Timeout close)\"\n    type: \"fact\"\n  - claim: \"Moving VAD detection to first-byte arrival saved ~80ms on the server side.\"\n    source: \"workflows\u002Fopenai-partner-network\u002Flanson-live-technical-architecture.md (v3 Optimization Techniques)\"\n    type: \"metric\"\ncreated_at: \"2026-08-27\"\nupdated_at: \"2026-08-27\"\n---\n\n# How to stream microphone audio over WebSocket\n\n> **TL;DR** — The realtime API wants **PCM16LE, 16kHz, mono, in ~100ms frames**. Your browser hands you float32 at 44.1\u002F48kHz stereo. The integration that breaks is almost never the WebSocket — it's the audio pipeline in between: wrong sample rate, resampling in the wrong place, frames that are too big, or ignoring backpressure. Set the sample rate at the `AudioContext`, convert to PCM16LE in an `AudioWorklet`, frame at 100ms, send binary, and respect `lanson.throttled`.\n\n## Problem\n\nYou have a working WebSocket to the realtime endpoint. You start sending mic audio. One of three things goes wrong:\n\n- **The transcript is garbage or empty.** The sample rate or channel count didn't match what the server expects, so the audio is misinterpreted.\n- **It works in the demo, stutters in production.** Frames are too big (CPU spikes) or too small (churn), or you're paying base64 overhead you don't need.\n- **The connection drops under load.** You kept sending faster than the upstream could drain, hit backpressure, and the socket closed — with no idea why.\n\nAll three are pipeline problems, not API problems. Get the pipeline right and the WebSocket is boring, which is exactly what you want.\n\n## Architecture\n\nThe pipeline has five stages. Each has one job and one common failure mode:\n\n```text\nmic ──► AudioContext(16kHz) ──► AudioWorklet ──► 100ms frames ──► WS (binary)\n       set rate HERE            float→PCM16LE      buffer to        respect\n       (not resample later)     + downmix mono     1600 samples     backpressure\n```\n\n1. **Capture at 16kHz.** Create the `AudioContext` with `sampleRate: 16000`. This is the single most important line. Resampling *after* capture (in the worklet) is a common source of drift and clicks; setting the rate at the context level lets the browser do it cleanly.\n2. **Convert to PCM16LE.** The worklet receives float32 in `[-1, 1]`; convert to 16-bit signed integers. Downmix to mono if the source is stereo.\n3. **Frame at 100ms.** At 16kHz, 100ms = **1600 samples = 3200 bytes**. This is the recommended sweet spot — 50ms is lower latency but higher CPU, 200ms+ adds noticeable latency. Hard cap is **1 MiB per frame** (exceeding it closes the socket with `1009`).\n4. **Send binary, not base64.** Base64 text frames add ~33% overhead — fine for debugging, wasteful in production. Send raw `ArrayBuffer` frames.\n5. **Respect backpressure.** When the upstream can't keep up, the server drops frames and sends `lanson.throttled` (soft, 1 MiB in-flight) or closes with `1013` (hard, 8 MiB). Pause sending when throttled; don't fight it.\n\n## Code\n\n### 1. The AudioWorklet (convert + frame)\n\n```javascript\n\u002F\u002F audio-processor.js — loaded via ctx.audioWorklet.addModule()\nclass PCMProcessor extends AudioWorkletProcessor {\n  constructor() {\n    super();\n    this.pending = [];\n    this.frameSize = 1600; \u002F\u002F 100ms @ 16kHz\n  }\n  process(inputs) {\n    const input = inputs[0];\n    if (!input || input.length === 0) return true;\n    const channel = input[0]; \u002F\u002F mono (AudioContext downmixes if source is stereo)\n\n    for (let i = 0; i \u003C channel.length; i++) this.pending.push(channel[i]);\n\n    \u002F\u002F Emit complete 100ms frames as PCM16LE\n    while (this.pending.length >= this.frameSize) {\n      const frame = this.pending.splice(0, this.frameSize);\n      const pcm16 = new Int16Array(frame.length);\n      for (let j = 0; j \u003C frame.length; j++) {\n        const s = Math.max(-1, Math.min(1, frame[j]));\n        pcm16[j] = s \u003C 0 ? s * 0x8000 : s * 0x7fff;\n      }\n      this.port.postMessage(pcm16.buffer, [pcm16.buffer]);\n    }\n    return true;\n  }\n}\nregisterProcessor(\"pcm-processor\", PCMProcessor);\n```\n\n### 2. Main thread (capture + send)\n\n```javascript\nconst ctx = new AudioContext({ sampleRate: 16000 }); \u002F\u002F ← the important line\nconst stream = await navigator.mediaDevices.getUserMedia({ audio: { channelCount: 1 } });\n\n\u002F\u002F Browser auth: session token from your backend (never the secret key)\nconst { token } = await fetch(\"\u002Fyour-backend\u002Fsession-token\", { method: \"POST\" }).then(r => r.json());\nconst ws = new WebSocket(`wss:\u002F\u002Faudio.lansonai.com\u002Fv1\u002Faudio\u002Ftranscriptions\u002Fstream?access_token=${token}`);\n\nlet throttled = false;\n\nawait ctx.audioWorklet.addModule(\"\u002Faudio-processor.js\");\nconst source = ctx.createMediaStreamSource(stream);\nconst node = new AudioWorkletNode(ctx, \"pcm-processor\");\n\n\u002F\u002F Keep the graph active without playing audio back\nconst mute = ctx.createGain();\nmute.gain.value = 0;\nsource.connect(node);\nnode.connect(mute);\nmute.connect(ctx.destination);\n\nnode.port.onmessage = (e) => {\n  if (throttled) return;                 \u002F\u002F backpressure: pause, don't queue\n  if (ws.readyState === WebSocket.OPEN) ws.send(e.data); \u002F\u002F binary PCM16LE frame\n};\n\nws.onmessage = (e) => {\n  const ev = JSON.parse(e.data);\n  if (ev.type === \"lanson.throttled\") { throttled = true; \u002F* resume after a beat *\u002F }\n  if (ev.type === \"conversation.item.input_audio_transcription.completed\") {\n    throttled = false;\n    console.log(`[${ev.utterance_index}] ${ev.text}`);\n  }\n};\n```\n\n> Server-side (Node) is simpler: use the `Authorization: Bearer sk-...` header and send binary PCM frames directly — no `AudioContext`, no worklet. Just keep sending to avoid the idle timeout.\n\n### 3. Reconnect by close code\n\n```javascript\nws.onclose = (e) => {\n  const { code } = e;\n  if (code === 1000) return;                 \u002F\u002F normal close — don't reconnect\n  if (code === 4408 || code === 1008) {      \u002F\u002F idle \u002F duration limit — reconnect now\n    reconnect();\n  } else if (code === 1013) {                \u002F\u002F upstream \u002F backpressure — back off\n    reconnectWithBackoff();\n  } else {\n    reconnectWithBackoff();\n  }\n  \u002F\u002F After reconnect: re-send session.update to restore configuration\n};\n```\n\n## Result\n\n- **Correct audio, every time.** 16kHz set at the context, PCM16LE mono, 100ms frames — the server gets exactly what it expects.\n- **No production stutter.** Binary frames (no base64 tax), 100ms framing (CPU\u002Flatency balance), backpressure respected (no 1013 drops under load).\n- **Self-healing.** Reconnect logic keyed to close codes means idle timeouts and upstream blips recover without user action.\n\n## Why it matters\n\n1. **Set the sample rate at the `AudioContext`, not in the worklet.** This is the highest-leverage line in the whole pipeline. Resampling float32→16kHz by hand in the worklet is where drift, clicks, and \"it works on my machine\" bugs come from. Let the browser's context do the rate conversion; your worklet only does float→int and framing.\n\n2. **Frame size is a tradeoff you should choose deliberately, not by accident.** 100ms is the sweet spot, but the real lesson is that frame size couples latency to CPU. Smaller frames = lower latency + more work; bigger frames = the opposite. Pick based on your device, and know the 1 MiB hard cap exists.\n\n3. **Backpressure is a feature, not an error.** Most tutorials treat a dropped frame or a `1013` close as a bug. It's the server telling you it can't drain as fast as you're producing. The correct response is to *pause and resume*, not to buffer everything and crash the socket. Treat your audio pipeline as a producer that must respect the consumer's drain rate — the same principle that applies to any streaming system.\n\n4. **Every stage should start at the earliest signal.** This lesson runs in both directions. On our server side, simply moving VAD detection from \"wait for a full buffer\" to \"first-byte arrival\" cut ~80ms. Your client side follows the same rule: don't hold audio longer than you must. The 100ms frame exists precisely so the server gets signal early enough to work with — the pipeline is a relay, and every handoff that waits is latency someone will feel.\n\n---\n\n### Further reading (Docs)\n\n- [Audio Input](https:\u002F\u002Fdocs.lansonai.com\u002Frealtime\u002Faudio-input) — 格式 \u002F 帧大小 \u002F backpressure\n- [Connection Lifecycle](https:\u002F\u002Fdocs.lansonai.com\u002Frealtime\u002Fconnection-lifecycle) — 连接阶段 \u002F close codes\n- [Reconnect a Live Session](https:\u002F\u002Fdocs.lansonai.com\u002Fguides\u002Freconnection) — 重连设计\n- [Realtime Quickstart](https:\u002F\u002Fdocs.lansonai.com\u002Frealtime\u002Fquickstart) — 最小可跑示例\n\n### Related cookbook recipes\n\n- [How to choose between realtime and batch transcription](\u002Fcookbook\u002Fchoose-realtime-vs-batch) — 为什么用 Realtime 平面\n- [How to build realtime captions that don't jitter](\u002Fcookbook\u002Fstable-captions-no-jitter) — 音频进来之后，渲染侧怎么做\n",1790059118939]