The realtime API accepts PCM16LE audio. This page covers format requirements, frame size, and latency tradeoffs.
Format requirements
| Property | Requirement | |---|---| | Encoding | PCM16LE (16-bit signed little-endian) | | Sample rate | 16000 Hz | | Channels | 1 (mono) |
::callout{icon="i-lucide-triangle-alert" color="amber"} Audio not matching this format will be rejected or produce incorrect results. Convert with ffmpeg first. ::
Converting audio
ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav
Transport methods
Text frames (base64)
{ "type": "input_audio_buffer.append", "audio": "<PCM16LE base64>" }
Has ~33% base64 overhead. Good for debugging and quick integration.
Binary frames (raw PCM)
Send raw ArrayBuffer directly. No encoding overhead. Recommended for production.
Frame size and latency
| Frame size | Interval | Data size | Latency impact | |---|---|---|---| | 50ms | 50ms | 1600 bytes | Lowest latency, higher CPU overhead | | 100ms | 100ms | 3200 bytes | Recommended balance | | 200ms | 200ms | 6400 bytes | Increased latency, fewer frames | | 500ms | 500ms | 16000 bytes | Noticeable latency |
100ms frames (3200 bytes) is the recommended sweet spot.
Max frame size
1 MiB (1,048,576 bytes). Exceeding this returns audio_frame_too_large and closes the connection (1009).
Browser capture
const ctx = new AudioContext({ sampleRate: 16000 });
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const source = ctx.createMediaStreamSource(stream);
// Use AudioWorklet or ScriptProcessorNode to convert float samples to PCM16LE
// Ensure 16000 Hz sample rate and mono
Server-side capture
Authorization: Bearer sk-... headerBackpressure
When the upstream cannot keep up:
| Condition | Behavior |
|---|---|
| Soft threshold (1 MiB in-flight) | Drop frames, send lanson.throttled |
| Hard threshold (8 MiB in-flight) | Close connection (1013) |
Monitor lanson.throttled events and reduce send rate accordingly.