Audio Input

PreviousNext

PCM format, sample rate, channels, chunk size, and latency tradeoffs.

The realtime API accepts PCM16LE audio. This page covers format requirements, frame size, and latency tradeoffs.

Format requirements

PropertyRequirement
EncodingPCM16LE (16-bit signed little-endian)
Sample rate16000 Hz
Channels1 (mono)

Converting audio

ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

Transport methods

Text frames (base64) — supported

{ "type": "input_audio_buffer.append", "audio": "<PCM16LE base64>" }

This is the transport supported by the public gateway today. It adds ~33% base64 overhead. Send 100ms frames (3200 bytes of PCM before encoding) as JSON text frames.

Binary frames (raw PCM) — not supported today

Frame size and latency

Frame sizeIntervalData sizeLatency impact
50ms50ms1600 bytesLowest latency, higher CPU overhead
100ms100ms3200 bytesRecommended balance
200ms200ms6400 bytesIncreased latency, fewer frames
500ms500ms16000 bytesNoticeable latency

100ms frames (3200 bytes) is the recommended sweet spot.

Max frame size

1 MiB (1,048,576 bytes). Exceeding this returns audio_frame_too_large and closes the connection (1009).

Browser capture

const ctx = new AudioContext({ sampleRate: 16000 });
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const source = ctx.createMediaStreamSource(stream);
// Use AudioWorklet or ScriptProcessorNode to convert float samples to PCM16LE
// Ensure 16000 Hz sample rate and mono

Server-side capture

  • Use Authorization: Bearer sk-... header
  • Send base64 JSON frames (input_audio_buffer.append), not binary frames
  • Keep sending to avoid idle timeout

Backpressure

When the client is not reading events fast enough, the gateway drops audio frames rather than accumulating memory:

ConditionBehavior
Client buffered events ≥ 8 MiBDrop audio frames, send lanson.throttled (client_backpressure)

Monitor lanson.throttled events and reduce send rate accordingly.