Messages the client sends over the WebSocket connection.
Overview
| type | Transport | Purpose |
|---|---|---|
input_audio_buffer.append | Text frame (JSON) | Send base64-encoded PCM audio (supported) |
input_audio_buffer.flush | Text frame (JSON) | Accepted compatibility message; no guaranteed segment finalization |
flush | Text frame (JSON) | Accepted alias; same limitation |
session.update | Text frame (JSON) | Set language, refinement, target_languages, or translate_to |
input_audio_buffer.append
Send base64-encoded PCM16LE audio data.
{
"type": "input_audio_buffer.append",
"audio": "<PCM16LE base64>"
}| Field | Type | Required | Description |
|---|---|---|---|
type | string | yes | Must be "input_audio_buffer.append" |
audio | string | yes | Base64-encoded PCM16LE audio data |
Errors:
invalid_audio:audiois empty or not valid base64
Binary audio frames — not supported today
The public gateway currently accepts binary WebSocket frames but does not transcribe them. Only base64 JSON input_audio_buffer.append frames produce transcription events today. Do not use binary frames in production until this notice is removed.
input_audio_buffer.flush
The message is accepted for compatibility, but the current public Realtime session handler does not implement a manual-flush action. Do not rely on it to end an utterance, deliver pending results, or produce a full-session final transcript.
{ "type": "input_audio_buffer.flush" }"flush" is an accepted alias with the same limitation. Continue streaming audio, including natural silence, for server-side segmentation. Keep the socket open while expected utterance results are in flight.
session.update
Optionally set the audio language before sending audio. Can also be sent at any time after connection to switch language.
{
"type": "session.update",
"language": "zh"
}Supported fields
| Field | Type | Description |
|---|---|---|
language | string | null | Audio language. A concrete code (e.g. zh, en) is recommended for accuracy; "auto", null, or omitting it enables detection. |
refinement | boolean | string | Opt in to per-utterance AI refinement before delivery. true selects the default lite profile; "lite" / "pro" select a profile explicitly; false disables it. Off by default and fixed before audio. Lite is included in the base transcription price; Pro is billed as the realtime refinement add-on. Requires gateway availability. |
target_languages | string | Translation targets (BCP-47, max 8). Fixed for the session; send before audio. |
translate_to | string | Single-language shorthand for target_languages. Fixed for the session. |
language can change at any time. target_languages / translate_to are session-fixed: the first accepted value wins, and later changes return session_features_locked.
The gateway confirms the applied state with a session.updated event.
Switching language mid-session
- A concrete code always wins: the session uses it immediately and does not run detection.
- Sending
"auto"(ornull) resets detection state and re-triggers detection on subsequent speech. Use this when the language changes or becomes unknown. - Sending
"auto"again while already in auto mode also resets and re-detects. - Transcripts already delivered keep their language; the new setting applies to speech transcribed after the update.
You can detect the language yourself with POST /v1/audio/language-detection and send the result explicitly, or let the session detect it via "auto". An explicit language is more accurate and avoids detection latency.
Tenant isolation
Identity and credential fields (request_id, session_id, user_id) are injected by the gateway and cannot be overridden by the client.
Errors
bad_json: JSON parse failurebad_message: Not a JSON object or unsupportedtypeinvalid_audio: Invalid audio datainvalid_target_languages: No valid BCP-47 tag intarget_languagestranslation_unavailable: Translation is not configured on this gatewayrefinement_unavailable: Correction is not configured on this gatewaysession_features_locked: Tried to change translation targets after the session feature set was frozen
Related
- Server Events — server-pushed events
- Session Configuration — session config details
- Audio Input — audio format reference