Configure session parameters via session.update messages.
language is optional. Passing a concrete code is recommended when you know it because it improves accuracy. If you omit it or send auto, the service auto-detects the language. You can also call POST /v1/audio/language-detection first and pass the detected code.
When to configure
- Before audio (recommended): send
session.updateright after receivingsession.created - During session: update parameters at any time
Supported options
| Field | Type | Description |
|---|---|---|
language | string | null | Audio source language. A concrete code (e.g. zh, en) is recommended for accuracy; "auto", null, or omitting it enables detection. |
refinement | boolean | string | Opt in to per-utterance AI refinement. true selects the default lite profile; "lite" / "pro" select a profile explicitly; false disables it. Fixed before audio. |
target_languages | string | Translation targets (BCP-47, max 8). Fixed for the session; send before audio. |
translate_to | string | Single-language shorthand for target_languages. Fixed for the session. |
language can change at any time. refinement, target_languages, and translate_to are session-fixed: set them before audio, and later changes return session_features_locked.
The gateway confirms every applied session.update with a session.updated event containing the effective language, target_languages, and features.
Enabling refinement
session.created.features.asr_correction tells you whether refinement is available on the gateway. It is off by default; opt in before sending audio:
{ "type": "session.update", "refinement": true }true selects the default Lite profile. To pick a profile explicitly:
{ "type": "session.update", "refinement": "pro" }| Profile | Billing | Behavior |
|---|---|---|
lite | Included in the base transcription price | Short prompt, two segments of context, tight character and completion budgets. |
pro | Realtime refinement add-on | Full prompt, ten segments of context, and a rolling guide. |
Both profiles skip very short slices — fewer than 3 CJK characters or 3 Latin words — and deliver the raw text unchanged. Such slices are rarely misheard on their own and usually lack the context needed to recover them. A skipped slice is still kept as context for the next utterance, so a later correction can still use it.
The gateway replies with session.updated and features.refinement set to the selected profile id ("lite" or "pro"). If correction is not configured, the gateway returns refinement_unavailable; an unknown profile returns invalid_refinement_profile; if audio has already started, it returns session_features_locked.
Language modes and switching
| Client action | Behavior |
|---|---|
session.update with a concrete code | Use that language immediately. The client value always wins; no detection is run. |
session.update with "auto" / null / no update | Auto-detect the language. Detection runs on the next speech audio. |
Explicit → auto mid-session | Clear the cached language/votes and re-run detection on subsequent speech. This is how a session that started in English can switch after another language appears. |
auto → explicit | Use the explicit language immediately and stop detection. |
| Explicit A → explicit B | Use B immediately. |
auto → auto again | Treat as a detection reset and re-run detection. |
Transcripts already delivered keep their language. The new setting applies to speech transcribed after the update.
Two ways to use auto
- Client-side detection — call
POST /v1/audio/language-detectionwith a short clip, then send the returned code viasession.update. This is the most predictable option and avoids detection inside the stream. - Server-side fallback — omit
languageor send"auto". The session runs its own basic detection strategy (probe/vote on speech utterances) and caches the result until the client changes the language again.
Passing a known language is still recommended for accuracy and lower latency. If detection cannot decide, the session reports a detection error and retries on later speech; an empty transcript is not marked safe for downstream use.
What the client receives
The detected language is delivered on each conversation.item.input_audio_transcription.completed event:
| Field | Meaning for the UI |
|---|---|
language | The language used for this utterance (detected in auto mode) |
language_source | client (explicit) or auto (detected) |
language_provisional | true while auto-detection is still voting; show “detecting” or a provisional badge |
language_detection_error | Detection failed; show “unknown” and retry on later speech |
There is no separate client event for “language detected”; update the language indicator when the completed event arrives. Use POST /v1/audio/language-detection if you need the language before audio.
Audio configuration
Fixed by the gateway, client cannot change:
{ "format": "pcm_s16le", "sample_rate": 16000, "channels": 1 }Examples
Set language
{ "type": "session.update", "language": "zh" }Enable translation
{
"type": "session.update",
"language": "zh",
"target_languages": ["en", "ja"]
}Send this before the first audio frame. The target set is frozen for the session; see Realtime Translation.
Switch language at runtime
{ "type": "session.update", "language": "en" }Switch to auto mid-session
{ "type": "session.update", "language": "auto" }Send this when the language changes or becomes unknown. It resets detection and re-detects on the next speech audio.
Switch back to an explicit language
{ "type": "session.update", "language": "ja" }Tenant isolation
Identity and credential fields (request_id, session_id, user_id) are injected by the gateway and cannot be overridden by the client.
Related
- Client Messages — message format
- Server Events — server-pushed events
- Live Translation — translation concepts
On This Page
When to configureSupported optionsEnabling refinementLanguage modes and switchingTwo ways to use autoWhat the client receivesAudio configurationExamplesSet languageEnable translationSwitch language at runtimeSwitch to auto mid-sessionSwitch back to an explicit languageTenant isolationRelated