Session Configuration

PreviousNext

language and other session-level options.

Configure session parameters via session.update messages.

When to configure

  • Before audio (recommended): send session.update right after receiving session.created
  • During session: update parameters at any time

Supported options

FieldTypeDescription
languagestring | nullAudio source language. A concrete code (e.g. zh, en) is recommended for accuracy; "auto", null, or omitting it enables detection.
refinementboolean | stringOpt in to per-utterance AI refinement. true selects the default lite profile; "lite" / "pro" select a profile explicitly; false disables it. Fixed before audio.
target_languagesstringTranslation targets (BCP-47, max 8). Fixed for the session; send before audio.
translate_tostringSingle-language shorthand for target_languages. Fixed for the session.

language can change at any time. refinement, target_languages, and translate_to are session-fixed: set them before audio, and later changes return session_features_locked.

The gateway confirms every applied session.update with a session.updated event containing the effective language, target_languages, and features.

Enabling refinement

session.created.features.asr_correction tells you whether refinement is available on the gateway. It is off by default; opt in before sending audio:

{ "type": "session.update", "refinement": true }

true selects the default Lite profile. To pick a profile explicitly:

{ "type": "session.update", "refinement": "pro" }
ProfileBillingBehavior
liteIncluded in the base transcription priceShort prompt, two segments of context, tight character and completion budgets.
proRealtime refinement add-onFull prompt, ten segments of context, and a rolling guide.

Both profiles skip very short slices — fewer than 3 CJK characters or 3 Latin words — and deliver the raw text unchanged. Such slices are rarely misheard on their own and usually lack the context needed to recover them. A skipped slice is still kept as context for the next utterance, so a later correction can still use it.

The gateway replies with session.updated and features.refinement set to the selected profile id ("lite" or "pro"). If correction is not configured, the gateway returns refinement_unavailable; an unknown profile returns invalid_refinement_profile; if audio has already started, it returns session_features_locked.

Language modes and switching

Client actionBehavior
session.update with a concrete codeUse that language immediately. The client value always wins; no detection is run.
session.update with "auto" / null / no updateAuto-detect the language. Detection runs on the next speech audio.
Explicit → auto mid-sessionClear the cached language/votes and re-run detection on subsequent speech. This is how a session that started in English can switch after another language appears.
auto → explicitUse the explicit language immediately and stop detection.
Explicit A → explicit BUse B immediately.
auto → auto againTreat as a detection reset and re-run detection.

Transcripts already delivered keep their language. The new setting applies to speech transcribed after the update.

Two ways to use auto

  1. Client-side detection — call POST /v1/audio/language-detection with a short clip, then send the returned code via session.update. This is the most predictable option and avoids detection inside the stream.
  2. Server-side fallback — omit language or send "auto". The session runs its own basic detection strategy (probe/vote on speech utterances) and caches the result until the client changes the language again.

Passing a known language is still recommended for accuracy and lower latency. If detection cannot decide, the session reports a detection error and retries on later speech; an empty transcript is not marked safe for downstream use.

What the client receives

The detected language is delivered on each conversation.item.input_audio_transcription.completed event:

FieldMeaning for the UI
languageThe language used for this utterance (detected in auto mode)
language_sourceclient (explicit) or auto (detected)
language_provisionaltrue while auto-detection is still voting; show “detecting” or a provisional badge
language_detection_errorDetection failed; show “unknown” and retry on later speech

There is no separate client event for “language detected”; update the language indicator when the completed event arrives. Use POST /v1/audio/language-detection if you need the language before audio.

Audio configuration

Fixed by the gateway, client cannot change:

{ "format": "pcm_s16le", "sample_rate": 16000, "channels": 1 }

Examples

Set language

{ "type": "session.update", "language": "zh" }

Enable translation

{
  "type": "session.update",
  "language": "zh",
  "target_languages": ["en", "ja"]
}

Send this before the first audio frame. The target set is frozen for the session; see Realtime Translation.

Switch language at runtime

{ "type": "session.update", "language": "en" }

Switch to auto mid-session

{ "type": "session.update", "language": "auto" }

Send this when the language changes or becomes unknown. It resets detection and re-detects on the next speech audio.

Switch back to an explicit language

{ "type": "session.update", "language": "ja" }

Tenant isolation

Identity and credential fields (request_id, session_id, user_id) are injected by the gateway and cannot be overridden by the client.