Realtime Translation

PreviousNext

Source and translated text correspondence, language config, switching, latency.

Real-time translation: get translated text alongside transcription while speech is happening.

Configure translation

Send session.update right after session.created, before the first audio frame:

{
  "type": "session.update",
  "language": "zh",
  "target_languages": ["en", "ja"]
}
  • language: source language of the audio (optional; auto enables detection)
  • target_languages: array of BCP-47 target tags. Up to 8 languages per session.
  • translate_to: single-language shorthand, equivalent to target_languages: [value]

Both zh_CN and zh-CN are accepted and normalized. Invalid tags are dropped; if no valid tag remains the gateway returns invalid_target_languages.

The gateway confirms the applied state with a session.updated event:

{
  "type": "session.updated",
  "request_id": "req_...",
  "session_id": "sess_...",
  "language": "zh",
  "language_source": "client",
  "target_languages": ["en", "ja"],
  "features": {
    "asr_correction": false,
    "refinement": false,
    "translation": true
  }
}

Session-fixed features

Translation is a session-level feature. The first accepted target_languages value is frozen before audio. If you send a different set later — or try to enable translation after audio has started — the gateway returns:

{
  "type": "error",
  "code": "session_features_locked",
  "message": "Translation target languages are fixed for the session; start a new session to change them."
}

This keeps billing deterministic: every audio second in the session is billed with the same feature set.

Translation output

Each target language produces its own live.translation.completed event. Events arrive independently, so a slow language never blocks the source transcript or another language:

{
  "type": "live.translation.completed",
  "request_id": "req_...",
  "session_id": "sess_...",
  "utterance_index": 3,
  "language": "en",
  "text": "We are shipping the new version next week.",
  "source_text": "我们下周发布新版本",
  "source_language": "zh",
  "latency_ms": 180
}
  • text: translated text for this language
  • source_text: the raw ASR line the translation is based on
  • utterance_index: matches the conversation.item.input_audio_transcription.completed event for the same utterance

Translation runs in parallel with correction. It does not delay the source transcript event.

Source and translated text correspondence

Use utterance_index to pair the source transcript with each translation. The source transcript arrives first; translations follow as they complete.

Switching languages

The source language can change at any time during the session:

{ "type": "session.update", "language": "en" }

New utterances after the switch use the new source language. Transcripts already delivered keep their previous language.

The translation target set cannot change mid-session. Start a new session if the target languages change.

Latency

Translation is an additional LLM call per utterance per target language. It runs off the transcript hot path, so the source transcript latency is unchanged. Typical translation latency is a few hundred milliseconds; it depends on audio length, context, and provider load.

Pricing

Realtime translation is billed per target language × processed audio duration. See Pricing for the current rate card.

  • 1 target language: base realtime rate + 1 × translation rate
  • 2 target languages: base realtime rate + 2 × translation rate
  • Translation is only charged when target_languages was accepted for the session

Multilingual / code-switching

  • Language code: set a concrete language such as en or zh when known; auto is also accepted and enables detection. A concrete code is more accurate.
  • Mixed language: upstream supports code-switching, accuracy depends on model and audio quality
  • Chinese aliases: zh-cn, zh-tw, mandarin etc. are normalized to zh

UI strategy

  1. Receive conversation.item.input_audio_transcription.completed and display the source text
  2. Receive live.translation.completed for each target language
  3. Match by utterance_index and display each translation on its own line
  4. No client-side replacement is needed; each utterance is independent

See Build Live Translation for a detailed guide.