SpyBara
Go Premium

guides/voice-agents.md 2026-05-06 00:01 UTC to 2026-05-07 21:57 UTC

This page contains 59 additions and 58 deletions.

2026
Tue 5 23:00 Thu 7 21:57 Wed 20 06:35

Voice agents

import { Bolt, Cube, Desktop, Phone, } from "@components/react/oai/platform/ui/Icon.react";

Voice agents turn the same agent concepts into spoken, low-latency interactions. The key design choice is deciding whether the model should work directly with live audio or whether your application should explicitly chain speech-to-text, text reasoning, and text-to-speech.

Choose the right architecture

Architecture Best for Why
Speech-to-speech with live audio sessions Natural, low-latency conversations The model handles live audio input and output directly
Chained voice pipeline Predictable workflows or extending an existing text agent Your app keeps explicit control over transcription, text reasoning, and speech output

Agent Builder doesn't currently support voice workflows, so voice stays an SDK-first surface.

The examples below are intentionally different architectures, not matching language tabs. The TypeScript and Python libraries expose different voice helpers today:

  • In TypeScript, the fastest path to a browser-based voice assistant is a RealtimeAgent and RealtimeSession.
  • In Python, the simplest path to extending an existing text agent into voice is a chained VoicePipeline.

Build a speech-to-speech voice agent

Use the live audio API path when the interaction should feel conversational and immediate. This is the best starting point for voice agents that need barge-in, low first-audio latency, natural turn taking, and realtime tool use.

The usual browser flow is:

  1. Your application server creates an ephemeral client secret for the live audio session.
  2. Your frontend creates a RealtimeSession.
  3. The session connects over WebRTC in the browser or WebSocket on the server.
  4. The agent handles audio turns, tools, interruptions, and handoffs inside that session.

From there, attach tools, handoffs, and guardrails to the RealtimeAgent the same way you would attach them to a text agent. Keep audio transport concerns in the session layer, and keep business logic in the agent definition.

Start with the transport docs when you need lower-level control:

Build a chained voice workflow

Use the chained path when you want stronger control over intermediate text, existing text-agent reuse, or a simpler extension path from a non-voice workflow. In that design, your application explicitly manages:

  1. speech-to-text
  2. the agent workflow itself
  3. text-to-speech

This is often the better fit for support flows, approval-heavy flows, or cases where you want durable transcripts and deterministic logic between each stage.

Use this path when each stage needs to be visible or replaceable. For example, you might store the transcript, run policy checks before the text agent responds, call internal systems, then generate speech only after the workflow reaches an approved answer.

Voice agents still use the same core agent building blocks

The voice surface changes the transport and audio loop, but the core workflow decisions are the same:

The practical rule is: choose the audio architecture first, then design the rest of the agent workflow the same way you would for text.

Next steps

Choose the right realtime or audio guide for your use case. Work with the Realtime session lifecycle and event model. Connect browser and mobile audio directly to a Realtime session. Tune reasoning, preambles, tools, entity capture, and voice behavior.