1# Realtime API1# Realtime and audio
2 2
3import {3import {
4 Bolt,
5 Phone,
6 Cube,4 Cube,
7 Desktop,5 Desktop,
6 Phone,
8} from "@components/react/oai/platform/ui/Icon.react";7} from "@components/react/oai/platform/ui/Icon.react";
9 8
10 9Start with the outcome you want to build. Realtime sessions are best for live audio that needs low latency. Request-based audio APIs are best for files, bounded requests, or generated speech that doesn't need a live session.
11 10
12The OpenAI Realtime API enables low-latency communication with [models](https://developers.openai.com/api/docs/models) that natively support speech-to-speech interactions as well as multimodal inputs (audio, images, and text) and outputs (audio and text). These APIs can also be used for [realtime audio transcription](https://developers.openai.com/api/docs/guides/realtime-transcription).11## Common use cases
13 12
14## Voice agents13<div className="w-full max-w-full overflow-hidden">
15 14 </div>
16One of the most common use cases for the Realtime API is building voice agents for speech-to-speech model interactions in the browser. Our recommended starting point for these applications is the on-site [Voice agents](https://developers.openai.com/api/docs/guides/voice-agents) guide, which uses a [WebRTC connection](https://developers.openai.com/api/docs/guides/realtime-webrtc) to the Realtime model in the browser, and [WebSocket](https://developers.openai.com/api/docs/guides/realtime-websocket) when used on the server.15
17 16## Understand different architectures
18```js17
19 18<table>
20 19 <thead>
21const agent = new RealtimeAgent({20 <tr>
22 name: "Assistant",21 <th>Goal</th>
23 instructions: "You are a helpful assistant.",22 <th>Model or API</th>
24});23 <th>Start here</th>
25 24 </tr>
26const session = new RealtimeSession(agent);25 </thead>
27 26 <tbody>
28// Automatically connects your microphone and audio output27 <tr>
29await session.connect({28 <td>Build a low-latency voice agent</td>
30 apiKey: "<client-api-key>",29 <td className="whitespace-nowrap">
31});30 <a href="/api/docs/models/gpt-realtime-2">
32```31 <code>gpt-realtime-2</code>
33 32 </a>
34<a href="/api/docs/guides/voice-agents#speech-to-speech-realtime-architecture">33 </td>
35 34 <td>
36 35 <a href="/api/docs/guides/voice-agents">Voice agents</a>
37<span slot="icon">36 </td>
38 </span>37 </tr>
39 See the speech-to-speech path for building Realtime voice agents in the38 <tr>
40 browser.39 <td>Translate live speech into another language</td>
41 40 <td className="whitespace-nowrap">
42 41 <a href="/api/docs/models/gpt-realtime-translate">
43</a>42 <code>gpt-realtime-translate</code>
44 43 </a>
45To use the Realtime API directly outside the context of voice agents, check out the other connection options below.44 </td>
46 45 <td>
47## Connection methods46 <a href="/api/docs/guides/realtime-translation">Realtime translation</a>
48 47 </td>
49While building [voice agents with the Agents SDK](https://developers.openai.com/api/docs/guides/voice-agents) is the fastest path to one specific type of application, the Realtime API provides an entire suite of flexible tools for a variety of use cases.48 </tr>
50 49 <tr>
51There are three primary supported interfaces for the Realtime API:50 <td>Transcribe live audio into streaming text</td>
51 <td className="whitespace-nowrap">
52 <a href="/api/docs/models/gpt-realtime-whisper">
53 <code>gpt-realtime-whisper</code>
54 </a>
55 </td>
56 <td>
57 <a href="/api/docs/guides/realtime-transcription">
58 Realtime transcription
59 </a>
60 </td>
61 </tr>
62 <tr>
63 <td>Transcribe files or bounded audio requests</td>
64 <td>Audio transcription models</td>
65 <td>
66 <a href="/api/docs/guides/speech-to-text">Speech to text</a>
67 </td>
68 </tr>
69 <tr>
70 <td>Generate speech from text</td>
71 <td>Speech generation models</td>
72 <td>
73 <a href="/api/docs/guides/text-to-speech">Text to speech</a>
74 </td>
75 </tr>
76 <tr>
77 <td>Add audio to an existing Chat Completions app</td>
78 <td>Audio-capable chat models</td>
79 <td>
80 <a href="/api/docs/guides/audio#add-audio-to-your-existing-application">
81 Audio and speech
82 </a>
83 </td>
84 </tr>
85 </tbody>
86</table>
87
88## Choose a realtime session
89
90Realtime sessions keep a connection open while your application sends audio, receives events, and updates session state.
91
92<table>
93 <thead>
94 <tr>
95 <th>Session type</th>
96 <th>Use when</th>
97 <th>Endpoint or pattern</th>
98 </tr>
99 </thead>
100 <tbody>
101 <tr>
102 <td>Voice-agent session</td>
103 <td>
104 The model should respond to the user, call tools, and manage
105 conversation state.
106 </td>
107 <td>
108 Conversation session on <code>/v1/realtime</code>
109 </td>
110 </tr>
111 <tr>
112 <td>Translation session</td>
113 <td>The app should continuously translate speech as it arrives.</td>
114 <td>
115 Continuous translation session on <code>/v1/realtime/translations</code>
116 </td>
117 </tr>
118 <tr>
119 <td>Transcription session</td>
120 <td>
121 The app needs streaming transcript deltas without model-generated spoken
122 responses.
123 </td>
124 <td>Transcription session that emits transcript deltas</td>
125 </tr>
126 </tbody>
127</table>
128
129Use a voice-agent session when your application needs an assistant that responds to the user. Use a translation session when your application needs an interpreter that translates the speaker. Use a transcription session when your application needs text from audio without model-generated responses.
130
131### Voice-agent sessions
132
133Voice-agent sessions use the standard Realtime API conversation lifecycle. The client connects to `/v1/realtime`, sends audio or text, and listens for model responses, tool calls, and session events.
134
135For most browser voice agents, start with the [Voice agents](https://developers.openai.com/api/docs/guides/voice-agents) guide. It uses the Agents SDK with WebRTC for browser audio and can connect to server-side tools.
136
137Realtime 2 adds reasoning to speech-to-speech workflows. Start with
138 `reasoning.effort` set to `low` for most production voice agents, then adjust
139 based on latency tolerance and task complexity. Use the [Realtime prompting
140 guide](https://developers.openai.com/api/docs/guides/realtime-models-prompting) to tune reasoning,
141 preambles, tool use, unclear audio, and exact entity capture.
142
143### Translation sessions
144
145Realtime translation uses a dedicated translation endpoint instead of the standard voice-agent endpoint. Translation sessions are continuous: the client streams audio into the session, and the service streams translated audio and transcript deltas out.
146
147Translation sessions don't use the normal assistant turn lifecycle. Don't call `response.create`, and don't wait for the client to commit a user turn before translation begins. For browser media, use WebRTC. For server media pipelines such as phone calls or broadcast ingest, use WebSockets.
148
149See [Realtime translation](https://developers.openai.com/api/docs/guides/realtime-translation) for the dedicated endpoint, session configuration, and architecture patterns.
150
151### Transcription sessions
152
153You can transcribe audio in more than one way. Use a realtime transcription session when your application needs live transcript deltas from streaming audio. Use the [Speech to text](https://developers.openai.com/api/docs/guides/speech-to-text) guide for file uploads, request-based transcription, or diarization-focused workflows.
154
155For realtime transcription, `gpt-realtime-whisper` gives you controllable latency. Lower delay settings produce earlier partial text, while higher delay settings can improve transcript quality. Test with your real audio conditions, target languages, accents, and domain vocabulary before choosing a production default.
156
157See [Realtime transcription](https://developers.openai.com/api/docs/guides/realtime-transcription) for session configuration and event handling.
158
159## Choose a connection method
160
161Choose the transport based on where your application captures and plays audio:
52 162
53[163[
54 164
55<span slot="icon">165<span slot="icon">
56 </span>166 </span>
57 Ideal for browser and client-side interactions with a Realtime model.167 Use for browser and mobile clients that capture or play audio directly.
58 168
59](https://developers.openai.com/api/docs/guides/realtime-webrtc)169](https://developers.openai.com/api/docs/guides/realtime-webrtc)
60 170
62 172
63<span slot="icon">173<span slot="icon">
64 </span>174 </span>
65 Ideal for middle tier server-side applications with consistent low-latency175 Use when your server already receives raw audio from a media pipeline, call
66 network connections.176 system, or worker.
67 177
68](https://developers.openai.com/api/docs/guides/realtime-websocket)178](https://developers.openai.com/api/docs/guides/realtime-websocket)
69 179
71 181
72<span slot="icon">182<span slot="icon">
73 </span>183 </span>
74 Ideal for VoIP telephony connections.184 Use for telephony voice agents. Confirm model support before using SIP for
185 translation or transcription.
75 186
76](https://developers.openai.com/api/docs/guides/realtime-sip)187](https://developers.openai.com/api/docs/guides/realtime-sip)
77 188
78Depending on how you'd like to connect to a Realtime model, check out one of the connection guides above to get started. You'll learn how to initialize a Realtime session, and how to interact with a Realtime model using client and server events.189## Safety identifiers
79
80## API Usage
81
82Once connected to a realtime model using one of the methods above, learn how to interact with the model in these usage guides.
83
84- **[Prompting guide](https://developers.openai.com/api/docs/guides/realtime-models-prompting):** learn tips and best practices for prompting and steering Realtime models.
85- **[Managing conversations](https://developers.openai.com/api/docs/guides/realtime-conversations):** Learn about the Realtime session lifecycle and the key events that happen during a conversation.
86- **[MCP servers](https://developers.openai.com/api/docs/guides/realtime-mcp):** Connect remote MCP servers or connectors to a Realtime session and handle their event flow.
87- **[Webhooks and server-side controls](https://developers.openai.com/api/docs/guides/realtime-server-controls):** Learn how you can control a Realtime session on the server to call tools and implement guardrails.
88- **[Managing costs](https://developers.openai.com/api/docs/guides/realtime-costs):** Learn how to monitor and optimize your usage of the Realtime API.
89- **[Realtime audio transcription](https://developers.openai.com/api/docs/guides/realtime-transcription):** Transcribe audio streams in real time over a WebSocket connection.
90
91## Beta to GA migration
92
93There are a few key differences between the interfaces in the Realtime beta API and the recently released GA API. Expand the topics below for more information about migrating from the beta interface to GA.
94
95Beta header
96
97For REST API requests, WebSocket connections, and other interfaces with the Realtime API, beta users had to include the following header with each request:
98
99```
100OpenAI-Beta: realtime=v1
101```
102
103This header should be removed for requests to the GA interface. To retain the behavior of the beta API, you should continue to include this header.
104
105Generating ephemeral API keys
106
107In the beta interface, there were multiple endpoints for generating ephemeral keys for either Realtime sessions or transcription sessions. In the GA interface, there is only one REST API endpoint used to generate keys - [`POST /v1/realtime/client_secrets`](https://developers.openai.com/api/docs/api-reference/realtime-sessions/create-realtime-client-secret).
108
109To create a session and receive a client secret you can use to initialize a WebRTC or WebSocket connection on a client, you can request one like this using the appropriate session configuration:
110
111```javascript
112const sessionConfig = JSON.stringify({
113 session: {
114 type: "realtime",
115 model: "gpt-realtime",
116 audio: {
117 output: { voice: "marin" },
118 },
119 },
120});
121
122const response = await fetch(
123 "https://api.openai.com/v1/realtime/client_secrets",
124 {
125 method: "POST",
126 headers: {
127 Authorization: `Bearer ${apiKey}`,
128 "Content-Type": "application/json",
129 },
130 body: sessionConfig,
131 }
132);
133
134const data = await response.json();
135console.log(data.value); // e.g. ek_68af296e8e408191a1120ab6383263c2
136```
137
138These tokens can safely be used in client environments like browsers and mobile applications.
139
140New URL for WebRTC SDP data
141
142When initializing a WebRTC session in the browser, the URL for obtaining remote session information via SDP is now `/v1/realtime/calls`:
143
144```javascript
145const baseUrl = "https://api.openai.com/v1/realtime/calls";
146const model = "gpt-realtime";
147const sdpResponse = await fetch(baseUrl, {
148 method: "POST",
149 body: offer.sdp,
150 headers: {
151 Authorization: `Bearer YOUR_EPHEMERAL_KEY_HERE`,
152 "Content-Type": "application/sdp",
153 },
154});
155
156const sdp = await sdpResponse.text();
157const answer = { type: "answer", sdp };
158await pc.setRemoteDescription(answer);
159```
160
161New event names and shapes
162
163When creating or [updating](https://developers.openai.com/api/docs/api-reference/realtime_client_events/session/update) a Realtime session in the GA interface, you must now specify a session type, since now the same client event is used to create both speech-to-speech and transcription sessions. The options for the session type are:
164
165- `realtime` for speech-to-speech
166- `transcription` for realtime audio transcription
167
168```javascript
169
170
171const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime";
172const ws = new WebSocket(url, {
173 headers: {
174 Authorization: "Bearer " + process.env.OPENAI_API_KEY,
175 },
176});
177
178ws.on("open", function open() {
179 console.log("Connected to server.");
180
181 // Send client events over the WebSocket once connected
182 ws.send(
183 JSON.stringify({
184 type: "session.update",
185 session: {
186 type: "realtime",
187 instructions: "Be extra nice today!",
188 },
189 })
190 );
191});
192```
193
194Configuration for input modalities and other properties have moved as well,
195notably output audio configuration like model voice. [Check the API reference](https://developers.openai.com/api/docs/api-reference/realtime_client_events) for the latest event shapes.
196
197```javascript
198ws.on("open", function open() {
199 ws.send(
200 JSON.stringify({
201 type: "session.update",
202 session: {
203 type: "realtime",
204 model: "gpt-realtime",
205 audio: {
206 output: { voice: "marin" },
207 },
208 },
209 })
210 );
211});
212```
213
214Finally, some event names have changed to reflect their new position in the event data model:
215
216- **`response.text.delta` → `response.output_text.delta`**
217- **`response.audio.delta` → `response.output_audio.delta`**
218- **`response.audio_transcript.delta` → `response.output_audio_transcript.delta`**
219
220New conversation item events
221
222For `response.output_item`, the API has always had both `.added` and `.done` events, but for conversation level items the API previously only had `.created`, which by convention is emitted at the start when the item added.
223
224We have added a `.added` and `.done` event to allow better ergonomics for developers when receiving events that need some loading time (such as MCP tool listing or input audio transcriptions if these were to be modeled as items in the future).
225
226Current event shape for conversation items added:
227
228```javascript
229{
230 "event_id": "event_1920",
231 "type": "conversation.item.created",
232 "previous_item_id": "msg_002",
233 "item": Item
234}
235```
236 190
237New events to replace the above:191If your application identifies individual end users, include a [safety identifier](https://developers.openai.com/api/docs/guides/safety-best-practices#implement-safety-identifiers) with Realtime API requests. Safety identifiers are recommended but not required. They help OpenAI monitor and detect abuse while allowing enforcement to target an individual user rather than your entire organization. Use a stable, privacy-preserving value, such as a hashed internal user ID.
238
239```javascript
240{
241 "event_id": "event_1920",
242 "type": "conversation.item.added",
243 "previous_item_id": "msg_002",
244 "item": Item
245}
246```
247
248```javascript
249{
250 "event_id": "event_1920",
251 "type": "conversation.item.done",
252 "previous_item_id": "msg_002",
253 "item": Item
254}
255```
256 192
257Input and output item changes193For Realtime API requests, send the identifier in the `OpenAI-Safety-Identifier` header. When using ephemeral tokens, set the header on the server-side request that creates the client secret so the identifier is bound to that session. When connecting from a trusted server with WebSocket or the unified WebRTC interface, set the header on the connection request.
258 194
259### All Items195Safety identifiers do not carry over from Responses API requests or from other sessions. If you use the Responses API `safety_identifier` parameter elsewhere in your application, pass the same stable value separately when you create or connect each Realtime session.
260 196
261Realtime API sets an `object=realtime.item` param on all items in the GA interface.197## Related guides
262 198
263### Function Call Output199- [Realtime prompting guide](https://developers.openai.com/api/docs/guides/realtime-models-prompting): Prompt and tune Realtime voice models.
200- [Managing conversations](https://developers.openai.com/api/docs/guides/realtime-conversations): Work with the Realtime session lifecycle.
201- [Realtime translation](https://developers.openai.com/api/docs/guides/realtime-translation): Translate live speech with a dedicated translation session.
202- [Realtime transcription](https://developers.openai.com/api/docs/guides/realtime-transcription): Stream live transcript deltas from audio.
203- [Realtime with tools](https://developers.openai.com/api/docs/guides/realtime-mcp): Connect function tools, MCP servers, and connectors to a Realtime session.
204- [Webhooks and server-side controls](https://developers.openai.com/api/docs/guides/realtime-server-controls): Control Realtime sessions from your server.
205- [Managing costs](https://developers.openai.com/api/docs/guides/realtime-costs): Track and optimize Realtime API usage.
264 206
265`status` : Realtime now accepts a no-op `status` field for the function call output item param. This aligns with the Responses API implementation.207Use [Audio and speech](https://developers.openai.com/api/docs/guides/audio) for the core concepts behind
266 208 audio input, audio output, streaming, latency, transcripts, and speech
267### Message209 generation. Use this overview when you are ready to choose an implementation
268 210 path.
269**Assistant Message Content**
270
271The `type` properties of output assistant messages now align with the Responses API:
272
273- `type=text` → `type=output_text` (no change to `text` field name)
274- `type=audio` → `type=output_audio` (no change to `audio` field name)
