Documentation 2026-07-04 16:59 UTC to 2026-07-06 22:58 UTC

2 2 

3Agents are applications that plan, call tools, collaborate across specialists, and keep enough state to complete multi-step work.3Agents are applications that plan, call tools, collaborate across specialists, and keep enough state to complete multi-step work.

4 4 

5- Use the [**Responses API**](https://developers.openai.com/api/reference/responses/overview) when one model call plus tools and application-owned logic is enough.

6- Use the **Agents SDK** pages when your application owns orchestration, tool execution, approvals, and state.

7 

8## Get your first agent running5## Get your first agent running

9 6 

10Start with the [Agents SDK quickstart](https://developers.openai.com/api/docs/guides/agents/quickstart) to install the SDK, define one agent, and run it. Once that works, return here to choose the next capability your application needs.7Start with the [Agents SDK quickstart](https://developers.openai.com/api/docs/guides/agents/quickstart) to install the SDK, define one agent, and run it. Once that works, return here to choose the next capability your application needs.

62 59 

63## Build with the SDK60## Build with the SDK

64 61 

65Use the SDK track when your server owns orchestration, tool execution, state, and approvals. That path is the best fit when you want:62Use the SDK track when your server owns deployment, tool implementations, state storage, and approval decisions, while the SDK runs the agent loop and invokes those tools. That path is the best fit when you want:

66 63 

67- typed application code in TypeScript or Python64- typed application code in TypeScript or Python

68- direct control over tools, MCP servers, and runtime behavior65- direct control over tools, MCP servers, and runtime behavior

75- Use [Agent definitions](https://developers.openai.com/api/docs/guides/agents/define-agents) and [Models and providers](https://developers.openai.com/api/docs/guides/agents/models) to shape one specialist cleanly.72- Use [Agent definitions](https://developers.openai.com/api/docs/guides/agents/define-agents) and [Models and providers](https://developers.openai.com/api/docs/guides/agents/models) to shape one specialist cleanly.

76- Continue to [Running agents](https://developers.openai.com/api/docs/guides/agents/running-agents), [Orchestration and handoffs](https://developers.openai.com/api/docs/guides/agents/orchestration), and [Guardrails and human review](https://developers.openai.com/api/docs/guides/agents/guardrails-approvals) as the workflow grows more complex.73- Continue to [Running agents](https://developers.openai.com/api/docs/guides/agents/running-agents), [Orchestration and handoffs](https://developers.openai.com/api/docs/guides/agents/orchestration), and [Guardrails and human review](https://developers.openai.com/api/docs/guides/agents/guardrails-approvals) as the workflow grows more complex.

77- Use [Results and state](https://developers.openai.com/api/docs/guides/agents/results) and [Integrations and observability](https://developers.openai.com/api/docs/guides/agents/integrations-observability) when application logic depends on the run object or deeper visibility into behavior.74- Use [Results and state](https://developers.openai.com/api/docs/guides/agents/results) and [Integrations and observability](https://developers.openai.com/api/docs/guides/agents/integrations-observability) when application logic depends on the run object or deeper visibility into behavior.

75 

76## Agents SDK vs. Responses API

77 

78Use the Responses API when you want to own the loop. Use the Agents SDK when you want the SDK to run it.

79 

80### Choose the Responses API when

81 

82- You want direct control over model interactions, output items, tools, state, and orchestration, whether the workflow takes one call or many.

83- You want to implement custom tool routing, loops, or branching directly in your application.

84 

85In the [Responses function-calling flow](https://developers.openai.com/api/docs/guides/function-calling#the-tool-calling-flow), your application receives function calls, executes them, returns their output, and calls the model again.

86 

87For example, a Responses API workflow might search a knowledge base and generate a cited answer.

88 

89### Choose the Agents SDK when

90 

91- You want the SDK to manage the agent loop and recurring orchestration such as repeated tool calls or branching.

92- Different specialists need different instructions, tools, or policies.

93- You want built-in sessions, tracing, guardrails, or resumable approval flows.

94 

95The [Agents SDK runner](https://developers.openai.com/api/docs/guides/agents/running-agents#the-agent-loop) performs the tool loop, switches agents after handoffs, and stops when the run finishes or pauses for approval.

96 

97For example, an Agents SDK workflow might investigate a support request, hand it to the correct specialist, call internal systems, request approval for a refund, and record the result.

98 

99### Compare the Responses API and Agents SDK

100 

101| | Responses API | Agents SDK |

102| -------------------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

103| **Best for** | Custom model-powered features and workflows | Bounded conversational or transactional workflows with defined tools and recurring orchestration patterns |

104| **Core abstraction** | A model response | An agent run |

105| **Tools** | Platform tools, function calling, and remote [Model Context Protocol (MCP)](https://developers.openai.com/api/docs/guides/tools-connectors-mcp) | Platform tools attached to reusable agents, plus tool wrappers, local MCP connections, and [agents as tools](https://developers.openai.com/api/docs/guides/agents/orchestration#use-agents-as-tools-for-manager-style-workflows) |

106| **Workflow orchestration** | You manage custom loops and branching | The SDK provides the agent loop and lifecycle |

107| **Multi-agent workflows** | Build routing and delegation yourself | Built-in agents-as-tools and [handoffs](https://developers.openai.com/api/docs/guides/agents/orchestration#use-handoffs-for-delegated-ownership) |

108| **State** | Manual history, response chaining, or [Conversations](https://developers.openai.com/api/docs/guides/conversation-state#using-the-conversations-api) | The same options, plus [SDK sessions and resumable run state](https://developers.openai.com/api/docs/guides/agents/running-agents#choose-one-conversation-strategy) |

109| **Safety and approvals** | Tool-specific approvals; you build broader controls | Input, output, and tool [guardrails plus resumable approval flows](https://developers.openai.com/api/docs/guides/agents/guardrails-approvals) |

110| **Debugging and tracing** | Response objects and API logs | [Built-in traces](https://developers.openai.com/api/docs/guides/agents/integrations-observability#tracing) across model calls, tools, agents, guardrails, and handoffs |

43 43 

44## Add audio to your existing application44## Add audio to your existing application

45 45 

46Models such as [`gpt-realtime-2`](https://developers.openai.com/api/docs/models/gpt-realtime-2) and [`gpt-audio-1.5`](https://developers.openai.com/api/docs/models/gpt-audio-1.5) are natively multimodal, meaning they can understand and generate audio and text as input and output.46Models such as [`gpt-realtime-2.1`](https://developers.openai.com/api/docs/models/gpt-realtime-2.1) and [`gpt-audio-1.5`](https://developers.openai.com/api/docs/models/gpt-audio-1.5) are natively multimodal, meaning they can understand and generate audio and text as input and output.

47 47 

48For live browser speech-to-speech interactions, start with a realtime session in the JavaScript SDK:48For live browser speech-to-speech interactions, start with a realtime session in the JavaScript SDK:

49 49 

58});58});

59 59 

60const session = new RealtimeSession(agent, {60const session = new RealtimeSession(agent, {

61 model: "gpt-realtime-2",61 model: "gpt-realtime-2.1",

62});62});

63 63 

64await session.connect({64await session.connect({

321,000 rows per sheet and adds model-generated summary and header metadata so the321,000 rows per sheet and adds model-generated summary and header metadata so the

33model can work from a smaller, structured view of the data.33model can work from a smaller, structured view of the data.

34 34 

35## PDF detail levels

36 

37For PDF inputs in the Responses API, set the optional `detail` field on an

38`input_file` item to control how page images are processed. `low` is the default

39and uses fewer input tokens. Use `high` when you need more visual detail from PDF

40page images, such as dense charts, small print, or diagrams.

41 

42The `detail` setting only affects PDF page image processing. Text extracted from

43the PDF is still included. Chat Completions file inputs don't support `detail`.

44 

45A minimal Responses API request body with explicit high detail looks like this:

46 

47```json

48{

49 "model": "gpt-4.1",

50 "input": [

51 {

52 "role": "user",

53 "content": [

54 {

55 "type": "input_file",

56 "filename": "document.pdf",

57 "file_data": "data:application/pdf;base64,...",

58 "detail": "high"

59 },

60 {

61 "type": "input_text",

62 "text": "Summarize this document."

63 }

64 ]

65 }

66 ]

67}

68```

69 

35## Accepted file types70## Accepted file types

36 71 

37The following table lists common file types accepted in `input_file`. The full72The following table lists common file types accepted in `input_file`. The full

387 422 

388Keep these constraints in mind when you use file inputs:423Keep these constraints in mind when you use file inputs:

389 424 

390- **Token usage:** PDF parsing includes both extracted text and page images in context, which can increase token usage. Before deploying at scale, review pricing and token implications. [More on pricing](https://developers.openai.com/api/docs/pricing).425- **Token usage:** PDF parsing includes both extracted text and page images in context, which can increase token usage. In the Responses API, set `detail` to `low` (the default) or `high` to control how much visual detail is processed for PDF page images. Before deploying at scale, review pricing and token implications. [More on pricing](https://developers.openai.com/api/docs/pricing).

391- **File size limits:** A single request can include more than one file, but each file must be under 50 MB. The combined limit across all files in the request is 50 MB.426- **File size limits:** A single request can include more than one file, but each file must be under 50 MB. The combined limit across all files in the request is 50 MB.

392- **Supported models:** PDF parsing that includes text and page images requires models with vision capabilities, such as `gpt-4o` and later models.427- **Supported models:** PDF parsing that includes text and page images requires models with vision capabilities, such as `gpt-4o` and later models.

393- **File upload purpose:** You can upload files with any supported [purpose](https://developers.openai.com/api/docs/api-reference/files/create#files-create-purpose), but use `user_data` for files you plan to pass as model inputs.428- **File upload purpose:** You can upload files with any supported [purpose](https://developers.openai.com/api/docs/api-reference/files/create#files-create-purpose), but use `user_data` for files you plan to pass as model inputs.

61 61 

62![Function Calling Diagram Steps](https://cdn.openai.com/API/docs/images/function-calling-diagram-steps.png)62![Function Calling Diagram Steps](https://cdn.openai.com/API/docs/images/function-calling-diagram-steps.png)

63 63 

64With Responses, your application can continue this flow for as many tool calls as the task requires. If you want a framework that packages recurring orchestration around that loop, see [how the Responses API compares with the Agents SDK](https://developers.openai.com/api/docs/guides/agents#agents-sdk-vs-responses-api).

65 

64## Function tool example66## Function tool example

65 67 

66Let's look at an end-to-end tool calling flow for a `get_horoscope` function that gets a daily horoscope for an astrological sign.68Let's look at an end-to-end tool calling flow for a `get_horoscope` function that gets a daily horoscope for an astrological sign.

21 <tr>21 <tr>

22 <td>Build a low-latency voice agent</td>22 <td>Build a low-latency voice agent</td>

23 <td className="whitespace-nowrap">23 <td className="whitespace-nowrap">

24 [`gpt-realtime-2`](https://developers.openai.com/api/docs/models/gpt-realtime-2)24 [`gpt-realtime-2.1`](https://developers.openai.com/api/docs/models/gpt-realtime-2.1)

25 </td>25 </td>

26 <td>26 <td>

27 <a href="/api/docs/guides/voice-agents">Voice agents</a>27 <a href="/api/docs/guides/voice-agents">Voice agents</a>

1# Realtime conversations1# Realtime conversations

2 2 

3Once you have connected to the Realtime API through either [WebRTC](https://developers.openai.com/api/docs/guides/realtime-webrtc) or [WebSocket](https://developers.openai.com/api/docs/guides/realtime-websocket), you can call a Realtime model (such as [`gpt-realtime-2`](https://developers.openai.com/api/docs/models/gpt-realtime-2)) to have speech-to-speech conversations. Doing so will require you to **send client events** to initiate actions, and **listen for server events** to respond to actions taken by the Realtime API.3Once you have connected to the Realtime API through either [WebRTC](https://developers.openai.com/api/docs/guides/realtime-webrtc) or [WebSocket](https://developers.openai.com/api/docs/guides/realtime-websocket), you can call a Realtime model (such as [`gpt-realtime-2.1`](https://developers.openai.com/api/docs/models/gpt-realtime-2.1)) to have speech-to-speech conversations. Doing so will require you to **send client events** to initiate actions, and **listen for server events** to respond to actions taken by the Realtime API.

4 4 

5This guide will walk through the event flows required to use model capabilities like audio and text generation, image input, and function calling, and how to think about the state of a Realtime Session.5This guide will walk through the event flows required to use model capabilities like audio and text generation, image input, and function calling, and how to think about the state of a Realtime Session.

6 6 

40 type: "session.update",40 type: "session.update",

41 session: {41 session: {

42 type: "realtime",42 type: "realtime",

43 model: "gpt-realtime-2",43 model: "gpt-realtime-2.1",

44 // Lock the output to audio (set to ["text"] if you want text without audio)44 // Lock the output to audio (set to ["text"] if you want text without audio)

45 output_modalities: ["audio"],45 output_modalities: ["audio"],

46 audio: {46 audio: {

82 "type": "session.update",82 "type": "session.update",

83 session: {83 session: {

84 type: "realtime",84 type: "realtime",

85 model: "gpt-realtime-2",85 model: "gpt-realtime-2.1",

86 # Lock the output to audio (add "text" if you also want text)86 # Lock the output to audio (add "text" if you also want text)

87 output_modalities: ["audio"],87 output_modalities: ["audio"],

88 audio: {88 audio: {

28 type: "session.update",28 type: "session.update",

29 session: {29 session: {

30 type: "realtime",30 type: "realtime",

31 model: "gpt-realtime-2",31 model: "gpt-realtime-2.1",

32 tools: [32 tools: [

33 {33 {

34 type: "function",34 type: "function",

58 "type": "session.update",58 "type": "session.update",

59 "session": {59 "session": {

60 "type": "realtime",60 "type": "realtime",

61 "model": "gpt-realtime-2",61 "model": "gpt-realtime-2.1",

62 "tools": [62 "tools": [

63 {63 {

64 "type": "function",64 "type": "function",

150 type: "session.update",150 type: "session.update",

151 session: {151 session: {

152 type: "realtime",152 type: "realtime",

153 model: "gpt-realtime-2",153 model: "gpt-realtime-2.1",

154 output_modalities: ["text"],154 output_modalities: ["text"],

155 tools: [155 tools: [

156 {156 {

172 "type": "session.update",172 "type": "session.update",

173 "session": {173 "session": {

174 "type": "realtime",174 "type": "realtime",

175 "model": "gpt-realtime-2",175 "model": "gpt-realtime-2.1",

176 "output_modalities": ["text"],176 "output_modalities": ["text"],

177 "tools": [177 "tools": [

178 {178 {

204 type: "session.update",204 type: "session.update",

205 session: {205 session: {

206 type: "realtime",206 type: "realtime",

207 model: "gpt-realtime-2",207 model: "gpt-realtime-2.1",

208 output_modalities: ["text"],208 output_modalities: ["text"],

209 tools: [209 tools: [

210 {210 {

227 "type": "session.update",227 "type": "session.update",

228 "session": {228 "session": {

229 "type": "realtime",229 "type": "realtime",

230 "model": "gpt-realtime-2",230 "model": "gpt-realtime-2.1",

231 "output_modalities": ["text"],231 "output_modalities": ["text"],

232 "tools": [232 "tools": [

233 {233 {

67 -H "Content-Type: application/json" \67 -H "Content-Type: application/json" \

68 -d '{68 -d '{

69 "type": "realtime",69 "type": "realtime",

70 "model": "gpt-realtime-2",70 "model": "gpt-realtime-2.1",

71 "instructions": "You are Alex, a friendly concierge for Example Corp."71 "instructions": "You are Alex, a friendly concierge for Example Corp."

72 }'72 }'

73```73```

222call_accept = {222call_accept = {

223 "type": "realtime",223 "type": "realtime",

224 "instructions": "You are a support agent.",224 "instructions": "You are a support agent.",

225 "model": "gpt-realtime-2",225 "model": "gpt-realtime-2.1",

226}226}

227 227 

228response_create = {228response_create = {

2 2 

3Realtime translation lets you stream source audio into a dedicated translation session and receive translated audio plus transcript deltas while the speaker is still talking. Use it for live interpretation, multilingual calls, broadcasts, meetings, lessons, and video rooms.3Realtime translation lets you stream source audio into a dedicated translation session and receive translated audio plus transcript deltas while the speaker is still talking. Use it for live interpretation, multilingual calls, broadcasts, meetings, lessons, and video rooms.

4 4 

5Use [`gpt-realtime-translate`](https://developers.openai.com/api/docs/models/gpt-realtime-translate) when your application should translate what a human says. If you need an assistant that answers questions, calls tools, and manages a conversation, use [`gpt-realtime-2`](https://developers.openai.com/api/docs/models/gpt-realtime-2) with a standard Realtime session instead.5Use [`gpt-realtime-translate`](https://developers.openai.com/api/docs/models/gpt-realtime-translate) when your application should translate what a human says. If you need an assistant that answers questions, calls tools, and manages a conversation, use [`gpt-realtime-2.1`](https://developers.openai.com/api/docs/models/gpt-realtime-2.1) with a standard Realtime session instead.

6 6 

7## How translation sessions differ7## How translation sessions differ

8 8 

37 37 

38const sessionConfig = JSON.stringify({38const sessionConfig = JSON.stringify({

39 type: "realtime",39 type: "realtime",

40 model: "gpt-realtime-2",40 model: "gpt-realtime-2.1",

41 audio: { output: { voice: "marin" } },41 audio: { output: { voice: "marin" } },

42});42});

43 43 

141const sessionConfig = JSON.stringify({141const sessionConfig = JSON.stringify({

142 session: {142 session: {

143 type: "realtime",143 type: "realtime",

144 model: "gpt-realtime-2",144 model: "gpt-realtime-2.1",

145 audio: {145 audio: {

146 output: {146 output: {

147 voice: "marin",147 voice: "marin",

21```javascript21```javascript

22import WebSocket from "ws";22import WebSocket from "ws";

23 23 

24const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2";24const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1";

25const ws = new WebSocket(url, {25const ws = new WebSocket(url, {

26 headers: {26 headers: {

27 Authorization: "Bearer " + process.env.OPENAI_API_KEY,27 Authorization: "Bearer " + process.env.OPENAI_API_KEY,

53 53 

54OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY")54OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY")

55 55 

56url = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2"56url = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1"

57headers = [57headers = [

58 "Authorization: Bearer " + OPENAI_API_KEY,58 "Authorization: Bearer " + OPENAI_API_KEY,

59 "OpenAI-Safety-Identifier: hashed-user-id",59 "OpenAI-Safety-Identifier: hashed-user-id",

93*/93*/

94 94 

95const ws = new WebSocket(95const ws = new WebSocket(

96 "wss://api.openai.com/v1/realtime?model=gpt-realtime-2",96 "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1",

97 [97 [

98 "realtime",98 "realtime",

99 // Auth99 // Auth

126```javascript126```javascript

127import WebSocket from "ws";127import WebSocket from "ws";

128 128 

129const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2";129const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1";

130const ws = new WebSocket(url, {130const ws = new WebSocket(url, {

131 headers: {131 headers: {

132 Authorization: "Bearer " + process.env.OPENAI_API_KEY,132 Authorization: "Bearer " + process.env.OPENAI_API_KEY,

97-d '{97-d '{

98"session": {98"session": {

99"type": "realtime",99"type": "realtime",

100"model": "gpt-realtime-2"100"model": "gpt-realtime-2.1"

101}101}

102}'102}'

103```103```

42});42});

43 43 

44const session = new RealtimeSession(agent, {44const session = new RealtimeSession(agent, {

45 model: "gpt-realtime-2",45 model: "gpt-realtime-2.1",

46});46});

47 47 

48await session.connect({48await session.connect({

196| `/v1/images/edits` | Images | All listed regions | United States, Europe (EEA + Switzerland) | `gpt-image-1`, `gpt-image-1.5`, `gpt-image-1-mini` | None | — |196| `/v1/images/edits` | Images | All listed regions | United States, Europe (EEA + Switzerland) | `gpt-image-1`, `gpt-image-1.5`, `gpt-image-1-mini` | None | — |

197| `/v1/images/generations` | Images | All listed regions | United States, Europe (EEA + Switzerland) | `dall-e-3`, `gpt-image-1`, `gpt-image-1.5`, `gpt-image-1-mini` | None | — |197| `/v1/images/generations` | Images | All listed regions | United States, Europe (EEA + Switzerland) | `dall-e-3`, `gpt-image-1`, `gpt-image-1.5`, `gpt-image-1-mini` | None | — |

198| `/v1/moderations` | Moderation | All listed regions | United States, Europe (EEA + Switzerland) | `omni-moderation-latest` | None | — |198| `/v1/moderations` | Moderation | All listed regions | United States, Europe (EEA + Switzerland) | `omni-moderation-latest` | None | — |

199| `/v1/realtime` | Realtime | United States, Europe (EEA + Switzerland) | United States, Europe (EEA + Switzerland) | `gpt-4o-realtime-preview-2025-06-03`, `gpt-realtime`, `gpt-realtime-1.5`, `gpt-realtime-mini`, `gpt-realtime-2` | None | — |199| `/v1/realtime` | Realtime | United States, Europe (EEA + Switzerland) | United States, Europe (EEA + Switzerland) | `gpt-4o-realtime-preview-2025-06-03`, `gpt-realtime`, `gpt-realtime-1.5`, `gpt-realtime-mini`, `gpt-realtime-2`, `gpt-realtime-2.1`, `gpt-realtime-2.1-mini` | None | — |

200| `/v1/realtime/transcription_sessions` | Realtime | United States, Europe (EEA + Switzerland) | United States, Europe (EEA + Switzerland) | `gpt-realtime-whisper` | None | — |200| `/v1/realtime/transcription_sessions` | Realtime | United States, Europe (EEA + Switzerland) | United States, Europe (EEA + Switzerland) | `gpt-realtime-whisper` | None | — |

201| `/v1/realtime/translations` | Realtime | United States, Europe (EEA + Switzerland) | United States, Europe (EEA + Switzerland) | `gpt-realtime-translate` | None | — |201| `/v1/realtime/translations` | Realtime | United States, Europe (EEA + Switzerland) | United States, Europe (EEA + Switzerland) | `gpt-realtime-translate` | None | — |

202| `/v1/realtime (legacy previews)` | Realtime | United States | United States | `gpt-4o-realtime-preview-2024-12-17`, `gpt-4o-realtime-preview-2024-10-01`, `gpt-4o-mini-realtime-preview-2024-12-17` | None | — |202| `/v1/realtime (legacy previews)` | Realtime | United States | United States | `gpt-4o-realtime-preview-2024-12-17`, `gpt-4o-realtime-preview-2024-10-01`, `gpt-4o-mini-realtime-preview-2024-12-17` | None | — |

65when your application needs code-first orchestration for agents, tools,65when your application needs code-first orchestration for agents, tools,

66handoffs, guardrails, tracing, or sandbox execution.66handoffs, guardrails, tracing, or sandbox execution.

67 67 

68If you are deciding between direct API requests and code-first orchestration,

69see [how the Responses API compares with the Agents SDK](https://developers.openai.com/api/docs/guides/agents#agents-sdk-vs-responses-api).

70 

68<a href="/api/docs/guides/agents/quickstart">71<a href="/api/docs/guides/agents/quickstart">

69 72

70 73