diff --git a/en/resources/realtime/index.md b/en/resources/realtime/index.md index 7c03c41..1c1ff6c 100644 --- a/en/resources/realtime/index.md +++ b/en/resources/realtime/index.md @@ -4,7 +4,7 @@ ### Audio Transcription -- `AudioTranscription = object { language, model, prompt }` +- `AudioTranscription object { language, model, prompt }` - `language: optional string` @@ -41,7 +41,7 @@ ### Conversation Created Event -- `ConversationCreatedEvent = object { conversation, event_id, type }` +- `ConversationCreatedEvent object { conversation, event_id, type }` Returned when a conversation is created. Emitted right after session creation. @@ -73,7 +73,7 @@ A single item within a Realtime conversation. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. @@ -123,7 +123,7 @@ - `"in_progress"` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` A user message item in a Realtime conversation. @@ -199,7 +199,7 @@ - `"in_progress"` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` An assistant message item in a Realtime conversation. @@ -259,7 +259,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` A function call item in a Realtime conversation. @@ -301,7 +301,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` A function call output item in a Realtime conversation. @@ -339,7 +339,7 @@ - `"in_progress"` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` A Realtime item responding to an MCP approval request. @@ -365,7 +365,7 @@ Optional reason for the decision. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `RealtimeMcpListTools object { server_label, tools, type, id }` A Realtime item listing tools available on an MCP server. @@ -403,7 +403,7 @@ The unique ID of the list. - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` A Realtime item representing an invocation of a tool on an MCP server. @@ -437,7 +437,7 @@ The error from the tool call, if any. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `RealtimeMcpProtocolError object { code, message, type }` - `code: number` @@ -447,7 +447,7 @@ - `"protocol_error"` - - `RealtimeMcpToolExecutionError = object { message, type }` + - `RealtimeMcpToolExecutionError object { message, type }` - `message: string` @@ -455,7 +455,7 @@ - `"tool_execution_error"` - - `RealtimeMcphttpError = object { code, message, type }` + - `RealtimeMcphttpError object { code, message, type }` - `code: number` @@ -469,7 +469,7 @@ The output from the tool call. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` A Realtime item requesting human approval of a tool invocation. @@ -497,7 +497,7 @@ ### Conversation Item Added -- `ConversationItemAdded = object { event_id, item, type, previous_item_id }` +- `ConversationItemAdded object { event_id, item, type, previous_item_id }` Sent by the server when an Item is added to the default Conversation. This can happen in several cases: @@ -515,7 +515,7 @@ A single item within a Realtime conversation. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. @@ -565,7 +565,7 @@ - `"in_progress"` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` A user message item in a Realtime conversation. @@ -641,7 +641,7 @@ - `"in_progress"` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` An assistant message item in a Realtime conversation. @@ -701,7 +701,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` A function call item in a Realtime conversation. @@ -743,7 +743,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` A function call output item in a Realtime conversation. @@ -781,7 +781,7 @@ - `"in_progress"` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` A Realtime item responding to an MCP approval request. @@ -807,7 +807,7 @@ Optional reason for the decision. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `RealtimeMcpListTools object { server_label, tools, type, id }` A Realtime item listing tools available on an MCP server. @@ -845,7 +845,7 @@ The unique ID of the list. - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` A Realtime item representing an invocation of a tool on an MCP server. @@ -879,7 +879,7 @@ The error from the tool call, if any. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `RealtimeMcpProtocolError object { code, message, type }` - `code: number` @@ -889,7 +889,7 @@ - `"protocol_error"` - - `RealtimeMcpToolExecutionError = object { message, type }` + - `RealtimeMcpToolExecutionError object { message, type }` - `message: string` @@ -897,7 +897,7 @@ - `"tool_execution_error"` - - `RealtimeMcphttpError = object { code, message, type }` + - `RealtimeMcphttpError object { code, message, type }` - `code: number` @@ -911,7 +911,7 @@ The output from the tool call. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` A Realtime item requesting human approval of a tool invocation. @@ -950,7 +950,7 @@ ### Conversation Item Create Event -- `ConversationItemCreateEvent = object { item, type, event_id, previous_item_id }` +- `ConversationItemCreateEvent object { item, type, event_id, previous_item_id }` Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a @@ -964,7 +964,7 @@ A single item within a Realtime conversation. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. @@ -1014,7 +1014,7 @@ - `"in_progress"` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` A user message item in a Realtime conversation. @@ -1090,7 +1090,7 @@ - `"in_progress"` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` An assistant message item in a Realtime conversation. @@ -1150,7 +1150,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` A function call item in a Realtime conversation. @@ -1192,7 +1192,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` A function call output item in a Realtime conversation. @@ -1230,7 +1230,7 @@ - `"in_progress"` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` A Realtime item responding to an MCP approval request. @@ -1256,7 +1256,7 @@ Optional reason for the decision. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `RealtimeMcpListTools object { server_label, tools, type, id }` A Realtime item listing tools available on an MCP server. @@ -1294,7 +1294,7 @@ The unique ID of the list. - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` A Realtime item representing an invocation of a tool on an MCP server. @@ -1328,7 +1328,7 @@ The error from the tool call, if any. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `RealtimeMcpProtocolError object { code, message, type }` - `code: number` @@ -1338,7 +1338,7 @@ - `"protocol_error"` - - `RealtimeMcpToolExecutionError = object { message, type }` + - `RealtimeMcpToolExecutionError object { message, type }` - `message: string` @@ -1346,7 +1346,7 @@ - `"tool_execution_error"` - - `RealtimeMcphttpError = object { code, message, type }` + - `RealtimeMcphttpError object { code, message, type }` - `code: number` @@ -1360,7 +1360,7 @@ The output from the tool call. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` A Realtime item requesting human approval of a tool invocation. @@ -1406,7 +1406,7 @@ ### Conversation Item Created Event -- `ConversationItemCreatedEvent = object { event_id, item, type, previous_item_id }` +- `ConversationItemCreatedEvent object { event_id, item, type, previous_item_id }` Returned when a conversation item is created. There are several scenarios that produce this event: @@ -1427,7 +1427,7 @@ A single item within a Realtime conversation. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. @@ -1477,7 +1477,7 @@ - `"in_progress"` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` A user message item in a Realtime conversation. @@ -1553,7 +1553,7 @@ - `"in_progress"` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` An assistant message item in a Realtime conversation. @@ -1613,7 +1613,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` A function call item in a Realtime conversation. @@ -1655,7 +1655,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` A function call output item in a Realtime conversation. @@ -1693,7 +1693,7 @@ - `"in_progress"` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` A Realtime item responding to an MCP approval request. @@ -1719,7 +1719,7 @@ Optional reason for the decision. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `RealtimeMcpListTools object { server_label, tools, type, id }` A Realtime item listing tools available on an MCP server. @@ -1757,7 +1757,7 @@ The unique ID of the list. - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` A Realtime item representing an invocation of a tool on an MCP server. @@ -1791,7 +1791,7 @@ The error from the tool call, if any. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `RealtimeMcpProtocolError object { code, message, type }` - `code: number` @@ -1801,7 +1801,7 @@ - `"protocol_error"` - - `RealtimeMcpToolExecutionError = object { message, type }` + - `RealtimeMcpToolExecutionError object { message, type }` - `message: string` @@ -1809,7 +1809,7 @@ - `"tool_execution_error"` - - `RealtimeMcphttpError = object { code, message, type }` + - `RealtimeMcphttpError object { code, message, type }` - `code: number` @@ -1823,7 +1823,7 @@ The output from the tool call. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` A Realtime item requesting human approval of a tool invocation. @@ -1863,7 +1863,7 @@ ### Conversation Item Delete Event -- `ConversationItemDeleteEvent = object { item_id, type, event_id }` +- `ConversationItemDeleteEvent object { item_id, type, event_id }` Send this event when you want to remove any item from the conversation history. The server will respond with a `conversation.item.deleted` event, @@ -1886,7 +1886,7 @@ ### Conversation Item Deleted Event -- `ConversationItemDeletedEvent = object { event_id, item_id, type }` +- `ConversationItemDeletedEvent object { event_id, item_id, type }` Returned when an item in the conversation is deleted by the client with a `conversation.item.delete` event. This event is used to synchronize the @@ -1908,7 +1908,7 @@ ### Conversation Item Done -- `ConversationItemDone = object { event_id, item, type, previous_item_id }` +- `ConversationItemDone object { event_id, item, type, previous_item_id }` Returned when a conversation item is finalized. @@ -1922,7 +1922,7 @@ A single item within a Realtime conversation. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. @@ -1972,7 +1972,7 @@ - `"in_progress"` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` A user message item in a Realtime conversation. @@ -2048,7 +2048,7 @@ - `"in_progress"` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` An assistant message item in a Realtime conversation. @@ -2108,7 +2108,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` A function call item in a Realtime conversation. @@ -2150,7 +2150,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` A function call output item in a Realtime conversation. @@ -2188,7 +2188,7 @@ - `"in_progress"` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` A Realtime item responding to an MCP approval request. @@ -2214,7 +2214,7 @@ Optional reason for the decision. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `RealtimeMcpListTools object { server_label, tools, type, id }` A Realtime item listing tools available on an MCP server. @@ -2252,7 +2252,7 @@ The unique ID of the list. - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` A Realtime item representing an invocation of a tool on an MCP server. @@ -2286,7 +2286,7 @@ The error from the tool call, if any. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `RealtimeMcpProtocolError object { code, message, type }` - `code: number` @@ -2296,7 +2296,7 @@ - `"protocol_error"` - - `RealtimeMcpToolExecutionError = object { message, type }` + - `RealtimeMcpToolExecutionError object { message, type }` - `message: string` @@ -2304,7 +2304,7 @@ - `"tool_execution_error"` - - `RealtimeMcphttpError = object { code, message, type }` + - `RealtimeMcphttpError object { code, message, type }` - `code: number` @@ -2318,7 +2318,7 @@ The output from the tool call. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` A Realtime item requesting human approval of a tool invocation. @@ -2357,7 +2357,7 @@ ### Conversation Item Input Audio Transcription Completed Event -- `ConversationItemInputAudioTranscriptionCompletedEvent = object { content_index, event_id, item_id, 4 more }` +- `ConversationItemInputAudioTranscriptionCompletedEvent object { content_index, event_id, item_id, 4 more }` This event is the output of audio transcription for user audio written to the user audio buffer. Transcription begins when the input audio buffer is @@ -2397,7 +2397,7 @@ Usage statistics for the transcription, this is billed according to the ASR model's pricing rather than the realtime model's pricing. - - `TokenUsage = object { input_tokens, output_tokens, total_tokens, 2 more }` + - `TokenUsage object { input_tokens, output_tokens, total_tokens, 2 more }` Usage statistics for models billed by token usage. @@ -2431,7 +2431,7 @@ Number of text tokens billed for this request. - - `DurationUsage = object { seconds, type }` + - `DurationUsage object { seconds, type }` Usage statistics for models billed by audio input duration. @@ -2463,7 +2463,7 @@ ### Conversation Item Input Audio Transcription Delta Event -- `ConversationItemInputAudioTranscriptionDeltaEvent = object { event_id, item_id, type, 3 more }` +- `ConversationItemInputAudioTranscriptionDeltaEvent object { event_id, item_id, type, 3 more }` Returned when the text value of an input audio transcription content part is updated with incremental transcription results. @@ -2507,7 +2507,7 @@ ### Conversation Item Input Audio Transcription Failed Event -- `ConversationItemInputAudioTranscriptionFailedEvent = object { content_index, error, event_id, 2 more }` +- `ConversationItemInputAudioTranscriptionFailedEvent object { content_index, error, event_id, 2 more }` Returned when input audio transcription is configured, and a transcription request for a user message failed. These events are separate from other @@ -2554,7 +2554,7 @@ ### Conversation Item Input Audio Transcription Segment -- `ConversationItemInputAudioTranscriptionSegment = object { id, content_index, end, 6 more }` +- `ConversationItemInputAudioTranscriptionSegment object { id, content_index, end, 6 more }` Returned when an input audio transcription segment is identified for an item. @@ -2598,7 +2598,7 @@ ### Conversation Item Retrieve Event -- `ConversationItemRetrieveEvent = object { item_id, type, event_id }` +- `ConversationItemRetrieveEvent object { item_id, type, event_id }` Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD. The server will respond with a `conversation.item.retrieved` event, @@ -2621,7 +2621,7 @@ ### Conversation Item Truncate Event -- `ConversationItemTruncateEvent = object { audio_end_ms, content_index, item_id, 2 more }` +- `ConversationItemTruncateEvent object { audio_end_ms, content_index, item_id, 2 more }` Send this event to truncate a previous assistant message’s audio. The server will produce audio faster than realtime, so this event is useful when the user @@ -2662,7 +2662,7 @@ ### Conversation Item Truncated Event -- `ConversationItemTruncatedEvent = object { audio_end_ms, content_index, event_id, 2 more }` +- `ConversationItemTruncatedEvent object { audio_end_ms, content_index, event_id, 2 more }` Returned when an earlier assistant audio message item is truncated by the client with a `conversation.item.truncate` event. This event is used to @@ -2695,7 +2695,7 @@ ### Conversation Item With Reference -- `ConversationItemWithReference = object { id, arguments, call_id, 7 more }` +- `ConversationItemWithReference object { id, arguments, call_id, 7 more }` The item to add to the conversation. @@ -2807,7 +2807,7 @@ ### Input Audio Buffer Append Event -- `InputAudioBufferAppendEvent = object { audio, type, event_id }` +- `InputAudioBufferAppendEvent object { audio, type, event_id }` Send this event to append audio bytes to the input audio buffer. The audio buffer is temporary storage you can write to and later commit. A "commit" will create a new @@ -2840,7 +2840,7 @@ ### Input Audio Buffer Clear Event -- `InputAudioBufferClearEvent = object { type, event_id }` +- `InputAudioBufferClearEvent object { type, event_id }` Send this event to clear the audio bytes in the buffer. The server will respond with an `input_audio_buffer.cleared` event. @@ -2857,7 +2857,7 @@ ### Input Audio Buffer Cleared Event -- `InputAudioBufferClearedEvent = object { event_id, type }` +- `InputAudioBufferClearedEvent object { event_id, type }` Returned when the input audio buffer is cleared by the client with a `input_audio_buffer.clear` event. @@ -2874,7 +2874,7 @@ ### Input Audio Buffer Commit Event -- `InputAudioBufferCommitEvent = object { type, event_id }` +- `InputAudioBufferCommitEvent object { type, event_id }` Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically. @@ -2892,7 +2892,7 @@ ### Input Audio Buffer Committed Event -- `InputAudioBufferCommittedEvent = object { event_id, item_id, type, previous_item_id }` +- `InputAudioBufferCommittedEvent object { event_id, item_id, type, previous_item_id }` Returned when an input audio buffer is committed, either by the client or automatically in server VAD mode. The `item_id` property is the ID of the user @@ -2920,7 +2920,7 @@ ### Input Audio Buffer Dtmf Event Received Event -- `InputAudioBufferDtmfEventReceivedEvent = object { event, received_at, type }` +- `InputAudioBufferDtmfEventReceivedEvent object { event, received_at, type }` **SIP Only:** Returned when an DTMF event is received. A DTMF event is a message that represents a telephone keypad press (0–9, *, #, A–D). The `event` property @@ -2943,7 +2943,7 @@ ### Input Audio Buffer Speech Started Event -- `InputAudioBufferSpeechStartedEvent = object { audio_start_ms, event_id, item_id, type }` +- `InputAudioBufferSpeechStartedEvent object { audio_start_ms, event_id, item_id, type }` Sent by the server when in `server_vad` mode to indicate that speech has been detected in the audio buffer. This can happen any time audio is added to the @@ -2979,7 +2979,7 @@ ### Input Audio Buffer Speech Stopped Event -- `InputAudioBufferSpeechStoppedEvent = object { audio_end_ms, event_id, item_id, type }` +- `InputAudioBufferSpeechStoppedEvent object { audio_end_ms, event_id, item_id, type }` Returned in `server_vad` mode when the server detects the end of speech in the audio buffer. The server will also send an `conversation.item.created` @@ -3007,7 +3007,7 @@ ### Input Audio Buffer Timeout Triggered -- `InputAudioBufferTimeoutTriggered = object { audio_end_ms, audio_start_ms, event_id, 2 more }` +- `InputAudioBufferTimeoutTriggered object { audio_end_ms, audio_start_ms, event_id, 2 more }` Returned when the Server VAD timeout is triggered for the input audio buffer. This is configured with `idle_timeout_ms` in the `turn_detection` settings of the session, and it indicates that @@ -3047,7 +3047,7 @@ ### Log Prob Properties -- `LogProbProperties = object { token, bytes, logprob }` +- `LogProbProperties object { token, bytes, logprob }` A log probability object. @@ -3065,7 +3065,7 @@ ### Mcp List Tools Completed -- `McpListToolsCompleted = object { event_id, item_id, type }` +- `McpListToolsCompleted object { event_id, item_id, type }` Returned when listing MCP tools has completed for an item. @@ -3085,7 +3085,7 @@ ### Mcp List Tools Failed -- `McpListToolsFailed = object { event_id, item_id, type }` +- `McpListToolsFailed object { event_id, item_id, type }` Returned when listing MCP tools has failed for an item. @@ -3105,7 +3105,7 @@ ### Mcp List Tools In Progress -- `McpListToolsInProgress = object { event_id, item_id, type }` +- `McpListToolsInProgress object { event_id, item_id, type }` Returned when listing MCP tools is in progress for an item. @@ -3135,7 +3135,7 @@ ### Output Audio Buffer Clear Event -- `OutputAudioBufferClearEvent = object { type, event_id }` +- `OutputAudioBufferClearEvent object { type, event_id }` **WebRTC/SIP Only:** Emit to cut off the current audio response. This will trigger the server to stop generating audio and emit a `output_audio_buffer.cleared` event. This @@ -3155,7 +3155,7 @@ ### Rate Limits Updated Event -- `RateLimitsUpdatedEvent = object { event_id, rate_limits, type }` +- `RateLimitsUpdatedEvent object { event_id, rate_limits, type }` Emitted at the beginning of a Response to indicate the updated rate limits. When a Response is created some tokens will be "reserved" for the output @@ -3198,7 +3198,7 @@ ### Realtime Audio Config -- `RealtimeAudioConfig = object { input, output }` +- `RealtimeAudioConfig object { input, output }` Configuration for input and output audio. @@ -3208,7 +3208,7 @@ The format of the input audio. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -3224,7 +3224,7 @@ - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -3234,7 +3234,7 @@ - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -3303,7 +3303,7 @@ Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. @@ -3357,7 +3357,7 @@ higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` Server-side semantic turn detection which uses a model to determine when the user has finished speaking. @@ -3394,42 +3394,6 @@ The format of the output audio. - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - `speed: optional number` The speed of the model's spoken response as a multiple of the original speed. @@ -3471,7 +3435,7 @@ - `"cedar"` - - `ID = object { id }` + - `ID object { id }` Custom voice reference. @@ -3481,13 +3445,13 @@ ### Realtime Audio Config Input -- `RealtimeAudioConfigInput = object { format, noise_reduction, transcription, turn_detection }` +- `RealtimeAudioConfigInput object { format, noise_reduction, transcription, turn_detection }` - `format: optional RealtimeAudioFormats` The format of the input audio. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -3503,7 +3467,7 @@ - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -3513,7 +3477,7 @@ - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -3582,7 +3546,7 @@ Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. @@ -3636,7 +3600,7 @@ higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` Server-side semantic turn detection which uses a model to determine when the user has finished speaking. @@ -3669,13 +3633,13 @@ ### Realtime Audio Config Output -- `RealtimeAudioConfigOutput = object { format, speed, voice }` +- `RealtimeAudioConfigOutput object { format, speed, voice }` - `format: optional RealtimeAudioFormats` The format of the output audio. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -3691,7 +3655,7 @@ - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -3701,7 +3665,7 @@ - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -3752,7 +3716,7 @@ - `"cedar"` - - `ID = object { id }` + - `ID object { id }` Custom voice reference. @@ -3766,7 +3730,7 @@ The PCM audio format. Only a 24kHz sample rate is supported. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -3782,7 +3746,7 @@ - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -3792,7 +3756,7 @@ - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -3812,7 +3776,7 @@ Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. @@ -3866,7 +3830,7 @@ higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` Server-side semantic turn detection which uses a model to determine when the user has finished speaking. @@ -3903,7 +3867,7 @@ A realtime client event. - - `ConversationItemCreateEvent = object { item, type, event_id, previous_item_id }` + - `ConversationItemCreateEvent object { item, type, event_id, previous_item_id }` Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a @@ -3917,7 +3881,7 @@ A single item within a Realtime conversation. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. @@ -3967,7 +3931,7 @@ - `"in_progress"` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` A user message item in a Realtime conversation. @@ -4043,7 +4007,7 @@ - `"in_progress"` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` An assistant message item in a Realtime conversation. @@ -4103,7 +4067,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` A function call item in a Realtime conversation. @@ -4145,7 +4109,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` A function call output item in a Realtime conversation. @@ -4183,7 +4147,7 @@ - `"in_progress"` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` A Realtime item responding to an MCP approval request. @@ -4209,7 +4173,7 @@ Optional reason for the decision. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `RealtimeMcpListTools object { server_label, tools, type, id }` A Realtime item listing tools available on an MCP server. @@ -4247,7 +4211,7 @@ The unique ID of the list. - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` A Realtime item representing an invocation of a tool on an MCP server. @@ -4281,7 +4245,7 @@ The error from the tool call, if any. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `RealtimeMcpProtocolError object { code, message, type }` - `code: number` @@ -4291,7 +4255,7 @@ - `"protocol_error"` - - `RealtimeMcpToolExecutionError = object { message, type }` + - `RealtimeMcpToolExecutionError object { message, type }` - `message: string` @@ -4299,7 +4263,7 @@ - `"tool_execution_error"` - - `RealtimeMcphttpError = object { code, message, type }` + - `RealtimeMcphttpError object { code, message, type }` - `code: number` @@ -4313,7 +4277,7 @@ The output from the tool call. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` A Realtime item requesting human approval of a tool invocation. @@ -4357,7 +4321,7 @@ If set to an existing ID, it allows an item to be inserted mid-conversation. If the ID cannot be found, an error will be returned and the item will not be added. - - `ConversationItemDeleteEvent = object { item_id, type, event_id }` + - `ConversationItemDeleteEvent object { item_id, type, event_id }` Send this event when you want to remove any item from the conversation history. The server will respond with a `conversation.item.deleted` event, @@ -4378,7 +4342,7 @@ Optional client-generated ID used to identify this event. - - `ConversationItemRetrieveEvent = object { item_id, type, event_id }` + - `ConversationItemRetrieveEvent object { item_id, type, event_id }` Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD. The server will respond with a `conversation.item.retrieved` event, @@ -4399,7 +4363,7 @@ Optional client-generated ID used to identify this event. - - `ConversationItemTruncateEvent = object { audio_end_ms, content_index, item_id, 2 more }` + - `ConversationItemTruncateEvent object { audio_end_ms, content_index, item_id, 2 more }` Send this event to truncate a previous assistant message’s audio. The server will produce audio faster than realtime, so this event is useful when the user @@ -4438,7 +4402,7 @@ Optional client-generated ID used to identify this event. - - `InputAudioBufferAppendEvent = object { audio, type, event_id }` + - `InputAudioBufferAppendEvent object { audio, type, event_id }` Send this event to append audio bytes to the input audio buffer. The audio buffer is temporary storage you can write to and later commit. A "commit" will create a new @@ -4469,7 +4433,7 @@ Optional client-generated ID used to identify this event. - - `InputAudioBufferClearEvent = object { type, event_id }` + - `InputAudioBufferClearEvent object { type, event_id }` Send this event to clear the audio bytes in the buffer. The server will respond with an `input_audio_buffer.cleared` event. @@ -4484,7 +4448,7 @@ Optional client-generated ID used to identify this event. - - `OutputAudioBufferClearEvent = object { type, event_id }` + - `OutputAudioBufferClearEvent object { type, event_id }` **WebRTC/SIP Only:** Emit to cut off the current audio response. This will trigger the server to stop generating audio and emit a `output_audio_buffer.cleared` event. This @@ -4502,7 +4466,7 @@ The unique ID of the client event used for error handling. - - `InputAudioBufferCommitEvent = object { type, event_id }` + - `InputAudioBufferCommitEvent object { type, event_id }` Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically. @@ -4518,7 +4482,7 @@ Optional client-generated ID used to identify this event. - - `ResponseCancelEvent = object { type, event_id, response_id }` + - `ResponseCancelEvent object { type, event_id, response_id }` Send this event to cancel an in-progress response. The server will respond with a `response.done` event with a status of `response.status=cancelled`. If @@ -4541,7 +4505,7 @@ A specific response ID to cancel - if not provided, will cancel an in-progress response in the default conversation. - - `ResponseCreateEvent = object { type, event_id, response }` + - `ResponseCreateEvent object { type, event_id, response }` This event instructs the server to create a Response, which means triggering model inference. When in Server VAD mode, the server will create Responses @@ -4593,7 +4557,7 @@ The format of the output audio. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -4609,7 +4573,7 @@ - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -4619,7 +4583,7 @@ - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -4662,7 +4626,7 @@ - `"cedar"` - - `ID = object { id }` + - `ID object { id }` Custom voice reference. @@ -4700,645 +4664,751 @@ Note that this can include references to items that previously appeared in the session using their id. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. - - `content: array of object { text, type }` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` - The content of the message. + A user message item in a Realtime conversation. - - `text: optional string` + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` - The text content. + An assistant message item in a Realtime conversation. - - `type: optional "input_text"` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` - The content type. Always `input_text` for system messages. + A function call item in a Realtime conversation. - - `"input_text"` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` - - `role: "system"` + A function call output item in a Realtime conversation. - The role of the message sender. Always `system`. + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` - - `"system"` + A Realtime item responding to an MCP approval request. - - `type: "message"` + - `RealtimeMcpListTools object { server_label, tools, type, id }` - The type of the item. Always `message`. + A Realtime item listing tools available on an MCP server. - - `"message"` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` - - `id: optional string` + A Realtime item representing an invocation of a tool on an MCP server. - The unique ID of the item. This may be provided by the client or generated by the server. + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` - - `object: optional "realtime.item"` + A Realtime item requesting human approval of a tool invocation. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `instructions: optional string` - - `"realtime.item"` + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `max_output_tokens: optional number or "inf"` - The status of the item. Has no effect on the conversation. + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - - `"completed"` + - `number` - - `"incomplete"` + - `"inf"` - - `"in_progress"` + - `"inf"` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `metadata: optional Metadata` - A user message item in a Realtime conversation. + Set of 16 key-value pairs that can be attached to an object. This can be + useful for storing additional information about the object in a structured + format, and querying for objects via API or the dashboard. - - `content: array of object { audio, detail, image_url, 3 more }` + Keys are strings with a maximum length of 64 characters. Values are strings + with a maximum length of 512 characters. - The content of the message. + - `output_modalities: optional array of "text" or "audio"` - - `audio: optional string` + The set of modalities the model used to respond, currently the only possible values are + `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the + output to mode `text` will disable audio output from the model. - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `"text"` - - `detail: optional "auto" or "low" or "high"` + - `"audio"` - The detail level of the image (for `input_image`). `auto` will default to `high`. + - `prompt: optional ResponsePrompt` - - `"auto"` + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - - `"low"` + - `id: string` - - `"high"` + The unique identifier of the prompt template to use. - - `image_url: optional string` + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - - `text: optional string` + - `string` - The text content (for `input_text`). + - `ResponseInputText object { text, type }` - - `transcript: optional string` + A text input to the model. - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + - `text: string` - - `type: optional "input_text" or "input_audio" or "input_image"` + The text input to the model. - The content type (`input_text`, `input_audio`, or `input_image`). + - `type: "input_text"` + + The type of the input item. Always `input_text`. - `"input_text"` - - `"input_audio"` + - `ResponseInputImage object { detail, type, file_id, image_url }` - - `"input_image"` + An image input to the model. Learn about [image inputs](/docs/guides/vision). - - `role: "user"` + - `detail: "low" or "high" or "auto" or "original"` - The role of the message sender. Always `user`. + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - `"user"` + - `"low"` - - `type: "message"` + - `"high"` - The type of the item. Always `message`. + - `"auto"` - - `"message"` + - `"original"` - - `id: optional string` + - `type: "input_image"` - The unique ID of the item. This may be provided by the client or generated by the server. + The type of the input item. Always `input_image`. - - `object: optional "realtime.item"` + - `"input_image"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `file_id: optional string` - - `"realtime.item"` + The ID of the file to be sent to the model. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `image_url: optional string` - The status of the item. Has no effect on the conversation. + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - `"completed"` + - `ResponseInputFile object { type, detail, file_data, 3 more }` - - `"incomplete"` + A file input to the model. - - `"in_progress"` + - `type: "input_file"` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + The type of the input item. Always `input_file`. - An assistant message item in a Realtime conversation. + - `"input_file"` - - `content: array of object { audio, text, transcript, type }` + - `detail: optional "low" or "high"` - The content of the message. + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - `audio: optional string` + - `"low"` - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `"high"` - - `text: optional string` + - `file_data: optional string` - The text content. + The content of the file to be sent to the model. - - `transcript: optional string` + - `file_id: optional string` - The transcript of the audio content, this will always be present if the output type is `audio`. + The ID of the file to be sent to the model. - - `type: optional "output_text" or "output_audio"` + - `file_url: optional string` - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + The URL of the file to be sent to the model. - - `"output_text"` + - `filename: optional string` - - `"output_audio"` + The name of the file to be sent to the model. - - `role: "assistant"` + - `version: optional string` - The role of the message sender. Always `assistant`. + Optional version of the prompt template. - - `"assistant"` + - `tool_choice: optional ToolChoiceOptions or ToolChoiceFunction or ToolChoiceMcp` - - `type: "message"` + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. - The type of the item. Always `message`. + - `ToolChoiceOptions = "none" or "auto" or "required"` - - `"message"` + Controls which (if any) tool is called by the model. - - `id: optional string` + `none` means the model will not call any tool and instead generates a message. - The unique ID of the item. This may be provided by the client or generated by the server. + `auto` means the model can pick between generating a message or calling one or + more tools. - - `object: optional "realtime.item"` + `required` means the model must call one or more tools. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"none"` - - `"realtime.item"` + - `"auto"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"required"` - The status of the item. Has no effect on the conversation. + - `ToolChoiceFunction object { name, type }` - - `"completed"` + Use this option to force the model to call a specific function. - - `"incomplete"` + - `name: string` - - `"in_progress"` + The name of the function to call. - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `type: "function"` - A function call item in a Realtime conversation. + For function calling, the type is always `function`. - - `arguments: string` + - `"function"` - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + - `ToolChoiceMcp object { server_label, type, name }` - - `name: string` + Use this option to force the model to call a specific tool on a remote MCP server. - The name of the function being called. + - `server_label: string` - - `type: "function_call"` + The label of the MCP server to use. - The type of the item. Always `function_call`. + - `type: "mcp"` - - `"function_call"` + For MCP tools, the type is always `mcp`. - - `id: optional string` + - `"mcp"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `name: optional string` - - `call_id: optional string` + The name of the tool to call on the server. - The ID of the function call. + - `tools: optional array of RealtimeFunctionTool or object { server_label, type, allowed_tools, 7 more }` - - `object: optional "realtime.item"` + Tools available to the model. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `RealtimeFunctionTool object { description, name, parameters, type }` - - `"realtime.item"` + - `description: optional string` - - `status: optional "completed" or "incomplete" or "in_progress"` + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - The status of the item. Has no effect on the conversation. + - `name: optional string` - - `"completed"` + The name of the function. - - `"incomplete"` + - `parameters: optional unknown` - - `"in_progress"` + Parameters of the function in JSON Schema. - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `type: optional "function"` - A function call output item in a Realtime conversation. + The type of the tool, i.e. `function`. - - `call_id: string` + - `"function"` - The ID of the function call this output is for. + - `McpTool object { server_label, type, allowed_tools, 7 more }` - - `output: string` + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - The output of the function call, this is free text and can contain any information or simply be empty. + - `server_label: string` - - `type: "function_call_output"` + A label for this MCP server, used to identify it in tool calls. - The type of the item. Always `function_call_output`. + - `type: "mcp"` - - `"function_call_output"` + The type of the MCP tool. Always `mcp`. - - `id: optional string` + - `"mcp"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `allowed_tools: optional array of string or object { read_only, tool_names }` - - `object: optional "realtime.item"` + List of allowed tool names or a filter object. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `McpAllowedTools = array of string` - - `"realtime.item"` + A string array of allowed tool names - - `status: optional "completed" or "incomplete" or "in_progress"` + - `McpToolFilter object { read_only, tool_names }` - The status of the item. Has no effect on the conversation. + A filter object to specify which tools are allowed. - - `"completed"` + - `read_only: optional boolean` - - `"incomplete"` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `"in_progress"` + - `tool_names: optional array of string` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + List of allowed tool names. - A Realtime item responding to an MCP approval request. + - `authorization: optional string` - - `id: string` + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. - The unique ID of the approval response. + - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - - `approval_request_id: string` + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](/docs/guides/tools-remote-mcp#connectors). - The ID of the approval request being answered. + Currently supported `connector_id` values are: - - `approve: boolean` + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` - Whether the request was approved. + - `"connector_dropbox"` - - `type: "mcp_approval_response"` + - `"connector_gmail"` - The type of the item. Always `mcp_approval_response`. + - `"connector_googlecalendar"` - - `"mcp_approval_response"` + - `"connector_googledrive"` - - `reason: optional string` + - `"connector_microsoftteams"` - Optional reason for the decision. + - `"connector_outlookcalendar"` - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `"connector_outlookemail"` - A Realtime item listing tools available on an MCP server. + - `"connector_sharepoint"` - - `server_label: string` + - `defer_loading: optional boolean` - The label of the MCP server. + Whether this MCP tool is deferred and discovered via tool search. - - `tools: array of object { input_schema, name, annotations, description }` + - `headers: optional map[string]` - The tools available on the server. + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. - - `input_schema: unknown` + - `require_approval: optional object { always, never } or "always" or "never"` - The JSON schema describing the tool's input. + Specify which of the MCP server's tools require approval. - - `name: string` + - `McpToolApprovalFilter object { always, never }` - The name of the tool. + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. - - `annotations: optional unknown` + - `always: optional object { read_only, tool_names }` - Additional annotations about the tool. + A filter object to specify which tools are allowed. - - `description: optional string` + - `read_only: optional boolean` - The description of the tool. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `type: "mcp_list_tools"` + - `tool_names: optional array of string` - The type of the item. Always `mcp_list_tools`. + List of allowed tool names. - - `"mcp_list_tools"` + - `never: optional object { read_only, tool_names }` - - `id: optional string` + A filter object to specify which tools are allowed. - The unique ID of the list. + - `read_only: optional boolean` - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - A Realtime item representing an invocation of a tool on an MCP server. + - `tool_names: optional array of string` - - `id: string` + List of allowed tool names. - The unique ID of the tool call. + - `McpToolApprovalSetting = "always" or "never"` - - `arguments: string` + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. - A JSON string of the arguments passed to the tool. + - `"always"` - - `name: string` + - `"never"` - The name of the tool that was run. + - `server_description: optional string` - - `server_label: string` + Optional description of the MCP server, used to provide more context. - The label of the MCP server running the tool. + - `server_url: optional string` - - `type: "mcp_call"` + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. - The type of the item. Always `mcp_call`. + - `SessionUpdateEvent object { session, type, event_id }` - - `"mcp_call"` + Send this event to update the session’s configuration. + The client may send this event at any time to update any field + except for `voice` and `model`. `voice` can be updated only if there have been no other audio outputs yet. - - `approval_request_id: optional string` + When the server receives a `session.update`, it will respond + with a `session.updated` event showing the full, effective configuration. + Only the fields that are present in the `session.update` are updated. To clear a field like + `instructions`, pass an empty string. To clear a field like `tools`, pass an empty array. + To clear a field like `turn_detection`, pass `null`. - The ID of an associated approval request, if any. + - `session: RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` + Update the Realtime session. Choose either a realtime + session or a transcription session. - The error from the tool call, if any. + - `RealtimeSessionCreateRequest object { type, audio, include, 9 more }` - - `RealtimeMcpProtocolError = object { code, message, type }` + Realtime session object configuration. - - `code: number` + - `type: "realtime"` - - `message: string` + The type of session to create. Always `realtime` for the Realtime API. - - `type: "protocol_error"` + - `"realtime"` - - `"protocol_error"` + - `audio: optional RealtimeAudioConfig` - - `RealtimeMcpToolExecutionError = object { message, type }` + Configuration for input and output audio. - - `message: string` + - `input: optional RealtimeAudioConfigInput` - - `type: "tool_execution_error"` + - `format: optional RealtimeAudioFormats` - - `"tool_execution_error"` + The format of the input audio. - - `RealtimeMcphttpError = object { code, message, type }` + - `noise_reduction: optional object { type }` - - `code: number` + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `message: string` + - `type: optional NoiseReductionType` - - `type: "http_error"` + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - `"http_error"` + - `"near_field"` - - `output: optional string` + - `"far_field"` - The output from the tool call. + - `transcription: optional AudioTranscription` - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - A Realtime item requesting human approval of a tool invocation. + - `language: optional string` - - `id: string` + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - The unique ID of the approval request. + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `arguments: string` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - A JSON string of arguments for the tool. + - `string` - - `name: string` + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - The name of the tool to run. + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `server_label: string` + - `"whisper-1"` - The label of the MCP server making the request. + - `"gpt-4o-mini-transcribe"` - - `type: "mcp_approval_request"` + - `"gpt-4o-mini-transcribe-2025-12-15"` - The type of the item. Always `mcp_approval_request`. + - `"gpt-4o-transcribe"` - - `"mcp_approval_request"` + - `"gpt-4o-transcribe-diarize"` - - `instructions: optional string` + - `prompt: optional string` - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - `max_output_tokens: optional number or "inf"` + - `turn_detection: optional RealtimeAudioInputTurnDetection` - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - `number` + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - `"inf"` + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `"inf"` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - - `metadata: optional Metadata` + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. + - `type: "server_vad"` - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `output_modalities: optional array of "text" or "audio"` + - `"server_vad"` - The set of modalities the model used to respond, currently the only possible values are - `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the - output to mode `text` will disable audio output from the model. + - `create_response: optional boolean` - - `"text"` + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - `"audio"` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `prompt: optional ResponsePrompt` + - `idle_timeout_ms: optional number` - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - - `id: string` + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - The unique identifier of the prompt template to use. + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` + - `interrupt_response: optional boolean` - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - `string` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `ResponseInputText = object { text, type }` + - `prefix_padding_ms: optional number` - A text input to the model. + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - - `text: string` + - `silence_duration_ms: optional number` - The text input to the model. + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - - `type: "input_text"` + - `threshold: optional number` - The type of the input item. Always `input_text`. + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - - `"input_text"` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - - `ResponseInputImage = object { detail, type, file_id, image_url }` + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - An image input to the model. Learn about [image inputs](/docs/guides/vision). + - `type: "semantic_vad"` - - `detail: "low" or "high" or "auto" or "original"` + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. + - `"semantic_vad"` + + - `create_response: optional boolean` + + Whether or not to automatically generate a response when a VAD stop event occurs. + + - `eagerness: optional "low" or "medium" or "high" or "auto"` + + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - `"low"` + - `"medium"` + - `"high"` - `"auto"` - - `"original"` + - `interrupt_response: optional boolean` - - `type: "input_image"` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - The type of the input item. Always `input_image`. + - `output: optional RealtimeAudioConfigOutput` - - `"input_image"` + - `format: optional RealtimeAudioFormats` - - `file_id: optional string` + The format of the output audio. - The ID of the file to be sent to the model. + - `speed: optional number` - - `image_url: optional string` + The speed of the model's spoken response as a multiple of the original speed. + 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. + This parameter is a post-processing adjustment to the audio after it is generated, it's + also possible to prompt the model to speak faster or slower. - - `ResponseInputFile = object { type, detail, file_data, 3 more }` + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - A file input to the model. + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. - - `type: "input_file"` + - `string` - The type of the input item. Always `input_file`. + - `"alloy" or "ash" or "ballad" or 7 more` - - `"input_file"` + - `"alloy"` - - `detail: optional "low" or "high"` + - `"ash"` - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. + - `"ballad"` - - `"low"` + - `"coral"` - - `"high"` + - `"echo"` - - `file_data: optional string` + - `"sage"` - The content of the file to be sent to the model. + - `"shimmer"` - - `file_id: optional string` + - `"verse"` - The ID of the file to be sent to the model. + - `"marin"` - - `file_url: optional string` + - `"cedar"` - The URL of the file to be sent to the model. + - `ID object { id }` - - `filename: optional string` + Custom voice reference. - The name of the file to be sent to the model. + - `id: string` - - `version: optional string` + The custom voice ID, e.g. `voice_1234`. - Optional version of the prompt template. + - `include: optional array of "item.input_audio_transcription.logprobs"` - - `tool_choice: optional ToolChoiceOptions or ToolChoiceFunction or ToolChoiceMcp` + Additional fields to include in server outputs. - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - `ToolChoiceOptions = "none" or "auto" or "required"` + - `"item.input_audio_transcription.logprobs"` - Controls which (if any) tool is called by the model. + - `instructions: optional string` - `none` means the model will not call any tool and instead generates a message. + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - `auto` means the model can pick between generating a message or calling one or - more tools. + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - `required` means the model must call one or more tools. + - `max_output_tokens: optional number or "inf"` - - `"none"` + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - - `"auto"` + - `number` - - `"required"` + - `"inf"` - - `ToolChoiceFunction = object { name, type }` + - `"inf"` - Use this option to force the model to call a specific function. + - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - `name: string` + The Realtime model used for this session. - The name of the function to call. + - `string` - - `type: "function"` + - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - For function calling, the type is always `function`. + The Realtime model used for this session. - - `"function"` + - `"gpt-realtime"` - - `ToolChoiceMcp = object { server_label, type, name }` + - `"gpt-realtime-1.5"` - Use this option to force the model to call a specific tool on a remote MCP server. + - `"gpt-realtime-2025-08-28"` - - `server_label: string` + - `"gpt-4o-realtime-preview"` - The label of the MCP server to use. + - `"gpt-4o-realtime-preview-2024-10-01"` - - `type: "mcp"` + - `"gpt-4o-realtime-preview-2024-12-17"` - For MCP tools, the type is always `mcp`. + - `"gpt-4o-realtime-preview-2025-06-03"` - - `"mcp"` + - `"gpt-4o-mini-realtime-preview"` - - `name: optional string` + - `"gpt-4o-mini-realtime-preview-2024-12-17"` - The name of the tool to call on the server. + - `"gpt-realtime-mini"` - - `tools: optional array of RealtimeFunctionTool or object { server_label, type, allowed_tools, 7 more }` + - `"gpt-realtime-mini-2025-10-06"` - Tools available to the model. + - `"gpt-realtime-mini-2025-12-15"` - - `RealtimeFunctionTool = object { description, name, parameters, type }` + - `"gpt-audio-1.5"` - - `description: optional string` + - `"gpt-audio-mini"` - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). + - `"gpt-audio-mini-2025-10-06"` - - `name: optional string` + - `"gpt-audio-mini-2025-12-15"` - The name of the function. + - `output_modalities: optional array of "text" or "audio"` - - `parameters: optional unknown` + The set of modalities the model can respond with. It defaults to `["audio"]`, indicating + that the model will respond with audio plus a transcript. `["text"]` can be used to make + the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - Parameters of the function in JSON Schema. + - `"text"` - - `type: optional "function"` + - `"audio"` - The type of the tool, i.e. `function`. + - `prompt: optional ResponsePrompt` - - `"function"` + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). + + - `tool_choice: optional RealtimeToolChoiceConfig` + + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. + + - `ToolChoiceOptions = "none" or "auto" or "required"` + + Controls which (if any) tool is called by the model. + + `none` means the model will not call any tool and instead generates a message. + + `auto` means the model can pick between generating a message or calling one or + more tools. + + `required` means the model must call one or more tools. + + - `ToolChoiceFunction object { name, type }` + + Use this option to force the model to call a specific function. + + - `ToolChoiceMcp object { server_label, type, name }` + + Use this option to force the model to call a specific tool on a remote MCP server. + + - `tools: optional RealtimeToolsConfig` - - `McpTool = object { server_label, type, allowed_tools, 7 more }` + Tools available to the model. + + - `RealtimeFunctionTool object { description, name, parameters, type }` + + - `McpTool object { server_label, type, allowed_tools, 7 more }` Give the model access to additional tools via remote Model Context Protocol (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). @@ -5361,7 +5431,7 @@ A string array of allowed tool names - - `McpToolFilter = object { read_only, tool_names }` + - `McpToolFilter object { read_only, tool_names }` A filter object to specify which tools are allowed. @@ -5427,7 +5497,7 @@ Specify which of the MCP server's tools require approval. - - `McpToolApprovalFilter = object { always, never }` + - `McpToolApprovalFilter object { always, never }` Specify which of the MCP server's tools require approval. Can be `always`, `never`, or a filter object associated with tools @@ -5480,131 +5550,114 @@ The URL for the MCP server. One of `server_url` or `connector_id` must be provided. - - `SessionUpdateEvent = object { session, type, event_id }` + - `tracing: optional RealtimeTracingConfig` - Send this event to update the session’s configuration. - The client may send this event at any time to update any field - except for `voice` and `model`. `voice` can be updated only if there have been no other audio outputs yet. + Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once + tracing is enabled for a session, the configuration cannot be modified. - When the server receives a `session.update`, it will respond - with a `session.updated` event showing the full, effective configuration. - Only the fields that are present in the `session.update` are updated. To clear a field like - `instructions`, pass an empty string. To clear a field like `tools`, pass an empty array. - To clear a field like `turn_detection`, pass `null`. + `auto` will create a trace for the session with default values for the + workflow name, group id, and metadata. - - `session: RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` + - `Auto = "auto"` - Update the Realtime session. Choose either a realtime - session or a transcription session. + Enables tracing and sets default values for tracing configuration options. Always `auto`. - - `RealtimeSessionCreateRequest = object { type, audio, include, 9 more }` + - `"auto"` - Realtime session object configuration. + - `TracingConfiguration object { group_id, metadata, workflow_name }` - - `type: "realtime"` + Granular configuration for tracing. - The type of session to create. Always `realtime` for the Realtime API. + - `group_id: optional string` - - `"realtime"` + The group id to attach to this trace to enable filtering and + grouping in the Traces Dashboard. - - `audio: optional RealtimeAudioConfig` + - `metadata: optional unknown` - Configuration for input and output audio. - - - `input: optional RealtimeAudioConfigInput` - - - `format: optional RealtimeAudioFormats` - - The format of the input audio. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. + The arbitrary metadata to attach to this trace to enable + filtering in the Traces Dashboard. - - `rate: optional 24000` + - `workflow_name: optional string` - The sample rate of the audio. Always `24000`. + The name of the workflow to attach to this trace. This is used to + name the trace in the Traces Dashboard. - - `24000` + - `truncation: optional RealtimeTruncation` - - `type: optional "audio/pcm"` + When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - The audio format. Always `audio/pcm`. + Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - - `"audio/pcm"` + Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - `PCMUAudioFormat = object { type }` + Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - The G.711 μ-law format. + - `"auto" or "disabled"` - - `type: optional "audio/pcmu"` + The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - The audio format. Always `audio/pcmu`. + - `"auto"` - - `"audio/pcmu"` + - `"disabled"` - - `PCMAAudioFormat = object { type }` + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` - The G.711 A-law format. + Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - - `type: optional "audio/pcma"` + - `retention_ratio: number` - The audio format. Always `audio/pcma`. + Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - `"audio/pcma"` + - `type: "retention_ratio"` - - `noise_reduction: optional object { type }` + Use retention ratio truncation. - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + - `"retention_ratio"` - - `type: optional NoiseReductionType` + - `token_limits: optional object { post_instructions }` - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - - `"near_field"` + - `post_instructions: optional number` - - `"far_field"` + Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - - `transcription: optional AudioTranscription` + - `RealtimeTranscriptionSessionCreateRequest object { type, audio, include }` - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + Realtime transcription session object configuration. - - `language: optional string` + - `type: "transcription"` - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. + The type of session to create. Always `transcription` for transcription sessions. - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `"transcription"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `audio: optional RealtimeTranscriptionSessionAudio` - - `string` + Configuration for input and output audio. - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `input: optional RealtimeTranscriptionSessionAudioInput` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `format: optional RealtimeAudioFormats` - - `"whisper-1"` + The PCM audio format. Only a 24kHz sample rate is supported. - - `"gpt-4o-mini-transcribe"` + - `noise_reduction: optional object { type }` - - `"gpt-4o-mini-transcribe-2025-12-15"` + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `"gpt-4o-transcribe"` + - `type: optional NoiseReductionType` - - `"gpt-4o-transcribe-diarize"` + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - `prompt: optional string` + - `transcription: optional AudioTranscription` - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - `turn_detection: optional RealtimeAudioInputTurnDetection` + - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. @@ -5612,7 +5665,7 @@ Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. @@ -5666,7 +5719,7 @@ higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` Server-side semantic turn detection which uses a model to determine when the user has finished speaking. @@ -5697,591 +5750,611 @@ Whether or not to automatically interrupt any ongoing response with output to the default conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - `output: optional RealtimeAudioConfigOutput` + - `include: optional array of "item.input_audio_transcription.logprobs"` - - `format: optional RealtimeAudioFormats` + Additional fields to include in server outputs. - The format of the output audio. + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - `PCMAudioFormat = object { rate, type }` + - `"item.input_audio_transcription.logprobs"` - The PCM audio format. Only a 24kHz sample rate is supported. + - `type: "session.update"` - - `rate: optional 24000` + The event type, must be `session.update`. - The sample rate of the audio. Always `24000`. + - `"session.update"` - - `24000` + - `event_id: optional string` - - `type: optional "audio/pcm"` + Optional client-generated ID used to identify this event. This is an arbitrary string that a client may assign. It will be passed back if there is an error with the event, but the corresponding `session.updated` event will not include it. - The audio format. Always `audio/pcm`. +### Realtime Conversation Item Assistant Message - - `"audio/pcm"` +- `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` - - `PCMUAudioFormat = object { type }` + An assistant message item in a Realtime conversation. - The G.711 μ-law format. + - `content: array of object { audio, text, transcript, type }` - - `type: optional "audio/pcmu"` + The content of the message. - The audio format. Always `audio/pcmu`. + - `audio: optional string` - - `"audio/pcmu"` + Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. - - `PCMAAudioFormat = object { type }` + - `text: optional string` - The G.711 A-law format. + The text content. - - `type: optional "audio/pcma"` + - `transcript: optional string` - The audio format. Always `audio/pcma`. + The transcript of the audio content, this will always be present if the output type is `audio`. - - `"audio/pcma"` + - `type: optional "output_text" or "output_audio"` - - `speed: optional number` + The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. - The speed of the model's spoken response as a multiple of the original speed. - 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. + - `"output_text"` - This parameter is a post-processing adjustment to the audio after it is generated, it's - also possible to prompt the model to speak faster or slower. + - `"output_audio"` - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` + - `role: "assistant"` - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with - an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed - during the session once the model has responded with audio at least once. - We recommend `marin` and `cedar` for best quality. + The role of the message sender. Always `assistant`. - - `string` + - `"assistant"` - - `"alloy" or "ash" or "ballad" or 7 more` + - `type: "message"` - - `"alloy"` + The type of the item. Always `message`. - - `"ash"` + - `"message"` - - `"ballad"` + - `id: optional string` - - `"coral"` + The unique ID of the item. This may be provided by the client or generated by the server. - - `"echo"` + - `object: optional "realtime.item"` - - `"sage"` + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - `"shimmer"` + - `"realtime.item"` - - `"verse"` + - `status: optional "completed" or "incomplete" or "in_progress"` - - `"marin"` + The status of the item. Has no effect on the conversation. - - `"cedar"` + - `"completed"` - - `ID = object { id }` + - `"incomplete"` - Custom voice reference. + - `"in_progress"` - - `id: string` +### Realtime Conversation Item Function Call - The custom voice ID, e.g. `voice_1234`. +- `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` - - `include: optional array of "item.input_audio_transcription.logprobs"` + A function call item in a Realtime conversation. - Additional fields to include in server outputs. + - `arguments: string` - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. - - `"item.input_audio_transcription.logprobs"` + - `name: string` - - `instructions: optional string` + The name of the function being called. - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. + - `type: "function_call"` - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. + The type of the item. Always `function_call`. - - `max_output_tokens: optional number or "inf"` + - `"function_call"` - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. + - `id: optional string` - - `number` + The unique ID of the item. This may be provided by the client or generated by the server. - - `"inf"` + - `call_id: optional string` - - `"inf"` + The ID of the function call. - - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` + - `object: optional "realtime.item"` - The Realtime model used for this session. + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - `string` + - `"realtime.item"` - - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` + - `status: optional "completed" or "incomplete" or "in_progress"` - The Realtime model used for this session. + The status of the item. Has no effect on the conversation. - - `"gpt-realtime"` + - `"completed"` - - `"gpt-realtime-1.5"` + - `"incomplete"` - - `"gpt-realtime-2025-08-28"` + - `"in_progress"` - - `"gpt-4o-realtime-preview"` +### Realtime Conversation Item Function Call Output - - `"gpt-4o-realtime-preview-2024-10-01"` +- `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` - - `"gpt-4o-realtime-preview-2024-12-17"` + A function call output item in a Realtime conversation. - - `"gpt-4o-realtime-preview-2025-06-03"` + - `call_id: string` - - `"gpt-4o-mini-realtime-preview"` + The ID of the function call this output is for. - - `"gpt-4o-mini-realtime-preview-2024-12-17"` + - `output: string` - - `"gpt-realtime-mini"` + The output of the function call, this is free text and can contain any information or simply be empty. - - `"gpt-realtime-mini-2025-10-06"` + - `type: "function_call_output"` - - `"gpt-realtime-mini-2025-12-15"` + The type of the item. Always `function_call_output`. - - `"gpt-audio-1.5"` + - `"function_call_output"` - - `"gpt-audio-mini"` + - `id: optional string` - - `"gpt-audio-mini-2025-10-06"` + The unique ID of the item. This may be provided by the client or generated by the server. - - `"gpt-audio-mini-2025-12-15"` + - `object: optional "realtime.item"` - - `output_modalities: optional array of "text" or "audio"` + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - The set of modalities the model can respond with. It defaults to `["audio"]`, indicating - that the model will respond with audio plus a transcript. `["text"]` can be used to make - the model respond with text only. It is not possible to request both `text` and `audio` at the same time. + - `"realtime.item"` - - `"text"` - - - `"audio"` + - `status: optional "completed" or "incomplete" or "in_progress"` - - `prompt: optional ResponsePrompt` + The status of the item. Has no effect on the conversation. - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). + - `"completed"` - - `id: string` + - `"incomplete"` - The unique identifier of the prompt template to use. + - `"in_progress"` - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` +### Realtime Conversation Item System Message - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. +- `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` - - `string` + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. - - `ResponseInputText = object { text, type }` + - `content: array of object { text, type }` - A text input to the model. + The content of the message. - - `text: string` + - `text: optional string` - The text input to the model. + The text content. - - `type: "input_text"` + - `type: optional "input_text"` - The type of the input item. Always `input_text`. + The content type. Always `input_text` for system messages. - `"input_text"` - - `ResponseInputImage = object { detail, type, file_id, image_url }` + - `role: "system"` - An image input to the model. Learn about [image inputs](/docs/guides/vision). + The role of the message sender. Always `system`. - - `detail: "low" or "high" or "auto" or "original"` + - `"system"` - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. + - `type: "message"` - - `"low"` + The type of the item. Always `message`. - - `"high"` + - `"message"` - - `"auto"` + - `id: optional string` - - `"original"` + The unique ID of the item. This may be provided by the client or generated by the server. - - `type: "input_image"` + - `object: optional "realtime.item"` - The type of the input item. Always `input_image`. + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - `"input_image"` + - `"realtime.item"` - - `file_id: optional string` + - `status: optional "completed" or "incomplete" or "in_progress"` - The ID of the file to be sent to the model. + The status of the item. Has no effect on the conversation. - - `image_url: optional string` + - `"completed"` - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. + - `"incomplete"` - - `ResponseInputFile = object { type, detail, file_data, 3 more }` + - `"in_progress"` - A file input to the model. +### Realtime Conversation Item User Message - - `type: "input_file"` +- `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` - The type of the input item. Always `input_file`. + A user message item in a Realtime conversation. - - `"input_file"` + - `content: array of object { audio, detail, image_url, 3 more }` - - `detail: optional "low" or "high"` + The content of the message. - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. + - `audio: optional string` + + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `detail: optional "auto" or "low" or "high"` + + The detail level of the image (for `input_image`). `auto` will default to `high`. + + - `"auto"` - `"low"` - `"high"` - - `file_data: optional string` + - `image_url: optional string` - The content of the file to be sent to the model. + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. - - `file_id: optional string` + - `text: optional string` - The ID of the file to be sent to the model. + The text content (for `input_text`). - - `file_url: optional string` + - `transcript: optional string` - The URL of the file to be sent to the model. + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. - - `filename: optional string` + - `type: optional "input_text" or "input_audio" or "input_image"` - The name of the file to be sent to the model. + The content type (`input_text`, `input_audio`, or `input_image`). - - `version: optional string` + - `"input_text"` - Optional version of the prompt template. + - `"input_audio"` - - `tool_choice: optional RealtimeToolChoiceConfig` + - `"input_image"` - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. + - `role: "user"` - - `ToolChoiceOptions = "none" or "auto" or "required"` + The role of the message sender. Always `user`. - Controls which (if any) tool is called by the model. + - `"user"` - `none` means the model will not call any tool and instead generates a message. + - `type: "message"` - `auto` means the model can pick between generating a message or calling one or - more tools. + The type of the item. Always `message`. - `required` means the model must call one or more tools. + - `"message"` - - `"none"` + - `id: optional string` - - `"auto"` + The unique ID of the item. This may be provided by the client or generated by the server. - - `"required"` + - `object: optional "realtime.item"` - - `ToolChoiceFunction = object { name, type }` + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - Use this option to force the model to call a specific function. + - `"realtime.item"` - - `name: string` + - `status: optional "completed" or "incomplete" or "in_progress"` - The name of the function to call. + The status of the item. Has no effect on the conversation. - - `type: "function"` + - `"completed"` - For function calling, the type is always `function`. + - `"incomplete"` - - `"function"` + - `"in_progress"` - - `ToolChoiceMcp = object { server_label, type, name }` +### Realtime Error - Use this option to force the model to call a specific tool on a remote MCP server. +- `RealtimeError object { message, type, code, 2 more }` - - `server_label: string` + Details of the error. - The label of the MCP server to use. + - `message: string` - - `type: "mcp"` + A human-readable error message. - For MCP tools, the type is always `mcp`. + - `type: string` - - `"mcp"` + The type of error (e.g., "invalid_request_error", "server_error"). - - `name: optional string` + - `code: optional string` - The name of the tool to call on the server. + Error code, if any. - - `tools: optional RealtimeToolsConfig` + - `event_id: optional string` - Tools available to the model. + The event_id of the client event that caused the error, if applicable. - - `RealtimeFunctionTool = object { description, name, parameters, type }` + - `param: optional string` - - `description: optional string` + Parameter related to the error, if any. - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). +### Realtime Error Event - - `name: optional string` +- `RealtimeErrorEvent object { error, event_id, type }` - The name of the function. + Returned when an error occurs, which could be a client problem or a server + problem. Most errors are recoverable and the session will stay open, we + recommend to implementors to monitor and log error messages by default. - - `parameters: optional unknown` + - `error: RealtimeError` - Parameters of the function in JSON Schema. + Details of the error. - - `type: optional "function"` + - `message: string` - The type of the tool, i.e. `function`. + A human-readable error message. - - `"function"` + - `type: string` - - `McpTool = object { server_label, type, allowed_tools, 7 more }` + The type of error (e.g., "invalid_request_error", "server_error"). - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). + - `code: optional string` - - `server_label: string` + Error code, if any. - A label for this MCP server, used to identify it in tool calls. + - `event_id: optional string` - - `type: "mcp"` + The event_id of the client event that caused the error, if applicable. - The type of the MCP tool. Always `mcp`. + - `param: optional string` - - `"mcp"` + Parameter related to the error, if any. - - `allowed_tools: optional array of string or object { read_only, tool_names }` + - `event_id: string` - List of allowed tool names or a filter object. + The unique ID of the server event. - - `McpAllowedTools = array of string` + - `type: "error"` - A string array of allowed tool names + The event type, must be `error`. - - `McpToolFilter = object { read_only, tool_names }` + - `"error"` - A filter object to specify which tools are allowed. +### Realtime Function Tool - - `read_only: optional boolean` +- `RealtimeFunctionTool object { description, name, parameters, type }` - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `description: optional string` - - `tool_names: optional array of string` + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - List of allowed tool names. + - `name: optional string` - - `authorization: optional string` + The name of the function. - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. + - `parameters: optional unknown` - - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` + Parameters of the function in JSON Schema. - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](/docs/guides/tools-remote-mcp#connectors). + - `type: optional "function"` - Currently supported `connector_id` values are: + The type of the tool, i.e. `function`. - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` + - `"function"` - - `"connector_dropbox"` +### Realtime Mcp Approval Request - - `"connector_gmail"` +- `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` - - `"connector_googlecalendar"` + A Realtime item requesting human approval of a tool invocation. - - `"connector_googledrive"` + - `id: string` - - `"connector_microsoftteams"` + The unique ID of the approval request. - - `"connector_outlookcalendar"` + - `arguments: string` - - `"connector_outlookemail"` + A JSON string of arguments for the tool. - - `"connector_sharepoint"` + - `name: string` - - `defer_loading: optional boolean` + The name of the tool to run. - Whether this MCP tool is deferred and discovered via tool search. + - `server_label: string` - - `headers: optional map[string]` + The label of the MCP server making the request. - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. + - `type: "mcp_approval_request"` - - `require_approval: optional object { always, never } or "always" or "never"` + The type of the item. Always `mcp_approval_request`. - Specify which of the MCP server's tools require approval. + - `"mcp_approval_request"` - - `McpToolApprovalFilter = object { always, never }` +### Realtime Mcp Approval Response - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. +- `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` - - `always: optional object { read_only, tool_names }` + A Realtime item responding to an MCP approval request. - A filter object to specify which tools are allowed. + - `id: string` - - `read_only: optional boolean` + The unique ID of the approval response. - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `approval_request_id: string` - - `tool_names: optional array of string` + The ID of the approval request being answered. - List of allowed tool names. + - `approve: boolean` - - `never: optional object { read_only, tool_names }` + Whether the request was approved. - A filter object to specify which tools are allowed. + - `type: "mcp_approval_response"` - - `read_only: optional boolean` + The type of the item. Always `mcp_approval_response`. - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `"mcp_approval_response"` - - `tool_names: optional array of string` + - `reason: optional string` - List of allowed tool names. + Optional reason for the decision. - - `McpToolApprovalSetting = "always" or "never"` +### Realtime Mcp List Tools - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. +- `RealtimeMcpListTools object { server_label, tools, type, id }` - - `"always"` + A Realtime item listing tools available on an MCP server. - - `"never"` + - `server_label: string` - - `server_description: optional string` + The label of the MCP server. - Optional description of the MCP server, used to provide more context. + - `tools: array of object { input_schema, name, annotations, description }` - - `server_url: optional string` + The tools available on the server. - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. + - `input_schema: unknown` - - `tracing: optional RealtimeTracingConfig` + The JSON schema describing the tool's input. - Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once - tracing is enabled for a session, the configuration cannot be modified. + - `name: string` - `auto` will create a trace for the session with default values for the - workflow name, group id, and metadata. + The name of the tool. - - `Auto = "auto"` + - `annotations: optional unknown` - Enables tracing and sets default values for tracing configuration options. Always `auto`. + Additional annotations about the tool. - - `"auto"` + - `description: optional string` - - `TracingConfiguration = object { group_id, metadata, workflow_name }` + The description of the tool. - Granular configuration for tracing. + - `type: "mcp_list_tools"` - - `group_id: optional string` + The type of the item. Always `mcp_list_tools`. - The group id to attach to this trace to enable filtering and - grouping in the Traces Dashboard. + - `"mcp_list_tools"` - - `metadata: optional unknown` + - `id: optional string` - The arbitrary metadata to attach to this trace to enable - filtering in the Traces Dashboard. + The unique ID of the list. - - `workflow_name: optional string` +### Realtime Mcp Protocol Error - The name of the workflow to attach to this trace. This is used to - name the trace in the Traces Dashboard. +- `RealtimeMcpProtocolError object { code, message, type }` - - `truncation: optional RealtimeTruncation` + - `code: number` - When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. + - `message: string` - Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. + - `type: "protocol_error"` - Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. + - `"protocol_error"` - Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. +### Realtime Mcp Tool Call - - `"auto" or "disabled"` +- `RealtimeMcpToolCall object { id, arguments, name, 5 more }` - The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. + A Realtime item representing an invocation of a tool on an MCP server. - - `"auto"` + - `id: string` - - `"disabled"` + The unique ID of the tool call. - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` + - `arguments: string` - Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. + A JSON string of the arguments passed to the tool. - - `retention_ratio: number` + - `name: string` - Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. + The name of the tool that was run. - - `type: "retention_ratio"` + - `server_label: string` - Use retention ratio truncation. + The label of the MCP server running the tool. - - `"retention_ratio"` + - `type: "mcp_call"` - - `token_limits: optional object { post_instructions }` + The type of the item. Always `mcp_call`. - Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. + - `"mcp_call"` - - `post_instructions: optional number` + - `approval_request_id: optional string` - Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. + The ID of an associated approval request, if any. - - `RealtimeTranscriptionSessionCreateRequest = object { type, audio, include }` + - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` - Realtime transcription session object configuration. + The error from the tool call, if any. - - `type: "transcription"` + - `RealtimeMcpProtocolError object { code, message, type }` - The type of session to create. Always `transcription` for transcription sessions. + - `code: number` - - `"transcription"` + - `message: string` - - `audio: optional RealtimeTranscriptionSessionAudio` + - `type: "protocol_error"` - Configuration for input and output audio. + - `"protocol_error"` - - `input: optional RealtimeTranscriptionSessionAudioInput` + - `RealtimeMcpToolExecutionError object { message, type }` + + - `message: string` + + - `type: "tool_execution_error"` + + - `"tool_execution_error"` + + - `RealtimeMcphttpError object { code, message, type }` + + - `code: number` + + - `message: string` + + - `type: "http_error"` + + - `"http_error"` + + - `output: optional string` + + The output from the tool call. + +### Realtime Mcp Tool Execution Error + +- `RealtimeMcpToolExecutionError object { message, type }` + + - `message: string` + + - `type: "tool_execution_error"` + + - `"tool_execution_error"` + +### Realtime Mcphttp Error + +- `RealtimeMcphttpError object { code, message, type }` + + - `code: number` + + - `message: string` + + - `type: "http_error"` + + - `"http_error"` + +### Realtime Response + +- `RealtimeResponse object { id, audio, conversation_id, 8 more }` + + The response resource. + + - `id: optional string` + + The unique ID of the response, will look like `resp_1234`. + + - `audio: optional object { output }` + + Configuration for audio output. + + - `output: optional object { format, voice }` - `format: optional RealtimeAudioFormats` - The PCM audio format. Only a 24kHz sample rate is supported. + The format of the output audio. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -6297,7 +6370,7 @@ - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -6307,7 +6380,7 @@ - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -6317,171 +6390,210 @@ - `"audio/pcma"` - - `noise_reduction: optional object { type }` + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more` - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + The voice the model uses to respond. Voice cannot be changed during the + session once the model has responded with audio at least once. Current + voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, + `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for + best quality. - - `type: optional NoiseReductionType` + - `string` - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + - `"alloy" or "ash" or "ballad" or 7 more` - - `"near_field"` + The voice the model uses to respond. Voice cannot be changed during the + session once the model has responded with audio at least once. Current + voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, + `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for + best quality. - - `"far_field"` + - `"alloy"` - - `transcription: optional AudioTranscription` + - `"ash"` - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + - `"ballad"` - - `language: optional string` + - `"coral"` - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. + - `"echo"` - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `"sage"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `"shimmer"` - - `string` + - `"verse"` - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `"marin"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `"cedar"` - - `"whisper-1"` + - `conversation_id: optional string` - - `"gpt-4o-mini-transcribe"` + Which conversation the response is added to, determined by the `conversation` + field in the `response.create` event. If `auto`, the response will be added to + the default conversation and the value of `conversation_id` will be an id like + `conv_1234`. If `none`, the response will not be added to any conversation and + the value of `conversation_id` will be `null`. If responses are being triggered + automatically by VAD the response will be added to the default conversation - - `"gpt-4o-mini-transcribe-2025-12-15"` + - `max_output_tokens: optional number or "inf"` - - `"gpt-4o-transcribe"` + Maximum number of output tokens for a single assistant response, + inclusive of tool calls, that was used in this response. - - `"gpt-4o-transcribe-diarize"` + - `number` - - `prompt: optional string` + - `"inf"` - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + - `"inf"` - - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` + - `metadata: optional Metadata` - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + Set of 16 key-value pairs that can be attached to an object. This can be + useful for storing additional information about the object in a structured + format, and querying for objects via API or the dashboard. - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + Keys are strings with a maximum length of 64 characters. Values are strings + with a maximum length of 512 characters. - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + - `object: optional "realtime.response"` - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + The object type, must be `realtime.response`. - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. + - `"realtime.response"` - - `type: "server_vad"` + - `output: optional array of ConversationItem` - Type of turn detection, `server_vad` to turn on simple Server VAD. + The list of output items generated by the response. - - `"server_vad"` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` - - `create_response: optional boolean` + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + - `content: array of object { text, type }` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + The content of the message. - - `idle_timeout_ms: optional number` + - `text: optional string` - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. + The text content. - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. + - `type: optional "input_text"` - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. + The content type. Always `input_text` for system messages. - - `interrupt_response: optional boolean` + - `"input_text"` - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + - `role: "system"` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + The role of the message sender. Always `system`. - - `prefix_padding_ms: optional number` + - `"system"` - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. + - `type: "message"` - - `silence_duration_ms: optional number` + The type of the item. Always `message`. - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. + - `"message"` - - `threshold: optional number` + - `id: optional string` - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. + The unique ID of the item. This may be provided by the client or generated by the server. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `object: optional "realtime.item"` - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - `type: "semantic_vad"` + - `"realtime.item"` - Type of turn detection, `semantic_vad` to turn on Semantic VAD. + - `status: optional "completed" or "incomplete" or "in_progress"` - - `"semantic_vad"` + The status of the item. Has no effect on the conversation. - - `create_response: optional boolean` + - `"completed"` - Whether or not to automatically generate a response when a VAD stop event occurs. + - `"incomplete"` - - `eagerness: optional "low" or "medium" or "high" or "auto"` + - `"in_progress"` - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` - - `"low"` + A user message item in a Realtime conversation. - - `"medium"` + - `content: array of object { audio, detail, image_url, 3 more }` - - `"high"` + The content of the message. + + - `audio: optional string` + + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `detail: optional "auto" or "low" or "high"` + + The detail level of the image (for `input_image`). `auto` will default to `high`. - `"auto"` - - `interrupt_response: optional boolean` + - `"low"` - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + - `"high"` - - `include: optional array of "item.input_audio_transcription.logprobs"` + - `image_url: optional string` - Additional fields to include in server outputs. + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + - `text: optional string` - - `"item.input_audio_transcription.logprobs"` + The text content (for `input_text`). - - `type: "session.update"` + - `transcript: optional string` - The event type, must be `session.update`. + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. - - `"session.update"` + - `type: optional "input_text" or "input_audio" or "input_image"` - - `event_id: optional string` + The content type (`input_text`, `input_audio`, or `input_image`). - Optional client-generated ID used to identify this event. This is an arbitrary string that a client may assign. It will be passed back if there is an error with the event, but the corresponding `session.updated` event will not include it. + - `"input_text"` -### Realtime Conversation Item Assistant Message + - `"input_audio"` + + - `"input_image"` + + - `role: "user"` + + The role of the message sender. Always `user`. -- `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `"user"` + + - `type: "message"` + + The type of the item. Always `message`. + + - `"message"` + + - `id: optional string` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `object: optional "realtime.item"` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `"realtime.item"` + + - `status: optional "completed" or "incomplete" or "in_progress"` + + The status of the item. Has no effect on the conversation. + + - `"completed"` + + - `"incomplete"` + + - `"in_progress"` + + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` An assistant message item in a Realtime conversation. @@ -6541,9 +6653,7 @@ - `"in_progress"` -### Realtime Conversation Item Function Call - -- `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` A function call item in a Realtime conversation. @@ -6585,9 +6695,7 @@ - `"in_progress"` -### Realtime Conversation Item Function Call Output - -- `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` A function call output item in a Realtime conversation. @@ -6625,439 +6733,398 @@ - `"in_progress"` -### Realtime Conversation Item System Message + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` -- `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + A Realtime item responding to an MCP approval request. - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + - `id: string` - - `content: array of object { text, type }` + The unique ID of the approval response. - The content of the message. + - `approval_request_id: string` - - `text: optional string` + The ID of the approval request being answered. - The text content. + - `approve: boolean` - - `type: optional "input_text"` + Whether the request was approved. - The content type. Always `input_text` for system messages. + - `type: "mcp_approval_response"` - - `"input_text"` + The type of the item. Always `mcp_approval_response`. - - `role: "system"` + - `"mcp_approval_response"` - The role of the message sender. Always `system`. + - `reason: optional string` - - `"system"` + Optional reason for the decision. - - `type: "message"` + - `RealtimeMcpListTools object { server_label, tools, type, id }` - The type of the item. Always `message`. + A Realtime item listing tools available on an MCP server. - - `"message"` + - `server_label: string` - - `id: optional string` + The label of the MCP server. - The unique ID of the item. This may be provided by the client or generated by the server. + - `tools: array of object { input_schema, name, annotations, description }` - - `object: optional "realtime.item"` + The tools available on the server. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `input_schema: unknown` - - `"realtime.item"` + The JSON schema describing the tool's input. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `name: string` - The status of the item. Has no effect on the conversation. + The name of the tool. - - `"completed"` + - `annotations: optional unknown` - - `"incomplete"` + Additional annotations about the tool. - - `"in_progress"` + - `description: optional string` -### Realtime Conversation Item User Message + The description of the tool. -- `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `type: "mcp_list_tools"` - A user message item in a Realtime conversation. + The type of the item. Always `mcp_list_tools`. - - `content: array of object { audio, detail, image_url, 3 more }` + - `"mcp_list_tools"` - The content of the message. + - `id: optional string` - - `audio: optional string` + The unique ID of the list. - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` - - `detail: optional "auto" or "low" or "high"` + A Realtime item representing an invocation of a tool on an MCP server. - The detail level of the image (for `input_image`). `auto` will default to `high`. + - `id: string` - - `"auto"` + The unique ID of the tool call. - - `"low"` + - `arguments: string` - - `"high"` + A JSON string of the arguments passed to the tool. - - `image_url: optional string` + - `name: string` - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + The name of the tool that was run. - - `text: optional string` + - `server_label: string` - The text content (for `input_text`). + The label of the MCP server running the tool. - - `transcript: optional string` + - `type: "mcp_call"` - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + The type of the item. Always `mcp_call`. - - `type: optional "input_text" or "input_audio" or "input_image"` + - `"mcp_call"` - The content type (`input_text`, `input_audio`, or `input_image`). + - `approval_request_id: optional string` - - `"input_text"` + The ID of an associated approval request, if any. - - `"input_audio"` + - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` - - `"input_image"` + The error from the tool call, if any. - - `role: "user"` + - `RealtimeMcpProtocolError object { code, message, type }` - The role of the message sender. Always `user`. + - `code: number` - - `"user"` + - `message: string` - - `type: "message"` + - `type: "protocol_error"` - The type of the item. Always `message`. + - `"protocol_error"` - - `"message"` + - `RealtimeMcpToolExecutionError object { message, type }` - - `id: optional string` + - `message: string` - The unique ID of the item. This may be provided by the client or generated by the server. + - `type: "tool_execution_error"` - - `object: optional "realtime.item"` + - `"tool_execution_error"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `RealtimeMcphttpError object { code, message, type }` - - `"realtime.item"` + - `code: number` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `message: string` - The status of the item. Has no effect on the conversation. + - `type: "http_error"` - - `"completed"` + - `"http_error"` - - `"incomplete"` + - `output: optional string` - - `"in_progress"` + The output from the tool call. -### Realtime Error + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` -- `RealtimeError = object { message, type, code, 2 more }` + A Realtime item requesting human approval of a tool invocation. - Details of the error. + - `id: string` - - `message: string` + The unique ID of the approval request. - A human-readable error message. + - `arguments: string` - - `type: string` + A JSON string of arguments for the tool. - The type of error (e.g., "invalid_request_error", "server_error"). + - `name: string` - - `code: optional string` + The name of the tool to run. - Error code, if any. + - `server_label: string` - - `event_id: optional string` + The label of the MCP server making the request. - The event_id of the client event that caused the error, if applicable. + - `type: "mcp_approval_request"` - - `param: optional string` + The type of the item. Always `mcp_approval_request`. - Parameter related to the error, if any. + - `"mcp_approval_request"` -### Realtime Error Event + - `output_modalities: optional array of "text" or "audio"` -- `RealtimeErrorEvent = object { error, event_id, type }` + The set of modalities the model used to respond, currently the only possible values are + `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the + output to mode `text` will disable audio output from the model. - Returned when an error occurs, which could be a client problem or a server - problem. Most errors are recoverable and the session will stay open, we - recommend to implementors to monitor and log error messages by default. + - `"text"` - - `error: RealtimeError` + - `"audio"` - Details of the error. + - `status: optional "completed" or "cancelled" or "failed" or 2 more` - - `message: string` + The final status of the response (`completed`, `cancelled`, `failed`, or + `incomplete`, `in_progress`). - A human-readable error message. + - `"completed"` - - `type: string` + - `"cancelled"` - The type of error (e.g., "invalid_request_error", "server_error"). + - `"failed"` - - `code: optional string` + - `"incomplete"` - Error code, if any. + - `"in_progress"` - - `event_id: optional string` + - `status_details: optional RealtimeResponseStatus` - The event_id of the client event that caused the error, if applicable. + Additional details about the status. - - `param: optional string` + - `error: optional object { code, type }` - Parameter related to the error, if any. + A description of the error that caused the response to fail, + populated when the `status` is `failed`. - - `event_id: string` + - `code: optional string` - The unique ID of the server event. + Error code, if any. - - `type: "error"` + - `type: optional string` - The event type, must be `error`. + The type of error. - - `"error"` + - `reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"` -### Realtime Function Tool + The reason the Response did not complete. For a `cancelled` Response, one of `turn_detected` (the server VAD detected a new start of speech) or `client_cancelled` (the client sent a cancel event). For an `incomplete` Response, one of `max_output_tokens` or `content_filter` (the server-side safety filter activated and cut off the response). -- `RealtimeFunctionTool = object { description, name, parameters, type }` + - `"turn_detected"` - - `description: optional string` + - `"client_cancelled"` - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). + - `"max_output_tokens"` - - `name: optional string` + - `"content_filter"` - The name of the function. + - `type: optional "completed" or "cancelled" or "failed" or "incomplete"` - - `parameters: optional unknown` + The type of error that caused the response to fail, corresponding + with the `status` field (`completed`, `cancelled`, `incomplete`, + `failed`). - Parameters of the function in JSON Schema. + - `"completed"` - - `type: optional "function"` + - `"cancelled"` - The type of the tool, i.e. `function`. + - `"failed"` - - `"function"` + - `"incomplete"` -### Realtime Mcp Approval Request + - `usage: optional RealtimeResponseUsage` -- `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + Usage statistics for the Response, this will correspond to billing. A + Realtime API session will maintain a conversation context and append new + Items to the Conversation, thus output from previous turns (text and + audio tokens) will become the input for later turns. - A Realtime item requesting human approval of a tool invocation. + - `input_token_details: optional RealtimeResponseUsageInputTokenDetails` - - `id: string` + Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. - The unique ID of the approval request. + - `audio_tokens: optional number` - - `arguments: string` + The number of audio tokens used as input for the Response. - A JSON string of arguments for the tool. + - `cached_tokens: optional number` - - `name: string` + The number of cached tokens used as input for the Response. - The name of the tool to run. + - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` - - `server_label: string` + Details about the cached tokens used as input for the Response. - The label of the MCP server making the request. + - `audio_tokens: optional number` - - `type: "mcp_approval_request"` + The number of cached audio tokens used as input for the Response. - The type of the item. Always `mcp_approval_request`. + - `image_tokens: optional number` - - `"mcp_approval_request"` + The number of cached image tokens used as input for the Response. -### Realtime Mcp Approval Response + - `text_tokens: optional number` -- `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + The number of cached text tokens used as input for the Response. - A Realtime item responding to an MCP approval request. + - `image_tokens: optional number` - - `id: string` + The number of image tokens used as input for the Response. - The unique ID of the approval response. + - `text_tokens: optional number` - - `approval_request_id: string` + The number of text tokens used as input for the Response. - The ID of the approval request being answered. + - `input_tokens: optional number` - - `approve: boolean` + The number of input tokens used in the Response, including text and + audio tokens. - Whether the request was approved. + - `output_token_details: optional RealtimeResponseUsageOutputTokenDetails` - - `type: "mcp_approval_response"` + Details about the output tokens used in the Response. - The type of the item. Always `mcp_approval_response`. + - `audio_tokens: optional number` - - `"mcp_approval_response"` + The number of audio tokens used in the Response. - - `reason: optional string` + - `text_tokens: optional number` - Optional reason for the decision. + The number of text tokens used in the Response. -### Realtime Mcp List Tools + - `output_tokens: optional number` -- `RealtimeMcpListTools = object { server_label, tools, type, id }` + The number of output tokens sent in the Response, including text and + audio tokens. - A Realtime item listing tools available on an MCP server. + - `total_tokens: optional number` - - `server_label: string` + The total number of tokens in the Response including input and output + text and audio tokens. - The label of the MCP server. +### Realtime Response Create Audio Output - - `tools: array of object { input_schema, name, annotations, description }` +- `RealtimeResponseCreateAudioOutput object { output }` - The tools available on the server. + Configuration for audio input and output. - - `input_schema: unknown` + - `output: optional object { format, voice }` - The JSON schema describing the tool's input. + - `format: optional RealtimeAudioFormats` - - `name: string` + The format of the output audio. - The name of the tool. + - `PCMAudioFormat object { rate, type }` - - `annotations: optional unknown` + The PCM audio format. Only a 24kHz sample rate is supported. - Additional annotations about the tool. + - `rate: optional 24000` - - `description: optional string` + The sample rate of the audio. Always `24000`. - The description of the tool. + - `24000` - - `type: "mcp_list_tools"` + - `type: optional "audio/pcm"` - The type of the item. Always `mcp_list_tools`. + The audio format. Always `audio/pcm`. - - `"mcp_list_tools"` + - `"audio/pcm"` - - `id: optional string` + - `PCMUAudioFormat object { type }` - The unique ID of the list. + The G.711 μ-law format. -### Realtime Mcp Protocol Error + - `type: optional "audio/pcmu"` -- `RealtimeMcpProtocolError = object { code, message, type }` + The audio format. Always `audio/pcmu`. - - `code: number` + - `"audio/pcmu"` - - `message: string` + - `PCMAAudioFormat object { type }` - - `type: "protocol_error"` + The G.711 A-law format. - - `"protocol_error"` + - `type: optional "audio/pcma"` -### Realtime Mcp Tool Call + The audio format. Always `audio/pcma`. -- `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `"audio/pcma"` - A Realtime item representing an invocation of a tool on an MCP server. + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - - `id: string` + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. - The unique ID of the tool call. + - `string` - - `arguments: string` + - `"alloy" or "ash" or "ballad" or 7 more` - A JSON string of the arguments passed to the tool. + - `"alloy"` - - `name: string` + - `"ash"` - The name of the tool that was run. + - `"ballad"` - - `server_label: string` + - `"coral"` - The label of the MCP server running the tool. + - `"echo"` - - `type: "mcp_call"` + - `"sage"` - The type of the item. Always `mcp_call`. + - `"shimmer"` - - `"mcp_call"` - - - `approval_request_id: optional string` - - The ID of an associated approval request, if any. - - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` - - The error from the tool call, if any. - - - `RealtimeMcpProtocolError = object { code, message, type }` - - - `code: number` - - - `message: string` - - - `type: "protocol_error"` - - - `"protocol_error"` - - - `RealtimeMcpToolExecutionError = object { message, type }` - - - `message: string` - - - `type: "tool_execution_error"` - - - `"tool_execution_error"` - - - `RealtimeMcphttpError = object { code, message, type }` - - - `code: number` - - - `message: string` - - - `type: "http_error"` - - - `"http_error"` - - - `output: optional string` - - The output from the tool call. - -### Realtime Mcp Tool Execution Error - -- `RealtimeMcpToolExecutionError = object { message, type }` - - - `message: string` - - - `type: "tool_execution_error"` - - - `"tool_execution_error"` - -### Realtime Mcphttp Error - -- `RealtimeMcphttpError = object { code, message, type }` + - `"verse"` - - `code: number` + - `"marin"` - - `message: string` + - `"cedar"` - - `type: "http_error"` + - `ID object { id }` - - `"http_error"` + Custom voice reference. -### Realtime Response + - `id: string` -- `RealtimeResponse = object { id, audio, conversation_id, 8 more }` + The custom voice ID, e.g. `voice_1234`. - The response resource. +### Realtime Response Create Params - - `id: optional string` +- `RealtimeResponseCreateParams object { audio, conversation, input, 7 more }` - The unique ID of the response, will look like `resp_1234`. + Create a new Realtime response with these parameters - - `audio: optional object { output }` + - `audio: optional RealtimeResponseCreateAudioOutput` - Configuration for audio output. + Configuration for audio input and output. - `output: optional object { format, voice }` @@ -7065,7 +7132,7 @@ The format of the output audio. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -7081,7 +7148,7 @@ - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -7091,7 +7158,7 @@ - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -7101,24 +7168,19 @@ - `"audio/pcma"` - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more` + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - The voice the model uses to respond. Voice cannot be changed during the - session once the model has responded with audio at least once. Current - voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, - `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for - best quality. + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. - `string` - `"alloy" or "ash" or "ballad" or 7 more` - The voice the model uses to respond. Voice cannot be changed during the - session once the model has responded with audio at least once. Current - voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, - `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for - best quality. - - `"alloy"` - `"ash"` @@ -7139,46 +7201,45 @@ - `"cedar"` - - `conversation_id: optional string` - - Which conversation the response is added to, determined by the `conversation` - field in the `response.create` event. If `auto`, the response will be added to - the default conversation and the value of `conversation_id` will be an id like - `conv_1234`. If `none`, the response will not be added to any conversation and - the value of `conversation_id` will be `null`. If responses are being triggered - automatically by VAD the response will be added to the default conversation - - - `max_output_tokens: optional number or "inf"` + - `ID object { id }` - Maximum number of output tokens for a single assistant response, - inclusive of tool calls, that was used in this response. + Custom voice reference. - - `number` + - `id: string` - - `"inf"` + The custom voice ID, e.g. `voice_1234`. - - `"inf"` + - `conversation: optional string or "auto" or "none"` - - `metadata: optional Metadata` + Controls which conversation the response is added to. Currently supports + `auto` and `none`, with `auto` as the default value. The `auto` value + means that the contents of the response will be added to the default + conversation. Set this to `none` to create an out-of-band response which + will not add items to default conversation. - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. + - `string` - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. + - `"auto" or "none"` - - `object: optional "realtime.response"` + Controls which conversation the response is added to. Currently supports + `auto` and `none`, with `auto` as the default value. The `auto` value + means that the contents of the response will be added to the default + conversation. Set this to `none` to create an out-of-band response which + will not add items to default conversation. - The object type, must be `realtime.response`. + - `"auto"` - - `"realtime.response"` + - `"none"` - - `output: optional array of ConversationItem` + - `input: optional array of ConversationItem` - The list of output items generated by the response. + Input items to include in the prompt for the model. Using this field + creates a new context for this Response instead of using the default + conversation. An empty array `[]` will clear the context for this Response. + Note that this can include references to items that previously appeared in the session + using their id. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. @@ -7228,7 +7289,7 @@ - `"in_progress"` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` A user message item in a Realtime conversation. @@ -7304,7 +7365,7 @@ - `"in_progress"` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` An assistant message item in a Realtime conversation. @@ -7364,7 +7425,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` A function call item in a Realtime conversation. @@ -7406,7 +7467,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` A function call output item in a Realtime conversation. @@ -7444,7 +7505,7 @@ - `"in_progress"` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` A Realtime item responding to an MCP approval request. @@ -7470,7 +7531,7 @@ Optional reason for the decision. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `RealtimeMcpListTools object { server_label, tools, type, id }` A Realtime item listing tools available on an MCP server. @@ -7508,7 +7569,7 @@ The unique ID of the list. - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` A Realtime item representing an invocation of a tool on an MCP server. @@ -7542,7 +7603,7 @@ The error from the tool call, if any. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `RealtimeMcpProtocolError object { code, message, type }` - `code: number` @@ -7552,7 +7613,7 @@ - `"protocol_error"` - - `RealtimeMcpToolExecutionError = object { message, type }` + - `RealtimeMcpToolExecutionError object { message, type }` - `message: string` @@ -7560,7 +7621,7 @@ - `"tool_execution_error"` - - `RealtimeMcphttpError = object { code, message, type }` + - `RealtimeMcphttpError object { code, message, type }` - `code: number` @@ -7574,7 +7635,7 @@ The output from the tool call. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` A Realtime item requesting human approval of a tool invocation. @@ -7600,6 +7661,33 @@ - `"mcp_approval_request"` + - `instructions: optional string` + + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. + + - `max_output_tokens: optional number or "inf"` + + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. + + - `number` + + - `"inf"` + + - `"inf"` + + - `metadata: optional Metadata` + + Set of 16 key-value pairs that can be attached to an object. This can be + useful for storing additional information about the object in a structured + format, and querying for objects via API or the dashboard. + + Keys are strings with a maximum length of 64 characters. Values are strings + with a maximum length of 512 characters. + - `output_modalities: optional array of "text" or "audio"` The set of modalities the model used to respond, currently the only possible values are @@ -7610,347 +7698,550 @@ - `"audio"` - - `status: optional "completed" or "cancelled" or "failed" or 2 more` + - `prompt: optional ResponsePrompt` - The final status of the response (`completed`, `cancelled`, `failed`, or - `incomplete`, `in_progress`). + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - - `"completed"` + - `id: string` - - `"cancelled"` + The unique identifier of the prompt template to use. - - `"failed"` + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - `"incomplete"` + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - - `"in_progress"` + - `string` - - `status_details: optional RealtimeResponseStatus` + - `ResponseInputText object { text, type }` - Additional details about the status. + A text input to the model. - - `error: optional object { code, type }` + - `text: string` - A description of the error that caused the response to fail, - populated when the `status` is `failed`. + The text input to the model. - - `code: optional string` + - `type: "input_text"` - Error code, if any. + The type of the input item. Always `input_text`. - - `type: optional string` + - `"input_text"` - The type of error. + - `ResponseInputImage object { detail, type, file_id, image_url }` - - `reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"` + An image input to the model. Learn about [image inputs](/docs/guides/vision). - The reason the Response did not complete. For a `cancelled` Response, one of `turn_detected` (the server VAD detected a new start of speech) or `client_cancelled` (the client sent a cancel event). For an `incomplete` Response, one of `max_output_tokens` or `content_filter` (the server-side safety filter activated and cut off the response). + - `detail: "low" or "high" or "auto" or "original"` - - `"turn_detected"` + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - `"client_cancelled"` + - `"low"` - - `"max_output_tokens"` + - `"high"` - - `"content_filter"` + - `"auto"` - - `type: optional "completed" or "cancelled" or "failed" or "incomplete"` + - `"original"` - The type of error that caused the response to fail, corresponding - with the `status` field (`completed`, `cancelled`, `incomplete`, - `failed`). + - `type: "input_image"` - - `"completed"` + The type of the input item. Always `input_image`. - - `"cancelled"` + - `"input_image"` - - `"failed"` + - `file_id: optional string` - - `"incomplete"` + The ID of the file to be sent to the model. - - `usage: optional RealtimeResponseUsage` + - `image_url: optional string` - Usage statistics for the Response, this will correspond to billing. A - Realtime API session will maintain a conversation context and append new - Items to the Conversation, thus output from previous turns (text and - audio tokens) will become the input for later turns. + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - `input_token_details: optional RealtimeResponseUsageInputTokenDetails` + - `ResponseInputFile object { type, detail, file_data, 3 more }` - Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. + A file input to the model. - - `audio_tokens: optional number` + - `type: "input_file"` - The number of audio tokens used as input for the Response. + The type of the input item. Always `input_file`. - - `cached_tokens: optional number` + - `"input_file"` - The number of cached tokens used as input for the Response. + - `detail: optional "low" or "high"` - - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - Details about the cached tokens used as input for the Response. + - `"low"` - - `audio_tokens: optional number` + - `"high"` - The number of cached audio tokens used as input for the Response. + - `file_data: optional string` - - `image_tokens: optional number` + The content of the file to be sent to the model. - The number of cached image tokens used as input for the Response. + - `file_id: optional string` - - `text_tokens: optional number` + The ID of the file to be sent to the model. - The number of cached text tokens used as input for the Response. + - `file_url: optional string` - - `image_tokens: optional number` + The URL of the file to be sent to the model. - The number of image tokens used as input for the Response. + - `filename: optional string` - - `text_tokens: optional number` + The name of the file to be sent to the model. - The number of text tokens used as input for the Response. + - `version: optional string` - - `input_tokens: optional number` + Optional version of the prompt template. - The number of input tokens used in the Response, including text and - audio tokens. + - `tool_choice: optional ToolChoiceOptions or ToolChoiceFunction or ToolChoiceMcp` - - `output_token_details: optional RealtimeResponseUsageOutputTokenDetails` + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. - Details about the output tokens used in the Response. + - `ToolChoiceOptions = "none" or "auto" or "required"` - - `audio_tokens: optional number` + Controls which (if any) tool is called by the model. - The number of audio tokens used in the Response. + `none` means the model will not call any tool and instead generates a message. - - `text_tokens: optional number` + `auto` means the model can pick between generating a message or calling one or + more tools. - The number of text tokens used in the Response. + `required` means the model must call one or more tools. - - `output_tokens: optional number` + - `"none"` - The number of output tokens sent in the Response, including text and - audio tokens. + - `"auto"` - - `total_tokens: optional number` + - `"required"` - The total number of tokens in the Response including input and output - text and audio tokens. + - `ToolChoiceFunction object { name, type }` -### Realtime Response Create Audio Output + Use this option to force the model to call a specific function. -- `RealtimeResponseCreateAudioOutput = object { output }` + - `name: string` - Configuration for audio input and output. + The name of the function to call. - - `output: optional object { format, voice }` + - `type: "function"` - - `format: optional RealtimeAudioFormats` + For function calling, the type is always `function`. - The format of the output audio. + - `"function"` - - `PCMAudioFormat = object { rate, type }` + - `ToolChoiceMcp object { server_label, type, name }` - The PCM audio format. Only a 24kHz sample rate is supported. + Use this option to force the model to call a specific tool on a remote MCP server. - - `rate: optional 24000` + - `server_label: string` - The sample rate of the audio. Always `24000`. + The label of the MCP server to use. - - `24000` + - `type: "mcp"` - - `type: optional "audio/pcm"` + For MCP tools, the type is always `mcp`. - The audio format. Always `audio/pcm`. + - `"mcp"` - - `"audio/pcm"` + - `name: optional string` - - `PCMUAudioFormat = object { type }` + The name of the tool to call on the server. - The G.711 μ-law format. + - `tools: optional array of RealtimeFunctionTool or object { server_label, type, allowed_tools, 7 more }` - - `type: optional "audio/pcmu"` + Tools available to the model. - The audio format. Always `audio/pcmu`. + - `RealtimeFunctionTool object { description, name, parameters, type }` - - `"audio/pcmu"` + - `description: optional string` - - `PCMAAudioFormat = object { type }` + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - The G.711 A-law format. + - `name: optional string` - - `type: optional "audio/pcma"` + The name of the function. - The audio format. Always `audio/pcma`. + - `parameters: optional unknown` - - `"audio/pcma"` + Parameters of the function in JSON Schema. - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` + - `type: optional "function"` - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with - an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed - during the session once the model has responded with audio at least once. - We recommend `marin` and `cedar` for best quality. + The type of the tool, i.e. `function`. - - `string` + - `"function"` - - `"alloy" or "ash" or "ballad" or 7 more` + - `McpTool object { server_label, type, allowed_tools, 7 more }` - - `"alloy"` + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - - `"ash"` + - `server_label: string` - - `"ballad"` + A label for this MCP server, used to identify it in tool calls. - - `"coral"` + - `type: "mcp"` - - `"echo"` + The type of the MCP tool. Always `mcp`. - - `"sage"` + - `"mcp"` - - `"shimmer"` + - `allowed_tools: optional array of string or object { read_only, tool_names }` - - `"verse"` + List of allowed tool names or a filter object. - - `"marin"` + - `McpAllowedTools = array of string` - - `"cedar"` + A string array of allowed tool names - - `ID = object { id }` + - `McpToolFilter object { read_only, tool_names }` - Custom voice reference. + A filter object to specify which tools are allowed. - - `id: string` + - `read_only: optional boolean` - The custom voice ID, e.g. `voice_1234`. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. -### Realtime Response Create Params + - `tool_names: optional array of string` -- `RealtimeResponseCreateParams = object { audio, conversation, input, 7 more }` + List of allowed tool names. - Create a new Realtime response with these parameters + - `authorization: optional string` - - `audio: optional RealtimeResponseCreateAudioOutput` + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. - Configuration for audio input and output. + - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - - `output: optional object { format, voice }` + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](/docs/guides/tools-remote-mcp#connectors). - - `format: optional RealtimeAudioFormats` + Currently supported `connector_id` values are: - The format of the output audio. + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` - - `PCMAudioFormat = object { rate, type }` + - `"connector_dropbox"` - The PCM audio format. Only a 24kHz sample rate is supported. + - `"connector_gmail"` - - `rate: optional 24000` + - `"connector_googlecalendar"` - The sample rate of the audio. Always `24000`. + - `"connector_googledrive"` - - `24000` + - `"connector_microsoftteams"` - - `type: optional "audio/pcm"` + - `"connector_outlookcalendar"` - The audio format. Always `audio/pcm`. + - `"connector_outlookemail"` - - `"audio/pcm"` + - `"connector_sharepoint"` - - `PCMUAudioFormat = object { type }` + - `defer_loading: optional boolean` - The G.711 μ-law format. + Whether this MCP tool is deferred and discovered via tool search. - - `type: optional "audio/pcmu"` + - `headers: optional map[string]` - The audio format. Always `audio/pcmu`. + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. - - `"audio/pcmu"` + - `require_approval: optional object { always, never } or "always" or "never"` - - `PCMAAudioFormat = object { type }` + Specify which of the MCP server's tools require approval. - The G.711 A-law format. + - `McpToolApprovalFilter object { always, never }` - - `type: optional "audio/pcma"` + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. - The audio format. Always `audio/pcma`. + - `always: optional object { read_only, tool_names }` - - `"audio/pcma"` + A filter object to specify which tools are allowed. - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` + - `read_only: optional boolean` - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with - an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed - during the session once the model has responded with audio at least once. - We recommend `marin` and `cedar` for best quality. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `string` + - `tool_names: optional array of string` - - `"alloy" or "ash" or "ballad" or 7 more` + List of allowed tool names. - - `"alloy"` + - `never: optional object { read_only, tool_names }` - - `"ash"` + A filter object to specify which tools are allowed. - - `"ballad"` + - `read_only: optional boolean` - - `"coral"` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `"echo"` + - `tool_names: optional array of string` - - `"sage"` + List of allowed tool names. - - `"shimmer"` + - `McpToolApprovalSetting = "always" or "never"` - - `"verse"` + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. - - `"marin"` + - `"always"` - - `"cedar"` + - `"never"` - - `ID = object { id }` + - `server_description: optional string` - Custom voice reference. + Optional description of the MCP server, used to provide more context. - - `id: string` + - `server_url: optional string` - The custom voice ID, e.g. `voice_1234`. + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. - - `conversation: optional string or "auto" or "none"` +### Realtime Response Status - Controls which conversation the response is added to. Currently supports - `auto` and `none`, with `auto` as the default value. The `auto` value - means that the contents of the response will be added to the default - conversation. Set this to `none` to create an out-of-band response which - will not add items to default conversation. +- `RealtimeResponseStatus object { error, reason, type }` - - `string` + Additional details about the status. - - `"auto" or "none"` + - `error: optional object { code, type }` - Controls which conversation the response is added to. Currently supports - `auto` and `none`, with `auto` as the default value. The `auto` value - means that the contents of the response will be added to the default - conversation. Set this to `none` to create an out-of-band response which - will not add items to default conversation. + A description of the error that caused the response to fail, + populated when the `status` is `failed`. - - `"auto"` + - `code: optional string` - - `"none"` + Error code, if any. - - `input: optional array of ConversationItem` + - `type: optional string` - Input items to include in the prompt for the model. Using this field - creates a new context for this Response instead of using the default - conversation. An empty array `[]` will clear the context for this Response. - Note that this can include references to items that previously appeared in the session - using their id. + The type of error. + + - `reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"` + + The reason the Response did not complete. For a `cancelled` Response, one of `turn_detected` (the server VAD detected a new start of speech) or `client_cancelled` (the client sent a cancel event). For an `incomplete` Response, one of `max_output_tokens` or `content_filter` (the server-side safety filter activated and cut off the response). + + - `"turn_detected"` + + - `"client_cancelled"` + + - `"max_output_tokens"` + + - `"content_filter"` + + - `type: optional "completed" or "cancelled" or "failed" or "incomplete"` + + The type of error that caused the response to fail, corresponding + with the `status` field (`completed`, `cancelled`, `incomplete`, + `failed`). + + - `"completed"` + + - `"cancelled"` + + - `"failed"` + + - `"incomplete"` + +### Realtime Response Usage + +- `RealtimeResponseUsage object { input_token_details, input_tokens, output_token_details, 2 more }` + + Usage statistics for the Response, this will correspond to billing. A + Realtime API session will maintain a conversation context and append new + Items to the Conversation, thus output from previous turns (text and + audio tokens) will become the input for later turns. + + - `input_token_details: optional RealtimeResponseUsageInputTokenDetails` + + Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. + + - `audio_tokens: optional number` + + The number of audio tokens used as input for the Response. + + - `cached_tokens: optional number` + + The number of cached tokens used as input for the Response. + + - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` + + Details about the cached tokens used as input for the Response. + + - `audio_tokens: optional number` + + The number of cached audio tokens used as input for the Response. + + - `image_tokens: optional number` + + The number of cached image tokens used as input for the Response. + + - `text_tokens: optional number` + + The number of cached text tokens used as input for the Response. + + - `image_tokens: optional number` + + The number of image tokens used as input for the Response. + + - `text_tokens: optional number` + + The number of text tokens used as input for the Response. + + - `input_tokens: optional number` + + The number of input tokens used in the Response, including text and + audio tokens. + + - `output_token_details: optional RealtimeResponseUsageOutputTokenDetails` + + Details about the output tokens used in the Response. + + - `audio_tokens: optional number` + + The number of audio tokens used in the Response. + + - `text_tokens: optional number` + + The number of text tokens used in the Response. + + - `output_tokens: optional number` + + The number of output tokens sent in the Response, including text and + audio tokens. + + - `total_tokens: optional number` + + The total number of tokens in the Response including input and output + text and audio tokens. + +### Realtime Response Usage Input Token Details + +- `RealtimeResponseUsageInputTokenDetails object { audio_tokens, cached_tokens, cached_tokens_details, 2 more }` + + Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. + + - `audio_tokens: optional number` + + The number of audio tokens used as input for the Response. + + - `cached_tokens: optional number` + + The number of cached tokens used as input for the Response. + + - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` + + Details about the cached tokens used as input for the Response. + + - `audio_tokens: optional number` + + The number of cached audio tokens used as input for the Response. + + - `image_tokens: optional number` + + The number of cached image tokens used as input for the Response. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `text_tokens: optional number` + + The number of cached text tokens used as input for the Response. + + - `image_tokens: optional number` + + The number of image tokens used as input for the Response. + + - `text_tokens: optional number` + + The number of text tokens used as input for the Response. + +### Realtime Response Usage Output Token Details + +- `RealtimeResponseUsageOutputTokenDetails object { audio_tokens, text_tokens }` + + Details about the output tokens used in the Response. + + - `audio_tokens: optional number` + + The number of audio tokens used in the Response. + + - `text_tokens: optional number` + + The number of text tokens used in the Response. + +### Realtime Server Event + +- `RealtimeServerEvent = ConversationCreatedEvent or ConversationItemCreatedEvent or ConversationItemDeletedEvent or 43 more` + + A realtime server event. + + - `ConversationCreatedEvent object { conversation, event_id, type }` + + Returned when a conversation is created. Emitted right after session creation. + + - `conversation: object { id, object }` + + The conversation resource. + + - `id: optional string` + + The unique ID of the conversation. + + - `object: optional string` + + The object type, must be `realtime.conversation`. + + - `event_id: string` + + The unique ID of the server event. + + - `type: "conversation.created"` + + The event type, must be `conversation.created`. + + - `"conversation.created"` + + - `ConversationItemCreatedEvent object { event_id, item, type, previous_item_id }` + + Returned when a conversation item is created. There are several scenarios that produce this event: + + - The server is generating a Response, which if successful will produce + either one or two Items, which will be of type `message` + (role `assistant`) or type `function_call`. + - The input audio buffer has been committed, either by the client or the + server (in `server_vad` mode). The server will take the content of the + input audio buffer and add it to a new user message Item. + - The client has sent a `conversation.item.create` event to add a new Item + to the Conversation. + + - `event_id: string` + + The unique ID of the server event. + + - `item: ConversationItem` + + A single item within a Realtime conversation. + + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. @@ -8000,7 +8291,7 @@ - `"in_progress"` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` A user message item in a Realtime conversation. @@ -8076,7 +8367,7 @@ - `"in_progress"` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` An assistant message item in a Realtime conversation. @@ -8136,7 +8427,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` A function call item in a Realtime conversation. @@ -8178,7 +8469,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` A function call output item in a Realtime conversation. @@ -8216,7 +8507,7 @@ - `"in_progress"` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` A Realtime item responding to an MCP approval request. @@ -8242,7 +8533,7 @@ Optional reason for the decision. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `RealtimeMcpListTools object { server_label, tools, type, id }` A Realtime item listing tools available on an MCP server. @@ -8280,7 +8571,7 @@ The unique ID of the list. - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` A Realtime item representing an invocation of a tool on an MCP server. @@ -8314,7 +8605,7 @@ The error from the tool call, if any. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `RealtimeMcpProtocolError object { code, message, type }` - `code: number` @@ -8324,7 +8615,7 @@ - `"protocol_error"` - - `RealtimeMcpToolExecutionError = object { message, type }` + - `RealtimeMcpToolExecutionError object { message, type }` - `message: string` @@ -8332,7 +8623,7 @@ - `"tool_execution_error"` - - `RealtimeMcphttpError = object { code, message, type }` + - `RealtimeMcphttpError object { code, message, type }` - `code: number` @@ -8346,7 +8637,7 @@ The output from the tool call. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` A Realtime item requesting human approval of a tool invocation. @@ -8372,1215 +8663,1209 @@ - `"mcp_approval_request"` - - `instructions: optional string` - - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - - `max_output_tokens: optional number or "inf"` + - `type: "conversation.item.created"` - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. + The event type, must be `conversation.item.created`. - - `number` + - `"conversation.item.created"` - - `"inf"` + - `previous_item_id: optional string` - - `"inf"` + The ID of the preceding item in the Conversation context, allows the + client to understand the order of the conversation. Can be `null` if the + item has no predecessor. - - `metadata: optional Metadata` + - `ConversationItemDeletedEvent object { event_id, item_id, type }` - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. + Returned when an item in the conversation is deleted by the client with a + `conversation.item.delete` event. This event is used to synchronize the + server's understanding of the conversation history with the client's view. - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. + - `event_id: string` - - `output_modalities: optional array of "text" or "audio"` + The unique ID of the server event. - The set of modalities the model used to respond, currently the only possible values are - `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the - output to mode `text` will disable audio output from the model. + - `item_id: string` - - `"text"` + The ID of the item that was deleted. - - `"audio"` + - `type: "conversation.item.deleted"` - - `prompt: optional ResponsePrompt` + The event type, must be `conversation.item.deleted`. - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). + - `"conversation.item.deleted"` - - `id: string` + - `ConversationItemInputAudioTranscriptionCompletedEvent object { content_index, event_id, item_id, 4 more }` - The unique identifier of the prompt template to use. + This event is the output of audio transcription for user audio written to the + user audio buffer. Transcription begins when the input audio buffer is + committed by the client or server (when VAD is enabled). Transcription runs + asynchronously with Response creation, so this event may come before or after + the Response events. - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` + Realtime API models accept audio natively, and thus input transcription is a + separate process run on a separate ASR (Automatic Speech Recognition) model. + The transcript may diverge somewhat from the model's interpretation, and + should be treated as a rough guide. - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. + - `content_index: number` - - `string` + The index of the content part containing the audio. - - `ResponseInputText = object { text, type }` + - `event_id: string` - A text input to the model. + The unique ID of the server event. - - `text: string` + - `item_id: string` - The text input to the model. + The ID of the item containing the audio that is being transcribed. - - `type: "input_text"` + - `transcript: string` - The type of the input item. Always `input_text`. + The transcribed text. - - `"input_text"` + - `type: "conversation.item.input_audio_transcription.completed"` - - `ResponseInputImage = object { detail, type, file_id, image_url }` + The event type, must be + `conversation.item.input_audio_transcription.completed`. - An image input to the model. Learn about [image inputs](/docs/guides/vision). + - `"conversation.item.input_audio_transcription.completed"` - - `detail: "low" or "high" or "auto" or "original"` + - `usage: object { input_tokens, output_tokens, total_tokens, 2 more } or object { seconds, type }` - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. + Usage statistics for the transcription, this is billed according to the ASR model's pricing rather than the realtime model's pricing. - - `"low"` + - `TokenUsage object { input_tokens, output_tokens, total_tokens, 2 more }` - - `"high"` + Usage statistics for models billed by token usage. - - `"auto"` + - `input_tokens: number` - - `"original"` + Number of input tokens billed for this request. - - `type: "input_image"` + - `output_tokens: number` - The type of the input item. Always `input_image`. + Number of output tokens generated. - - `"input_image"` + - `total_tokens: number` - - `file_id: optional string` + Total number of tokens used (input + output). - The ID of the file to be sent to the model. + - `type: "tokens"` - - `image_url: optional string` + The type of the usage object. Always `tokens` for this variant. - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. + - `"tokens"` - - `ResponseInputFile = object { type, detail, file_data, 3 more }` + - `input_token_details: optional object { audio_tokens, text_tokens }` - A file input to the model. + Details about the input tokens billed for this request. - - `type: "input_file"` + - `audio_tokens: optional number` - The type of the input item. Always `input_file`. + Number of audio tokens billed for this request. - - `"input_file"` + - `text_tokens: optional number` - - `detail: optional "low" or "high"` + Number of text tokens billed for this request. - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. + - `DurationUsage object { seconds, type }` - - `"low"` + Usage statistics for models billed by audio input duration. - - `"high"` + - `seconds: number` - - `file_data: optional string` + Duration of the input audio in seconds. - The content of the file to be sent to the model. + - `type: "duration"` - - `file_id: optional string` + The type of the usage object. Always `duration` for this variant. - The ID of the file to be sent to the model. + - `"duration"` - - `file_url: optional string` + - `logprobs: optional array of LogProbProperties` - The URL of the file to be sent to the model. + The log probabilities of the transcription. - - `filename: optional string` + - `token: string` - The name of the file to be sent to the model. + The token that was used to generate the log probability. - - `version: optional string` + - `bytes: array of number` - Optional version of the prompt template. + The bytes that were used to generate the log probability. - - `tool_choice: optional ToolChoiceOptions or ToolChoiceFunction or ToolChoiceMcp` + - `logprob: number` - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. + The log probability of the token. - - `ToolChoiceOptions = "none" or "auto" or "required"` + - `ConversationItemInputAudioTranscriptionDeltaEvent object { event_id, item_id, type, 3 more }` - Controls which (if any) tool is called by the model. + Returned when the text value of an input audio transcription content part is updated with incremental transcription results. - `none` means the model will not call any tool and instead generates a message. + - `event_id: string` - `auto` means the model can pick between generating a message or calling one or - more tools. + The unique ID of the server event. - `required` means the model must call one or more tools. + - `item_id: string` - - `"none"` + The ID of the item containing the audio that is being transcribed. - - `"auto"` + - `type: "conversation.item.input_audio_transcription.delta"` - - `"required"` + The event type, must be `conversation.item.input_audio_transcription.delta`. - - `ToolChoiceFunction = object { name, type }` + - `"conversation.item.input_audio_transcription.delta"` - Use this option to force the model to call a specific function. + - `content_index: optional number` - - `name: string` + The index of the content part in the item's content array. - The name of the function to call. + - `delta: optional string` - - `type: "function"` + The text delta. - For function calling, the type is always `function`. + - `logprobs: optional array of LogProbProperties` - - `"function"` + The log probabilities of the transcription. These can be enabled by configurating the session with `"include": ["item.input_audio_transcription.logprobs"]`. Each entry in the array corresponds a log probability of which token would be selected for this chunk of transcription. This can help to identify if it was possible there were multiple valid options for a given chunk of transcription. - - `ToolChoiceMcp = object { server_label, type, name }` + - `token: string` - Use this option to force the model to call a specific tool on a remote MCP server. + The token that was used to generate the log probability. - - `server_label: string` + - `bytes: array of number` - The label of the MCP server to use. + The bytes that were used to generate the log probability. - - `type: "mcp"` + - `logprob: number` - For MCP tools, the type is always `mcp`. + The log probability of the token. - - `"mcp"` + - `ConversationItemInputAudioTranscriptionFailedEvent object { content_index, error, event_id, 2 more }` - - `name: optional string` + Returned when input audio transcription is configured, and a transcription + request for a user message failed. These events are separate from other + `error` events so that the client can identify the related Item. - The name of the tool to call on the server. + - `content_index: number` - - `tools: optional array of RealtimeFunctionTool or object { server_label, type, allowed_tools, 7 more }` + The index of the content part containing the audio. - Tools available to the model. + - `error: object { code, message, param, type }` - - `RealtimeFunctionTool = object { description, name, parameters, type }` + Details of the transcription error. - - `description: optional string` + - `code: optional string` - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). + Error code, if any. - - `name: optional string` + - `message: optional string` - The name of the function. + A human-readable error message. - - `parameters: optional unknown` + - `param: optional string` - Parameters of the function in JSON Schema. + Parameter related to the error, if any. - - `type: optional "function"` + - `type: optional string` - The type of the tool, i.e. `function`. + The type of error. - - `"function"` + - `event_id: string` - - `McpTool = object { server_label, type, allowed_tools, 7 more }` + The unique ID of the server event. - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). + - `item_id: string` - - `server_label: string` + The ID of the user message item. - A label for this MCP server, used to identify it in tool calls. + - `type: "conversation.item.input_audio_transcription.failed"` - - `type: "mcp"` + The event type, must be + `conversation.item.input_audio_transcription.failed`. - The type of the MCP tool. Always `mcp`. + - `"conversation.item.input_audio_transcription.failed"` - - `"mcp"` + - `ConversationItemRetrieved object { event_id, item, type }` - - `allowed_tools: optional array of string or object { read_only, tool_names }` + Returned when a conversation item is retrieved with `conversation.item.retrieve`. This is provided as a way to fetch the server's representation of an item, for example to get access to the post-processed audio data after noise cancellation and VAD. It includes the full content of the Item, including audio data. - List of allowed tool names or a filter object. + - `event_id: string` - - `McpAllowedTools = array of string` + The unique ID of the server event. - A string array of allowed tool names + - `item: ConversationItem` - - `McpToolFilter = object { read_only, tool_names }` + A single item within a Realtime conversation. - A filter object to specify which tools are allowed. + - `type: "conversation.item.retrieved"` - - `read_only: optional boolean` + The event type, must be `conversation.item.retrieved`. - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `"conversation.item.retrieved"` - - `tool_names: optional array of string` + - `ConversationItemTruncatedEvent object { audio_end_ms, content_index, event_id, 2 more }` - List of allowed tool names. + Returned when an earlier assistant audio message item is truncated by the + client with a `conversation.item.truncate` event. This event is used to + synchronize the server's understanding of the audio with the client's playback. - - `authorization: optional string` + This action will truncate the audio and remove the server-side text transcript + to ensure there is no text in the context that hasn't been heard by the user. - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. + - `audio_end_ms: number` - - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` + The duration up to which the audio was truncated, in milliseconds. - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](/docs/guides/tools-remote-mcp#connectors). + - `content_index: number` - Currently supported `connector_id` values are: + The index of the content part that was truncated. - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` + - `event_id: string` - - `"connector_dropbox"` + The unique ID of the server event. - - `"connector_gmail"` + - `item_id: string` - - `"connector_googlecalendar"` + The ID of the assistant message item that was truncated. - - `"connector_googledrive"` + - `type: "conversation.item.truncated"` - - `"connector_microsoftteams"` + The event type, must be `conversation.item.truncated`. - - `"connector_outlookcalendar"` + - `"conversation.item.truncated"` - - `"connector_outlookemail"` + - `RealtimeErrorEvent object { error, event_id, type }` - - `"connector_sharepoint"` + Returned when an error occurs, which could be a client problem or a server + problem. Most errors are recoverable and the session will stay open, we + recommend to implementors to monitor and log error messages by default. - - `defer_loading: optional boolean` + - `error: RealtimeError` - Whether this MCP tool is deferred and discovered via tool search. + Details of the error. - - `headers: optional map[string]` + - `message: string` - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. + A human-readable error message. - - `require_approval: optional object { always, never } or "always" or "never"` + - `type: string` - Specify which of the MCP server's tools require approval. + The type of error (e.g., "invalid_request_error", "server_error"). - - `McpToolApprovalFilter = object { always, never }` + - `code: optional string` - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. + Error code, if any. - - `always: optional object { read_only, tool_names }` + - `event_id: optional string` - A filter object to specify which tools are allowed. + The event_id of the client event that caused the error, if applicable. - - `read_only: optional boolean` + - `param: optional string` - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + Parameter related to the error, if any. - - `tool_names: optional array of string` + - `event_id: string` - List of allowed tool names. + The unique ID of the server event. - - `never: optional object { read_only, tool_names }` + - `type: "error"` - A filter object to specify which tools are allowed. + The event type, must be `error`. - - `read_only: optional boolean` + - `"error"` - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `InputAudioBufferClearedEvent object { event_id, type }` - - `tool_names: optional array of string` + Returned when the input audio buffer is cleared by the client with a + `input_audio_buffer.clear` event. - List of allowed tool names. + - `event_id: string` - - `McpToolApprovalSetting = "always" or "never"` + The unique ID of the server event. - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. + - `type: "input_audio_buffer.cleared"` - - `"always"` + The event type, must be `input_audio_buffer.cleared`. - - `"never"` + - `"input_audio_buffer.cleared"` - - `server_description: optional string` + - `InputAudioBufferCommittedEvent object { event_id, item_id, type, previous_item_id }` - Optional description of the MCP server, used to provide more context. + Returned when an input audio buffer is committed, either by the client or + automatically in server VAD mode. The `item_id` property is the ID of the user + message item that will be created, thus a `conversation.item.created` event + will also be sent to the client. - - `server_url: optional string` + - `event_id: string` - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. + The unique ID of the server event. -### Realtime Response Status + - `item_id: string` -- `RealtimeResponseStatus = object { error, reason, type }` + The ID of the user message item that will be created. - Additional details about the status. + - `type: "input_audio_buffer.committed"` - - `error: optional object { code, type }` + The event type, must be `input_audio_buffer.committed`. - A description of the error that caused the response to fail, - populated when the `status` is `failed`. + - `"input_audio_buffer.committed"` - - `code: optional string` + - `previous_item_id: optional string` - Error code, if any. + The ID of the preceding item after which the new item will be inserted. + Can be `null` if the item has no predecessor. - - `type: optional string` + - `InputAudioBufferDtmfEventReceivedEvent object { event, received_at, type }` - The type of error. + **SIP Only:** Returned when an DTMF event is received. A DTMF event is a message that + represents a telephone keypad press (0–9, *, #, A–D). The `event` property + is the keypad that the user press. The `received_at` is the UTC Unix Timestamp + that the server received the event. - - `reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"` + - `event: string` - The reason the Response did not complete. For a `cancelled` Response, one of `turn_detected` (the server VAD detected a new start of speech) or `client_cancelled` (the client sent a cancel event). For an `incomplete` Response, one of `max_output_tokens` or `content_filter` (the server-side safety filter activated and cut off the response). + The telephone keypad that was pressed by the user. - - `"turn_detected"` + - `received_at: number` - - `"client_cancelled"` + UTC Unix Timestamp when DTMF Event was received by server. - - `"max_output_tokens"` + - `type: "input_audio_buffer.dtmf_event_received"` - - `"content_filter"` + The event type, must be `input_audio_buffer.dtmf_event_received`. - - `type: optional "completed" or "cancelled" or "failed" or "incomplete"` + - `"input_audio_buffer.dtmf_event_received"` - The type of error that caused the response to fail, corresponding - with the `status` field (`completed`, `cancelled`, `incomplete`, - `failed`). + - `InputAudioBufferSpeechStartedEvent object { audio_start_ms, event_id, item_id, type }` - - `"completed"` + Sent by the server when in `server_vad` mode to indicate that speech has been + detected in the audio buffer. This can happen any time audio is added to the + buffer (unless speech is already detected). The client may want to use this + event to interrupt audio playback or provide visual feedback to the user. - - `"cancelled"` + The client should expect to receive a `input_audio_buffer.speech_stopped` event + when speech stops. The `item_id` property is the ID of the user message item + that will be created when speech stops and will also be included in the + `input_audio_buffer.speech_stopped` event (unless the client manually commits + the audio buffer during VAD activation). - - `"failed"` + - `audio_start_ms: number` - - `"incomplete"` + Milliseconds from the start of all audio written to the buffer during the + session when speech was first detected. This will correspond to the + beginning of audio sent to the model, and thus includes the + `prefix_padding_ms` configured in the Session. -### Realtime Response Usage + - `event_id: string` -- `RealtimeResponseUsage = object { input_token_details, input_tokens, output_token_details, 2 more }` + The unique ID of the server event. - Usage statistics for the Response, this will correspond to billing. A - Realtime API session will maintain a conversation context and append new - Items to the Conversation, thus output from previous turns (text and - audio tokens) will become the input for later turns. + - `item_id: string` - - `input_token_details: optional RealtimeResponseUsageInputTokenDetails` + The ID of the user message item that will be created when speech stops. - Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. + - `type: "input_audio_buffer.speech_started"` - - `audio_tokens: optional number` + The event type, must be `input_audio_buffer.speech_started`. - The number of audio tokens used as input for the Response. + - `"input_audio_buffer.speech_started"` - - `cached_tokens: optional number` + - `InputAudioBufferSpeechStoppedEvent object { audio_end_ms, event_id, item_id, type }` - The number of cached tokens used as input for the Response. + Returned in `server_vad` mode when the server detects the end of speech in + the audio buffer. The server will also send an `conversation.item.created` + event with the user message item that is created from the audio buffer. - - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` + - `audio_end_ms: number` - Details about the cached tokens used as input for the Response. + Milliseconds since the session started when speech stopped. This will + correspond to the end of audio sent to the model, and thus includes the + `min_silence_duration_ms` configured in the Session. - - `audio_tokens: optional number` + - `event_id: string` - The number of cached audio tokens used as input for the Response. + The unique ID of the server event. - - `image_tokens: optional number` + - `item_id: string` - The number of cached image tokens used as input for the Response. + The ID of the user message item that will be created. - - `text_tokens: optional number` + - `type: "input_audio_buffer.speech_stopped"` - The number of cached text tokens used as input for the Response. + The event type, must be `input_audio_buffer.speech_stopped`. - - `image_tokens: optional number` + - `"input_audio_buffer.speech_stopped"` - The number of image tokens used as input for the Response. + - `RateLimitsUpdatedEvent object { event_id, rate_limits, type }` - - `text_tokens: optional number` + Emitted at the beginning of a Response to indicate the updated rate limits. + When a Response is created some tokens will be "reserved" for the output + tokens, the rate limits shown here reflect that reservation, which is then + adjusted accordingly once the Response is completed. - The number of text tokens used as input for the Response. + - `event_id: string` - - `input_tokens: optional number` + The unique ID of the server event. - The number of input tokens used in the Response, including text and - audio tokens. + - `rate_limits: array of object { limit, name, remaining, reset_seconds }` - - `output_token_details: optional RealtimeResponseUsageOutputTokenDetails` + List of rate limit information. - Details about the output tokens used in the Response. + - `limit: optional number` - - `audio_tokens: optional number` + The maximum allowed value for the rate limit. - The number of audio tokens used in the Response. + - `name: optional "requests" or "tokens"` - - `text_tokens: optional number` + The name of the rate limit (`requests`, `tokens`). - The number of text tokens used in the Response. + - `"requests"` - - `output_tokens: optional number` + - `"tokens"` - The number of output tokens sent in the Response, including text and - audio tokens. + - `remaining: optional number` - - `total_tokens: optional number` + The remaining value before the limit is reached. - The total number of tokens in the Response including input and output - text and audio tokens. + - `reset_seconds: optional number` -### Realtime Response Usage Input Token Details + Seconds until the rate limit resets. -- `RealtimeResponseUsageInputTokenDetails = object { audio_tokens, cached_tokens, cached_tokens_details, 2 more }` + - `type: "rate_limits.updated"` - Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. + The event type, must be `rate_limits.updated`. - - `audio_tokens: optional number` + - `"rate_limits.updated"` - The number of audio tokens used as input for the Response. + - `ResponseAudioDeltaEvent object { content_index, delta, event_id, 4 more }` - - `cached_tokens: optional number` + Returned when the model-generated audio is updated. - The number of cached tokens used as input for the Response. + - `content_index: number` - - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` + The index of the content part in the item's content array. - Details about the cached tokens used as input for the Response. + - `delta: string` - - `audio_tokens: optional number` + Base64-encoded audio data delta. - The number of cached audio tokens used as input for the Response. + - `event_id: string` - - `image_tokens: optional number` + The unique ID of the server event. - The number of cached image tokens used as input for the Response. + - `item_id: string` - - `text_tokens: optional number` + The ID of the item. - The number of cached text tokens used as input for the Response. + - `output_index: number` - - `image_tokens: optional number` + The index of the output item in the response. - The number of image tokens used as input for the Response. + - `response_id: string` - - `text_tokens: optional number` + The ID of the response. - The number of text tokens used as input for the Response. + - `type: "response.output_audio.delta"` -### Realtime Response Usage Output Token Details + The event type, must be `response.output_audio.delta`. -- `RealtimeResponseUsageOutputTokenDetails = object { audio_tokens, text_tokens }` + - `"response.output_audio.delta"` - Details about the output tokens used in the Response. + - `ResponseAudioDoneEvent object { content_index, event_id, item_id, 3 more }` - - `audio_tokens: optional number` + Returned when the model-generated audio is done. Also emitted when a Response + is interrupted, incomplete, or cancelled. - The number of audio tokens used in the Response. + - `content_index: number` - - `text_tokens: optional number` + The index of the content part in the item's content array. - The number of text tokens used in the Response. + - `event_id: string` -### Realtime Server Event + The unique ID of the server event. -- `RealtimeServerEvent = ConversationCreatedEvent or ConversationItemCreatedEvent or ConversationItemDeletedEvent or 43 more` + - `item_id: string` - A realtime server event. + The ID of the item. - - `ConversationCreatedEvent = object { conversation, event_id, type }` + - `output_index: number` - Returned when a conversation is created. Emitted right after session creation. + The index of the output item in the response. - - `conversation: object { id, object }` + - `response_id: string` - The conversation resource. + The ID of the response. - - `id: optional string` + - `type: "response.output_audio.done"` - The unique ID of the conversation. + The event type, must be `response.output_audio.done`. - - `object: optional string` + - `"response.output_audio.done"` - The object type, must be `realtime.conversation`. + - `ResponseAudioTranscriptDeltaEvent object { content_index, delta, event_id, 4 more }` - - `event_id: string` - - The unique ID of the server event. - - - `type: "conversation.created"` - - The event type, must be `conversation.created`. + Returned when the model-generated transcription of audio output is updated. - - `"conversation.created"` + - `content_index: number` - - `ConversationItemCreatedEvent = object { event_id, item, type, previous_item_id }` + The index of the content part in the item's content array. - Returned when a conversation item is created. There are several scenarios that produce this event: + - `delta: string` - - The server is generating a Response, which if successful will produce - either one or two Items, which will be of type `message` - (role `assistant`) or type `function_call`. - - The input audio buffer has been committed, either by the client or the - server (in `server_vad` mode). The server will take the content of the - input audio buffer and add it to a new user message Item. - - The client has sent a `conversation.item.create` event to add a new Item - to the Conversation. + The transcript delta. - `event_id: string` The unique ID of the server event. - - `item: ConversationItem` + - `item_id: string` - A single item within a Realtime conversation. + The ID of the item. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `output_index: number` - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + The index of the output item in the response. - - `content: array of object { text, type }` + - `response_id: string` - The content of the message. + The ID of the response. - - `text: optional string` + - `type: "response.output_audio_transcript.delta"` - The text content. + The event type, must be `response.output_audio_transcript.delta`. - - `type: optional "input_text"` + - `"response.output_audio_transcript.delta"` - The content type. Always `input_text` for system messages. + - `ResponseAudioTranscriptDoneEvent object { content_index, event_id, item_id, 4 more }` - - `"input_text"` + Returned when the model-generated transcription of audio output is done + streaming. Also emitted when a Response is interrupted, incomplete, or + cancelled. - - `role: "system"` + - `content_index: number` - The role of the message sender. Always `system`. + The index of the content part in the item's content array. - - `"system"` + - `event_id: string` - - `type: "message"` + The unique ID of the server event. - The type of the item. Always `message`. + - `item_id: string` - - `"message"` + The ID of the item. - - `id: optional string` + - `output_index: number` - The unique ID of the item. This may be provided by the client or generated by the server. + The index of the output item in the response. - - `object: optional "realtime.item"` + - `response_id: string` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + The ID of the response. - - `"realtime.item"` + - `transcript: string` - - `status: optional "completed" or "incomplete" or "in_progress"` + The final transcript of the audio. - The status of the item. Has no effect on the conversation. + - `type: "response.output_audio_transcript.done"` - - `"completed"` + The event type, must be `response.output_audio_transcript.done`. - - `"incomplete"` + - `"response.output_audio_transcript.done"` - - `"in_progress"` + - `ResponseContentPartAddedEvent object { content_index, event_id, item_id, 4 more }` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + Returned when a new content part is added to an assistant message item during + response generation. - A user message item in a Realtime conversation. + - `content_index: number` - - `content: array of object { audio, detail, image_url, 3 more }` + The index of the content part in the item's content array. - The content of the message. + - `event_id: string` - - `audio: optional string` + The unique ID of the server event. - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `item_id: string` - - `detail: optional "auto" or "low" or "high"` + The ID of the item to which the content part was added. - The detail level of the image (for `input_image`). `auto` will default to `high`. + - `output_index: number` - - `"auto"` + The index of the output item in the response. - - `"low"` + - `part: object { audio, text, transcript, type }` - - `"high"` + The content part that was added. - - `image_url: optional string` + - `audio: optional string` - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + Base64-encoded audio data (if type is "audio"). - `text: optional string` - The text content (for `input_text`). + The text content (if type is "text"). - `transcript: optional string` - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. - - - `type: optional "input_text" or "input_audio" or "input_image"` - - The content type (`input_text`, `input_audio`, or `input_image`). - - - `"input_text"` - - - `"input_audio"` + The transcript of the audio (if type is "audio"). - - `"input_image"` + - `type: optional "audio" or "text"` - - `role: "user"` + The content type ("text", "audio"). - The role of the message sender. Always `user`. + - `"audio"` - - `"user"` + - `"text"` - - `type: "message"` + - `response_id: string` - The type of the item. Always `message`. + The ID of the response. - - `"message"` + - `type: "response.content_part.added"` - - `id: optional string` + The event type, must be `response.content_part.added`. - The unique ID of the item. This may be provided by the client or generated by the server. + - `"response.content_part.added"` - - `object: optional "realtime.item"` + - `ResponseContentPartDoneEvent object { content_index, event_id, item_id, 4 more }` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Returned when a content part is done streaming in an assistant message item. + Also emitted when a Response is interrupted, incomplete, or cancelled. - - `"realtime.item"` + - `content_index: number` - - `status: optional "completed" or "incomplete" or "in_progress"` + The index of the content part in the item's content array. - The status of the item. Has no effect on the conversation. + - `event_id: string` - - `"completed"` + The unique ID of the server event. - - `"incomplete"` + - `item_id: string` - - `"in_progress"` + The ID of the item. - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `output_index: number` - An assistant message item in a Realtime conversation. + The index of the output item in the response. - - `content: array of object { audio, text, transcript, type }` + - `part: object { audio, text, transcript, type }` - The content of the message. + The content part that is done. - `audio: optional string` - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + Base64-encoded audio data (if type is "audio"). - `text: optional string` - The text content. + The text content (if type is "text"). - `transcript: optional string` - The transcript of the audio content, this will always be present if the output type is `audio`. + The transcript of the audio (if type is "audio"). - - `type: optional "output_text" or "output_audio"` + - `type: optional "audio" or "text"` - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + The content type ("text", "audio"). - - `"output_text"` + - `"audio"` - - `"output_audio"` + - `"text"` - - `role: "assistant"` + - `response_id: string` - The role of the message sender. Always `assistant`. + The ID of the response. - - `"assistant"` + - `type: "response.content_part.done"` - - `type: "message"` + The event type, must be `response.content_part.done`. - The type of the item. Always `message`. + - `"response.content_part.done"` - - `"message"` + - `ResponseCreatedEvent object { event_id, response, type }` - - `id: optional string` + Returned when a new Response is created. The first event of response creation, + where the response is in an initial state of `in_progress`. - The unique ID of the item. This may be provided by the client or generated by the server. + - `event_id: string` - - `object: optional "realtime.item"` + The unique ID of the server event. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `response: RealtimeResponse` - - `"realtime.item"` + The response resource. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `id: optional string` - The status of the item. Has no effect on the conversation. + The unique ID of the response, will look like `resp_1234`. - - `"completed"` + - `audio: optional object { output }` - - `"incomplete"` + Configuration for audio output. - - `"in_progress"` + - `output: optional object { format, voice }` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `format: optional RealtimeAudioFormats` - A function call item in a Realtime conversation. + The format of the output audio. - - `arguments: string` + - `PCMAudioFormat object { rate, type }` - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + The PCM audio format. Only a 24kHz sample rate is supported. - - `name: string` + - `rate: optional 24000` - The name of the function being called. + The sample rate of the audio. Always `24000`. - - `type: "function_call"` + - `24000` - The type of the item. Always `function_call`. + - `type: optional "audio/pcm"` - - `"function_call"` + The audio format. Always `audio/pcm`. - - `id: optional string` + - `"audio/pcm"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `PCMUAudioFormat object { type }` - - `call_id: optional string` + The G.711 μ-law format. - The ID of the function call. + - `type: optional "audio/pcmu"` - - `object: optional "realtime.item"` + The audio format. Always `audio/pcmu`. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"audio/pcmu"` - - `"realtime.item"` + - `PCMAAudioFormat object { type }` - - `status: optional "completed" or "incomplete" or "in_progress"` + The G.711 A-law format. - The status of the item. Has no effect on the conversation. + - `type: optional "audio/pcma"` - - `"completed"` + The audio format. Always `audio/pcma`. - - `"incomplete"` + - `"audio/pcma"` - - `"in_progress"` + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + The voice the model uses to respond. Voice cannot be changed during the + session once the model has responded with audio at least once. Current + voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, + `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for + best quality. - A function call output item in a Realtime conversation. + - `string` - - `call_id: string` + - `"alloy" or "ash" or "ballad" or 7 more` - The ID of the function call this output is for. + The voice the model uses to respond. Voice cannot be changed during the + session once the model has responded with audio at least once. Current + voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, + `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for + best quality. - - `output: string` + - `"alloy"` - The output of the function call, this is free text and can contain any information or simply be empty. + - `"ash"` - - `type: "function_call_output"` + - `"ballad"` - The type of the item. Always `function_call_output`. + - `"coral"` - - `"function_call_output"` + - `"echo"` - - `id: optional string` + - `"sage"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `"shimmer"` - - `object: optional "realtime.item"` + - `"verse"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"marin"` - - `"realtime.item"` + - `"cedar"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `conversation_id: optional string` - The status of the item. Has no effect on the conversation. + Which conversation the response is added to, determined by the `conversation` + field in the `response.create` event. If `auto`, the response will be added to + the default conversation and the value of `conversation_id` will be an id like + `conv_1234`. If `none`, the response will not be added to any conversation and + the value of `conversation_id` will be `null`. If responses are being triggered + automatically by VAD the response will be added to the default conversation - - `"completed"` + - `max_output_tokens: optional number or "inf"` - - `"incomplete"` + Maximum number of output tokens for a single assistant response, + inclusive of tool calls, that was used in this response. - - `"in_progress"` + - `number` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `"inf"` - A Realtime item responding to an MCP approval request. + - `"inf"` - - `id: string` + - `metadata: optional Metadata` - The unique ID of the approval response. + Set of 16 key-value pairs that can be attached to an object. This can be + useful for storing additional information about the object in a structured + format, and querying for objects via API or the dashboard. - - `approval_request_id: string` + Keys are strings with a maximum length of 64 characters. Values are strings + with a maximum length of 512 characters. - The ID of the approval request being answered. + - `object: optional "realtime.response"` - - `approve: boolean` + The object type, must be `realtime.response`. - Whether the request was approved. + - `"realtime.response"` - - `type: "mcp_approval_response"` + - `output: optional array of ConversationItem` - The type of the item. Always `mcp_approval_response`. + The list of output items generated by the response. - - `"mcp_approval_response"` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` - - `reason: optional string` + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. - Optional reason for the decision. + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + A user message item in a Realtime conversation. - A Realtime item listing tools available on an MCP server. + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` - - `server_label: string` + An assistant message item in a Realtime conversation. - The label of the MCP server. + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` - - `tools: array of object { input_schema, name, annotations, description }` + A function call item in a Realtime conversation. - The tools available on the server. + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` - - `input_schema: unknown` + A function call output item in a Realtime conversation. - The JSON schema describing the tool's input. + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` - - `name: string` + A Realtime item responding to an MCP approval request. - The name of the tool. + - `RealtimeMcpListTools object { server_label, tools, type, id }` - - `annotations: optional unknown` + A Realtime item listing tools available on an MCP server. - Additional annotations about the tool. + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` - - `description: optional string` + A Realtime item representing an invocation of a tool on an MCP server. - The description of the tool. + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` - - `type: "mcp_list_tools"` + A Realtime item requesting human approval of a tool invocation. - The type of the item. Always `mcp_list_tools`. + - `output_modalities: optional array of "text" or "audio"` - - `"mcp_list_tools"` + The set of modalities the model used to respond, currently the only possible values are + `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the + output to mode `text` will disable audio output from the model. - - `id: optional string` + - `"text"` - The unique ID of the list. + - `"audio"` - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `status: optional "completed" or "cancelled" or "failed" or 2 more` - A Realtime item representing an invocation of a tool on an MCP server. + The final status of the response (`completed`, `cancelled`, `failed`, or + `incomplete`, `in_progress`). - - `id: string` + - `"completed"` - The unique ID of the tool call. + - `"cancelled"` - - `arguments: string` + - `"failed"` - A JSON string of the arguments passed to the tool. + - `"incomplete"` - - `name: string` + - `"in_progress"` - The name of the tool that was run. + - `status_details: optional RealtimeResponseStatus` - - `server_label: string` + Additional details about the status. - The label of the MCP server running the tool. + - `error: optional object { code, type }` - - `type: "mcp_call"` + A description of the error that caused the response to fail, + populated when the `status` is `failed`. - The type of the item. Always `mcp_call`. + - `code: optional string` - - `"mcp_call"` + Error code, if any. - - `approval_request_id: optional string` + - `type: optional string` - The ID of an associated approval request, if any. + The type of error. - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` + - `reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"` - The error from the tool call, if any. + The reason the Response did not complete. For a `cancelled` Response, one of `turn_detected` (the server VAD detected a new start of speech) or `client_cancelled` (the client sent a cancel event). For an `incomplete` Response, one of `max_output_tokens` or `content_filter` (the server-side safety filter activated and cut off the response). - - `RealtimeMcpProtocolError = object { code, message, type }` + - `"turn_detected"` - - `code: number` + - `"client_cancelled"` - - `message: string` + - `"max_output_tokens"` - - `type: "protocol_error"` + - `"content_filter"` - - `"protocol_error"` + - `type: optional "completed" or "cancelled" or "failed" or "incomplete"` - - `RealtimeMcpToolExecutionError = object { message, type }` + The type of error that caused the response to fail, corresponding + with the `status` field (`completed`, `cancelled`, `incomplete`, + `failed`). - - `message: string` + - `"completed"` - - `type: "tool_execution_error"` + - `"cancelled"` - - `"tool_execution_error"` + - `"failed"` - - `RealtimeMcphttpError = object { code, message, type }` + - `"incomplete"` - - `code: number` + - `usage: optional RealtimeResponseUsage` - - `message: string` + Usage statistics for the Response, this will correspond to billing. A + Realtime API session will maintain a conversation context and append new + Items to the Conversation, thus output from previous turns (text and + audio tokens) will become the input for later turns. - - `type: "http_error"` + - `input_token_details: optional RealtimeResponseUsageInputTokenDetails` - - `"http_error"` + Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. - - `output: optional string` + - `audio_tokens: optional number` - The output from the tool call. + The number of audio tokens used as input for the Response. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `cached_tokens: optional number` - A Realtime item requesting human approval of a tool invocation. + The number of cached tokens used as input for the Response. - - `id: string` + - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` - The unique ID of the approval request. + Details about the cached tokens used as input for the Response. - - `arguments: string` + - `audio_tokens: optional number` - A JSON string of arguments for the tool. + The number of cached audio tokens used as input for the Response. - - `name: string` + - `image_tokens: optional number` - The name of the tool to run. + The number of cached image tokens used as input for the Response. - - `server_label: string` + - `text_tokens: optional number` - The label of the MCP server making the request. + The number of cached text tokens used as input for the Response. - - `type: "mcp_approval_request"` + - `image_tokens: optional number` - The type of the item. Always `mcp_approval_request`. + The number of image tokens used as input for the Response. - - `"mcp_approval_request"` + - `text_tokens: optional number` - - `type: "conversation.item.created"` + The number of text tokens used as input for the Response. - The event type, must be `conversation.item.created`. + - `input_tokens: optional number` - - `"conversation.item.created"` + The number of input tokens used in the Response, including text and + audio tokens. - - `previous_item_id: optional string` + - `output_token_details: optional RealtimeResponseUsageOutputTokenDetails` - The ID of the preceding item in the Conversation context, allows the - client to understand the order of the conversation. Can be `null` if the - item has no predecessor. + Details about the output tokens used in the Response. - - `ConversationItemDeletedEvent = object { event_id, item_id, type }` + - `audio_tokens: optional number` - Returned when an item in the conversation is deleted by the client with a - `conversation.item.delete` event. This event is used to synchronize the - server's understanding of the conversation history with the client's view. + The number of audio tokens used in the Response. - - `event_id: string` + - `text_tokens: optional number` - The unique ID of the server event. + The number of text tokens used in the Response. - - `item_id: string` + - `output_tokens: optional number` - The ID of the item that was deleted. + The number of output tokens sent in the Response, including text and + audio tokens. - - `type: "conversation.item.deleted"` + - `total_tokens: optional number` - The event type, must be `conversation.item.deleted`. + The total number of tokens in the Response including input and output + text and audio tokens. - - `"conversation.item.deleted"` + - `type: "response.created"` - - `ConversationItemInputAudioTranscriptionCompletedEvent = object { content_index, event_id, item_id, 4 more }` + The event type, must be `response.created`. - This event is the output of audio transcription for user audio written to the - user audio buffer. Transcription begins when the input audio buffer is - committed by the client or server (when VAD is enabled). Transcription runs - asynchronously with Response creation, so this event may come before or after - the Response events. + - `"response.created"` - Realtime API models accept audio natively, and thus input transcription is a - separate process run on a separate ASR (Automatic Speech Recognition) model. - The transcript may diverge somewhat from the model's interpretation, and - should be treated as a rough guide. + - `ResponseDoneEvent object { event_id, response, type }` - - `content_index: number` + Returned when a Response is done streaming. Always emitted, no matter the + final state. The Response object included in the `response.done` event will + include all output Items in the Response but will omit the raw audio data. - The index of the content part containing the audio. + Clients should check the `status` field of the Response to determine if it was successful + (`completed`) or if there was another outcome: `cancelled`, `failed`, or `incomplete`. + + A response will contain all output items that were generated during the response, excluding + any audio content. - `event_id: string` The unique ID of the server event. - - `item_id: string` + - `response: RealtimeResponse` - The ID of the item containing the audio that is being transcribed. + The response resource. - - `transcript: string` + - `type: "response.done"` - The transcribed text. + The event type, must be `response.done`. - - `type: "conversation.item.input_audio_transcription.completed"` + - `"response.done"` - The event type, must be - `conversation.item.input_audio_transcription.completed`. + - `ResponseFunctionCallArgumentsDeltaEvent object { call_id, delta, event_id, 4 more }` - - `"conversation.item.input_audio_transcription.completed"` + Returned when the model-generated function call arguments are updated. - - `usage: object { input_tokens, output_tokens, total_tokens, 2 more } or object { seconds, type }` + - `call_id: string` - Usage statistics for the transcription, this is billed according to the ASR model's pricing rather than the realtime model's pricing. + The ID of the function call. - - `TokenUsage = object { input_tokens, output_tokens, total_tokens, 2 more }` + - `delta: string` - Usage statistics for models billed by token usage. + The arguments delta as a JSON string. - - `input_tokens: number` + - `event_id: string` - Number of input tokens billed for this request. + The unique ID of the server event. - - `output_tokens: number` + - `item_id: string` - Number of output tokens generated. + The ID of the function call item. - - `total_tokens: number` + - `output_index: number` - Total number of tokens used (input + output). + The index of the output item in the response. - - `type: "tokens"` + - `response_id: string` - The type of the usage object. Always `tokens` for this variant. + The ID of the response. - - `"tokens"` + - `type: "response.function_call_arguments.delta"` - - `input_token_details: optional object { audio_tokens, text_tokens }` + The event type, must be `response.function_call_arguments.delta`. - Details about the input tokens billed for this request. + - `"response.function_call_arguments.delta"` - - `audio_tokens: optional number` + - `ResponseFunctionCallArgumentsDoneEvent object { arguments, call_id, event_id, 5 more }` - Number of audio tokens billed for this request. + Returned when the model-generated function call arguments are done streaming. + Also emitted when a Response is interrupted, incomplete, or cancelled. - - `text_tokens: optional number` + - `arguments: string` - Number of text tokens billed for this request. + The final arguments as a JSON string. - - `DurationUsage = object { seconds, type }` + - `call_id: string` - Usage statistics for models billed by audio input duration. + The ID of the function call. - - `seconds: number` + - `event_id: string` - Duration of the input audio in seconds. + The unique ID of the server event. - - `type: "duration"` + - `item_id: string` - The type of the usage object. Always `duration` for this variant. + The ID of the function call item. - - `"duration"` + - `name: string` - - `logprobs: optional array of LogProbProperties` + The name of the function that was called. - The log probabilities of the transcription. + - `output_index: number` - - `token: string` + The index of the output item in the response. - The token that was used to generate the log probability. + - `response_id: string` - - `bytes: array of number` + The ID of the response. - The bytes that were used to generate the log probability. + - `type: "response.function_call_arguments.done"` - - `logprob: number` + The event type, must be `response.function_call_arguments.done`. - The log probability of the token. + - `"response.function_call_arguments.done"` - - `ConversationItemInputAudioTranscriptionDeltaEvent = object { event_id, item_id, type, 3 more }` + - `ResponseOutputItemAddedEvent object { event_id, item, output_index, 2 more }` - Returned when the text value of an input audio transcription content part is updated with incremental transcription results. + Returned when a new Item is created during Response generation. - `event_id: string` The unique ID of the server event. - - `item_id: string` - - The ID of the item containing the audio that is being transcribed. - - - `type: "conversation.item.input_audio_transcription.delta"` - - The event type, must be `conversation.item.input_audio_transcription.delta`. + - `item: ConversationItem` - - `"conversation.item.input_audio_transcription.delta"` + A single item within a Realtime conversation. - - `content_index: optional number` + - `output_index: number` - The index of the content part in the item's content array. + The index of the output item in the Response. - - `delta: optional string` + - `response_id: string` - The text delta. + The ID of the Response to which the item belongs. - - `logprobs: optional array of LogProbProperties` + - `type: "response.output_item.added"` - The log probabilities of the transcription. These can be enabled by configurating the session with `"include": ["item.input_audio_transcription.logprobs"]`. Each entry in the array corresponds a log probability of which token would be selected for this chunk of transcription. This can help to identify if it was possible there were multiple valid options for a given chunk of transcription. + The event type, must be `response.output_item.added`. - - `token: string` + - `"response.output_item.added"` - The token that was used to generate the log probability. + - `ResponseOutputItemDoneEvent object { event_id, item, output_index, 2 more }` - - `bytes: array of number` + Returned when an Item is done streaming. Also emitted when a Response is + interrupted, incomplete, or cancelled. - The bytes that were used to generate the log probability. + - `event_id: string` - - `logprob: number` + The unique ID of the server event. - The log probability of the token. + - `item: ConversationItem` - - `ConversationItemInputAudioTranscriptionFailedEvent = object { content_index, error, event_id, 2 more }` + A single item within a Realtime conversation. - Returned when input audio transcription is configured, and a transcription - request for a user message failed. These events are separate from other - `error` events so that the client can identify the related Item. + - `output_index: number` - - `content_index: number` + The index of the output item in the Response. - The index of the content part containing the audio. + - `response_id: string` - - `error: object { code, message, param, type }` + The ID of the Response to which the item belongs. - Details of the transcription error. + - `type: "response.output_item.done"` - - `code: optional string` + The event type, must be `response.output_item.done`. - Error code, if any. + - `"response.output_item.done"` - - `message: optional string` + - `ResponseTextDeltaEvent object { content_index, delta, event_id, 4 more }` - A human-readable error message. + Returned when the text value of an "output_text" content part is updated. - - `param: optional string` + - `content_index: number` - Parameter related to the error, if any. + The index of the content part in the item's content array. - - `type: optional string` + - `delta: string` - The type of error. + The text delta. - `event_id: string` @@ -9588,1717 +9873,1764 @@ - `item_id: string` - The ID of the user message item. - - - `type: "conversation.item.input_audio_transcription.failed"` + The ID of the item. - The event type, must be - `conversation.item.input_audio_transcription.failed`. + - `output_index: number` - - `"conversation.item.input_audio_transcription.failed"` + The index of the output item in the response. - - `ConversationItemRetrieved = object { event_id, item, type }` + - `response_id: string` - Returned when a conversation item is retrieved with `conversation.item.retrieve`. This is provided as a way to fetch the server's representation of an item, for example to get access to the post-processed audio data after noise cancellation and VAD. It includes the full content of the Item, including audio data. + The ID of the response. - - `event_id: string` + - `type: "response.output_text.delta"` - The unique ID of the server event. + The event type, must be `response.output_text.delta`. - - `item: ConversationItem` + - `"response.output_text.delta"` - A single item within a Realtime conversation. + - `ResponseTextDoneEvent object { content_index, event_id, item_id, 4 more }` - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + Returned when the text value of an "output_text" content part is done streaming. Also + emitted when a Response is interrupted, incomplete, or cancelled. - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + - `content_index: number` - - `content: array of object { text, type }` + The index of the content part in the item's content array. - The content of the message. + - `event_id: string` - - `text: optional string` + The unique ID of the server event. - The text content. + - `item_id: string` - - `type: optional "input_text"` + The ID of the item. - The content type. Always `input_text` for system messages. + - `output_index: number` - - `"input_text"` + The index of the output item in the response. - - `role: "system"` + - `response_id: string` - The role of the message sender. Always `system`. + The ID of the response. - - `"system"` + - `text: string` - - `type: "message"` + The final text content. - The type of the item. Always `message`. + - `type: "response.output_text.done"` - - `"message"` + The event type, must be `response.output_text.done`. - - `id: optional string` + - `"response.output_text.done"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `SessionCreatedEvent object { event_id, session, type }` - - `object: optional "realtime.item"` + Returned when a Session is created. Emitted automatically when a new + connection is established as the first server event. This event will contain + the default Session configuration. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `event_id: string` - - `"realtime.item"` + The unique ID of the server event. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `session: RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` - The status of the item. Has no effect on the conversation. + The session configuration. - - `"completed"` + - `RealtimeSessionCreateRequest object { type, audio, include, 9 more }` - - `"incomplete"` + Realtime session object configuration. - - `"in_progress"` + - `type: "realtime"` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + The type of session to create. Always `realtime` for the Realtime API. - A user message item in a Realtime conversation. + - `"realtime"` - - `content: array of object { audio, detail, image_url, 3 more }` + - `audio: optional RealtimeAudioConfig` - The content of the message. + Configuration for input and output audio. - - `audio: optional string` + - `input: optional RealtimeAudioConfigInput` - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `format: optional RealtimeAudioFormats` - - `detail: optional "auto" or "low" or "high"` + The format of the input audio. - The detail level of the image (for `input_image`). `auto` will default to `high`. + - `noise_reduction: optional object { type }` - - `"auto"` + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `"low"` + - `type: optional NoiseReductionType` - - `"high"` + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - `image_url: optional string` + - `"near_field"` - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + - `"far_field"` - - `text: optional string` + - `transcription: optional AudioTranscription` - The text content (for `input_text`). + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - `transcript: optional string` + - `language: optional string` - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - - `type: optional "input_text" or "input_audio" or "input_image"` + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - The content type (`input_text`, `input_audio`, or `input_image`). + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `"input_text"` + - `string` - - `"input_audio"` + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `"input_image"` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `role: "user"` + - `"whisper-1"` - The role of the message sender. Always `user`. + - `"gpt-4o-mini-transcribe"` - - `"user"` + - `"gpt-4o-mini-transcribe-2025-12-15"` - - `type: "message"` + - `"gpt-4o-transcribe"` - The type of the item. Always `message`. + - `"gpt-4o-transcribe-diarize"` - - `"message"` + - `prompt: optional string` - - `id: optional string` + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - The unique ID of the item. This may be provided by the client or generated by the server. + - `turn_detection: optional RealtimeAudioInputTurnDetection` - - `object: optional "realtime.item"` + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - `"realtime.item"` + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - The status of the item. Has no effect on the conversation. + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - `"completed"` + - `type: "server_vad"` - - `"incomplete"` + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `"in_progress"` + - `"server_vad"` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `create_response: optional boolean` - An assistant message item in a Realtime conversation. + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - `content: array of object { audio, text, transcript, type }` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - The content of the message. + - `idle_timeout_ms: optional number` - - `audio: optional string` + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - - `text: optional string` + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - The text content. + - `interrupt_response: optional boolean` - - `transcript: optional string` + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - The transcript of the audio content, this will always be present if the output type is `audio`. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `type: optional "output_text" or "output_audio"` + - `prefix_padding_ms: optional number` - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - - `"output_text"` + - `silence_duration_ms: optional number` - - `"output_audio"` + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - - `role: "assistant"` + - `threshold: optional number` - The role of the message sender. Always `assistant`. + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - - `"assistant"` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - - `type: "message"` + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - The type of the item. Always `message`. + - `type: "semantic_vad"` - - `"message"` + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - `id: optional string` + - `"semantic_vad"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `create_response: optional boolean` - - `object: optional "realtime.item"` + Whether or not to automatically generate a response when a VAD stop event occurs. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `eagerness: optional "low" or "medium" or "high" or "auto"` - - `"realtime.item"` + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"low"` - The status of the item. Has no effect on the conversation. + - `"medium"` - - `"completed"` + - `"high"` - - `"incomplete"` + - `"auto"` - - `"in_progress"` + - `interrupt_response: optional boolean` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - A function call item in a Realtime conversation. + - `output: optional RealtimeAudioConfigOutput` - - `arguments: string` + - `format: optional RealtimeAudioFormats` - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + The format of the output audio. - - `name: string` + - `speed: optional number` - The name of the function being called. + The speed of the model's spoken response as a multiple of the original speed. + 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - - `type: "function_call"` + This parameter is a post-processing adjustment to the audio after it is generated, it's + also possible to prompt the model to speak faster or slower. - The type of the item. Always `function_call`. + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - - `"function_call"` + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. - - `id: optional string` + - `string` - The unique ID of the item. This may be provided by the client or generated by the server. + - `"alloy" or "ash" or "ballad" or 7 more` - - `call_id: optional string` + - `"alloy"` - The ID of the function call. + - `"ash"` - - `object: optional "realtime.item"` + - `"ballad"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"coral"` - - `"realtime.item"` + - `"echo"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"sage"` - The status of the item. Has no effect on the conversation. + - `"shimmer"` - - `"completed"` + - `"verse"` - - `"incomplete"` + - `"marin"` - - `"in_progress"` + - `"cedar"` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `ID object { id }` - A function call output item in a Realtime conversation. + Custom voice reference. - - `call_id: string` + - `id: string` - The ID of the function call this output is for. + The custom voice ID, e.g. `voice_1234`. - - `output: string` + - `include: optional array of "item.input_audio_transcription.logprobs"` - The output of the function call, this is free text and can contain any information or simply be empty. + Additional fields to include in server outputs. - - `type: "function_call_output"` + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - The type of the item. Always `function_call_output`. + - `"item.input_audio_transcription.logprobs"` - - `"function_call_output"` + - `instructions: optional string` - - `id: optional string` + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - The unique ID of the item. This may be provided by the client or generated by the server. + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - `object: optional "realtime.item"` + - `max_output_tokens: optional number or "inf"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - - `"realtime.item"` + - `number` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"inf"` - The status of the item. Has no effect on the conversation. + - `"inf"` - - `"completed"` + - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - `"incomplete"` + The Realtime model used for this session. - - `"in_progress"` + - `string` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - A Realtime item responding to an MCP approval request. + The Realtime model used for this session. - - `id: string` + - `"gpt-realtime"` - The unique ID of the approval response. + - `"gpt-realtime-1.5"` - - `approval_request_id: string` + - `"gpt-realtime-2025-08-28"` - The ID of the approval request being answered. + - `"gpt-4o-realtime-preview"` - - `approve: boolean` + - `"gpt-4o-realtime-preview-2024-10-01"` - Whether the request was approved. + - `"gpt-4o-realtime-preview-2024-12-17"` - - `type: "mcp_approval_response"` + - `"gpt-4o-realtime-preview-2025-06-03"` - The type of the item. Always `mcp_approval_response`. + - `"gpt-4o-mini-realtime-preview"` - - `"mcp_approval_response"` + - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - `reason: optional string` + - `"gpt-realtime-mini"` - Optional reason for the decision. + - `"gpt-realtime-mini-2025-10-06"` - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `"gpt-realtime-mini-2025-12-15"` - A Realtime item listing tools available on an MCP server. + - `"gpt-audio-1.5"` - - `server_label: string` + - `"gpt-audio-mini"` - The label of the MCP server. + - `"gpt-audio-mini-2025-10-06"` - - `tools: array of object { input_schema, name, annotations, description }` + - `"gpt-audio-mini-2025-12-15"` - The tools available on the server. + - `output_modalities: optional array of "text" or "audio"` - - `input_schema: unknown` + The set of modalities the model can respond with. It defaults to `["audio"]`, indicating + that the model will respond with audio plus a transcript. `["text"]` can be used to make + the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - The JSON schema describing the tool's input. + - `"text"` - - `name: string` + - `"audio"` - The name of the tool. + - `prompt: optional ResponsePrompt` - - `annotations: optional unknown` + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - Additional annotations about the tool. + - `id: string` - - `description: optional string` + The unique identifier of the prompt template to use. - The description of the tool. + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - `type: "mcp_list_tools"` + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - The type of the item. Always `mcp_list_tools`. + - `string` - - `"mcp_list_tools"` + - `ResponseInputText object { text, type }` - - `id: optional string` + A text input to the model. - The unique ID of the list. + - `text: string` - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + The text input to the model. - A Realtime item representing an invocation of a tool on an MCP server. + - `type: "input_text"` - - `id: string` + The type of the input item. Always `input_text`. - The unique ID of the tool call. + - `"input_text"` - - `arguments: string` + - `ResponseInputImage object { detail, type, file_id, image_url }` - A JSON string of the arguments passed to the tool. + An image input to the model. Learn about [image inputs](/docs/guides/vision). - - `name: string` + - `detail: "low" or "high" or "auto" or "original"` - The name of the tool that was run. + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - `server_label: string` + - `"low"` - The label of the MCP server running the tool. + - `"high"` - - `type: "mcp_call"` + - `"auto"` - The type of the item. Always `mcp_call`. + - `"original"` - - `"mcp_call"` + - `type: "input_image"` - - `approval_request_id: optional string` + The type of the input item. Always `input_image`. - The ID of an associated approval request, if any. + - `"input_image"` - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` + - `file_id: optional string` - The error from the tool call, if any. + The ID of the file to be sent to the model. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `image_url: optional string` - - `code: number` + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - `message: string` + - `ResponseInputFile object { type, detail, file_data, 3 more }` - - `type: "protocol_error"` + A file input to the model. - - `"protocol_error"` + - `type: "input_file"` - - `RealtimeMcpToolExecutionError = object { message, type }` + The type of the input item. Always `input_file`. - - `message: string` + - `"input_file"` - - `type: "tool_execution_error"` + - `detail: optional "low" or "high"` - - `"tool_execution_error"` + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - `RealtimeMcphttpError = object { code, message, type }` + - `"low"` - - `code: number` + - `"high"` - - `message: string` + - `file_data: optional string` - - `type: "http_error"` + The content of the file to be sent to the model. - - `"http_error"` + - `file_id: optional string` - - `output: optional string` + The ID of the file to be sent to the model. - The output from the tool call. + - `file_url: optional string` - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + The URL of the file to be sent to the model. - A Realtime item requesting human approval of a tool invocation. + - `filename: optional string` - - `id: string` + The name of the file to be sent to the model. - The unique ID of the approval request. + - `version: optional string` - - `arguments: string` + Optional version of the prompt template. - A JSON string of arguments for the tool. + - `tool_choice: optional RealtimeToolChoiceConfig` - - `name: string` + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. - The name of the tool to run. + - `ToolChoiceOptions = "none" or "auto" or "required"` - - `server_label: string` + Controls which (if any) tool is called by the model. - The label of the MCP server making the request. + `none` means the model will not call any tool and instead generates a message. - - `type: "mcp_approval_request"` + `auto` means the model can pick between generating a message or calling one or + more tools. - The type of the item. Always `mcp_approval_request`. + `required` means the model must call one or more tools. - - `"mcp_approval_request"` + - `"none"` - - `type: "conversation.item.retrieved"` + - `"auto"` - The event type, must be `conversation.item.retrieved`. + - `"required"` - - `"conversation.item.retrieved"` + - `ToolChoiceFunction object { name, type }` - - `ConversationItemTruncatedEvent = object { audio_end_ms, content_index, event_id, 2 more }` + Use this option to force the model to call a specific function. - Returned when an earlier assistant audio message item is truncated by the - client with a `conversation.item.truncate` event. This event is used to - synchronize the server's understanding of the audio with the client's playback. + - `name: string` - This action will truncate the audio and remove the server-side text transcript - to ensure there is no text in the context that hasn't been heard by the user. + The name of the function to call. - - `audio_end_ms: number` + - `type: "function"` - The duration up to which the audio was truncated, in milliseconds. + For function calling, the type is always `function`. - - `content_index: number` + - `"function"` - The index of the content part that was truncated. + - `ToolChoiceMcp object { server_label, type, name }` - - `event_id: string` + Use this option to force the model to call a specific tool on a remote MCP server. - The unique ID of the server event. + - `server_label: string` - - `item_id: string` + The label of the MCP server to use. - The ID of the assistant message item that was truncated. + - `type: "mcp"` - - `type: "conversation.item.truncated"` + For MCP tools, the type is always `mcp`. - The event type, must be `conversation.item.truncated`. + - `"mcp"` - - `"conversation.item.truncated"` + - `name: optional string` - - `RealtimeErrorEvent = object { error, event_id, type }` + The name of the tool to call on the server. - Returned when an error occurs, which could be a client problem or a server - problem. Most errors are recoverable and the session will stay open, we - recommend to implementors to monitor and log error messages by default. + - `tools: optional RealtimeToolsConfig` - - `error: RealtimeError` + Tools available to the model. - Details of the error. + - `RealtimeFunctionTool object { description, name, parameters, type }` - - `message: string` + - `description: optional string` - A human-readable error message. + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - - `type: string` + - `name: optional string` - The type of error (e.g., "invalid_request_error", "server_error"). + The name of the function. - - `code: optional string` + - `parameters: optional unknown` - Error code, if any. + Parameters of the function in JSON Schema. - - `event_id: optional string` + - `type: optional "function"` - The event_id of the client event that caused the error, if applicable. + The type of the tool, i.e. `function`. - - `param: optional string` + - `"function"` - Parameter related to the error, if any. + - `McpTool object { server_label, type, allowed_tools, 7 more }` - - `event_id: string` + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - The unique ID of the server event. + - `server_label: string` - - `type: "error"` + A label for this MCP server, used to identify it in tool calls. - The event type, must be `error`. + - `type: "mcp"` - - `"error"` + The type of the MCP tool. Always `mcp`. - - `InputAudioBufferClearedEvent = object { event_id, type }` + - `"mcp"` - Returned when the input audio buffer is cleared by the client with a - `input_audio_buffer.clear` event. + - `allowed_tools: optional array of string or object { read_only, tool_names }` - - `event_id: string` + List of allowed tool names or a filter object. - The unique ID of the server event. + - `McpAllowedTools = array of string` - - `type: "input_audio_buffer.cleared"` + A string array of allowed tool names - The event type, must be `input_audio_buffer.cleared`. + - `McpToolFilter object { read_only, tool_names }` - - `"input_audio_buffer.cleared"` + A filter object to specify which tools are allowed. - - `InputAudioBufferCommittedEvent = object { event_id, item_id, type, previous_item_id }` + - `read_only: optional boolean` - Returned when an input audio buffer is committed, either by the client or - automatically in server VAD mode. The `item_id` property is the ID of the user - message item that will be created, thus a `conversation.item.created` event - will also be sent to the client. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `event_id: string` + - `tool_names: optional array of string` - The unique ID of the server event. + List of allowed tool names. - - `item_id: string` + - `authorization: optional string` - The ID of the user message item that will be created. + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. - - `type: "input_audio_buffer.committed"` + - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - The event type, must be `input_audio_buffer.committed`. + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](/docs/guides/tools-remote-mcp#connectors). - - `"input_audio_buffer.committed"` + Currently supported `connector_id` values are: - - `previous_item_id: optional string` + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` - The ID of the preceding item after which the new item will be inserted. - Can be `null` if the item has no predecessor. + - `"connector_dropbox"` - - `InputAudioBufferDtmfEventReceivedEvent = object { event, received_at, type }` + - `"connector_gmail"` - **SIP Only:** Returned when an DTMF event is received. A DTMF event is a message that - represents a telephone keypad press (0–9, *, #, A–D). The `event` property - is the keypad that the user press. The `received_at` is the UTC Unix Timestamp - that the server received the event. + - `"connector_googlecalendar"` - - `event: string` + - `"connector_googledrive"` - The telephone keypad that was pressed by the user. + - `"connector_microsoftteams"` - - `received_at: number` + - `"connector_outlookcalendar"` - UTC Unix Timestamp when DTMF Event was received by server. + - `"connector_outlookemail"` - - `type: "input_audio_buffer.dtmf_event_received"` + - `"connector_sharepoint"` - The event type, must be `input_audio_buffer.dtmf_event_received`. + - `defer_loading: optional boolean` - - `"input_audio_buffer.dtmf_event_received"` + Whether this MCP tool is deferred and discovered via tool search. - - `InputAudioBufferSpeechStartedEvent = object { audio_start_ms, event_id, item_id, type }` + - `headers: optional map[string]` - Sent by the server when in `server_vad` mode to indicate that speech has been - detected in the audio buffer. This can happen any time audio is added to the - buffer (unless speech is already detected). The client may want to use this - event to interrupt audio playback or provide visual feedback to the user. + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. - The client should expect to receive a `input_audio_buffer.speech_stopped` event - when speech stops. The `item_id` property is the ID of the user message item - that will be created when speech stops and will also be included in the - `input_audio_buffer.speech_stopped` event (unless the client manually commits - the audio buffer during VAD activation). + - `require_approval: optional object { always, never } or "always" or "never"` - - `audio_start_ms: number` + Specify which of the MCP server's tools require approval. - Milliseconds from the start of all audio written to the buffer during the - session when speech was first detected. This will correspond to the - beginning of audio sent to the model, and thus includes the - `prefix_padding_ms` configured in the Session. + - `McpToolApprovalFilter object { always, never }` - - `event_id: string` + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. - The unique ID of the server event. + - `always: optional object { read_only, tool_names }` - - `item_id: string` + A filter object to specify which tools are allowed. - The ID of the user message item that will be created when speech stops. + - `read_only: optional boolean` - - `type: "input_audio_buffer.speech_started"` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - The event type, must be `input_audio_buffer.speech_started`. + - `tool_names: optional array of string` - - `"input_audio_buffer.speech_started"` + List of allowed tool names. - - `InputAudioBufferSpeechStoppedEvent = object { audio_end_ms, event_id, item_id, type }` + - `never: optional object { read_only, tool_names }` - Returned in `server_vad` mode when the server detects the end of speech in - the audio buffer. The server will also send an `conversation.item.created` - event with the user message item that is created from the audio buffer. + A filter object to specify which tools are allowed. - - `audio_end_ms: number` + - `read_only: optional boolean` - Milliseconds since the session started when speech stopped. This will - correspond to the end of audio sent to the model, and thus includes the - `min_silence_duration_ms` configured in the Session. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `event_id: string` + - `tool_names: optional array of string` - The unique ID of the server event. + List of allowed tool names. - - `item_id: string` + - `McpToolApprovalSetting = "always" or "never"` - The ID of the user message item that will be created. + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. - - `type: "input_audio_buffer.speech_stopped"` + - `"always"` - The event type, must be `input_audio_buffer.speech_stopped`. + - `"never"` - - `"input_audio_buffer.speech_stopped"` + - `server_description: optional string` - - `RateLimitsUpdatedEvent = object { event_id, rate_limits, type }` + Optional description of the MCP server, used to provide more context. - Emitted at the beginning of a Response to indicate the updated rate limits. - When a Response is created some tokens will be "reserved" for the output - tokens, the rate limits shown here reflect that reservation, which is then - adjusted accordingly once the Response is completed. + - `server_url: optional string` - - `event_id: string` + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. - The unique ID of the server event. + - `tracing: optional RealtimeTracingConfig` - - `rate_limits: array of object { limit, name, remaining, reset_seconds }` + Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once + tracing is enabled for a session, the configuration cannot be modified. - List of rate limit information. + `auto` will create a trace for the session with default values for the + workflow name, group id, and metadata. - - `limit: optional number` + - `Auto = "auto"` - The maximum allowed value for the rate limit. + Enables tracing and sets default values for tracing configuration options. Always `auto`. - - `name: optional "requests" or "tokens"` + - `"auto"` - The name of the rate limit (`requests`, `tokens`). + - `TracingConfiguration object { group_id, metadata, workflow_name }` - - `"requests"` + Granular configuration for tracing. - - `"tokens"` + - `group_id: optional string` - - `remaining: optional number` + The group id to attach to this trace to enable filtering and + grouping in the Traces Dashboard. - The remaining value before the limit is reached. + - `metadata: optional unknown` - - `reset_seconds: optional number` + The arbitrary metadata to attach to this trace to enable + filtering in the Traces Dashboard. - Seconds until the rate limit resets. + - `workflow_name: optional string` - - `type: "rate_limits.updated"` + The name of the workflow to attach to this trace. This is used to + name the trace in the Traces Dashboard. - The event type, must be `rate_limits.updated`. + - `truncation: optional RealtimeTruncation` - - `"rate_limits.updated"` + When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - - `ResponseAudioDeltaEvent = object { content_index, delta, event_id, 4 more }` + Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - Returned when the model-generated audio is updated. + Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - `content_index: number` + Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - The index of the content part in the item's content array. + - `"auto" or "disabled"` - - `delta: string` + The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - Base64-encoded audio data delta. + - `"auto"` - - `event_id: string` + - `"disabled"` - The unique ID of the server event. + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` - - `item_id: string` + Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - The ID of the item. + - `retention_ratio: number` - - `output_index: number` + Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - The index of the output item in the response. + - `type: "retention_ratio"` - - `response_id: string` + Use retention ratio truncation. - The ID of the response. + - `"retention_ratio"` - - `type: "response.output_audio.delta"` + - `token_limits: optional object { post_instructions }` - The event type, must be `response.output_audio.delta`. + Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - - `"response.output_audio.delta"` + - `post_instructions: optional number` - - `ResponseAudioDoneEvent = object { content_index, event_id, item_id, 3 more }` + Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - Returned when the model-generated audio is done. Also emitted when a Response - is interrupted, incomplete, or cancelled. + - `RealtimeTranscriptionSessionCreateRequest object { type, audio, include }` - - `content_index: number` + Realtime transcription session object configuration. - The index of the content part in the item's content array. + - `type: "transcription"` - - `event_id: string` + The type of session to create. Always `transcription` for transcription sessions. - The unique ID of the server event. + - `"transcription"` - - `item_id: string` + - `audio: optional RealtimeTranscriptionSessionAudio` - The ID of the item. + Configuration for input and output audio. - - `output_index: number` + - `input: optional RealtimeTranscriptionSessionAudioInput` - The index of the output item in the response. + - `format: optional RealtimeAudioFormats` - - `response_id: string` + The PCM audio format. Only a 24kHz sample rate is supported. - The ID of the response. + - `noise_reduction: optional object { type }` - - `type: "response.output_audio.done"` + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - The event type, must be `response.output_audio.done`. + - `type: optional NoiseReductionType` - - `"response.output_audio.done"` + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - `ResponseAudioTranscriptDeltaEvent = object { content_index, delta, event_id, 4 more }` + - `transcription: optional AudioTranscription` - Returned when the model-generated transcription of audio output is updated. + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - `content_index: number` + - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` - The index of the content part in the item's content array. + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - `delta: string` + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - The transcript delta. + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `event_id: string` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - The unique ID of the server event. + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - `item_id: string` + - `type: "server_vad"` - The ID of the item. + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `output_index: number` + - `"server_vad"` - The index of the output item in the response. + - `create_response: optional boolean` - - `response_id: string` + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - The ID of the response. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `type: "response.output_audio_transcript.delta"` + - `idle_timeout_ms: optional number` - The event type, must be `response.output_audio_transcript.delta`. + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - - `"response.output_audio_transcript.delta"` + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - - `ResponseAudioTranscriptDoneEvent = object { content_index, event_id, item_id, 4 more }` + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - Returned when the model-generated transcription of audio output is done - streaming. Also emitted when a Response is interrupted, incomplete, or - cancelled. + - `interrupt_response: optional boolean` - - `content_index: number` + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - The index of the content part in the item's content array. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `event_id: string` + - `prefix_padding_ms: optional number` - The unique ID of the server event. + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - - `item_id: string` + - `silence_duration_ms: optional number` - The ID of the item. + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - - `output_index: number` + - `threshold: optional number` - The index of the output item in the response. + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - - `response_id: string` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - The ID of the response. + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - `transcript: string` + - `type: "semantic_vad"` - The final transcript of the audio. + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - `type: "response.output_audio_transcript.done"` + - `"semantic_vad"` - The event type, must be `response.output_audio_transcript.done`. + - `create_response: optional boolean` - - `"response.output_audio_transcript.done"` + Whether or not to automatically generate a response when a VAD stop event occurs. - - `ResponseContentPartAddedEvent = object { content_index, event_id, item_id, 4 more }` + - `eagerness: optional "low" or "medium" or "high" or "auto"` - Returned when a new content part is added to an assistant message item during - response generation. + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - `content_index: number` + - `"low"` - The index of the content part in the item's content array. + - `"medium"` - - `event_id: string` + - `"high"` - The unique ID of the server event. + - `"auto"` - - `item_id: string` + - `interrupt_response: optional boolean` - The ID of the item to which the content part was added. + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - `output_index: number` + - `include: optional array of "item.input_audio_transcription.logprobs"` - The index of the output item in the response. + Additional fields to include in server outputs. - - `part: object { audio, text, transcript, type }` + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - The content part that was added. + - `"item.input_audio_transcription.logprobs"` - - `audio: optional string` + - `type: "session.created"` - Base64-encoded audio data (if type is "audio"). + The event type, must be `session.created`. - - `text: optional string` + - `"session.created"` - The text content (if type is "text"). + - `SessionUpdatedEvent object { event_id, session, type }` - - `transcript: optional string` + Returned when a session is updated with a `session.update` event, unless + there is an error. - The transcript of the audio (if type is "audio"). + - `event_id: string` - - `type: optional "audio" or "text"` + The unique ID of the server event. - The content type ("text", "audio"). - - - `"audio"` + - `session: RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` - - `"text"` + The session configuration. - - `response_id: string` + - `RealtimeSessionCreateRequest object { type, audio, include, 9 more }` - The ID of the response. + Realtime session object configuration. - - `type: "response.content_part.added"` + - `RealtimeTranscriptionSessionCreateRequest object { type, audio, include }` - The event type, must be `response.content_part.added`. + Realtime transcription session object configuration. - - `"response.content_part.added"` + - `type: "session.updated"` - - `ResponseContentPartDoneEvent = object { content_index, event_id, item_id, 4 more }` + The event type, must be `session.updated`. - Returned when a content part is done streaming in an assistant message item. - Also emitted when a Response is interrupted, incomplete, or cancelled. + - `"session.updated"` - - `content_index: number` + - `OutputAudioBufferStarted object { event_id, response_id, type }` - The index of the content part in the item's content array. + **WebRTC/SIP Only:** Emitted when the server begins streaming audio to the client. This event is + emitted after an audio content part has been added (`response.content_part.added`) + to the response. + [Learn more](/docs/guides/realtime-conversations#client-and-server-events-for-audio-in-webrtc). - `event_id: string` The unique ID of the server event. - - `item_id: string` + - `response_id: string` - The ID of the item. + The unique ID of the response that produced the audio. - - `output_index: number` + - `type: "output_audio_buffer.started"` - The index of the output item in the response. + The event type, must be `output_audio_buffer.started`. - - `part: object { audio, text, transcript, type }` + - `"output_audio_buffer.started"` - The content part that is done. + - `OutputAudioBufferStopped object { event_id, response_id, type }` - - `audio: optional string` + **WebRTC/SIP Only:** Emitted when the output audio buffer has been completely drained on the server, + and no more audio is forthcoming. This event is emitted after the full response + data has been sent to the client (`response.done`). + [Learn more](/docs/guides/realtime-conversations#client-and-server-events-for-audio-in-webrtc). - Base64-encoded audio data (if type is "audio"). + - `event_id: string` - - `text: optional string` + The unique ID of the server event. - The text content (if type is "text"). + - `response_id: string` - - `transcript: optional string` + The unique ID of the response that produced the audio. - The transcript of the audio (if type is "audio"). + - `type: "output_audio_buffer.stopped"` - - `type: optional "audio" or "text"` + The event type, must be `output_audio_buffer.stopped`. - The content type ("text", "audio"). + - `"output_audio_buffer.stopped"` - - `"audio"` + - `OutputAudioBufferCleared object { event_id, response_id, type }` - - `"text"` + **WebRTC/SIP Only:** Emitted when the output audio buffer is cleared. This happens either in VAD + mode when the user has interrupted (`input_audio_buffer.speech_started`), + or when the client has emitted the `output_audio_buffer.clear` event to manually + cut off the current audio response. + [Learn more](/docs/guides/realtime-conversations#client-and-server-events-for-audio-in-webrtc). + + - `event_id: string` + + The unique ID of the server event. - `response_id: string` - The ID of the response. + The unique ID of the response that produced the audio. - - `type: "response.content_part.done"` + - `type: "output_audio_buffer.cleared"` - The event type, must be `response.content_part.done`. + The event type, must be `output_audio_buffer.cleared`. - - `"response.content_part.done"` + - `"output_audio_buffer.cleared"` - - `ResponseCreatedEvent = object { event_id, response, type }` + - `ConversationItemAdded object { event_id, item, type, previous_item_id }` - Returned when a new Response is created. The first event of response creation, - where the response is in an initial state of `in_progress`. + Sent by the server when an Item is added to the default Conversation. This can happen in several cases: + + - When the client sends a `conversation.item.create` event. + - When the input audio buffer is committed. In this case the item will be a user message containing the audio from the buffer. + - When the model is generating a Response. In this case the `conversation.item.added` event will be sent when the model starts generating a specific Item, and thus it will not yet have any content (and `status` will be `in_progress`). + + The event will include the full content of the Item (except when model is generating a Response) except for audio data, which can be retrieved separately with a `conversation.item.retrieve` event if necessary. - `event_id: string` The unique ID of the server event. - - `response: RealtimeResponse` + - `item: ConversationItem` - The response resource. + A single item within a Realtime conversation. - - `id: optional string` + - `type: "conversation.item.added"` - The unique ID of the response, will look like `resp_1234`. + The event type, must be `conversation.item.added`. - - `audio: optional object { output }` + - `"conversation.item.added"` - Configuration for audio output. + - `previous_item_id: optional string` - - `output: optional object { format, voice }` + The ID of the item that precedes this one, if any. This is used to + maintain ordering when items are inserted. - - `format: optional RealtimeAudioFormats` + - `ConversationItemDone object { event_id, item, type, previous_item_id }` - The format of the output audio. + Returned when a conversation item is finalized. - - `PCMAudioFormat = object { rate, type }` + The event will include the full content of the Item except for audio data, which can be retrieved separately with a `conversation.item.retrieve` event if needed. - The PCM audio format. Only a 24kHz sample rate is supported. + - `event_id: string` - - `rate: optional 24000` + The unique ID of the server event. - The sample rate of the audio. Always `24000`. + - `item: ConversationItem` - - `24000` + A single item within a Realtime conversation. - - `type: optional "audio/pcm"` + - `type: "conversation.item.done"` - The audio format. Always `audio/pcm`. + The event type, must be `conversation.item.done`. - - `"audio/pcm"` + - `"conversation.item.done"` - - `PCMUAudioFormat = object { type }` + - `previous_item_id: optional string` - The G.711 μ-law format. + The ID of the item that precedes this one, if any. This is used to + maintain ordering when items are inserted. - - `type: optional "audio/pcmu"` + - `InputAudioBufferTimeoutTriggered object { audio_end_ms, audio_start_ms, event_id, 2 more }` - The audio format. Always `audio/pcmu`. + Returned when the Server VAD timeout is triggered for the input audio buffer. This is configured + with `idle_timeout_ms` in the `turn_detection` settings of the session, and it indicates that + there hasn't been any speech detected for the configured duration. - - `"audio/pcmu"` + The `audio_start_ms` and `audio_end_ms` fields indicate the segment of audio after the last + model response up to the triggering time, as an offset from the beginning of audio written + to the input audio buffer. This means it demarcates the segment of audio that was silent and + the difference between the start and end values will roughly match the configured timeout. - - `PCMAAudioFormat = object { type }` + The empty audio will be committed to the conversation as an `input_audio` item (there will be a + `input_audio_buffer.committed` event) and a model response will be generated. There may be speech + that didn't trigger VAD but is still detected by the model, so the model may respond with + something relevant to the conversation or a prompt to continue speaking. - The G.711 A-law format. + - `audio_end_ms: number` - - `type: optional "audio/pcma"` + Millisecond offset of audio written to the input audio buffer at the time the timeout was triggered. - The audio format. Always `audio/pcma`. + - `audio_start_ms: number` - - `"audio/pcma"` + Millisecond offset of audio written to the input audio buffer that was after the playback time of the last model response. - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more` + - `event_id: string` - The voice the model uses to respond. Voice cannot be changed during the - session once the model has responded with audio at least once. Current - voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, - `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for - best quality. + The unique ID of the server event. - - `string` + - `item_id: string` - - `"alloy" or "ash" or "ballad" or 7 more` + The ID of the item associated with this segment. - The voice the model uses to respond. Voice cannot be changed during the - session once the model has responded with audio at least once. Current - voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, - `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for - best quality. + - `type: "input_audio_buffer.timeout_triggered"` - - `"alloy"` + The event type, must be `input_audio_buffer.timeout_triggered`. - - `"ash"` + - `"input_audio_buffer.timeout_triggered"` - - `"ballad"` + - `ConversationItemInputAudioTranscriptionSegment object { id, content_index, end, 6 more }` - - `"coral"` + Returned when an input audio transcription segment is identified for an item. - - `"echo"` + - `id: string` - - `"sage"` + The segment identifier. - - `"shimmer"` + - `content_index: number` - - `"verse"` + The index of the input audio content part within the item. - - `"marin"` + - `end: number` - - `"cedar"` + End time of the segment in seconds. - - `conversation_id: optional string` + - `event_id: string` - Which conversation the response is added to, determined by the `conversation` - field in the `response.create` event. If `auto`, the response will be added to - the default conversation and the value of `conversation_id` will be an id like - `conv_1234`. If `none`, the response will not be added to any conversation and - the value of `conversation_id` will be `null`. If responses are being triggered - automatically by VAD the response will be added to the default conversation + The unique ID of the server event. - - `max_output_tokens: optional number or "inf"` + - `item_id: string` - Maximum number of output tokens for a single assistant response, - inclusive of tool calls, that was used in this response. + The ID of the item containing the input audio content. - - `number` + - `speaker: string` - - `"inf"` + The detected speaker label for this segment. - - `"inf"` + - `start: number` - - `metadata: optional Metadata` + Start time of the segment in seconds. - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. + - `text: string` - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. + The text for this segment. - - `object: optional "realtime.response"` + - `type: "conversation.item.input_audio_transcription.segment"` - The object type, must be `realtime.response`. + The event type, must be `conversation.item.input_audio_transcription.segment`. - - `"realtime.response"` + - `"conversation.item.input_audio_transcription.segment"` - - `output: optional array of ConversationItem` + - `McpListToolsInProgress object { event_id, item_id, type }` - The list of output items generated by the response. + Returned when listing MCP tools is in progress for an item. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `event_id: string` - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + The unique ID of the server event. - - `content: array of object { text, type }` + - `item_id: string` - The content of the message. + The ID of the MCP list tools item. - - `text: optional string` + - `type: "mcp_list_tools.in_progress"` - The text content. + The event type, must be `mcp_list_tools.in_progress`. - - `type: optional "input_text"` + - `"mcp_list_tools.in_progress"` - The content type. Always `input_text` for system messages. + - `McpListToolsCompleted object { event_id, item_id, type }` - - `"input_text"` + Returned when listing MCP tools has completed for an item. - - `role: "system"` + - `event_id: string` - The role of the message sender. Always `system`. + The unique ID of the server event. - - `"system"` + - `item_id: string` - - `type: "message"` + The ID of the MCP list tools item. - The type of the item. Always `message`. + - `type: "mcp_list_tools.completed"` - - `"message"` + The event type, must be `mcp_list_tools.completed`. - - `id: optional string` + - `"mcp_list_tools.completed"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `McpListToolsFailed object { event_id, item_id, type }` - - `object: optional "realtime.item"` + Returned when listing MCP tools has failed for an item. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `event_id: string` - - `"realtime.item"` + The unique ID of the server event. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `item_id: string` - The status of the item. Has no effect on the conversation. + The ID of the MCP list tools item. - - `"completed"` + - `type: "mcp_list_tools.failed"` - - `"incomplete"` + The event type, must be `mcp_list_tools.failed`. - - `"in_progress"` + - `"mcp_list_tools.failed"` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `ResponseMcpCallArgumentsDelta object { delta, event_id, item_id, 4 more }` - A user message item in a Realtime conversation. + Returned when MCP tool call arguments are updated during response generation. - - `content: array of object { audio, detail, image_url, 3 more }` + - `delta: string` - The content of the message. + The JSON-encoded arguments delta. - - `audio: optional string` + - `event_id: string` - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + The unique ID of the server event. - - `detail: optional "auto" or "low" or "high"` + - `item_id: string` - The detail level of the image (for `input_image`). `auto` will default to `high`. + The ID of the MCP tool call item. - - `"auto"` + - `output_index: number` - - `"low"` + The index of the output item in the response. - - `"high"` + - `response_id: string` - - `image_url: optional string` + The ID of the response. - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + - `type: "response.mcp_call_arguments.delta"` - - `text: optional string` + The event type, must be `response.mcp_call_arguments.delta`. - The text content (for `input_text`). + - `"response.mcp_call_arguments.delta"` - - `transcript: optional string` + - `obfuscation: optional string` - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + If present, indicates the delta text was obfuscated. - - `type: optional "input_text" or "input_audio" or "input_image"` + - `ResponseMcpCallArgumentsDone object { arguments, event_id, item_id, 3 more }` - The content type (`input_text`, `input_audio`, or `input_image`). + Returned when MCP tool call arguments are finalized during response generation. - - `"input_text"` + - `arguments: string` - - `"input_audio"` + The final JSON-encoded arguments string. - - `"input_image"` + - `event_id: string` - - `role: "user"` + The unique ID of the server event. - The role of the message sender. Always `user`. + - `item_id: string` - - `"user"` + The ID of the MCP tool call item. - - `type: "message"` + - `output_index: number` - The type of the item. Always `message`. + The index of the output item in the response. - - `"message"` + - `response_id: string` - - `id: optional string` + The ID of the response. - The unique ID of the item. This may be provided by the client or generated by the server. + - `type: "response.mcp_call_arguments.done"` - - `object: optional "realtime.item"` + The event type, must be `response.mcp_call_arguments.done`. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"response.mcp_call_arguments.done"` - - `"realtime.item"` + - `ResponseMcpCallInProgress object { event_id, item_id, output_index, type }` - - `status: optional "completed" or "incomplete" or "in_progress"` + Returned when an MCP tool call has started and is in progress. - The status of the item. Has no effect on the conversation. + - `event_id: string` - - `"completed"` + The unique ID of the server event. - - `"incomplete"` + - `item_id: string` - - `"in_progress"` + The ID of the MCP tool call item. - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `output_index: number` - An assistant message item in a Realtime conversation. + The index of the output item in the response. - - `content: array of object { audio, text, transcript, type }` + - `type: "response.mcp_call.in_progress"` - The content of the message. + The event type, must be `response.mcp_call.in_progress`. - - `audio: optional string` + - `"response.mcp_call.in_progress"` - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `ResponseMcpCallCompleted object { event_id, item_id, output_index, type }` - - `text: optional string` + Returned when an MCP tool call has completed successfully. - The text content. + - `event_id: string` - - `transcript: optional string` + The unique ID of the server event. - The transcript of the audio content, this will always be present if the output type is `audio`. + - `item_id: string` - - `type: optional "output_text" or "output_audio"` + The ID of the MCP tool call item. - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + - `output_index: number` - - `"output_text"` + The index of the output item in the response. - - `"output_audio"` + - `type: "response.mcp_call.completed"` - - `role: "assistant"` + The event type, must be `response.mcp_call.completed`. - The role of the message sender. Always `assistant`. + - `"response.mcp_call.completed"` - - `"assistant"` + - `ResponseMcpCallFailed object { event_id, item_id, output_index, type }` - - `type: "message"` + Returned when an MCP tool call has failed. - The type of the item. Always `message`. + - `event_id: string` - - `"message"` + The unique ID of the server event. - - `id: optional string` + - `item_id: string` - The unique ID of the item. This may be provided by the client or generated by the server. + The ID of the MCP tool call item. - - `object: optional "realtime.item"` + - `output_index: number` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + The index of the output item in the response. - - `"realtime.item"` + - `type: "response.mcp_call.failed"` - - `status: optional "completed" or "incomplete" or "in_progress"` + The event type, must be `response.mcp_call.failed`. - The status of the item. Has no effect on the conversation. + - `"response.mcp_call.failed"` - - `"completed"` +### Realtime Session - - `"incomplete"` +- `RealtimeSession object { id, expires_at, include, 17 more }` - - `"in_progress"` + Realtime session object for the beta interface. - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `id: optional string` - A function call item in a Realtime conversation. + Unique identifier for the session that looks like `sess_1234567890abcdef`. - - `arguments: string` + - `expires_at: optional number` - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + Expiration timestamp for the session, in seconds since epoch. - - `name: string` + - `include: optional array of "item.input_audio_transcription.logprobs"` - The name of the function being called. + Additional fields to include in server outputs. - - `type: "function_call"` + - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - The type of the item. Always `function_call`. + - `"item.input_audio_transcription.logprobs"` - - `"function_call"` + - `input_audio_format: optional "pcm16" or "g711_ulaw" or "g711_alaw"` - - `id: optional string` + The format of input audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. + For `pcm16`, input audio must be 16-bit PCM at a 24kHz sample rate, + single channel (mono), and little-endian byte order. - The unique ID of the item. This may be provided by the client or generated by the server. + - `"pcm16"` - - `call_id: optional string` + - `"g711_ulaw"` - The ID of the function call. + - `"g711_alaw"` - - `object: optional "realtime.item"` + - `input_audio_noise_reduction: optional object { type }` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `"realtime.item"` + - `type: optional NoiseReductionType` - - `status: optional "completed" or "incomplete" or "in_progress"` + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - The status of the item. Has no effect on the conversation. + - `"near_field"` - - `"completed"` + - `"far_field"` - - `"incomplete"` + - `input_audio_transcription: optional AudioTranscription` - - `"in_progress"` + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](https://platform.openai.com/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `language: optional string` - A function call output item in a Realtime conversation. + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - - `call_id: string` + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - The ID of the function call this output is for. + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `output: string` + - `string` - The output of the function call, this is free text and can contain any information or simply be empty. + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `type: "function_call_output"` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - The type of the item. Always `function_call_output`. + - `"whisper-1"` - - `"function_call_output"` + - `"gpt-4o-mini-transcribe"` - - `id: optional string` + - `"gpt-4o-mini-transcribe-2025-12-15"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `"gpt-4o-transcribe"` - - `object: optional "realtime.item"` + - `"gpt-4o-transcribe-diarize"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `prompt: optional string` - - `"realtime.item"` + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - `status: optional "completed" or "incomplete" or "in_progress"` + - `instructions: optional string` - The status of the item. Has no effect on the conversation. + The default system instructions (i.e. system message) prepended to model + calls. This field allows the client to guide the model on desired + responses. The model can be instructed on response content and format, + (e.g. "be extremely succinct", "act friendly", "here are examples of good + responses") and on audio behavior (e.g. "talk quickly", "inject emotion + into your voice", "laugh frequently"). The instructions are not + guaranteed to be followed by the model, but they provide guidance to the + model on the desired behavior. - - `"completed"` + Note that the server sets default instructions which will be used if this + field is not set and are visible in the `session.created` event at the + start of the session. - - `"incomplete"` + - `max_response_output_tokens: optional number or "inf"` - - `"in_progress"` + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `number` - A Realtime item responding to an MCP approval request. + - `"inf"` - - `id: string` + - `"inf"` - The unique ID of the approval response. + - `modalities: optional array of "text" or "audio"` - - `approval_request_id: string` + The set of modalities the model can respond with. To disable audio, + set this to ["text"]. - The ID of the approval request being answered. + - `"text"` - - `approve: boolean` + - `"audio"` - Whether the request was approved. + - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - `type: "mcp_approval_response"` + The Realtime model used for this session. - The type of the item. Always `mcp_approval_response`. + - `string` - - `"mcp_approval_response"` + - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - `reason: optional string` + The Realtime model used for this session. - Optional reason for the decision. + - `"gpt-realtime"` - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `"gpt-realtime-1.5"` - A Realtime item listing tools available on an MCP server. + - `"gpt-realtime-2025-08-28"` - - `server_label: string` + - `"gpt-4o-realtime-preview"` - The label of the MCP server. + - `"gpt-4o-realtime-preview-2024-10-01"` - - `tools: array of object { input_schema, name, annotations, description }` + - `"gpt-4o-realtime-preview-2024-12-17"` - The tools available on the server. + - `"gpt-4o-realtime-preview-2025-06-03"` - - `input_schema: unknown` + - `"gpt-4o-mini-realtime-preview"` - The JSON schema describing the tool's input. + - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - `name: string` + - `"gpt-realtime-mini"` - The name of the tool. + - `"gpt-realtime-mini-2025-10-06"` - - `annotations: optional unknown` + - `"gpt-realtime-mini-2025-12-15"` - Additional annotations about the tool. + - `"gpt-audio-1.5"` - - `description: optional string` + - `"gpt-audio-mini"` - The description of the tool. + - `"gpt-audio-mini-2025-10-06"` - - `type: "mcp_list_tools"` + - `"gpt-audio-mini-2025-12-15"` - The type of the item. Always `mcp_list_tools`. + - `object: optional "realtime.session"` - - `"mcp_list_tools"` + The object type. Always `realtime.session`. - - `id: optional string` + - `"realtime.session"` - The unique ID of the list. + - `output_audio_format: optional "pcm16" or "g711_ulaw" or "g711_alaw"` - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + The format of output audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. + For `pcm16`, output audio is sampled at a rate of 24kHz. - A Realtime item representing an invocation of a tool on an MCP server. + - `"pcm16"` - - `id: string` + - `"g711_ulaw"` - The unique ID of the tool call. + - `"g711_alaw"` - - `arguments: string` + - `prompt: optional ResponsePrompt` - A JSON string of the arguments passed to the tool. + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - - `name: string` + - `id: string` - The name of the tool that was run. + The unique identifier of the prompt template to use. - - `server_label: string` + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - The label of the MCP server running the tool. + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - - `type: "mcp_call"` + - `string` - The type of the item. Always `mcp_call`. + - `ResponseInputText object { text, type }` - - `"mcp_call"` + A text input to the model. - - `approval_request_id: optional string` + - `text: string` - The ID of an associated approval request, if any. + The text input to the model. - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` + - `type: "input_text"` - The error from the tool call, if any. + The type of the input item. Always `input_text`. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `"input_text"` - - `code: number` + - `ResponseInputImage object { detail, type, file_id, image_url }` - - `message: string` + An image input to the model. Learn about [image inputs](/docs/guides/vision). - - `type: "protocol_error"` + - `detail: "low" or "high" or "auto" or "original"` - - `"protocol_error"` + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - `RealtimeMcpToolExecutionError = object { message, type }` + - `"low"` - - `message: string` + - `"high"` - - `type: "tool_execution_error"` + - `"auto"` - - `"tool_execution_error"` + - `"original"` - - `RealtimeMcphttpError = object { code, message, type }` + - `type: "input_image"` - - `code: number` + The type of the input item. Always `input_image`. - - `message: string` + - `"input_image"` - - `type: "http_error"` + - `file_id: optional string` - - `"http_error"` + The ID of the file to be sent to the model. - - `output: optional string` + - `image_url: optional string` - The output from the tool call. + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `ResponseInputFile object { type, detail, file_data, 3 more }` - A Realtime item requesting human approval of a tool invocation. + A file input to the model. - - `id: string` + - `type: "input_file"` - The unique ID of the approval request. + The type of the input item. Always `input_file`. - - `arguments: string` + - `"input_file"` - A JSON string of arguments for the tool. + - `detail: optional "low" or "high"` - - `name: string` + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - The name of the tool to run. + - `"low"` - - `server_label: string` + - `"high"` - The label of the MCP server making the request. + - `file_data: optional string` - - `type: "mcp_approval_request"` + The content of the file to be sent to the model. - The type of the item. Always `mcp_approval_request`. + - `file_id: optional string` - - `"mcp_approval_request"` + The ID of the file to be sent to the model. - - `output_modalities: optional array of "text" or "audio"` + - `file_url: optional string` - The set of modalities the model used to respond, currently the only possible values are - `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the - output to mode `text` will disable audio output from the model. + The URL of the file to be sent to the model. - - `"text"` + - `filename: optional string` - - `"audio"` + The name of the file to be sent to the model. - - `status: optional "completed" or "cancelled" or "failed" or 2 more` + - `version: optional string` - The final status of the response (`completed`, `cancelled`, `failed`, or - `incomplete`, `in_progress`). + Optional version of the prompt template. - - `"completed"` + - `speed: optional number` - - `"cancelled"` + The speed of the model's spoken response. 1.0 is the default speed. 0.25 is + the minimum speed. 1.5 is the maximum speed. This value can only be changed + in between model turns, not while a response is in progress. - - `"failed"` + - `temperature: optional number` - - `"incomplete"` + Sampling temperature for the model, limited to [0.6, 1.2]. For audio models a temperature of 0.8 is highly recommended for best performance. - - `"in_progress"` + - `tool_choice: optional string` - - `status_details: optional RealtimeResponseStatus` + How the model chooses tools. Options are `auto`, `none`, `required`, or + specify a function. - Additional details about the status. + - `tools: optional array of RealtimeFunctionTool` - - `error: optional object { code, type }` + Tools (functions) available to the model. - A description of the error that caused the response to fail, - populated when the `status` is `failed`. + - `description: optional string` - - `code: optional string` + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - Error code, if any. + - `name: optional string` - - `type: optional string` + The name of the function. - The type of error. + - `parameters: optional unknown` - - `reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"` + Parameters of the function in JSON Schema. - The reason the Response did not complete. For a `cancelled` Response, one of `turn_detected` (the server VAD detected a new start of speech) or `client_cancelled` (the client sent a cancel event). For an `incomplete` Response, one of `max_output_tokens` or `content_filter` (the server-side safety filter activated and cut off the response). + - `type: optional "function"` - - `"turn_detected"` + The type of the tool, i.e. `function`. - - `"client_cancelled"` + - `"function"` - - `"max_output_tokens"` + - `tracing: optional "auto" or object { group_id, metadata, workflow_name }` - - `"content_filter"` + Configuration options for tracing. Set to null to disable tracing. Once + tracing is enabled for a session, the configuration cannot be modified. - - `type: optional "completed" or "cancelled" or "failed" or "incomplete"` + `auto` will create a trace for the session with default values for the + workflow name, group id, and metadata. - The type of error that caused the response to fail, corresponding - with the `status` field (`completed`, `cancelled`, `incomplete`, - `failed`). + - `"auto"` - - `"completed"` + Default tracing mode for the session. - - `"cancelled"` + - `"auto"` - - `"failed"` + - `TracingConfiguration object { group_id, metadata, workflow_name }` - - `"incomplete"` + Granular configuration for tracing. - - `usage: optional RealtimeResponseUsage` + - `group_id: optional string` - Usage statistics for the Response, this will correspond to billing. A - Realtime API session will maintain a conversation context and append new - Items to the Conversation, thus output from previous turns (text and - audio tokens) will become the input for later turns. + The group id to attach to this trace to enable filtering and + grouping in the traces dashboard. - - `input_token_details: optional RealtimeResponseUsageInputTokenDetails` + - `metadata: optional unknown` - Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. + The arbitrary metadata to attach to this trace to enable + filtering in the traces dashboard. - - `audio_tokens: optional number` + - `workflow_name: optional string` - The number of audio tokens used as input for the Response. + The name of the workflow to attach to this trace. This is used to + name the trace in the traces dashboard. - - `cached_tokens: optional number` + - `turn_detection: optional object { type, create_response, idle_timeout_ms, 4 more } or object { type, create_response, eagerness, interrupt_response }` - The number of cached tokens used as input for the Response. + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - Details about the cached tokens used as input for the Response. + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `audio_tokens: optional number` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - The number of cached audio tokens used as input for the Response. + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - `image_tokens: optional number` + - `type: "server_vad"` - The number of cached image tokens used as input for the Response. + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `text_tokens: optional number` + - `"server_vad"` - The number of cached text tokens used as input for the Response. + - `create_response: optional boolean` - - `image_tokens: optional number` + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - The number of image tokens used as input for the Response. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `text_tokens: optional number` + - `idle_timeout_ms: optional number` - The number of text tokens used as input for the Response. + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - - `input_tokens: optional number` + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - The number of input tokens used in the Response, including text and - audio tokens. - - - `output_token_details: optional RealtimeResponseUsageOutputTokenDetails` - - Details about the output tokens used in the Response. - - - `audio_tokens: optional number` - - The number of audio tokens used in the Response. - - - `text_tokens: optional number` - - The number of text tokens used in the Response. - - - `output_tokens: optional number` - - The number of output tokens sent in the Response, including text and - audio tokens. - - - `total_tokens: optional number` - - The total number of tokens in the Response including input and output - text and audio tokens. - - - `type: "response.created"` - - The event type, must be `response.created`. - - - `"response.created"` - - - `ResponseDoneEvent = object { event_id, response, type }` - - Returned when a Response is done streaming. Always emitted, no matter the - final state. The Response object included in the `response.done` event will - include all output Items in the Response but will omit the raw audio data. - - Clients should check the `status` field of the Response to determine if it was successful - (`completed`) or if there was another outcome: `cancelled`, `failed`, or `incomplete`. - - A response will contain all output items that were generated during the response, excluding - any audio content. - - - `event_id: string` - - The unique ID of the server event. - - - `response: RealtimeResponse` - - The response resource. - - - `id: optional string` + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - The unique ID of the response, will look like `resp_1234`. + - `interrupt_response: optional boolean` - - `audio: optional object { output }` + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - Configuration for audio output. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `output: optional object { format, voice }` + - `prefix_padding_ms: optional number` - - `format: optional RealtimeAudioFormats` + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - The format of the output audio. + - `silence_duration_ms: optional number` - - `PCMAudioFormat = object { rate, type }` + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - The PCM audio format. Only a 24kHz sample rate is supported. + - `threshold: optional number` - - `rate: optional 24000` + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - The sample rate of the audio. Always `24000`. + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - - `24000` + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - `type: optional "audio/pcm"` + - `type: "semantic_vad"` - The audio format. Always `audio/pcm`. + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - `"audio/pcm"` + - `"semantic_vad"` - - `PCMUAudioFormat = object { type }` + - `create_response: optional boolean` - The G.711 μ-law format. + Whether or not to automatically generate a response when a VAD stop event occurs. - - `type: optional "audio/pcmu"` + - `eagerness: optional "low" or "medium" or "high" or "auto"` - The audio format. Always `audio/pcmu`. + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - `"audio/pcmu"` + - `"low"` - - `PCMAAudioFormat = object { type }` + - `"medium"` - The G.711 A-law format. + - `"high"` - - `type: optional "audio/pcma"` + - `"auto"` - The audio format. Always `audio/pcma`. + - `interrupt_response: optional boolean` - - `"audio/pcma"` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more` The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, - `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for - best quality. + `shimmer`, and `verse`. - `string` @@ -11307,8 +11639,7 @@ The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, - `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for - best quality. + `shimmer`, and `verse`. - `"alloy"` @@ -11330,1772 +11661,1753 @@ - `"cedar"` - - `conversation_id: optional string` - - Which conversation the response is added to, determined by the `conversation` - field in the `response.create` event. If `auto`, the response will be added to - the default conversation and the value of `conversation_id` will be an id like - `conv_1234`. If `none`, the response will not be added to any conversation and - the value of `conversation_id` will be `null`. If responses are being triggered - automatically by VAD the response will be added to the default conversation +### Realtime Session Create Request - - `max_output_tokens: optional number or "inf"` +- `RealtimeSessionCreateRequest object { type, audio, include, 9 more }` - Maximum number of output tokens for a single assistant response, - inclusive of tool calls, that was used in this response. + Realtime session object configuration. - - `number` + - `type: "realtime"` - - `"inf"` + The type of session to create. Always `realtime` for the Realtime API. - - `"inf"` + - `"realtime"` - - `metadata: optional Metadata` + - `audio: optional RealtimeAudioConfig` - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. + Configuration for input and output audio. - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. + - `input: optional RealtimeAudioConfigInput` - - `object: optional "realtime.response"` + - `format: optional RealtimeAudioFormats` - The object type, must be `realtime.response`. + The format of the input audio. - - `"realtime.response"` + - `PCMAudioFormat object { rate, type }` - - `output: optional array of ConversationItem` + The PCM audio format. Only a 24kHz sample rate is supported. - The list of output items generated by the response. + - `rate: optional 24000` - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + The sample rate of the audio. Always `24000`. - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + - `24000` - - `content: array of object { text, type }` + - `type: optional "audio/pcm"` - The content of the message. + The audio format. Always `audio/pcm`. - - `text: optional string` + - `"audio/pcm"` - The text content. + - `PCMUAudioFormat object { type }` - - `type: optional "input_text"` + The G.711 μ-law format. - The content type. Always `input_text` for system messages. + - `type: optional "audio/pcmu"` - - `"input_text"` + The audio format. Always `audio/pcmu`. - - `role: "system"` + - `"audio/pcmu"` - The role of the message sender. Always `system`. + - `PCMAAudioFormat object { type }` - - `"system"` + The G.711 A-law format. - - `type: "message"` + - `type: optional "audio/pcma"` - The type of the item. Always `message`. + The audio format. Always `audio/pcma`. - - `"message"` + - `"audio/pcma"` - - `id: optional string` + - `noise_reduction: optional object { type }` - The unique ID of the item. This may be provided by the client or generated by the server. + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `object: optional "realtime.item"` + - `type: optional NoiseReductionType` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - `"realtime.item"` + - `"near_field"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"far_field"` - The status of the item. Has no effect on the conversation. + - `transcription: optional AudioTranscription` - - `"completed"` + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - `"incomplete"` + - `language: optional string` - - `"in_progress"` + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - A user message item in a Realtime conversation. + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `content: array of object { audio, detail, image_url, 3 more }` + - `string` - The content of the message. + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `audio: optional string` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `"whisper-1"` - - `detail: optional "auto" or "low" or "high"` + - `"gpt-4o-mini-transcribe"` - The detail level of the image (for `input_image`). `auto` will default to `high`. + - `"gpt-4o-mini-transcribe-2025-12-15"` - - `"auto"` + - `"gpt-4o-transcribe"` - - `"low"` + - `"gpt-4o-transcribe-diarize"` - - `"high"` + - `prompt: optional string` - - `image_url: optional string` + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + - `turn_detection: optional RealtimeAudioInputTurnDetection` - - `text: optional string` + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - The text content (for `input_text`). + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - `transcript: optional string` + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - - `type: optional "input_text" or "input_audio" or "input_image"` + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - The content type (`input_text`, `input_audio`, or `input_image`). + - `type: "server_vad"` - - `"input_text"` + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `"input_audio"` + - `"server_vad"` - - `"input_image"` + - `create_response: optional boolean` - - `role: "user"` + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - The role of the message sender. Always `user`. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `"user"` + - `idle_timeout_ms: optional number` - - `type: "message"` + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - The type of the item. Always `message`. + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - - `"message"` + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - - `id: optional string` + - `interrupt_response: optional boolean` - The unique ID of the item. This may be provided by the client or generated by the server. + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - `object: optional "realtime.item"` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `prefix_padding_ms: optional number` - - `"realtime.item"` + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `silence_duration_ms: optional number` - The status of the item. Has no effect on the conversation. + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - - `"completed"` + - `threshold: optional number` - - `"incomplete"` + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - - `"in_progress"` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - An assistant message item in a Realtime conversation. + - `type: "semantic_vad"` - - `content: array of object { audio, text, transcript, type }` + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - The content of the message. + - `"semantic_vad"` - - `audio: optional string` + - `create_response: optional boolean` - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + Whether or not to automatically generate a response when a VAD stop event occurs. - - `text: optional string` + - `eagerness: optional "low" or "medium" or "high" or "auto"` - The text content. + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - `transcript: optional string` + - `"low"` - The transcript of the audio content, this will always be present if the output type is `audio`. + - `"medium"` - - `type: optional "output_text" or "output_audio"` + - `"high"` - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + - `"auto"` - - `"output_text"` + - `interrupt_response: optional boolean` - - `"output_audio"` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - `role: "assistant"` + - `output: optional RealtimeAudioConfigOutput` - The role of the message sender. Always `assistant`. + - `format: optional RealtimeAudioFormats` - - `"assistant"` + The format of the output audio. - - `type: "message"` + - `speed: optional number` - The type of the item. Always `message`. + The speed of the model's spoken response as a multiple of the original speed. + 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - - `"message"` + This parameter is a post-processing adjustment to the audio after it is generated, it's + also possible to prompt the model to speak faster or slower. - - `id: optional string` + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - The unique ID of the item. This may be provided by the client or generated by the server. + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. - - `object: optional "realtime.item"` + - `string` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"alloy" or "ash" or "ballad" or 7 more` - - `"realtime.item"` + - `"alloy"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"ash"` - The status of the item. Has no effect on the conversation. + - `"ballad"` - - `"completed"` + - `"coral"` - - `"incomplete"` + - `"echo"` - - `"in_progress"` + - `"sage"` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `"shimmer"` - A function call item in a Realtime conversation. + - `"verse"` - - `arguments: string` + - `"marin"` - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + - `"cedar"` - - `name: string` + - `ID object { id }` - The name of the function being called. + Custom voice reference. - - `type: "function_call"` + - `id: string` - The type of the item. Always `function_call`. + The custom voice ID, e.g. `voice_1234`. - - `"function_call"` + - `include: optional array of "item.input_audio_transcription.logprobs"` - - `id: optional string` + Additional fields to include in server outputs. - The unique ID of the item. This may be provided by the client or generated by the server. + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - `call_id: optional string` + - `"item.input_audio_transcription.logprobs"` - The ID of the function call. + - `instructions: optional string` - - `object: optional "realtime.item"` + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - `"realtime.item"` + - `max_output_tokens: optional number or "inf"` - - `status: optional "completed" or "incomplete" or "in_progress"` + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - The status of the item. Has no effect on the conversation. + - `number` - - `"completed"` + - `"inf"` - - `"incomplete"` + - `"inf"` - - `"in_progress"` + - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + The Realtime model used for this session. - A function call output item in a Realtime conversation. + - `string` - - `call_id: string` + - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - The ID of the function call this output is for. + The Realtime model used for this session. - - `output: string` + - `"gpt-realtime"` - The output of the function call, this is free text and can contain any information or simply be empty. + - `"gpt-realtime-1.5"` - - `type: "function_call_output"` + - `"gpt-realtime-2025-08-28"` - The type of the item. Always `function_call_output`. + - `"gpt-4o-realtime-preview"` - - `"function_call_output"` + - `"gpt-4o-realtime-preview-2024-10-01"` - - `id: optional string` + - `"gpt-4o-realtime-preview-2024-12-17"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `"gpt-4o-realtime-preview-2025-06-03"` - - `object: optional "realtime.item"` + - `"gpt-4o-mini-realtime-preview"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - `"realtime.item"` + - `"gpt-realtime-mini"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"gpt-realtime-mini-2025-10-06"` - The status of the item. Has no effect on the conversation. + - `"gpt-realtime-mini-2025-12-15"` - - `"completed"` + - `"gpt-audio-1.5"` - - `"incomplete"` + - `"gpt-audio-mini"` - - `"in_progress"` + - `"gpt-audio-mini-2025-10-06"` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `"gpt-audio-mini-2025-12-15"` - A Realtime item responding to an MCP approval request. + - `output_modalities: optional array of "text" or "audio"` - - `id: string` + The set of modalities the model can respond with. It defaults to `["audio"]`, indicating + that the model will respond with audio plus a transcript. `["text"]` can be used to make + the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - The unique ID of the approval response. + - `"text"` - - `approval_request_id: string` + - `"audio"` - The ID of the approval request being answered. + - `prompt: optional ResponsePrompt` - - `approve: boolean` + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - Whether the request was approved. + - `id: string` - - `type: "mcp_approval_response"` + The unique identifier of the prompt template to use. - The type of the item. Always `mcp_approval_response`. + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - `"mcp_approval_response"` + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - - `reason: optional string` + - `string` - Optional reason for the decision. + - `ResponseInputText object { text, type }` - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + A text input to the model. - A Realtime item listing tools available on an MCP server. + - `text: string` - - `server_label: string` + The text input to the model. - The label of the MCP server. + - `type: "input_text"` - - `tools: array of object { input_schema, name, annotations, description }` + The type of the input item. Always `input_text`. - The tools available on the server. + - `"input_text"` - - `input_schema: unknown` + - `ResponseInputImage object { detail, type, file_id, image_url }` - The JSON schema describing the tool's input. + An image input to the model. Learn about [image inputs](/docs/guides/vision). - - `name: string` + - `detail: "low" or "high" or "auto" or "original"` - The name of the tool. + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - `annotations: optional unknown` + - `"low"` - Additional annotations about the tool. + - `"high"` - - `description: optional string` + - `"auto"` - The description of the tool. + - `"original"` - - `type: "mcp_list_tools"` + - `type: "input_image"` - The type of the item. Always `mcp_list_tools`. + The type of the input item. Always `input_image`. - - `"mcp_list_tools"` + - `"input_image"` - - `id: optional string` + - `file_id: optional string` - The unique ID of the list. + The ID of the file to be sent to the model. - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `image_url: optional string` - A Realtime item representing an invocation of a tool on an MCP server. + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - `id: string` + - `ResponseInputFile object { type, detail, file_data, 3 more }` - The unique ID of the tool call. + A file input to the model. - - `arguments: string` + - `type: "input_file"` - A JSON string of the arguments passed to the tool. + The type of the input item. Always `input_file`. - - `name: string` + - `"input_file"` - The name of the tool that was run. + - `detail: optional "low" or "high"` - - `server_label: string` - - The label of the MCP server running the tool. + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - `type: "mcp_call"` + - `"low"` - The type of the item. Always `mcp_call`. + - `"high"` - - `"mcp_call"` + - `file_data: optional string` - - `approval_request_id: optional string` + The content of the file to be sent to the model. - The ID of an associated approval request, if any. + - `file_id: optional string` - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` + The ID of the file to be sent to the model. - The error from the tool call, if any. + - `file_url: optional string` - - `RealtimeMcpProtocolError = object { code, message, type }` + The URL of the file to be sent to the model. - - `code: number` + - `filename: optional string` - - `message: string` + The name of the file to be sent to the model. - - `type: "protocol_error"` + - `version: optional string` - - `"protocol_error"` + Optional version of the prompt template. - - `RealtimeMcpToolExecutionError = object { message, type }` + - `tool_choice: optional RealtimeToolChoiceConfig` - - `message: string` + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. - - `type: "tool_execution_error"` + - `ToolChoiceOptions = "none" or "auto" or "required"` - - `"tool_execution_error"` + Controls which (if any) tool is called by the model. - - `RealtimeMcphttpError = object { code, message, type }` + `none` means the model will not call any tool and instead generates a message. - - `code: number` + `auto` means the model can pick between generating a message or calling one or + more tools. - - `message: string` + `required` means the model must call one or more tools. - - `type: "http_error"` + - `"none"` - - `"http_error"` + - `"auto"` - - `output: optional string` + - `"required"` - The output from the tool call. + - `ToolChoiceFunction object { name, type }` - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + Use this option to force the model to call a specific function. - A Realtime item requesting human approval of a tool invocation. + - `name: string` - - `id: string` + The name of the function to call. - The unique ID of the approval request. + - `type: "function"` - - `arguments: string` + For function calling, the type is always `function`. - A JSON string of arguments for the tool. + - `"function"` - - `name: string` + - `ToolChoiceMcp object { server_label, type, name }` - The name of the tool to run. + Use this option to force the model to call a specific tool on a remote MCP server. - `server_label: string` - The label of the MCP server making the request. + The label of the MCP server to use. - - `type: "mcp_approval_request"` + - `type: "mcp"` - The type of the item. Always `mcp_approval_request`. + For MCP tools, the type is always `mcp`. - - `"mcp_approval_request"` + - `"mcp"` - - `output_modalities: optional array of "text" or "audio"` + - `name: optional string` - The set of modalities the model used to respond, currently the only possible values are - `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the - output to mode `text` will disable audio output from the model. + The name of the tool to call on the server. - - `"text"` + - `tools: optional RealtimeToolsConfig` - - `"audio"` + Tools available to the model. - - `status: optional "completed" or "cancelled" or "failed" or 2 more` + - `RealtimeFunctionTool object { description, name, parameters, type }` - The final status of the response (`completed`, `cancelled`, `failed`, or - `incomplete`, `in_progress`). + - `description: optional string` - - `"completed"` + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - - `"cancelled"` + - `name: optional string` - - `"failed"` + The name of the function. - - `"incomplete"` + - `parameters: optional unknown` - - `"in_progress"` + Parameters of the function in JSON Schema. - - `status_details: optional RealtimeResponseStatus` + - `type: optional "function"` - Additional details about the status. + The type of the tool, i.e. `function`. - - `error: optional object { code, type }` + - `"function"` - A description of the error that caused the response to fail, - populated when the `status` is `failed`. + - `McpTool object { server_label, type, allowed_tools, 7 more }` - - `code: optional string` + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - Error code, if any. + - `server_label: string` - - `type: optional string` + A label for this MCP server, used to identify it in tool calls. - The type of error. + - `type: "mcp"` - - `reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"` + The type of the MCP tool. Always `mcp`. - The reason the Response did not complete. For a `cancelled` Response, one of `turn_detected` (the server VAD detected a new start of speech) or `client_cancelled` (the client sent a cancel event). For an `incomplete` Response, one of `max_output_tokens` or `content_filter` (the server-side safety filter activated and cut off the response). + - `"mcp"` - - `"turn_detected"` + - `allowed_tools: optional array of string or object { read_only, tool_names }` - - `"client_cancelled"` + List of allowed tool names or a filter object. - - `"max_output_tokens"` + - `McpAllowedTools = array of string` - - `"content_filter"` + A string array of allowed tool names - - `type: optional "completed" or "cancelled" or "failed" or "incomplete"` + - `McpToolFilter object { read_only, tool_names }` - The type of error that caused the response to fail, corresponding - with the `status` field (`completed`, `cancelled`, `incomplete`, - `failed`). + A filter object to specify which tools are allowed. - - `"completed"` + - `read_only: optional boolean` - - `"cancelled"` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `"failed"` + - `tool_names: optional array of string` - - `"incomplete"` + List of allowed tool names. - - `usage: optional RealtimeResponseUsage` + - `authorization: optional string` - Usage statistics for the Response, this will correspond to billing. A - Realtime API session will maintain a conversation context and append new - Items to the Conversation, thus output from previous turns (text and - audio tokens) will become the input for later turns. + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. - - `input_token_details: optional RealtimeResponseUsageInputTokenDetails` + - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](/docs/guides/tools-remote-mcp#connectors). - - `audio_tokens: optional number` + Currently supported `connector_id` values are: - The number of audio tokens used as input for the Response. + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` - - `cached_tokens: optional number` + - `"connector_dropbox"` - The number of cached tokens used as input for the Response. + - `"connector_gmail"` - - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` + - `"connector_googlecalendar"` - Details about the cached tokens used as input for the Response. + - `"connector_googledrive"` - - `audio_tokens: optional number` + - `"connector_microsoftteams"` - The number of cached audio tokens used as input for the Response. + - `"connector_outlookcalendar"` - - `image_tokens: optional number` + - `"connector_outlookemail"` - The number of cached image tokens used as input for the Response. + - `"connector_sharepoint"` - - `text_tokens: optional number` + - `defer_loading: optional boolean` - The number of cached text tokens used as input for the Response. + Whether this MCP tool is deferred and discovered via tool search. - - `image_tokens: optional number` + - `headers: optional map[string]` - The number of image tokens used as input for the Response. + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. - - `text_tokens: optional number` + - `require_approval: optional object { always, never } or "always" or "never"` - The number of text tokens used as input for the Response. + Specify which of the MCP server's tools require approval. - - `input_tokens: optional number` + - `McpToolApprovalFilter object { always, never }` - The number of input tokens used in the Response, including text and - audio tokens. + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. - - `output_token_details: optional RealtimeResponseUsageOutputTokenDetails` + - `always: optional object { read_only, tool_names }` - Details about the output tokens used in the Response. + A filter object to specify which tools are allowed. - - `audio_tokens: optional number` + - `read_only: optional boolean` - The number of audio tokens used in the Response. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `text_tokens: optional number` + - `tool_names: optional array of string` - The number of text tokens used in the Response. + List of allowed tool names. - - `output_tokens: optional number` + - `never: optional object { read_only, tool_names }` - The number of output tokens sent in the Response, including text and - audio tokens. + A filter object to specify which tools are allowed. - - `total_tokens: optional number` + - `read_only: optional boolean` - The total number of tokens in the Response including input and output - text and audio tokens. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `type: "response.done"` + - `tool_names: optional array of string` - The event type, must be `response.done`. + List of allowed tool names. - - `"response.done"` + - `McpToolApprovalSetting = "always" or "never"` - - `ResponseFunctionCallArgumentsDeltaEvent = object { call_id, delta, event_id, 4 more }` + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. - Returned when the model-generated function call arguments are updated. + - `"always"` - - `call_id: string` + - `"never"` - The ID of the function call. + - `server_description: optional string` - - `delta: string` + Optional description of the MCP server, used to provide more context. - The arguments delta as a JSON string. + - `server_url: optional string` - - `event_id: string` + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. - The unique ID of the server event. + - `tracing: optional RealtimeTracingConfig` - - `item_id: string` + Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once + tracing is enabled for a session, the configuration cannot be modified. - The ID of the function call item. + `auto` will create a trace for the session with default values for the + workflow name, group id, and metadata. - - `output_index: number` + - `Auto = "auto"` - The index of the output item in the response. + Enables tracing and sets default values for tracing configuration options. Always `auto`. - - `response_id: string` + - `"auto"` - The ID of the response. + - `TracingConfiguration object { group_id, metadata, workflow_name }` - - `type: "response.function_call_arguments.delta"` + Granular configuration for tracing. - The event type, must be `response.function_call_arguments.delta`. + - `group_id: optional string` - - `"response.function_call_arguments.delta"` + The group id to attach to this trace to enable filtering and + grouping in the Traces Dashboard. - - `ResponseFunctionCallArgumentsDoneEvent = object { arguments, call_id, event_id, 5 more }` + - `metadata: optional unknown` - Returned when the model-generated function call arguments are done streaming. - Also emitted when a Response is interrupted, incomplete, or cancelled. + The arbitrary metadata to attach to this trace to enable + filtering in the Traces Dashboard. - - `arguments: string` + - `workflow_name: optional string` - The final arguments as a JSON string. + The name of the workflow to attach to this trace. This is used to + name the trace in the Traces Dashboard. - - `call_id: string` + - `truncation: optional RealtimeTruncation` - The ID of the function call. + When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - - `event_id: string` + Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - The unique ID of the server event. + Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - `item_id: string` + Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - The ID of the function call item. + - `"auto" or "disabled"` - - `name: string` + The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - The name of the function that was called. + - `"auto"` - - `output_index: number` + - `"disabled"` - The index of the output item in the response. + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` - - `response_id: string` + Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - The ID of the response. + - `retention_ratio: number` - - `type: "response.function_call_arguments.done"` + Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - The event type, must be `response.function_call_arguments.done`. + - `type: "retention_ratio"` - - `"response.function_call_arguments.done"` + Use retention ratio truncation. - - `ResponseOutputItemAddedEvent = object { event_id, item, output_index, 2 more }` + - `"retention_ratio"` - Returned when a new Item is created during Response generation. + - `token_limits: optional object { post_instructions }` - - `event_id: string` + Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - The unique ID of the server event. + - `post_instructions: optional number` - - `item: ConversationItem` + Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - A single item within a Realtime conversation. +### Realtime Tool Choice Config - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` +- `RealtimeToolChoiceConfig = ToolChoiceOptions or ToolChoiceFunction or ToolChoiceMcp` - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. - - `content: array of object { text, type }` + - `ToolChoiceOptions = "none" or "auto" or "required"` - The content of the message. + Controls which (if any) tool is called by the model. - - `text: optional string` + `none` means the model will not call any tool and instead generates a message. - The text content. + `auto` means the model can pick between generating a message or calling one or + more tools. - - `type: optional "input_text"` + `required` means the model must call one or more tools. - The content type. Always `input_text` for system messages. + - `"none"` - - `"input_text"` + - `"auto"` - - `role: "system"` + - `"required"` - The role of the message sender. Always `system`. + - `ToolChoiceFunction object { name, type }` - - `"system"` + Use this option to force the model to call a specific function. - - `type: "message"` + - `name: string` - The type of the item. Always `message`. + The name of the function to call. - - `"message"` + - `type: "function"` - - `id: optional string` + For function calling, the type is always `function`. - The unique ID of the item. This may be provided by the client or generated by the server. + - `"function"` - - `object: optional "realtime.item"` + - `ToolChoiceMcp object { server_label, type, name }` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Use this option to force the model to call a specific tool on a remote MCP server. - - `"realtime.item"` + - `server_label: string` - - `status: optional "completed" or "incomplete" or "in_progress"` + The label of the MCP server to use. - The status of the item. Has no effect on the conversation. + - `type: "mcp"` - - `"completed"` + For MCP tools, the type is always `mcp`. - - `"incomplete"` + - `"mcp"` - - `"in_progress"` + - `name: optional string` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + The name of the tool to call on the server. - A user message item in a Realtime conversation. +### Realtime Tools Config - - `content: array of object { audio, detail, image_url, 3 more }` +- `RealtimeToolsConfig = array of RealtimeToolsConfigUnion` - The content of the message. + Tools available to the model. - - `audio: optional string` + - `RealtimeFunctionTool object { description, name, parameters, type }` - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `description: optional string` - - `detail: optional "auto" or "low" or "high"` + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - The detail level of the image (for `input_image`). `auto` will default to `high`. + - `name: optional string` - - `"auto"` + The name of the function. - - `"low"` + - `parameters: optional unknown` - - `"high"` + Parameters of the function in JSON Schema. - - `image_url: optional string` + - `type: optional "function"` - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + The type of the tool, i.e. `function`. - - `text: optional string` + - `"function"` - The text content (for `input_text`). + - `McpTool object { server_label, type, allowed_tools, 7 more }` - - `transcript: optional string` + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + - `server_label: string` - - `type: optional "input_text" or "input_audio" or "input_image"` + A label for this MCP server, used to identify it in tool calls. - The content type (`input_text`, `input_audio`, or `input_image`). + - `type: "mcp"` - - `"input_text"` + The type of the MCP tool. Always `mcp`. - - `"input_audio"` + - `"mcp"` - - `"input_image"` + - `allowed_tools: optional array of string or object { read_only, tool_names }` - - `role: "user"` + List of allowed tool names or a filter object. - The role of the message sender. Always `user`. + - `McpAllowedTools = array of string` - - `"user"` + A string array of allowed tool names - - `type: "message"` + - `McpToolFilter object { read_only, tool_names }` - The type of the item. Always `message`. + A filter object to specify which tools are allowed. - - `"message"` + - `read_only: optional boolean` - - `id: optional string` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - The unique ID of the item. This may be provided by the client or generated by the server. + - `tool_names: optional array of string` - - `object: optional "realtime.item"` + List of allowed tool names. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `authorization: optional string` - - `"realtime.item"` + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - The status of the item. Has no effect on the conversation. + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](/docs/guides/tools-remote-mcp#connectors). - - `"completed"` + Currently supported `connector_id` values are: - - `"incomplete"` + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` - - `"in_progress"` + - `"connector_dropbox"` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `"connector_gmail"` - An assistant message item in a Realtime conversation. + - `"connector_googlecalendar"` - - `content: array of object { audio, text, transcript, type }` + - `"connector_googledrive"` - The content of the message. + - `"connector_microsoftteams"` - - `audio: optional string` + - `"connector_outlookcalendar"` - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `"connector_outlookemail"` - - `text: optional string` + - `"connector_sharepoint"` - The text content. + - `defer_loading: optional boolean` - - `transcript: optional string` + Whether this MCP tool is deferred and discovered via tool search. - The transcript of the audio content, this will always be present if the output type is `audio`. + - `headers: optional map[string]` - - `type: optional "output_text" or "output_audio"` + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + - `require_approval: optional object { always, never } or "always" or "never"` - - `"output_text"` + Specify which of the MCP server's tools require approval. - - `"output_audio"` + - `McpToolApprovalFilter object { always, never }` - - `role: "assistant"` + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. - The role of the message sender. Always `assistant`. + - `always: optional object { read_only, tool_names }` - - `"assistant"` + A filter object to specify which tools are allowed. - - `type: "message"` + - `read_only: optional boolean` - The type of the item. Always `message`. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `"message"` + - `tool_names: optional array of string` - - `id: optional string` + List of allowed tool names. - The unique ID of the item. This may be provided by the client or generated by the server. + - `never: optional object { read_only, tool_names }` - - `object: optional "realtime.item"` + A filter object to specify which tools are allowed. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `read_only: optional boolean` - - `"realtime.item"` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `tool_names: optional array of string` - The status of the item. Has no effect on the conversation. + List of allowed tool names. - - `"completed"` + - `McpToolApprovalSetting = "always" or "never"` - - `"incomplete"` + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. - - `"in_progress"` + - `"always"` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `"never"` - A function call item in a Realtime conversation. + - `server_description: optional string` - - `arguments: string` + Optional description of the MCP server, used to provide more context. - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + - `server_url: optional string` - - `name: string` + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. - The name of the function being called. +### Realtime Tools Config Union - - `type: "function_call"` +- `RealtimeToolsConfigUnion = RealtimeFunctionTool or object { server_label, type, allowed_tools, 7 more }` - The type of the item. Always `function_call`. + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - - `"function_call"` + - `RealtimeFunctionTool object { description, name, parameters, type }` - - `id: optional string` + - `description: optional string` - The unique ID of the item. This may be provided by the client or generated by the server. + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - - `call_id: optional string` + - `name: optional string` - The ID of the function call. + The name of the function. - - `object: optional "realtime.item"` + - `parameters: optional unknown` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Parameters of the function in JSON Schema. - - `"realtime.item"` + - `type: optional "function"` - - `status: optional "completed" or "incomplete" or "in_progress"` + The type of the tool, i.e. `function`. - The status of the item. Has no effect on the conversation. + - `"function"` - - `"completed"` + - `McpTool object { server_label, type, allowed_tools, 7 more }` - - `"incomplete"` + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - - `"in_progress"` + - `server_label: string` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + A label for this MCP server, used to identify it in tool calls. - A function call output item in a Realtime conversation. + - `type: "mcp"` - - `call_id: string` + The type of the MCP tool. Always `mcp`. - The ID of the function call this output is for. + - `"mcp"` - - `output: string` + - `allowed_tools: optional array of string or object { read_only, tool_names }` - The output of the function call, this is free text and can contain any information or simply be empty. + List of allowed tool names or a filter object. - - `type: "function_call_output"` + - `McpAllowedTools = array of string` - The type of the item. Always `function_call_output`. + A string array of allowed tool names - - `"function_call_output"` + - `McpToolFilter object { read_only, tool_names }` - - `id: optional string` + A filter object to specify which tools are allowed. - The unique ID of the item. This may be provided by the client or generated by the server. + - `read_only: optional boolean` - - `object: optional "realtime.item"` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `tool_names: optional array of string` - - `"realtime.item"` + List of allowed tool names. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `authorization: optional string` - The status of the item. Has no effect on the conversation. + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. - - `"completed"` + - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - - `"incomplete"` + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](/docs/guides/tools-remote-mcp#connectors). - - `"in_progress"` + Currently supported `connector_id` values are: - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` - A Realtime item responding to an MCP approval request. + - `"connector_dropbox"` - - `id: string` + - `"connector_gmail"` - The unique ID of the approval response. + - `"connector_googlecalendar"` - - `approval_request_id: string` + - `"connector_googledrive"` - The ID of the approval request being answered. + - `"connector_microsoftteams"` - - `approve: boolean` + - `"connector_outlookcalendar"` - Whether the request was approved. + - `"connector_outlookemail"` - - `type: "mcp_approval_response"` + - `"connector_sharepoint"` - The type of the item. Always `mcp_approval_response`. + - `defer_loading: optional boolean` - - `"mcp_approval_response"` + Whether this MCP tool is deferred and discovered via tool search. - - `reason: optional string` + - `headers: optional map[string]` - Optional reason for the decision. + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `require_approval: optional object { always, never } or "always" or "never"` - A Realtime item listing tools available on an MCP server. + Specify which of the MCP server's tools require approval. - - `server_label: string` + - `McpToolApprovalFilter object { always, never }` - The label of the MCP server. + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. - - `tools: array of object { input_schema, name, annotations, description }` + - `always: optional object { read_only, tool_names }` - The tools available on the server. + A filter object to specify which tools are allowed. - - `input_schema: unknown` + - `read_only: optional boolean` - The JSON schema describing the tool's input. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `name: string` + - `tool_names: optional array of string` - The name of the tool. + List of allowed tool names. - - `annotations: optional unknown` + - `never: optional object { read_only, tool_names }` - Additional annotations about the tool. + A filter object to specify which tools are allowed. - - `description: optional string` + - `read_only: optional boolean` - The description of the tool. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `type: "mcp_list_tools"` + - `tool_names: optional array of string` - The type of the item. Always `mcp_list_tools`. + List of allowed tool names. - - `"mcp_list_tools"` + - `McpToolApprovalSetting = "always" or "never"` - - `id: optional string` + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. - The unique ID of the list. + - `"always"` - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `"never"` - A Realtime item representing an invocation of a tool on an MCP server. + - `server_description: optional string` - - `id: string` + Optional description of the MCP server, used to provide more context. - The unique ID of the tool call. + - `server_url: optional string` - - `arguments: string` + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. - A JSON string of the arguments passed to the tool. +### Realtime Tracing Config - - `name: string` +- `RealtimeTracingConfig = "auto" or object { group_id, metadata, workflow_name }` - The name of the tool that was run. + Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once + tracing is enabled for a session, the configuration cannot be modified. - - `server_label: string` + `auto` will create a trace for the session with default values for the + workflow name, group id, and metadata. - The label of the MCP server running the tool. + - `Auto = "auto"` - - `type: "mcp_call"` + Enables tracing and sets default values for tracing configuration options. Always `auto`. - The type of the item. Always `mcp_call`. + - `"auto"` - - `"mcp_call"` + - `TracingConfiguration object { group_id, metadata, workflow_name }` - - `approval_request_id: optional string` + Granular configuration for tracing. - The ID of an associated approval request, if any. + - `group_id: optional string` - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` + The group id to attach to this trace to enable filtering and + grouping in the Traces Dashboard. - The error from the tool call, if any. + - `metadata: optional unknown` - - `RealtimeMcpProtocolError = object { code, message, type }` + The arbitrary metadata to attach to this trace to enable + filtering in the Traces Dashboard. - - `code: number` + - `workflow_name: optional string` - - `message: string` + The name of the workflow to attach to this trace. This is used to + name the trace in the Traces Dashboard. - - `type: "protocol_error"` +### Realtime Transcription Session Audio - - `"protocol_error"` +- `RealtimeTranscriptionSessionAudio object { input }` - - `RealtimeMcpToolExecutionError = object { message, type }` + Configuration for input and output audio. - - `message: string` + - `input: optional RealtimeTranscriptionSessionAudioInput` - - `type: "tool_execution_error"` + - `format: optional RealtimeAudioFormats` - - `"tool_execution_error"` + The PCM audio format. Only a 24kHz sample rate is supported. - - `RealtimeMcphttpError = object { code, message, type }` + - `PCMAudioFormat object { rate, type }` - - `code: number` + The PCM audio format. Only a 24kHz sample rate is supported. - - `message: string` + - `rate: optional 24000` - - `type: "http_error"` + The sample rate of the audio. Always `24000`. - - `"http_error"` + - `24000` - - `output: optional string` + - `type: optional "audio/pcm"` - The output from the tool call. + The audio format. Always `audio/pcm`. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `"audio/pcm"` - A Realtime item requesting human approval of a tool invocation. + - `PCMUAudioFormat object { type }` - - `id: string` + The G.711 μ-law format. - The unique ID of the approval request. + - `type: optional "audio/pcmu"` - - `arguments: string` + The audio format. Always `audio/pcmu`. - A JSON string of arguments for the tool. + - `"audio/pcmu"` - - `name: string` + - `PCMAAudioFormat object { type }` - The name of the tool to run. + The G.711 A-law format. - - `server_label: string` + - `type: optional "audio/pcma"` - The label of the MCP server making the request. + The audio format. Always `audio/pcma`. - - `type: "mcp_approval_request"` + - `"audio/pcma"` - The type of the item. Always `mcp_approval_request`. + - `noise_reduction: optional object { type }` - - `"mcp_approval_request"` + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `output_index: number` + - `type: optional NoiseReductionType` - The index of the output item in the Response. + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - `response_id: string` + - `"near_field"` - The ID of the Response to which the item belongs. + - `"far_field"` - - `type: "response.output_item.added"` + - `transcription: optional AudioTranscription` - The event type, must be `response.output_item.added`. + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - `"response.output_item.added"` + - `language: optional string` - - `ResponseOutputItemDoneEvent = object { event_id, item, output_index, 2 more }` + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - Returned when an Item is done streaming. Also emitted when a Response is - interrupted, incomplete, or cancelled. + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `event_id: string` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - The unique ID of the server event. + - `string` - - `item: ConversationItem` + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - A single item within a Realtime conversation. + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `"whisper-1"` - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + - `"gpt-4o-mini-transcribe"` - - `content: array of object { text, type }` + - `"gpt-4o-mini-transcribe-2025-12-15"` - The content of the message. + - `"gpt-4o-transcribe"` - - `text: optional string` + - `"gpt-4o-transcribe-diarize"` - The text content. + - `prompt: optional string` - - `type: optional "input_text"` + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - The content type. Always `input_text` for system messages. + - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` - - `"input_text"` + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - `role: "system"` + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - The role of the message sender. Always `system`. + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `"system"` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - - `type: "message"` + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - The type of the item. Always `message`. + - `type: "server_vad"` - - `"message"` + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `id: optional string` + - `"server_vad"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `create_response: optional boolean` - - `object: optional "realtime.item"` + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `"realtime.item"` + - `idle_timeout_ms: optional number` - - `status: optional "completed" or "incomplete" or "in_progress"` + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - The status of the item. Has no effect on the conversation. + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - - `"completed"` + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - - `"incomplete"` + - `interrupt_response: optional boolean` - - `"in_progress"` + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - A user message item in a Realtime conversation. + - `prefix_padding_ms: optional number` - - `content: array of object { audio, detail, image_url, 3 more }` + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - The content of the message. + - `silence_duration_ms: optional number` - - `audio: optional string` + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `threshold: optional number` - - `detail: optional "auto" or "low" or "high"` + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - The detail level of the image (for `input_image`). `auto` will default to `high`. + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - - `"auto"` + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - `"low"` + - `type: "semantic_vad"` - - `"high"` + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - `image_url: optional string` + - `"semantic_vad"` - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + - `create_response: optional boolean` - - `text: optional string` + Whether or not to automatically generate a response when a VAD stop event occurs. - The text content (for `input_text`). + - `eagerness: optional "low" or "medium" or "high" or "auto"` - - `transcript: optional string` + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + - `"low"` - - `type: optional "input_text" or "input_audio" or "input_image"` + - `"medium"` - The content type (`input_text`, `input_audio`, or `input_image`). + - `"high"` - - `"input_text"` + - `"auto"` - - `"input_audio"` + - `interrupt_response: optional boolean` - - `"input_image"` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - `role: "user"` +### Realtime Transcription Session Audio Input - The role of the message sender. Always `user`. +- `RealtimeTranscriptionSessionAudioInput object { format, noise_reduction, transcription, turn_detection }` - - `"user"` + - `format: optional RealtimeAudioFormats` - - `type: "message"` + The PCM audio format. Only a 24kHz sample rate is supported. - The type of the item. Always `message`. + - `PCMAudioFormat object { rate, type }` - - `"message"` + The PCM audio format. Only a 24kHz sample rate is supported. - - `id: optional string` + - `rate: optional 24000` - The unique ID of the item. This may be provided by the client or generated by the server. + The sample rate of the audio. Always `24000`. - - `object: optional "realtime.item"` + - `24000` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `type: optional "audio/pcm"` - - `"realtime.item"` + The audio format. Always `audio/pcm`. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"audio/pcm"` - The status of the item. Has no effect on the conversation. + - `PCMUAudioFormat object { type }` - - `"completed"` + The G.711 μ-law format. - - `"incomplete"` + - `type: optional "audio/pcmu"` - - `"in_progress"` + The audio format. Always `audio/pcmu`. - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `"audio/pcmu"` - An assistant message item in a Realtime conversation. + - `PCMAAudioFormat object { type }` - - `content: array of object { audio, text, transcript, type }` + The G.711 A-law format. - The content of the message. + - `type: optional "audio/pcma"` - - `audio: optional string` + The audio format. Always `audio/pcma`. - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `"audio/pcma"` - - `text: optional string` + - `noise_reduction: optional object { type }` - The text content. + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `transcript: optional string` + - `type: optional NoiseReductionType` - The transcript of the audio content, this will always be present if the output type is `audio`. + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - `type: optional "output_text" or "output_audio"` + - `"near_field"` - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + - `"far_field"` - - `"output_text"` + - `transcription: optional AudioTranscription` - - `"output_audio"` + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - `role: "assistant"` + - `language: optional string` - The role of the message sender. Always `assistant`. + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - - `"assistant"` + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `type: "message"` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - The type of the item. Always `message`. + - `string` - - `"message"` + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `id: optional string` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - The unique ID of the item. This may be provided by the client or generated by the server. + - `"whisper-1"` - - `object: optional "realtime.item"` + - `"gpt-4o-mini-transcribe"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"gpt-4o-mini-transcribe-2025-12-15"` - - `"realtime.item"` + - `"gpt-4o-transcribe"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"gpt-4o-transcribe-diarize"` - The status of the item. Has no effect on the conversation. + - `prompt: optional string` - - `"completed"` + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - `"incomplete"` + - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` - - `"in_progress"` + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - A function call item in a Realtime conversation. + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `arguments: string` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - `name: string` + - `type: "server_vad"` - The name of the function being called. + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `type: "function_call"` + - `"server_vad"` - The type of the item. Always `function_call`. + - `create_response: optional boolean` - - `"function_call"` + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - `id: optional string` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - The unique ID of the item. This may be provided by the client or generated by the server. + - `idle_timeout_ms: optional number` - - `call_id: optional string` + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - The ID of the function call. + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - - `object: optional "realtime.item"` + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `interrupt_response: optional boolean` - - `"realtime.item"` + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - `status: optional "completed" or "incomplete" or "in_progress"` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - The status of the item. Has no effect on the conversation. + - `prefix_padding_ms: optional number` - - `"completed"` + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - - `"incomplete"` + - `silence_duration_ms: optional number` - - `"in_progress"` + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `threshold: optional number` - A function call output item in a Realtime conversation. + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - - `call_id: string` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - The ID of the function call this output is for. + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - `output: string` + - `type: "semantic_vad"` - The output of the function call, this is free text and can contain any information or simply be empty. + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - `type: "function_call_output"` + - `"semantic_vad"` - The type of the item. Always `function_call_output`. + - `create_response: optional boolean` - - `"function_call_output"` + Whether or not to automatically generate a response when a VAD stop event occurs. - - `id: optional string` + - `eagerness: optional "low" or "medium" or "high" or "auto"` - The unique ID of the item. This may be provided by the client or generated by the server. + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - `object: optional "realtime.item"` + - `"low"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"medium"` - - `"realtime.item"` + - `"high"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"auto"` - The status of the item. Has no effect on the conversation. + - `interrupt_response: optional boolean` - - `"completed"` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - `"incomplete"` +### Realtime Transcription Session Audio Input Turn Detection - - `"in_progress"` +- `RealtimeTranscriptionSessionAudioInputTurnDetection = object { type, create_response, idle_timeout_ms, 4 more } or object { type, create_response, eagerness, interrupt_response }` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - A Realtime item responding to an MCP approval request. + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - `id: string` + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - The unique ID of the approval response. + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - - `approval_request_id: string` + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - The ID of the approval request being answered. + - `type: "server_vad"` - - `approve: boolean` + Type of turn detection, `server_vad` to turn on simple Server VAD. - Whether the request was approved. + - `"server_vad"` - - `type: "mcp_approval_response"` + - `create_response: optional boolean` - The type of the item. Always `mcp_approval_response`. + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - `"mcp_approval_response"` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `reason: optional string` + - `idle_timeout_ms: optional number` - Optional reason for the decision. + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - A Realtime item listing tools available on an MCP server. + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - - `server_label: string` + - `interrupt_response: optional boolean` - The label of the MCP server. + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - `tools: array of object { input_schema, name, annotations, description }` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - The tools available on the server. + - `prefix_padding_ms: optional number` - - `input_schema: unknown` + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - The JSON schema describing the tool's input. + - `silence_duration_ms: optional number` - - `name: string` + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - The name of the tool. + - `threshold: optional number` - - `annotations: optional unknown` + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - Additional annotations about the tool. + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - - `description: optional string` + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - The description of the tool. + - `type: "semantic_vad"` - - `type: "mcp_list_tools"` + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - The type of the item. Always `mcp_list_tools`. + - `"semantic_vad"` - - `"mcp_list_tools"` + - `create_response: optional boolean` - - `id: optional string` + Whether or not to automatically generate a response when a VAD stop event occurs. - The unique ID of the list. + - `eagerness: optional "low" or "medium" or "high" or "auto"` - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - A Realtime item representing an invocation of a tool on an MCP server. + - `"low"` - - `id: string` + - `"medium"` - The unique ID of the tool call. + - `"high"` - - `arguments: string` + - `"auto"` - A JSON string of the arguments passed to the tool. + - `interrupt_response: optional boolean` - - `name: string` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - The name of the tool that was run. +### Realtime Transcription Session Create Request - - `server_label: string` +- `RealtimeTranscriptionSessionCreateRequest object { type, audio, include }` - The label of the MCP server running the tool. + Realtime transcription session object configuration. - - `type: "mcp_call"` + - `type: "transcription"` - The type of the item. Always `mcp_call`. + The type of session to create. Always `transcription` for transcription sessions. - - `"mcp_call"` + - `"transcription"` - - `approval_request_id: optional string` + - `audio: optional RealtimeTranscriptionSessionAudio` - The ID of an associated approval request, if any. + Configuration for input and output audio. - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` + - `input: optional RealtimeTranscriptionSessionAudioInput` - The error from the tool call, if any. + - `format: optional RealtimeAudioFormats` - - `RealtimeMcpProtocolError = object { code, message, type }` + The PCM audio format. Only a 24kHz sample rate is supported. - - `code: number` + - `PCMAudioFormat object { rate, type }` - - `message: string` + The PCM audio format. Only a 24kHz sample rate is supported. - - `type: "protocol_error"` + - `rate: optional 24000` - - `"protocol_error"` + The sample rate of the audio. Always `24000`. - - `RealtimeMcpToolExecutionError = object { message, type }` + - `24000` - - `message: string` + - `type: optional "audio/pcm"` - - `type: "tool_execution_error"` + The audio format. Always `audio/pcm`. - - `"tool_execution_error"` + - `"audio/pcm"` - - `RealtimeMcphttpError = object { code, message, type }` + - `PCMUAudioFormat object { type }` - - `code: number` + The G.711 μ-law format. - - `message: string` + - `type: optional "audio/pcmu"` - - `type: "http_error"` + The audio format. Always `audio/pcmu`. - - `"http_error"` + - `"audio/pcmu"` - - `output: optional string` + - `PCMAAudioFormat object { type }` - The output from the tool call. + The G.711 A-law format. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `type: optional "audio/pcma"` - A Realtime item requesting human approval of a tool invocation. + The audio format. Always `audio/pcma`. - - `id: string` + - `"audio/pcma"` - The unique ID of the approval request. + - `noise_reduction: optional object { type }` - - `arguments: string` + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - A JSON string of arguments for the tool. + - `type: optional NoiseReductionType` - - `name: string` + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - The name of the tool to run. + - `"near_field"` - - `server_label: string` + - `"far_field"` - The label of the MCP server making the request. + - `transcription: optional AudioTranscription` - - `type: "mcp_approval_request"` + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - The type of the item. Always `mcp_approval_request`. + - `language: optional string` - - `"mcp_approval_request"` + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - - `output_index: number` + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - The index of the output item in the Response. + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `response_id: string` + - `string` - The ID of the Response to which the item belongs. + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `type: "response.output_item.done"` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - The event type, must be `response.output_item.done`. + - `"whisper-1"` - - `"response.output_item.done"` + - `"gpt-4o-mini-transcribe"` - - `ResponseTextDeltaEvent = object { content_index, delta, event_id, 4 more }` + - `"gpt-4o-mini-transcribe-2025-12-15"` - Returned when the text value of an "output_text" content part is updated. + - `"gpt-4o-transcribe"` - - `content_index: number` + - `"gpt-4o-transcribe-diarize"` - The index of the content part in the item's content array. + - `prompt: optional string` - - `delta: string` + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - The text delta. - - - `event_id: string` - - The unique ID of the server event. - - - `item_id: string` - - The ID of the item. - - - `output_index: number` - - The index of the output item in the response. - - - `response_id: string` - - The ID of the response. - - - `type: "response.output_text.delta"` - - The event type, must be `response.output_text.delta`. - - - `"response.output_text.delta"` - - - `ResponseTextDoneEvent = object { content_index, event_id, item_id, 4 more }` - - Returned when the text value of an "output_text" content part is done streaming. Also - emitted when a Response is interrupted, incomplete, or cancelled. - - - `content_index: number` - - The index of the content part in the item's content array. - - - `event_id: string` - - The unique ID of the server event. - - - `item_id: string` - - The ID of the item. - - - `output_index: number` - - The index of the output item in the response. - - - `response_id: string` - - The ID of the response. - - - `text: string` - - The final text content. - - - `type: "response.output_text.done"` - - The event type, must be `response.output_text.done`. - - - `"response.output_text.done"` - - - `SessionCreatedEvent = object { event_id, session, type }` - - Returned when a Session is created. Emitted automatically when a new - connection is established as the first server event. This event will contain - the default Session configuration. - - - `event_id: string` - - The unique ID of the server event. - - - `session: RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` - - The session configuration. - - - `RealtimeSessionCreateRequest = object { type, audio, include, 9 more }` - - Realtime session object configuration. - - - `type: "realtime"` - - The type of session to create. Always `realtime` for the Realtime API. - - - `"realtime"` - - - `audio: optional RealtimeAudioConfig` - - Configuration for input and output audio. - - - `input: optional RealtimeAudioConfigInput` - - - `format: optional RealtimeAudioFormats` - - The format of the input audio. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `noise_reduction: optional object { type }` - - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - - `type: optional NoiseReductionType` - - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - - `"near_field"` - - - `"far_field"` - - - `transcription: optional AudioTranscription` - - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `string` - - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - - `turn_detection: optional RealtimeAudioInputTurnDetection` + - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. @@ -13103,7 +13415,7 @@ Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. @@ -13157,7 +13469,7 @@ higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` Server-side semantic turn detection which uses a model to determine when the user has finished speaking. @@ -13188,1279 +13500,1070 @@ Whether or not to automatically interrupt any ongoing response with output to the default conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - `output: optional RealtimeAudioConfigOutput` + - `include: optional array of "item.input_audio_transcription.logprobs"` - - `format: optional RealtimeAudioFormats` + Additional fields to include in server outputs. - The format of the output audio. + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - `PCMAudioFormat = object { rate, type }` + - `"item.input_audio_transcription.logprobs"` - The PCM audio format. Only a 24kHz sample rate is supported. +### Realtime Truncation - - `rate: optional 24000` +- `RealtimeTruncation = "auto" or "disabled" or object { retention_ratio, type, token_limits }` - The sample rate of the audio. Always `24000`. + When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - - `24000` + Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - - `type: optional "audio/pcm"` + Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - The audio format. Always `audio/pcm`. + Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - - `"audio/pcm"` + - `"auto" or "disabled"` - - `PCMUAudioFormat = object { type }` + The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - The G.711 μ-law format. + - `"auto"` - - `type: optional "audio/pcmu"` + - `"disabled"` - The audio format. Always `audio/pcmu`. + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` - - `"audio/pcmu"` + Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - - `PCMAAudioFormat = object { type }` + - `retention_ratio: number` - The G.711 A-law format. + Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - `type: optional "audio/pcma"` + - `type: "retention_ratio"` - The audio format. Always `audio/pcma`. + Use retention ratio truncation. - - `"audio/pcma"` + - `"retention_ratio"` - - `speed: optional number` + - `token_limits: optional object { post_instructions }` - The speed of the model's spoken response as a multiple of the original speed. - 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. + Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - This parameter is a post-processing adjustment to the audio after it is generated, it's - also possible to prompt the model to speak faster or slower. + - `post_instructions: optional number` - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` + Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with - an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed - during the session once the model has responded with audio at least once. - We recommend `marin` and `cedar` for best quality. +### Response Audio Delta Event - - `string` +- `ResponseAudioDeltaEvent object { content_index, delta, event_id, 4 more }` - - `"alloy" or "ash" or "ballad" or 7 more` + Returned when the model-generated audio is updated. - - `"alloy"` + - `content_index: number` - - `"ash"` + The index of the content part in the item's content array. - - `"ballad"` + - `delta: string` - - `"coral"` + Base64-encoded audio data delta. - - `"echo"` + - `event_id: string` - - `"sage"` + The unique ID of the server event. - - `"shimmer"` + - `item_id: string` - - `"verse"` + The ID of the item. - - `"marin"` + - `output_index: number` - - `"cedar"` + The index of the output item in the response. - - `ID = object { id }` + - `response_id: string` - Custom voice reference. + The ID of the response. - - `id: string` + - `type: "response.output_audio.delta"` - The custom voice ID, e.g. `voice_1234`. + The event type, must be `response.output_audio.delta`. - - `include: optional array of "item.input_audio_transcription.logprobs"` + - `"response.output_audio.delta"` - Additional fields to include in server outputs. +### Response Audio Done Event - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. +- `ResponseAudioDoneEvent object { content_index, event_id, item_id, 3 more }` - - `"item.input_audio_transcription.logprobs"` + Returned when the model-generated audio is done. Also emitted when a Response + is interrupted, incomplete, or cancelled. - - `instructions: optional string` + - `content_index: number` - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. + The index of the content part in the item's content array. - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. + - `event_id: string` - - `max_output_tokens: optional number or "inf"` + The unique ID of the server event. - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. + - `item_id: string` - - `number` + The ID of the item. - - `"inf"` + - `output_index: number` - - `"inf"` + The index of the output item in the response. - - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` + - `response_id: string` - The Realtime model used for this session. + The ID of the response. - - `string` + - `type: "response.output_audio.done"` - - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` + The event type, must be `response.output_audio.done`. - The Realtime model used for this session. + - `"response.output_audio.done"` - - `"gpt-realtime"` +### Response Audio Transcript Delta Event - - `"gpt-realtime-1.5"` +- `ResponseAudioTranscriptDeltaEvent object { content_index, delta, event_id, 4 more }` - - `"gpt-realtime-2025-08-28"` + Returned when the model-generated transcription of audio output is updated. - - `"gpt-4o-realtime-preview"` + - `content_index: number` - - `"gpt-4o-realtime-preview-2024-10-01"` + The index of the content part in the item's content array. - - `"gpt-4o-realtime-preview-2024-12-17"` + - `delta: string` - - `"gpt-4o-realtime-preview-2025-06-03"` + The transcript delta. - - `"gpt-4o-mini-realtime-preview"` + - `event_id: string` - - `"gpt-4o-mini-realtime-preview-2024-12-17"` + The unique ID of the server event. - - `"gpt-realtime-mini"` + - `item_id: string` - - `"gpt-realtime-mini-2025-10-06"` + The ID of the item. - - `"gpt-realtime-mini-2025-12-15"` + - `output_index: number` - - `"gpt-audio-1.5"` + The index of the output item in the response. - - `"gpt-audio-mini"` + - `response_id: string` - - `"gpt-audio-mini-2025-10-06"` + The ID of the response. - - `"gpt-audio-mini-2025-12-15"` + - `type: "response.output_audio_transcript.delta"` - - `output_modalities: optional array of "text" or "audio"` + The event type, must be `response.output_audio_transcript.delta`. - The set of modalities the model can respond with. It defaults to `["audio"]`, indicating - that the model will respond with audio plus a transcript. `["text"]` can be used to make - the model respond with text only. It is not possible to request both `text` and `audio` at the same time. + - `"response.output_audio_transcript.delta"` - - `"text"` +### Response Audio Transcript Done Event - - `"audio"` +- `ResponseAudioTranscriptDoneEvent object { content_index, event_id, item_id, 4 more }` - - `prompt: optional ResponsePrompt` + Returned when the model-generated transcription of audio output is done + streaming. Also emitted when a Response is interrupted, incomplete, or + cancelled. - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). + - `content_index: number` - - `id: string` + The index of the content part in the item's content array. - The unique identifier of the prompt template to use. + - `event_id: string` - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` + The unique ID of the server event. - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. + - `item_id: string` - - `string` + The ID of the item. - - `ResponseInputText = object { text, type }` + - `output_index: number` - A text input to the model. + The index of the output item in the response. - - `text: string` + - `response_id: string` - The text input to the model. + The ID of the response. - - `type: "input_text"` + - `transcript: string` - The type of the input item. Always `input_text`. + The final transcript of the audio. - - `"input_text"` + - `type: "response.output_audio_transcript.done"` - - `ResponseInputImage = object { detail, type, file_id, image_url }` + The event type, must be `response.output_audio_transcript.done`. - An image input to the model. Learn about [image inputs](/docs/guides/vision). + - `"response.output_audio_transcript.done"` - - `detail: "low" or "high" or "auto" or "original"` +### Response Cancel Event - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. +- `ResponseCancelEvent object { type, event_id, response_id }` - - `"low"` + Send this event to cancel an in-progress response. The server will respond + with a `response.done` event with a status of `response.status=cancelled`. If + there is no response to cancel, the server will respond with an error. It's safe + to call `response.cancel` even if no response is in progress, an error will be + returned the session will remain unaffected. - - `"high"` + - `type: "response.cancel"` - - `"auto"` + The event type, must be `response.cancel`. - - `"original"` + - `"response.cancel"` - - `type: "input_image"` + - `event_id: optional string` - The type of the input item. Always `input_image`. + Optional client-generated ID used to identify this event. - - `"input_image"` + - `response_id: optional string` - - `file_id: optional string` + A specific response ID to cancel - if not provided, will cancel an + in-progress response in the default conversation. - The ID of the file to be sent to the model. +### Response Content Part Added Event - - `image_url: optional string` +- `ResponseContentPartAddedEvent object { content_index, event_id, item_id, 4 more }` - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. + Returned when a new content part is added to an assistant message item during + response generation. - - `ResponseInputFile = object { type, detail, file_data, 3 more }` + - `content_index: number` - A file input to the model. + The index of the content part in the item's content array. - - `type: "input_file"` + - `event_id: string` - The type of the input item. Always `input_file`. + The unique ID of the server event. - - `"input_file"` + - `item_id: string` - - `detail: optional "low" or "high"` + The ID of the item to which the content part was added. - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. + - `output_index: number` - - `"low"` + The index of the output item in the response. - - `"high"` + - `part: object { audio, text, transcript, type }` - - `file_data: optional string` + The content part that was added. - The content of the file to be sent to the model. + - `audio: optional string` - - `file_id: optional string` + Base64-encoded audio data (if type is "audio"). - The ID of the file to be sent to the model. + - `text: optional string` - - `file_url: optional string` + The text content (if type is "text"). - The URL of the file to be sent to the model. + - `transcript: optional string` - - `filename: optional string` + The transcript of the audio (if type is "audio"). - The name of the file to be sent to the model. + - `type: optional "audio" or "text"` - - `version: optional string` + The content type ("text", "audio"). - Optional version of the prompt template. + - `"audio"` - - `tool_choice: optional RealtimeToolChoiceConfig` + - `"text"` - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. + - `response_id: string` - - `ToolChoiceOptions = "none" or "auto" or "required"` + The ID of the response. - Controls which (if any) tool is called by the model. + - `type: "response.content_part.added"` - `none` means the model will not call any tool and instead generates a message. + The event type, must be `response.content_part.added`. - `auto` means the model can pick between generating a message or calling one or - more tools. + - `"response.content_part.added"` - `required` means the model must call one or more tools. +### Response Content Part Done Event - - `"none"` +- `ResponseContentPartDoneEvent object { content_index, event_id, item_id, 4 more }` - - `"auto"` + Returned when a content part is done streaming in an assistant message item. + Also emitted when a Response is interrupted, incomplete, or cancelled. - - `"required"` + - `content_index: number` - - `ToolChoiceFunction = object { name, type }` + The index of the content part in the item's content array. - Use this option to force the model to call a specific function. + - `event_id: string` - - `name: string` + The unique ID of the server event. - The name of the function to call. + - `item_id: string` - - `type: "function"` + The ID of the item. - For function calling, the type is always `function`. + - `output_index: number` - - `"function"` + The index of the output item in the response. - - `ToolChoiceMcp = object { server_label, type, name }` + - `part: object { audio, text, transcript, type }` - Use this option to force the model to call a specific tool on a remote MCP server. + The content part that is done. - - `server_label: string` + - `audio: optional string` - The label of the MCP server to use. + Base64-encoded audio data (if type is "audio"). - - `type: "mcp"` + - `text: optional string` - For MCP tools, the type is always `mcp`. + The text content (if type is "text"). - - `"mcp"` + - `transcript: optional string` - - `name: optional string` + The transcript of the audio (if type is "audio"). - The name of the tool to call on the server. + - `type: optional "audio" or "text"` - - `tools: optional RealtimeToolsConfig` + The content type ("text", "audio"). - Tools available to the model. + - `"audio"` - - `RealtimeFunctionTool = object { description, name, parameters, type }` + - `"text"` - - `description: optional string` + - `response_id: string` - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). + The ID of the response. - - `name: optional string` + - `type: "response.content_part.done"` - The name of the function. + The event type, must be `response.content_part.done`. - - `parameters: optional unknown` + - `"response.content_part.done"` - Parameters of the function in JSON Schema. +### Response Create Event - - `type: optional "function"` +- `ResponseCreateEvent object { type, event_id, response }` - The type of the tool, i.e. `function`. + This event instructs the server to create a Response, which means triggering + model inference. When in Server VAD mode, the server will create Responses + automatically. - - `"function"` + A Response will include at least one Item, and may have two, in which case + the second will be a function call. These Items will be appended to the + conversation history by default. - - `McpTool = object { server_label, type, allowed_tools, 7 more }` + The server will respond with a `response.created` event, events for Items + and content created, and finally a `response.done` event to indicate the + Response is complete. - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). + The `response.create` event includes inference configuration like + `instructions` and `tools`. If these are set, they will override the Session's + configuration for this Response only. - - `server_label: string` + Responses can be created out-of-band of the default Conversation, meaning that they can + have arbitrary input, and it's possible to disable writing the output to the Conversation. + Only one Response can write to the default Conversation at a time, but otherwise multiple + Responses can be created in parallel. The `metadata` field is a good way to disambiguate + multiple simultaneous Responses. - A label for this MCP server, used to identify it in tool calls. + Clients can set `conversation` to `none` to create a Response that does not write to the default + Conversation. Arbitrary input can be provided with the `input` field, which is an array accepting + raw Items and references to existing Items. - - `type: "mcp"` + - `type: "response.create"` - The type of the MCP tool. Always `mcp`. + The event type, must be `response.create`. - - `"mcp"` + - `"response.create"` - - `allowed_tools: optional array of string or object { read_only, tool_names }` + - `event_id: optional string` - List of allowed tool names or a filter object. + Optional client-generated ID used to identify this event. - - `McpAllowedTools = array of string` + - `response: optional RealtimeResponseCreateParams` - A string array of allowed tool names + Create a new Realtime response with these parameters - - `McpToolFilter = object { read_only, tool_names }` + - `audio: optional RealtimeResponseCreateAudioOutput` - A filter object to specify which tools are allowed. + Configuration for audio input and output. - - `read_only: optional boolean` + - `output: optional object { format, voice }` - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `format: optional RealtimeAudioFormats` - - `tool_names: optional array of string` + The format of the output audio. - List of allowed tool names. + - `PCMAudioFormat object { rate, type }` - - `authorization: optional string` + The PCM audio format. Only a 24kHz sample rate is supported. - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. + - `rate: optional 24000` - - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` + The sample rate of the audio. Always `24000`. - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](/docs/guides/tools-remote-mcp#connectors). + - `24000` - Currently supported `connector_id` values are: + - `type: optional "audio/pcm"` - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` + The audio format. Always `audio/pcm`. - - `"connector_dropbox"` + - `"audio/pcm"` - - `"connector_gmail"` + - `PCMUAudioFormat object { type }` - - `"connector_googlecalendar"` + The G.711 μ-law format. - - `"connector_googledrive"` + - `type: optional "audio/pcmu"` - - `"connector_microsoftteams"` + The audio format. Always `audio/pcmu`. - - `"connector_outlookcalendar"` + - `"audio/pcmu"` - - `"connector_outlookemail"` + - `PCMAAudioFormat object { type }` - - `"connector_sharepoint"` + The G.711 A-law format. - - `defer_loading: optional boolean` + - `type: optional "audio/pcma"` - Whether this MCP tool is deferred and discovered via tool search. + The audio format. Always `audio/pcma`. - - `headers: optional map[string]` + - `"audio/pcma"` - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - - `require_approval: optional object { always, never } or "always" or "never"` + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. - Specify which of the MCP server's tools require approval. + - `string` - - `McpToolApprovalFilter = object { always, never }` + - `"alloy" or "ash" or "ballad" or 7 more` - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. + - `"alloy"` - - `always: optional object { read_only, tool_names }` + - `"ash"` - A filter object to specify which tools are allowed. + - `"ballad"` - - `read_only: optional boolean` + - `"coral"` - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `"echo"` - - `tool_names: optional array of string` + - `"sage"` - List of allowed tool names. + - `"shimmer"` - - `never: optional object { read_only, tool_names }` + - `"verse"` - A filter object to specify which tools are allowed. + - `"marin"` - - `read_only: optional boolean` + - `"cedar"` - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `ID object { id }` - - `tool_names: optional array of string` + Custom voice reference. - List of allowed tool names. + - `id: string` - - `McpToolApprovalSetting = "always" or "never"` + The custom voice ID, e.g. `voice_1234`. - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. + - `conversation: optional string or "auto" or "none"` - - `"always"` + Controls which conversation the response is added to. Currently supports + `auto` and `none`, with `auto` as the default value. The `auto` value + means that the contents of the response will be added to the default + conversation. Set this to `none` to create an out-of-band response which + will not add items to default conversation. - - `"never"` + - `string` - - `server_description: optional string` + - `"auto" or "none"` - Optional description of the MCP server, used to provide more context. + Controls which conversation the response is added to. Currently supports + `auto` and `none`, with `auto` as the default value. The `auto` value + means that the contents of the response will be added to the default + conversation. Set this to `none` to create an out-of-band response which + will not add items to default conversation. - - `server_url: optional string` + - `"auto"` - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. + - `"none"` - - `tracing: optional RealtimeTracingConfig` + - `input: optional array of ConversationItem` - Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once - tracing is enabled for a session, the configuration cannot be modified. + Input items to include in the prompt for the model. Using this field + creates a new context for this Response instead of using the default + conversation. An empty array `[]` will clear the context for this Response. + Note that this can include references to items that previously appeared in the session + using their id. - `auto` will create a trace for the session with default values for the - workflow name, group id, and metadata. + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` - - `Auto = "auto"` + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. - Enables tracing and sets default values for tracing configuration options. Always `auto`. + - `content: array of object { text, type }` - - `"auto"` + The content of the message. - - `TracingConfiguration = object { group_id, metadata, workflow_name }` + - `text: optional string` - Granular configuration for tracing. + The text content. - - `group_id: optional string` + - `type: optional "input_text"` - The group id to attach to this trace to enable filtering and - grouping in the Traces Dashboard. + The content type. Always `input_text` for system messages. - - `metadata: optional unknown` + - `"input_text"` - The arbitrary metadata to attach to this trace to enable - filtering in the Traces Dashboard. + - `role: "system"` - - `workflow_name: optional string` + The role of the message sender. Always `system`. - The name of the workflow to attach to this trace. This is used to - name the trace in the Traces Dashboard. + - `"system"` - - `truncation: optional RealtimeTruncation` + - `type: "message"` - When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. + The type of the item. Always `message`. - Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. + - `"message"` - Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. + - `id: optional string` - Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. + The unique ID of the item. This may be provided by the client or generated by the server. - - `"auto" or "disabled"` + - `object: optional "realtime.item"` - The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - `"auto"` + - `"realtime.item"` - - `"disabled"` + - `status: optional "completed" or "incomplete" or "in_progress"` - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` + The status of the item. Has no effect on the conversation. - Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. + - `"completed"` - - `retention_ratio: number` + - `"incomplete"` - Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. + - `"in_progress"` - - `type: "retention_ratio"` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` - Use retention ratio truncation. + A user message item in a Realtime conversation. - - `"retention_ratio"` + - `content: array of object { audio, detail, image_url, 3 more }` - - `token_limits: optional object { post_instructions }` + The content of the message. - Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. + - `audio: optional string` - - `post_instructions: optional number` + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. - Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. + - `detail: optional "auto" or "low" or "high"` - - `RealtimeTranscriptionSessionCreateRequest = object { type, audio, include }` + The detail level of the image (for `input_image`). `auto` will default to `high`. - Realtime transcription session object configuration. + - `"auto"` - - `type: "transcription"` + - `"low"` - The type of session to create. Always `transcription` for transcription sessions. + - `"high"` - - `"transcription"` + - `image_url: optional string` - - `audio: optional RealtimeTranscriptionSessionAudio` + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. - Configuration for input and output audio. + - `text: optional string` - - `input: optional RealtimeTranscriptionSessionAudioInput` + The text content (for `input_text`). - - `format: optional RealtimeAudioFormats` + - `transcript: optional string` - The PCM audio format. Only a 24kHz sample rate is supported. + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. - - `PCMAudioFormat = object { rate, type }` + - `type: optional "input_text" or "input_audio" or "input_image"` - The PCM audio format. Only a 24kHz sample rate is supported. + The content type (`input_text`, `input_audio`, or `input_image`). - - `rate: optional 24000` + - `"input_text"` - The sample rate of the audio. Always `24000`. + - `"input_audio"` - - `24000` + - `"input_image"` - - `type: optional "audio/pcm"` + - `role: "user"` - The audio format. Always `audio/pcm`. + The role of the message sender. Always `user`. - - `"audio/pcm"` + - `"user"` - - `PCMUAudioFormat = object { type }` + - `type: "message"` - The G.711 μ-law format. + The type of the item. Always `message`. - - `type: optional "audio/pcmu"` + - `"message"` - The audio format. Always `audio/pcmu`. + - `id: optional string` - - `"audio/pcmu"` + The unique ID of the item. This may be provided by the client or generated by the server. - - `PCMAAudioFormat = object { type }` + - `object: optional "realtime.item"` - The G.711 A-law format. + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - `type: optional "audio/pcma"` + - `"realtime.item"` - The audio format. Always `audio/pcma`. + - `status: optional "completed" or "incomplete" or "in_progress"` - - `"audio/pcma"` + The status of the item. Has no effect on the conversation. - - `noise_reduction: optional object { type }` + - `"completed"` - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + - `"incomplete"` - - `type: optional NoiseReductionType` + - `"in_progress"` - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` - - `"near_field"` + An assistant message item in a Realtime conversation. - - `"far_field"` + - `content: array of object { audio, text, transcript, type }` - - `transcription: optional AudioTranscription` + The content of the message. - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + - `audio: optional string` - - `language: optional string` + Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. + - `text: optional string` - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + The text content. - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `transcript: optional string` - - `string` + The transcript of the audio content, this will always be present if the output type is `audio`. - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `type: optional "output_text" or "output_audio"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. - - `"whisper-1"` + - `"output_text"` - - `"gpt-4o-mini-transcribe"` + - `"output_audio"` - - `"gpt-4o-mini-transcribe-2025-12-15"` + - `role: "assistant"` - - `"gpt-4o-transcribe"` + The role of the message sender. Always `assistant`. - - `"gpt-4o-transcribe-diarize"` + - `"assistant"` - - `prompt: optional string` + - `type: "message"` - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + The type of the item. Always `message`. - - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` + - `"message"` - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + - `id: optional string` - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + The unique ID of the item. This may be provided by the client or generated by the server. - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + - `object: optional "realtime.item"` - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. + - `"realtime.item"` - - `type: "server_vad"` + - `status: optional "completed" or "incomplete" or "in_progress"` - Type of turn detection, `server_vad` to turn on simple Server VAD. + The status of the item. Has no effect on the conversation. - - `"server_vad"` + - `"completed"` - - `create_response: optional boolean` + - `"incomplete"` - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + - `"in_progress"` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` - - `idle_timeout_ms: optional number` + A function call item in a Realtime conversation. - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. + - `arguments: string` - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. + The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. + - `name: string` - - `interrupt_response: optional boolean` + The name of the function being called. - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + - `type: "function_call"` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + The type of the item. Always `function_call`. - - `prefix_padding_ms: optional number` + - `"function_call"` - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. + - `id: optional string` - - `silence_duration_ms: optional number` + The unique ID of the item. This may be provided by the client or generated by the server. - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. + - `call_id: optional string` - - `threshold: optional number` + The ID of the function call. - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. + - `object: optional "realtime.item"` - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. + - `"realtime.item"` - - `type: "semantic_vad"` + - `status: optional "completed" or "incomplete" or "in_progress"` - Type of turn detection, `semantic_vad` to turn on Semantic VAD. + The status of the item. Has no effect on the conversation. - - `"semantic_vad"` + - `"completed"` - - `create_response: optional boolean` + - `"incomplete"` - Whether or not to automatically generate a response when a VAD stop event occurs. + - `"in_progress"` - - `eagerness: optional "low" or "medium" or "high" or "auto"` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + A function call output item in a Realtime conversation. - - `"low"` + - `call_id: string` - - `"medium"` + The ID of the function call this output is for. - - `"high"` + - `output: string` - - `"auto"` + The output of the function call, this is free text and can contain any information or simply be empty. - - `interrupt_response: optional boolean` + - `type: "function_call_output"` - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + The type of the item. Always `function_call_output`. - - `include: optional array of "item.input_audio_transcription.logprobs"` + - `"function_call_output"` - Additional fields to include in server outputs. + - `id: optional string` - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + The unique ID of the item. This may be provided by the client or generated by the server. - - `"item.input_audio_transcription.logprobs"` + - `object: optional "realtime.item"` - - `type: "session.created"` + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - The event type, must be `session.created`. + - `"realtime.item"` - - `"session.created"` + - `status: optional "completed" or "incomplete" or "in_progress"` - - `SessionUpdatedEvent = object { event_id, session, type }` + The status of the item. Has no effect on the conversation. - Returned when a session is updated with a `session.update` event, unless - there is an error. + - `"completed"` - - `event_id: string` + - `"incomplete"` - The unique ID of the server event. + - `"in_progress"` - - `session: RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` - The session configuration. + A Realtime item responding to an MCP approval request. - - `RealtimeSessionCreateRequest = object { type, audio, include, 9 more }` + - `id: string` - Realtime session object configuration. + The unique ID of the approval response. - - `type: "realtime"` + - `approval_request_id: string` - The type of session to create. Always `realtime` for the Realtime API. + The ID of the approval request being answered. - - `"realtime"` + - `approve: boolean` - - `audio: optional RealtimeAudioConfig` + Whether the request was approved. - Configuration for input and output audio. + - `type: "mcp_approval_response"` - - `input: optional RealtimeAudioConfigInput` + The type of the item. Always `mcp_approval_response`. - - `format: optional RealtimeAudioFormats` + - `"mcp_approval_response"` - The format of the input audio. + - `reason: optional string` - - `PCMAudioFormat = object { rate, type }` + Optional reason for the decision. - The PCM audio format. Only a 24kHz sample rate is supported. + - `RealtimeMcpListTools object { server_label, tools, type, id }` - - `rate: optional 24000` + A Realtime item listing tools available on an MCP server. - The sample rate of the audio. Always `24000`. + - `server_label: string` - - `24000` + The label of the MCP server. - - `type: optional "audio/pcm"` + - `tools: array of object { input_schema, name, annotations, description }` - The audio format. Always `audio/pcm`. + The tools available on the server. - - `"audio/pcm"` + - `input_schema: unknown` - - `PCMUAudioFormat = object { type }` + The JSON schema describing the tool's input. - The G.711 μ-law format. + - `name: string` - - `type: optional "audio/pcmu"` + The name of the tool. - The audio format. Always `audio/pcmu`. + - `annotations: optional unknown` - - `"audio/pcmu"` + Additional annotations about the tool. - - `PCMAAudioFormat = object { type }` + - `description: optional string` - The G.711 A-law format. + The description of the tool. - - `type: optional "audio/pcma"` + - `type: "mcp_list_tools"` - The audio format. Always `audio/pcma`. + The type of the item. Always `mcp_list_tools`. - - `"audio/pcma"` + - `"mcp_list_tools"` - - `noise_reduction: optional object { type }` + - `id: optional string` - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + The unique ID of the list. - - `type: optional NoiseReductionType` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + A Realtime item representing an invocation of a tool on an MCP server. - - `"near_field"` + - `id: string` - - `"far_field"` + The unique ID of the tool call. - - `transcription: optional AudioTranscription` + - `arguments: string` - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + A JSON string of the arguments passed to the tool. - - `language: optional string` + - `name: string` - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. + The name of the tool that was run. - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `server_label: string` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + The label of the MCP server running the tool. - - `string` + - `type: "mcp_call"` - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + The type of the item. Always `mcp_call`. - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `"mcp_call"` - - `"whisper-1"` + - `approval_request_id: optional string` - - `"gpt-4o-mini-transcribe"` + The ID of an associated approval request, if any. - - `"gpt-4o-mini-transcribe-2025-12-15"` + - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` - - `"gpt-4o-transcribe"` + The error from the tool call, if any. - - `"gpt-4o-transcribe-diarize"` + - `RealtimeMcpProtocolError object { code, message, type }` - - `prompt: optional string` + - `code: number` - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + - `message: string` - - `turn_detection: optional RealtimeAudioInputTurnDetection` + - `type: "protocol_error"` - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + - `"protocol_error"` - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + - `RealtimeMcpToolExecutionError object { message, type }` - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + - `message: string` - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + - `type: "tool_execution_error"` - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. + - `"tool_execution_error"` - - `type: "server_vad"` + - `RealtimeMcphttpError object { code, message, type }` - Type of turn detection, `server_vad` to turn on simple Server VAD. + - `code: number` - - `"server_vad"` + - `message: string` - - `create_response: optional boolean` + - `type: "http_error"` - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + - `"http_error"` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + - `output: optional string` - - `idle_timeout_ms: optional number` + The output from the tool call. - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. + A Realtime item requesting human approval of a tool invocation. - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. + - `id: string` - - `interrupt_response: optional boolean` + The unique ID of the approval request. - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + - `arguments: string` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + A JSON string of arguments for the tool. - - `prefix_padding_ms: optional number` + - `name: string` - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. + The name of the tool to run. - - `silence_duration_ms: optional number` + - `server_label: string` - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. + The label of the MCP server making the request. - - `threshold: optional number` + - `type: "mcp_approval_request"` - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. + The type of the item. Always `mcp_approval_request`. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `"mcp_approval_request"` - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. + - `instructions: optional string` - - `type: "semantic_vad"` + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - Type of turn detection, `semantic_vad` to turn on Semantic VAD. + - `max_output_tokens: optional number or "inf"` - - `"semantic_vad"` + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - - `create_response: optional boolean` + - `number` - Whether or not to automatically generate a response when a VAD stop event occurs. + - `"inf"` - - `eagerness: optional "low" or "medium" or "high" or "auto"` + - `"inf"` - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + - `metadata: optional Metadata` - - `"low"` + Set of 16 key-value pairs that can be attached to an object. This can be + useful for storing additional information about the object in a structured + format, and querying for objects via API or the dashboard. - - `"medium"` + Keys are strings with a maximum length of 64 characters. Values are strings + with a maximum length of 512 characters. - - `"high"` + - `output_modalities: optional array of "text" or "audio"` - - `"auto"` + The set of modalities the model used to respond, currently the only possible values are + `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the + output to mode `text` will disable audio output from the model. - - `interrupt_response: optional boolean` + - `"text"` - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + - `"audio"` - - `output: optional RealtimeAudioConfigOutput` + - `prompt: optional ResponsePrompt` - - `format: optional RealtimeAudioFormats` + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - The format of the output audio. + - `id: string` - - `PCMAudioFormat = object { rate, type }` + The unique identifier of the prompt template to use. - The PCM audio format. Only a 24kHz sample rate is supported. + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - `rate: optional 24000` + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - The sample rate of the audio. Always `24000`. + - `string` - - `24000` + - `ResponseInputText object { text, type }` - - `type: optional "audio/pcm"` + A text input to the model. - The audio format. Always `audio/pcm`. + - `text: string` - - `"audio/pcm"` + The text input to the model. - - `PCMUAudioFormat = object { type }` + - `type: "input_text"` - The G.711 μ-law format. + The type of the input item. Always `input_text`. - - `type: optional "audio/pcmu"` + - `"input_text"` - The audio format. Always `audio/pcmu`. + - `ResponseInputImage object { detail, type, file_id, image_url }` - - `"audio/pcmu"` + An image input to the model. Learn about [image inputs](/docs/guides/vision). - - `PCMAAudioFormat = object { type }` + - `detail: "low" or "high" or "auto" or "original"` - The G.711 A-law format. + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - `type: optional "audio/pcma"` + - `"low"` - The audio format. Always `audio/pcma`. + - `"high"` - - `"audio/pcma"` + - `"auto"` - - `speed: optional number` + - `"original"` - The speed of the model's spoken response as a multiple of the original speed. - 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. + - `type: "input_image"` - This parameter is a post-processing adjustment to the audio after it is generated, it's - also possible to prompt the model to speak faster or slower. + The type of the input item. Always `input_image`. - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` + - `"input_image"` - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with - an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed - during the session once the model has responded with audio at least once. - We recommend `marin` and `cedar` for best quality. + - `file_id: optional string` - - `string` + The ID of the file to be sent to the model. - - `"alloy" or "ash" or "ballad" or 7 more` + - `image_url: optional string` - - `"alloy"` + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - `"ash"` + - `ResponseInputFile object { type, detail, file_data, 3 more }` - - `"ballad"` + A file input to the model. - - `"coral"` + - `type: "input_file"` - - `"echo"` + The type of the input item. Always `input_file`. - - `"sage"` + - `"input_file"` - - `"shimmer"` + - `detail: optional "low" or "high"` - - `"verse"` + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - `"marin"` + - `"low"` - - `"cedar"` + - `"high"` - - `ID = object { id }` + - `file_data: optional string` - Custom voice reference. + The content of the file to be sent to the model. - - `id: string` + - `file_id: optional string` - The custom voice ID, e.g. `voice_1234`. + The ID of the file to be sent to the model. - - `include: optional array of "item.input_audio_transcription.logprobs"` + - `file_url: optional string` - Additional fields to include in server outputs. + The URL of the file to be sent to the model. - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + - `filename: optional string` - - `"item.input_audio_transcription.logprobs"` + The name of the file to be sent to the model. - - `instructions: optional string` + - `version: optional string` - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. + Optional version of the prompt template. - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. + - `tool_choice: optional ToolChoiceOptions or ToolChoiceFunction or ToolChoiceMcp` - - `max_output_tokens: optional number or "inf"` + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. + - `ToolChoiceOptions = "none" or "auto" or "required"` - - `number` + Controls which (if any) tool is called by the model. - - `"inf"` + `none` means the model will not call any tool and instead generates a message. - - `"inf"` - - - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - The Realtime model used for this session. - - - `string` - - - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - The Realtime model used for this session. - - - `"gpt-realtime"` - - - `"gpt-realtime-1.5"` - - - `"gpt-realtime-2025-08-28"` - - - `"gpt-4o-realtime-preview"` - - - `"gpt-4o-realtime-preview-2024-10-01"` - - - `"gpt-4o-realtime-preview-2024-12-17"` - - - `"gpt-4o-realtime-preview-2025-06-03"` - - - `"gpt-4o-mini-realtime-preview"` - - - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - - `"gpt-realtime-mini"` - - - `"gpt-realtime-mini-2025-10-06"` - - - `"gpt-realtime-mini-2025-12-15"` - - - `"gpt-audio-1.5"` - - - `"gpt-audio-mini"` - - - `"gpt-audio-mini-2025-10-06"` - - - `"gpt-audio-mini-2025-12-15"` - - - `output_modalities: optional array of "text" or "audio"` - - The set of modalities the model can respond with. It defaults to `["audio"]`, indicating - that the model will respond with audio plus a transcript. `["text"]` can be used to make - the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - - - `"text"` - - - `"audio"` - - - `prompt: optional ResponsePrompt` - - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - - - `id: string` - - The unique identifier of the prompt template to use. - - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. - - - `string` - - - `ResponseInputText = object { text, type }` - - A text input to the model. - - - `text: string` - - The text input to the model. - - - `type: "input_text"` - - The type of the input item. Always `input_text`. - - - `"input_text"` - - - `ResponseInputImage = object { detail, type, file_id, image_url }` - - An image input to the model. Learn about [image inputs](/docs/guides/vision). - - - `detail: "low" or "high" or "auto" or "original"` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `"low"` - - - `"high"` - - - `"auto"` - - - `"original"` - - - `type: "input_image"` - - The type of the input item. Always `input_image`. - - - `"input_image"` - - - `file_id: optional string` - - The ID of the file to be sent to the model. - - - `image_url: optional string` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `ResponseInputFile = object { type, detail, file_data, 3 more }` - - A file input to the model. - - - `type: "input_file"` - - The type of the input item. Always `input_file`. - - - `"input_file"` - - - `detail: optional "low" or "high"` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `"low"` - - - `"high"` - - - `file_data: optional string` - - The content of the file to be sent to the model. - - - `file_id: optional string` - - The ID of the file to be sent to the model. - - - `file_url: optional string` - - The URL of the file to be sent to the model. - - - `filename: optional string` - - The name of the file to be sent to the model. - - - `version: optional string` - - Optional version of the prompt template. - - - `tool_choice: optional RealtimeToolChoiceConfig` - - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. - - - `ToolChoiceOptions = "none" or "auto" or "required"` - - Controls which (if any) tool is called by the model. - - `none` means the model will not call any tool and instead generates a message. - - `auto` means the model can pick between generating a message or calling one or - more tools. + `auto` means the model can pick between generating a message or calling one or + more tools. `required` means the model must call one or more tools. @@ -14470,7 +14573,7 @@ - `"required"` - - `ToolChoiceFunction = object { name, type }` + - `ToolChoiceFunction object { name, type }` Use this option to force the model to call a specific function. @@ -14484,7 +14587,7 @@ - `"function"` - - `ToolChoiceMcp = object { server_label, type, name }` + - `ToolChoiceMcp object { server_label, type, name }` Use this option to force the model to call a specific tool on a remote MCP server. @@ -14502,11 +14605,11 @@ The name of the tool to call on the server. - - `tools: optional RealtimeToolsConfig` + - `tools: optional array of RealtimeFunctionTool or object { server_label, type, allowed_tools, 7 more }` Tools available to the model. - - `RealtimeFunctionTool = object { description, name, parameters, type }` + - `RealtimeFunctionTool object { description, name, parameters, type }` - `description: optional string` @@ -14528,7 +14631,7 @@ - `"function"` - - `McpTool = object { server_label, type, allowed_tools, 7 more }` + - `McpTool object { server_label, type, allowed_tools, 7 more }` Give the model access to additional tools via remote Model Context Protocol (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). @@ -14551,7 +14654,7 @@ A string array of allowed tool names - - `McpToolFilter = object { read_only, tool_names }` + - `McpToolFilter object { read_only, tool_names }` A filter object to specify which tools are allowed. @@ -14617,7 +14720,7 @@ Specify which of the MCP server's tools require approval. - - `McpToolApprovalFilter = object { always, never }` + - `McpToolApprovalFilter object { always, never }` Specify which of the MCP server's tools require approval. Can be `always`, `never`, or a filter object associated with tools @@ -14670,100 +14773,36 @@ The URL for the MCP server. One of `server_url` or `connector_id` must be provided. - - `tracing: optional RealtimeTracingConfig` - - Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once - tracing is enabled for a session, the configuration cannot be modified. - - `auto` will create a trace for the session with default values for the - workflow name, group id, and metadata. - - - `Auto = "auto"` - - Enables tracing and sets default values for tracing configuration options. Always `auto`. - - - `"auto"` - - - `TracingConfiguration = object { group_id, metadata, workflow_name }` - - Granular configuration for tracing. - - - `group_id: optional string` - - The group id to attach to this trace to enable filtering and - grouping in the Traces Dashboard. - - - `metadata: optional unknown` - - The arbitrary metadata to attach to this trace to enable - filtering in the Traces Dashboard. - - - `workflow_name: optional string` - - The name of the workflow to attach to this trace. This is used to - name the trace in the Traces Dashboard. - - - `truncation: optional RealtimeTruncation` - - When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - - Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - - Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - - - `"auto" or "disabled"` - - The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - - - `"auto"` - - - `"disabled"` - - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` - - Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - - - `retention_ratio: number` - - Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - - `type: "retention_ratio"` - - Use retention ratio truncation. - - - `"retention_ratio"` - - - `token_limits: optional object { post_instructions }` +### Response Created Event - Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. +- `ResponseCreatedEvent object { event_id, response, type }` - - `post_instructions: optional number` + Returned when a new Response is created. The first event of response creation, + where the response is in an initial state of `in_progress`. - Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. + - `event_id: string` - - `RealtimeTranscriptionSessionCreateRequest = object { type, audio, include }` + The unique ID of the server event. - Realtime transcription session object configuration. + - `response: RealtimeResponse` - - `type: "transcription"` + The response resource. - The type of session to create. Always `transcription` for transcription sessions. + - `id: optional string` - - `"transcription"` + The unique ID of the response, will look like `resp_1234`. - - `audio: optional RealtimeTranscriptionSessionAudio` + - `audio: optional object { output }` - Configuration for input and output audio. + Configuration for audio output. - - `input: optional RealtimeTranscriptionSessionAudioInput` + - `output: optional object { format, voice }` - `format: optional RealtimeAudioFormats` - The PCM audio format. Only a 24kHz sample rate is supported. + The format of the output audio. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -14779,7 +14818,7 @@ - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -14789,7 +14828,7 @@ - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -14799,247 +14838,84 @@ - `"audio/pcma"` - - `noise_reduction: optional object { type }` + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more` - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - - `type: optional NoiseReductionType` - - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - - `"near_field"` - - - `"far_field"` - - - `transcription: optional AudioTranscription` - - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + The voice the model uses to respond. Voice cannot be changed during the + session once the model has responded with audio at least once. Current + voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, + `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for + best quality. - `string` - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` - - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` - - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - - `type: "server_vad"` - - Type of turn detection, `server_vad` to turn on simple Server VAD. - - - `"server_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `idle_timeout_ms: optional number` - - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. - - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. - - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `prefix_padding_ms: optional number` - - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. - - - `silence_duration_ms: optional number` - - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. - - - `threshold: optional number` - - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. - - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` - - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - - `type: "semantic_vad"` - - Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - - `"semantic_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. - - - `eagerness: optional "low" or "medium" or "high" or "auto"` - - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - - `"low"` - - - `"medium"` - - - `"high"` - - - `"auto"` - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - - `include: optional array of "item.input_audio_transcription.logprobs"` - - Additional fields to include in server outputs. - - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - - `"item.input_audio_transcription.logprobs"` - - - `type: "session.updated"` - - The event type, must be `session.updated`. - - - `"session.updated"` - - - `OutputAudioBufferStarted = object { event_id, response_id, type }` - - **WebRTC/SIP Only:** Emitted when the server begins streaming audio to the client. This event is - emitted after an audio content part has been added (`response.content_part.added`) - to the response. - [Learn more](/docs/guides/realtime-conversations#client-and-server-events-for-audio-in-webrtc). - - - `event_id: string` - - The unique ID of the server event. - - - `response_id: string` - - The unique ID of the response that produced the audio. - - - `type: "output_audio_buffer.started"` - - The event type, must be `output_audio_buffer.started`. - - - `"output_audio_buffer.started"` + - `"alloy" or "ash" or "ballad" or 7 more` - - `OutputAudioBufferStopped = object { event_id, response_id, type }` + The voice the model uses to respond. Voice cannot be changed during the + session once the model has responded with audio at least once. Current + voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, + `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for + best quality. - **WebRTC/SIP Only:** Emitted when the output audio buffer has been completely drained on the server, - and no more audio is forthcoming. This event is emitted after the full response - data has been sent to the client (`response.done`). - [Learn more](/docs/guides/realtime-conversations#client-and-server-events-for-audio-in-webrtc). + - `"alloy"` - - `event_id: string` + - `"ash"` - The unique ID of the server event. + - `"ballad"` - - `response_id: string` + - `"coral"` - The unique ID of the response that produced the audio. + - `"echo"` - - `type: "output_audio_buffer.stopped"` + - `"sage"` - The event type, must be `output_audio_buffer.stopped`. + - `"shimmer"` - - `"output_audio_buffer.stopped"` + - `"verse"` - - `OutputAudioBufferCleared = object { event_id, response_id, type }` + - `"marin"` - **WebRTC/SIP Only:** Emitted when the output audio buffer is cleared. This happens either in VAD - mode when the user has interrupted (`input_audio_buffer.speech_started`), - or when the client has emitted the `output_audio_buffer.clear` event to manually - cut off the current audio response. - [Learn more](/docs/guides/realtime-conversations#client-and-server-events-for-audio-in-webrtc). + - `"cedar"` - - `event_id: string` + - `conversation_id: optional string` - The unique ID of the server event. + Which conversation the response is added to, determined by the `conversation` + field in the `response.create` event. If `auto`, the response will be added to + the default conversation and the value of `conversation_id` will be an id like + `conv_1234`. If `none`, the response will not be added to any conversation and + the value of `conversation_id` will be `null`. If responses are being triggered + automatically by VAD the response will be added to the default conversation - - `response_id: string` + - `max_output_tokens: optional number or "inf"` - The unique ID of the response that produced the audio. + Maximum number of output tokens for a single assistant response, + inclusive of tool calls, that was used in this response. - - `type: "output_audio_buffer.cleared"` + - `number` - The event type, must be `output_audio_buffer.cleared`. + - `"inf"` - - `"output_audio_buffer.cleared"` + - `"inf"` - - `ConversationItemAdded = object { event_id, item, type, previous_item_id }` + - `metadata: optional Metadata` - Sent by the server when an Item is added to the default Conversation. This can happen in several cases: + Set of 16 key-value pairs that can be attached to an object. This can be + useful for storing additional information about the object in a structured + format, and querying for objects via API or the dashboard. - - When the client sends a `conversation.item.create` event. - - When the input audio buffer is committed. In this case the item will be a user message containing the audio from the buffer. - - When the model is generating a Response. In this case the `conversation.item.added` event will be sent when the model starts generating a specific Item, and thus it will not yet have any content (and `status` will be `in_progress`). + Keys are strings with a maximum length of 64 characters. Values are strings + with a maximum length of 512 characters. - The event will include the full content of the Item (except when model is generating a Response) except for audio data, which can be retrieved separately with a `conversation.item.retrieve` event if necessary. + - `object: optional "realtime.response"` - - `event_id: string` + The object type, must be `realtime.response`. - The unique ID of the server event. + - `"realtime.response"` - - `item: ConversationItem` + - `output: optional array of ConversationItem` - A single item within a Realtime conversation. + The list of output items generated by the response. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. @@ -15089,7 +14965,7 @@ - `"in_progress"` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` A user message item in a Realtime conversation. @@ -15165,7 +15041,7 @@ - `"in_progress"` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` An assistant message item in a Realtime conversation. @@ -15225,7 +15101,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` A function call item in a Realtime conversation. @@ -15267,7 +15143,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` A function call output item in a Realtime conversation. @@ -15305,7 +15181,7 @@ - `"in_progress"` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` A Realtime item responding to an MCP approval request. @@ -15331,7 +15207,7 @@ Optional reason for the decision. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `RealtimeMcpListTools object { server_label, tools, type, id }` A Realtime item listing tools available on an MCP server. @@ -15369,7 +15245,7 @@ The unique ID of the list. - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` A Realtime item representing an invocation of a tool on an MCP server. @@ -15403,7 +15279,7 @@ The error from the tool call, if any. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `RealtimeMcpProtocolError object { code, message, type }` - `code: number` @@ -15413,7 +15289,7 @@ - `"protocol_error"` - - `RealtimeMcpToolExecutionError = object { message, type }` + - `RealtimeMcpToolExecutionError object { message, type }` - `message: string` @@ -15421,7 +15297,7 @@ - `"tool_execution_error"` - - `RealtimeMcphttpError = object { code, message, type }` + - `RealtimeMcphttpError object { code, message, type }` - `code: number` @@ -15435,7 +15311,7 @@ The output from the tool call. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` A Realtime item requesting human approval of a tool invocation. @@ -15461,252 +15337,520 @@ - `"mcp_approval_request"` - - `type: "conversation.item.added"` + - `output_modalities: optional array of "text" or "audio"` - The event type, must be `conversation.item.added`. + The set of modalities the model used to respond, currently the only possible values are + `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the + output to mode `text` will disable audio output from the model. - - `"conversation.item.added"` + - `"text"` - - `previous_item_id: optional string` + - `"audio"` - The ID of the item that precedes this one, if any. This is used to - maintain ordering when items are inserted. + - `status: optional "completed" or "cancelled" or "failed" or 2 more` - - `ConversationItemDone = object { event_id, item, type, previous_item_id }` + The final status of the response (`completed`, `cancelled`, `failed`, or + `incomplete`, `in_progress`). - Returned when a conversation item is finalized. + - `"completed"` - The event will include the full content of the Item except for audio data, which can be retrieved separately with a `conversation.item.retrieve` event if needed. + - `"cancelled"` - - `event_id: string` + - `"failed"` - The unique ID of the server event. + - `"incomplete"` - - `item: ConversationItem` + - `"in_progress"` - A single item within a Realtime conversation. + - `status_details: optional RealtimeResponseStatus` - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + Additional details about the status. - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + - `error: optional object { code, type }` - - `content: array of object { text, type }` + A description of the error that caused the response to fail, + populated when the `status` is `failed`. - The content of the message. + - `code: optional string` - - `text: optional string` + Error code, if any. - The text content. + - `type: optional string` - - `type: optional "input_text"` + The type of error. - The content type. Always `input_text` for system messages. + - `reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"` - - `"input_text"` + The reason the Response did not complete. For a `cancelled` Response, one of `turn_detected` (the server VAD detected a new start of speech) or `client_cancelled` (the client sent a cancel event). For an `incomplete` Response, one of `max_output_tokens` or `content_filter` (the server-side safety filter activated and cut off the response). - - `role: "system"` + - `"turn_detected"` - The role of the message sender. Always `system`. + - `"client_cancelled"` - - `"system"` + - `"max_output_tokens"` - - `type: "message"` + - `"content_filter"` - The type of the item. Always `message`. + - `type: optional "completed" or "cancelled" or "failed" or "incomplete"` - - `"message"` + The type of error that caused the response to fail, corresponding + with the `status` field (`completed`, `cancelled`, `incomplete`, + `failed`). - - `id: optional string` + - `"completed"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `"cancelled"` - - `object: optional "realtime.item"` + - `"failed"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"incomplete"` - - `"realtime.item"` + - `usage: optional RealtimeResponseUsage` - - `status: optional "completed" or "incomplete" or "in_progress"` + Usage statistics for the Response, this will correspond to billing. A + Realtime API session will maintain a conversation context and append new + Items to the Conversation, thus output from previous turns (text and + audio tokens) will become the input for later turns. - The status of the item. Has no effect on the conversation. + - `input_token_details: optional RealtimeResponseUsageInputTokenDetails` - - `"completed"` + Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. - - `"incomplete"` + - `audio_tokens: optional number` - - `"in_progress"` + The number of audio tokens used as input for the Response. - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `cached_tokens: optional number` - A user message item in a Realtime conversation. + The number of cached tokens used as input for the Response. - - `content: array of object { audio, detail, image_url, 3 more }` + - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` - The content of the message. + Details about the cached tokens used as input for the Response. - - `audio: optional string` + - `audio_tokens: optional number` - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + The number of cached audio tokens used as input for the Response. - - `detail: optional "auto" or "low" or "high"` + - `image_tokens: optional number` - The detail level of the image (for `input_image`). `auto` will default to `high`. + The number of cached image tokens used as input for the Response. - - `"auto"` + - `text_tokens: optional number` - - `"low"` + The number of cached text tokens used as input for the Response. - - `"high"` + - `image_tokens: optional number` - - `image_url: optional string` + The number of image tokens used as input for the Response. - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + - `text_tokens: optional number` - - `text: optional string` + The number of text tokens used as input for the Response. - The text content (for `input_text`). + - `input_tokens: optional number` - - `transcript: optional string` + The number of input tokens used in the Response, including text and + audio tokens. - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + - `output_token_details: optional RealtimeResponseUsageOutputTokenDetails` - - `type: optional "input_text" or "input_audio" or "input_image"` + Details about the output tokens used in the Response. - The content type (`input_text`, `input_audio`, or `input_image`). + - `audio_tokens: optional number` - - `"input_text"` + The number of audio tokens used in the Response. - - `"input_audio"` + - `text_tokens: optional number` - - `"input_image"` + The number of text tokens used in the Response. - - `role: "user"` + - `output_tokens: optional number` - The role of the message sender. Always `user`. + The number of output tokens sent in the Response, including text and + audio tokens. - - `"user"` + - `total_tokens: optional number` - - `type: "message"` + The total number of tokens in the Response including input and output + text and audio tokens. - The type of the item. Always `message`. + - `type: "response.created"` - - `"message"` + The event type, must be `response.created`. - - `id: optional string` + - `"response.created"` - The unique ID of the item. This may be provided by the client or generated by the server. +### Response Done Event - - `object: optional "realtime.item"` +- `ResponseDoneEvent object { event_id, response, type }` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Returned when a Response is done streaming. Always emitted, no matter the + final state. The Response object included in the `response.done` event will + include all output Items in the Response but will omit the raw audio data. - - `"realtime.item"` + Clients should check the `status` field of the Response to determine if it was successful + (`completed`) or if there was another outcome: `cancelled`, `failed`, or `incomplete`. - - `status: optional "completed" or "incomplete" or "in_progress"` + A response will contain all output items that were generated during the response, excluding + any audio content. - The status of the item. Has no effect on the conversation. + - `event_id: string` - - `"completed"` + The unique ID of the server event. - - `"incomplete"` + - `response: RealtimeResponse` - - `"in_progress"` + The response resource. - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `id: optional string` - An assistant message item in a Realtime conversation. + The unique ID of the response, will look like `resp_1234`. - - `content: array of object { audio, text, transcript, type }` + - `audio: optional object { output }` - The content of the message. + Configuration for audio output. - - `audio: optional string` + - `output: optional object { format, voice }` - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `format: optional RealtimeAudioFormats` - - `text: optional string` + The format of the output audio. - The text content. + - `PCMAudioFormat object { rate, type }` - - `transcript: optional string` + The PCM audio format. Only a 24kHz sample rate is supported. - The transcript of the audio content, this will always be present if the output type is `audio`. + - `rate: optional 24000` - - `type: optional "output_text" or "output_audio"` + The sample rate of the audio. Always `24000`. - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + - `24000` - - `"output_text"` + - `type: optional "audio/pcm"` - - `"output_audio"` + The audio format. Always `audio/pcm`. - - `role: "assistant"` + - `"audio/pcm"` - The role of the message sender. Always `assistant`. + - `PCMUAudioFormat object { type }` - - `"assistant"` + The G.711 μ-law format. - - `type: "message"` + - `type: optional "audio/pcmu"` - The type of the item. Always `message`. + The audio format. Always `audio/pcmu`. - - `"message"` + - `"audio/pcmu"` - - `id: optional string` + - `PCMAAudioFormat object { type }` - The unique ID of the item. This may be provided by the client or generated by the server. + The G.711 A-law format. - - `object: optional "realtime.item"` + - `type: optional "audio/pcma"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + The audio format. Always `audio/pcma`. - - `"realtime.item"` + - `"audio/pcma"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more` - The status of the item. Has no effect on the conversation. + The voice the model uses to respond. Voice cannot be changed during the + session once the model has responded with audio at least once. Current + voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, + `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for + best quality. - - `"completed"` + - `string` - - `"incomplete"` + - `"alloy" or "ash" or "ballad" or 7 more` - - `"in_progress"` + The voice the model uses to respond. Voice cannot be changed during the + session once the model has responded with audio at least once. Current + voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, + `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for + best quality. - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `"alloy"` - A function call item in a Realtime conversation. + - `"ash"` - - `arguments: string` + - `"ballad"` - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + - `"coral"` - - `name: string` + - `"echo"` - The name of the function being called. + - `"sage"` - - `type: "function_call"` + - `"shimmer"` - The type of the item. Always `function_call`. + - `"verse"` - - `"function_call"` + - `"marin"` - - `id: optional string` + - `"cedar"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `conversation_id: optional string` - - `call_id: optional string` + Which conversation the response is added to, determined by the `conversation` + field in the `response.create` event. If `auto`, the response will be added to + the default conversation and the value of `conversation_id` will be an id like + `conv_1234`. If `none`, the response will not be added to any conversation and + the value of `conversation_id` will be `null`. If responses are being triggered + automatically by VAD the response will be added to the default conversation - The ID of the function call. + - `max_output_tokens: optional number or "inf"` - - `object: optional "realtime.item"` + Maximum number of output tokens for a single assistant response, + inclusive of tool calls, that was used in this response. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `number` - - `"realtime.item"` + - `"inf"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"inf"` - The status of the item. Has no effect on the conversation. + - `metadata: optional Metadata` + + Set of 16 key-value pairs that can be attached to an object. This can be + useful for storing additional information about the object in a structured + format, and querying for objects via API or the dashboard. + + Keys are strings with a maximum length of 64 characters. Values are strings + with a maximum length of 512 characters. + + - `object: optional "realtime.response"` + + The object type, must be `realtime.response`. + + - `"realtime.response"` + + - `output: optional array of ConversationItem` + + The list of output items generated by the response. + + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` + + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + + - `content: array of object { text, type }` + + The content of the message. + + - `text: optional string` + + The text content. + + - `type: optional "input_text"` + + The content type. Always `input_text` for system messages. + + - `"input_text"` + + - `role: "system"` + + The role of the message sender. Always `system`. + + - `"system"` + + - `type: "message"` + + The type of the item. Always `message`. + + - `"message"` + + - `id: optional string` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `object: optional "realtime.item"` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `"realtime.item"` + + - `status: optional "completed" or "incomplete" or "in_progress"` + + The status of the item. Has no effect on the conversation. + + - `"completed"` + + - `"incomplete"` + + - `"in_progress"` + + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` + + A user message item in a Realtime conversation. + + - `content: array of object { audio, detail, image_url, 3 more }` + + The content of the message. + + - `audio: optional string` + + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `detail: optional "auto" or "low" or "high"` + + The detail level of the image (for `input_image`). `auto` will default to `high`. + + - `"auto"` + + - `"low"` + + - `"high"` + + - `image_url: optional string` + + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + + - `text: optional string` + + The text content (for `input_text`). + + - `transcript: optional string` + + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + + - `type: optional "input_text" or "input_audio" or "input_image"` + + The content type (`input_text`, `input_audio`, or `input_image`). + + - `"input_text"` + + - `"input_audio"` + + - `"input_image"` + + - `role: "user"` + + The role of the message sender. Always `user`. + + - `"user"` + + - `type: "message"` + + The type of the item. Always `message`. + + - `"message"` + + - `id: optional string` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `object: optional "realtime.item"` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `"realtime.item"` + + - `status: optional "completed" or "incomplete" or "in_progress"` + + The status of the item. Has no effect on the conversation. + + - `"completed"` + + - `"incomplete"` + + - `"in_progress"` + + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` + + An assistant message item in a Realtime conversation. + + - `content: array of object { audio, text, transcript, type }` + + The content of the message. + + - `audio: optional string` + + Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `text: optional string` + + The text content. + + - `transcript: optional string` + + The transcript of the audio content, this will always be present if the output type is `audio`. + + - `type: optional "output_text" or "output_audio"` + + The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + + - `"output_text"` + + - `"output_audio"` + + - `role: "assistant"` + + The role of the message sender. Always `assistant`. + + - `"assistant"` + + - `type: "message"` + + The type of the item. Always `message`. + + - `"message"` + + - `id: optional string` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `object: optional "realtime.item"` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `"realtime.item"` + + - `status: optional "completed" or "incomplete" or "in_progress"` + + The status of the item. Has no effect on the conversation. + + - `"completed"` + + - `"incomplete"` + + - `"in_progress"` + + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` + + A function call item in a Realtime conversation. + + - `arguments: string` + + The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + + - `name: string` + + The name of the function being called. + + - `type: "function_call"` + + The type of the item. Always `function_call`. + + - `"function_call"` + + - `id: optional string` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `call_id: optional string` + + The ID of the function call. + + - `object: optional "realtime.item"` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `"realtime.item"` + + - `status: optional "completed" or "incomplete" or "in_progress"` + + The status of the item. Has no effect on the conversation. - `"completed"` @@ -15714,7 +15858,7 @@ - `"in_progress"` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` A function call output item in a Realtime conversation. @@ -15752,7 +15896,7 @@ - `"in_progress"` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` A Realtime item responding to an MCP approval request. @@ -15778,7 +15922,7 @@ Optional reason for the decision. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `RealtimeMcpListTools object { server_label, tools, type, id }` A Realtime item listing tools available on an MCP server. @@ -15816,7 +15960,7 @@ The unique ID of the list. - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` A Realtime item representing an invocation of a tool on an MCP server. @@ -15850,7 +15994,7 @@ The error from the tool call, if any. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `RealtimeMcpProtocolError object { code, message, type }` - `code: number` @@ -15860,7 +16004,7 @@ - `"protocol_error"` - - `RealtimeMcpToolExecutionError = object { message, type }` + - `RealtimeMcpToolExecutionError object { message, type }` - `message: string` @@ -15868,7 +16012,7 @@ - `"tool_execution_error"` - - `RealtimeMcphttpError = object { code, message, type }` + - `RealtimeMcphttpError object { code, message, type }` - `code: number` @@ -15882,7 +16026,7 @@ The output from the tool call. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` A Realtime item requesting human approval of a tool invocation. @@ -15908,118 +16052,163 @@ - `"mcp_approval_request"` - - `type: "conversation.item.done"` + - `output_modalities: optional array of "text" or "audio"` - The event type, must be `conversation.item.done`. + The set of modalities the model used to respond, currently the only possible values are + `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the + output to mode `text` will disable audio output from the model. - - `"conversation.item.done"` + - `"text"` - - `previous_item_id: optional string` + - `"audio"` - The ID of the item that precedes this one, if any. This is used to - maintain ordering when items are inserted. + - `status: optional "completed" or "cancelled" or "failed" or 2 more` - - `InputAudioBufferTimeoutTriggered = object { audio_end_ms, audio_start_ms, event_id, 2 more }` + The final status of the response (`completed`, `cancelled`, `failed`, or + `incomplete`, `in_progress`). - Returned when the Server VAD timeout is triggered for the input audio buffer. This is configured - with `idle_timeout_ms` in the `turn_detection` settings of the session, and it indicates that - there hasn't been any speech detected for the configured duration. + - `"completed"` - The `audio_start_ms` and `audio_end_ms` fields indicate the segment of audio after the last - model response up to the triggering time, as an offset from the beginning of audio written - to the input audio buffer. This means it demarcates the segment of audio that was silent and - the difference between the start and end values will roughly match the configured timeout. + - `"cancelled"` - The empty audio will be committed to the conversation as an `input_audio` item (there will be a - `input_audio_buffer.committed` event) and a model response will be generated. There may be speech - that didn't trigger VAD but is still detected by the model, so the model may respond with - something relevant to the conversation or a prompt to continue speaking. + - `"failed"` - - `audio_end_ms: number` + - `"incomplete"` - Millisecond offset of audio written to the input audio buffer at the time the timeout was triggered. + - `"in_progress"` - - `audio_start_ms: number` + - `status_details: optional RealtimeResponseStatus` - Millisecond offset of audio written to the input audio buffer that was after the playback time of the last model response. + Additional details about the status. - - `event_id: string` + - `error: optional object { code, type }` - The unique ID of the server event. + A description of the error that caused the response to fail, + populated when the `status` is `failed`. - - `item_id: string` + - `code: optional string` - The ID of the item associated with this segment. + Error code, if any. - - `type: "input_audio_buffer.timeout_triggered"` + - `type: optional string` - The event type, must be `input_audio_buffer.timeout_triggered`. + The type of error. - - `"input_audio_buffer.timeout_triggered"` + - `reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"` - - `ConversationItemInputAudioTranscriptionSegment = object { id, content_index, end, 6 more }` + The reason the Response did not complete. For a `cancelled` Response, one of `turn_detected` (the server VAD detected a new start of speech) or `client_cancelled` (the client sent a cancel event). For an `incomplete` Response, one of `max_output_tokens` or `content_filter` (the server-side safety filter activated and cut off the response). - Returned when an input audio transcription segment is identified for an item. + - `"turn_detected"` - - `id: string` + - `"client_cancelled"` - The segment identifier. + - `"max_output_tokens"` - - `content_index: number` + - `"content_filter"` - The index of the input audio content part within the item. + - `type: optional "completed" or "cancelled" or "failed" or "incomplete"` - - `end: number` + The type of error that caused the response to fail, corresponding + with the `status` field (`completed`, `cancelled`, `incomplete`, + `failed`). - End time of the segment in seconds. + - `"completed"` - - `event_id: string` + - `"cancelled"` - The unique ID of the server event. + - `"failed"` - - `item_id: string` + - `"incomplete"` - The ID of the item containing the input audio content. + - `usage: optional RealtimeResponseUsage` - - `speaker: string` + Usage statistics for the Response, this will correspond to billing. A + Realtime API session will maintain a conversation context and append new + Items to the Conversation, thus output from previous turns (text and + audio tokens) will become the input for later turns. - The detected speaker label for this segment. + - `input_token_details: optional RealtimeResponseUsageInputTokenDetails` - - `start: number` + Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. - Start time of the segment in seconds. + - `audio_tokens: optional number` - - `text: string` + The number of audio tokens used as input for the Response. - The text for this segment. + - `cached_tokens: optional number` - - `type: "conversation.item.input_audio_transcription.segment"` + The number of cached tokens used as input for the Response. - The event type, must be `conversation.item.input_audio_transcription.segment`. + - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` - - `"conversation.item.input_audio_transcription.segment"` + Details about the cached tokens used as input for the Response. - - `McpListToolsInProgress = object { event_id, item_id, type }` + - `audio_tokens: optional number` - Returned when listing MCP tools is in progress for an item. + The number of cached audio tokens used as input for the Response. - - `event_id: string` + - `image_tokens: optional number` - The unique ID of the server event. + The number of cached image tokens used as input for the Response. - - `item_id: string` + - `text_tokens: optional number` - The ID of the MCP list tools item. + The number of cached text tokens used as input for the Response. - - `type: "mcp_list_tools.in_progress"` + - `image_tokens: optional number` - The event type, must be `mcp_list_tools.in_progress`. + The number of image tokens used as input for the Response. - - `"mcp_list_tools.in_progress"` + - `text_tokens: optional number` + + The number of text tokens used as input for the Response. - - `McpListToolsCompleted = object { event_id, item_id, type }` + - `input_tokens: optional number` - Returned when listing MCP tools has completed for an item. + The number of input tokens used in the Response, including text and + audio tokens. + + - `output_token_details: optional RealtimeResponseUsageOutputTokenDetails` + + Details about the output tokens used in the Response. + + - `audio_tokens: optional number` + + The number of audio tokens used in the Response. + + - `text_tokens: optional number` + + The number of text tokens used in the Response. + + - `output_tokens: optional number` + + The number of output tokens sent in the Response, including text and + audio tokens. + + - `total_tokens: optional number` + + The total number of tokens in the Response including input and output + text and audio tokens. + + - `type: "response.done"` + + The event type, must be `response.done`. + + - `"response.done"` + +### Response Function Call Arguments Delta Event + +- `ResponseFunctionCallArgumentsDeltaEvent object { call_id, delta, event_id, 4 more }` + + Returned when the model-generated function call arguments are updated. + + - `call_id: string` + + The ID of the function call. + + - `delta: string` + + The arguments delta as a JSON string. - `event_id: string` @@ -16027,17 +16216,36 @@ - `item_id: string` - The ID of the MCP list tools item. + The ID of the function call item. - - `type: "mcp_list_tools.completed"` + - `output_index: number` - The event type, must be `mcp_list_tools.completed`. + The index of the output item in the response. - - `"mcp_list_tools.completed"` + - `response_id: string` - - `McpListToolsFailed = object { event_id, item_id, type }` + The ID of the response. - Returned when listing MCP tools has failed for an item. + - `type: "response.function_call_arguments.delta"` + + The event type, must be `response.function_call_arguments.delta`. + + - `"response.function_call_arguments.delta"` + +### Response Function Call Arguments Done Event + +- `ResponseFunctionCallArgumentsDoneEvent object { arguments, call_id, event_id, 5 more }` + + Returned when the model-generated function call arguments are done streaming. + Also emitted when a Response is interrupted, incomplete, or cancelled. + + - `arguments: string` + + The final arguments as a JSON string. + + - `call_id: string` + + The ID of the function call. - `event_id: string` @@ -16045,15 +16253,29 @@ - `item_id: string` - The ID of the MCP list tools item. + The ID of the function call item. - - `type: "mcp_list_tools.failed"` + - `name: string` - The event type, must be `mcp_list_tools.failed`. + The name of the function that was called. - - `"mcp_list_tools.failed"` + - `output_index: number` + + The index of the output item in the response. + + - `response_id: string` + + The ID of the response. + + - `type: "response.function_call_arguments.done"` + + The event type, must be `response.function_call_arguments.done`. + + - `"response.function_call_arguments.done"` + +### Response Mcp Call Arguments Delta - - `ResponseMcpCallArgumentsDelta = object { delta, event_id, item_id, 4 more }` +- `ResponseMcpCallArgumentsDelta object { delta, event_id, item_id, 4 more }` Returned when MCP tool call arguments are updated during response generation. @@ -16087,7 +16309,9 @@ If present, indicates the delta text was obfuscated. - - `ResponseMcpCallArgumentsDone = object { arguments, event_id, item_id, 3 more }` +### Response Mcp Call Arguments Done + +- `ResponseMcpCallArgumentsDone object { arguments, event_id, item_id, 3 more }` Returned when MCP tool call arguments are finalized during response generation. @@ -16117,9 +16341,11 @@ - `"response.mcp_call_arguments.done"` - - `ResponseMcpCallInProgress = object { event_id, item_id, output_index, type }` +### Response Mcp Call Completed + +- `ResponseMcpCallCompleted object { event_id, item_id, output_index, type }` - Returned when an MCP tool call has started and is in progress. + Returned when an MCP tool call has completed successfully. - `event_id: string` @@ -16133,15 +16359,17 @@ The index of the output item in the response. - - `type: "response.mcp_call.in_progress"` + - `type: "response.mcp_call.completed"` - The event type, must be `response.mcp_call.in_progress`. + The event type, must be `response.mcp_call.completed`. - - `"response.mcp_call.in_progress"` + - `"response.mcp_call.completed"` + +### Response Mcp Call Failed - - `ResponseMcpCallCompleted = object { event_id, item_id, output_index, type }` +- `ResponseMcpCallFailed object { event_id, item_id, output_index, type }` - Returned when an MCP tool call has completed successfully. + Returned when an MCP tool call has failed. - `event_id: string` @@ -16155,15 +16383,17 @@ The index of the output item in the response. - - `type: "response.mcp_call.completed"` + - `type: "response.mcp_call.failed"` - The event type, must be `response.mcp_call.completed`. + The event type, must be `response.mcp_call.failed`. - - `"response.mcp_call.completed"` + - `"response.mcp_call.failed"` + +### Response Mcp Call In Progress - - `ResponseMcpCallFailed = object { event_id, item_id, output_index, type }` +- `ResponseMcpCallInProgress object { event_id, item_id, output_index, type }` - Returned when an MCP tool call has failed. + Returned when an MCP tool call has started and is in progress. - `event_id: string` @@ -16177,1493 +16407,1494 @@ The index of the output item in the response. - - `type: "response.mcp_call.failed"` + - `type: "response.mcp_call.in_progress"` - The event type, must be `response.mcp_call.failed`. + The event type, must be `response.mcp_call.in_progress`. - - `"response.mcp_call.failed"` + - `"response.mcp_call.in_progress"` -### Realtime Session +### Response Output Item Added Event -- `RealtimeSession = object { id, expires_at, include, 17 more }` +- `ResponseOutputItemAddedEvent object { event_id, item, output_index, 2 more }` - Realtime session object for the beta interface. + Returned when a new Item is created during Response generation. - - `id: optional string` + - `event_id: string` - Unique identifier for the session that looks like `sess_1234567890abcdef`. + The unique ID of the server event. - - `expires_at: optional number` + - `item: ConversationItem` - Expiration timestamp for the session, in seconds since epoch. + A single item within a Realtime conversation. - - `include: optional array of "item.input_audio_transcription.logprobs"` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` - Additional fields to include in server outputs. + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. - - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + - `content: array of object { text, type }` - - `"item.input_audio_transcription.logprobs"` + The content of the message. - - `input_audio_format: optional "pcm16" or "g711_ulaw" or "g711_alaw"` + - `text: optional string` - The format of input audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. - For `pcm16`, input audio must be 16-bit PCM at a 24kHz sample rate, - single channel (mono), and little-endian byte order. + The text content. - - `"pcm16"` + - `type: optional "input_text"` - - `"g711_ulaw"` + The content type. Always `input_text` for system messages. - - `"g711_alaw"` + - `"input_text"` - - `input_audio_noise_reduction: optional object { type }` + - `role: "system"` - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + The role of the message sender. Always `system`. - - `type: optional NoiseReductionType` + - `"system"` - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + - `type: "message"` - - `"near_field"` + The type of the item. Always `message`. - - `"far_field"` + - `"message"` - - `input_audio_transcription: optional AudioTranscription` + - `id: optional string` - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](https://platform.openai.com/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + The unique ID of the item. This may be provided by the client or generated by the server. - - `language: optional string` + - `object: optional "realtime.item"` - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `"realtime.item"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `status: optional "completed" or "incomplete" or "in_progress"` - - `string` + The status of the item. Has no effect on the conversation. - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `"completed"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `"incomplete"` - - `"whisper-1"` + - `"in_progress"` - - `"gpt-4o-mini-transcribe"` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` - - `"gpt-4o-mini-transcribe-2025-12-15"` + A user message item in a Realtime conversation. - - `"gpt-4o-transcribe"` + - `content: array of object { audio, detail, image_url, 3 more }` - - `"gpt-4o-transcribe-diarize"` + The content of the message. - - `prompt: optional string` + - `audio: optional string` - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. - - `instructions: optional string` + - `detail: optional "auto" or "low" or "high"` - The default system instructions (i.e. system message) prepended to model - calls. This field allows the client to guide the model on desired - responses. The model can be instructed on response content and format, - (e.g. "be extremely succinct", "act friendly", "here are examples of good - responses") and on audio behavior (e.g. "talk quickly", "inject emotion - into your voice", "laugh frequently"). The instructions are not - guaranteed to be followed by the model, but they provide guidance to the - model on the desired behavior. + The detail level of the image (for `input_image`). `auto` will default to `high`. - Note that the server sets default instructions which will be used if this - field is not set and are visible in the `session.created` event at the - start of the session. + - `"auto"` - - `max_response_output_tokens: optional number or "inf"` + - `"low"` - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. + - `"high"` - - `number` + - `image_url: optional string` - - `"inf"` + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. - - `"inf"` + - `text: optional string` - - `modalities: optional array of "text" or "audio"` + The text content (for `input_text`). - The set of modalities the model can respond with. To disable audio, - set this to ["text"]. + - `transcript: optional string` - - `"text"` + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. - - `"audio"` + - `type: optional "input_text" or "input_audio" or "input_image"` - - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` + The content type (`input_text`, `input_audio`, or `input_image`). - The Realtime model used for this session. + - `"input_text"` - - `string` + - `"input_audio"` - - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` + - `"input_image"` - The Realtime model used for this session. + - `role: "user"` - - `"gpt-realtime"` + The role of the message sender. Always `user`. - - `"gpt-realtime-1.5"` + - `"user"` - - `"gpt-realtime-2025-08-28"` + - `type: "message"` - - `"gpt-4o-realtime-preview"` + The type of the item. Always `message`. - - `"gpt-4o-realtime-preview-2024-10-01"` + - `"message"` - - `"gpt-4o-realtime-preview-2024-12-17"` + - `id: optional string` - - `"gpt-4o-realtime-preview-2025-06-03"` + The unique ID of the item. This may be provided by the client or generated by the server. - - `"gpt-4o-mini-realtime-preview"` + - `object: optional "realtime.item"` - - `"gpt-4o-mini-realtime-preview-2024-12-17"` + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - `"gpt-realtime-mini"` + - `"realtime.item"` - - `"gpt-realtime-mini-2025-10-06"` + - `status: optional "completed" or "incomplete" or "in_progress"` - - `"gpt-realtime-mini-2025-12-15"` + The status of the item. Has no effect on the conversation. - - `"gpt-audio-1.5"` + - `"completed"` - - `"gpt-audio-mini"` + - `"incomplete"` - - `"gpt-audio-mini-2025-10-06"` + - `"in_progress"` - - `"gpt-audio-mini-2025-12-15"` + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` - - `object: optional "realtime.session"` + An assistant message item in a Realtime conversation. - The object type. Always `realtime.session`. + - `content: array of object { audio, text, transcript, type }` - - `"realtime.session"` + The content of the message. - - `output_audio_format: optional "pcm16" or "g711_ulaw" or "g711_alaw"` + - `audio: optional string` - The format of output audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. - For `pcm16`, output audio is sampled at a rate of 24kHz. + Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. - - `"pcm16"` + - `text: optional string` - - `"g711_ulaw"` + The text content. - - `"g711_alaw"` + - `transcript: optional string` - - `prompt: optional ResponsePrompt` + The transcript of the audio content, this will always be present if the output type is `audio`. - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). + - `type: optional "output_text" or "output_audio"` - - `id: string` + The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. - The unique identifier of the prompt template to use. + - `"output_text"` - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` + - `"output_audio"` - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. + - `role: "assistant"` - - `string` + The role of the message sender. Always `assistant`. - - `ResponseInputText = object { text, type }` + - `"assistant"` - A text input to the model. + - `type: "message"` - - `text: string` + The type of the item. Always `message`. - The text input to the model. + - `"message"` - - `type: "input_text"` + - `id: optional string` - The type of the input item. Always `input_text`. + The unique ID of the item. This may be provided by the client or generated by the server. - - `"input_text"` + - `object: optional "realtime.item"` - - `ResponseInputImage = object { detail, type, file_id, image_url }` + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - An image input to the model. Learn about [image inputs](/docs/guides/vision). + - `"realtime.item"` - - `detail: "low" or "high" or "auto" or "original"` + - `status: optional "completed" or "incomplete" or "in_progress"` - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. + The status of the item. Has no effect on the conversation. - - `"low"` + - `"completed"` - - `"high"` + - `"incomplete"` - - `"auto"` + - `"in_progress"` - - `"original"` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` - - `type: "input_image"` + A function call item in a Realtime conversation. - The type of the input item. Always `input_image`. + - `arguments: string` - - `"input_image"` + The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. - - `file_id: optional string` + - `name: string` - The ID of the file to be sent to the model. + The name of the function being called. - - `image_url: optional string` + - `type: "function_call"` - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. + The type of the item. Always `function_call`. - - `ResponseInputFile = object { type, detail, file_data, 3 more }` + - `"function_call"` - A file input to the model. + - `id: optional string` - - `type: "input_file"` + The unique ID of the item. This may be provided by the client or generated by the server. - The type of the input item. Always `input_file`. + - `call_id: optional string` - - `"input_file"` + The ID of the function call. - - `detail: optional "low" or "high"` + - `object: optional "realtime.item"` - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - `"low"` + - `"realtime.item"` - - `"high"` + - `status: optional "completed" or "incomplete" or "in_progress"` - - `file_data: optional string` + The status of the item. Has no effect on the conversation. - The content of the file to be sent to the model. + - `"completed"` - - `file_id: optional string` + - `"incomplete"` - The ID of the file to be sent to the model. + - `"in_progress"` - - `file_url: optional string` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` - The URL of the file to be sent to the model. + A function call output item in a Realtime conversation. - - `filename: optional string` + - `call_id: string` - The name of the file to be sent to the model. + The ID of the function call this output is for. - - `version: optional string` + - `output: string` - Optional version of the prompt template. + The output of the function call, this is free text and can contain any information or simply be empty. - - `speed: optional number` + - `type: "function_call_output"` - The speed of the model's spoken response. 1.0 is the default speed. 0.25 is - the minimum speed. 1.5 is the maximum speed. This value can only be changed - in between model turns, not while a response is in progress. + The type of the item. Always `function_call_output`. - - `temperature: optional number` + - `"function_call_output"` - Sampling temperature for the model, limited to [0.6, 1.2]. For audio models a temperature of 0.8 is highly recommended for best performance. + - `id: optional string` - - `tool_choice: optional string` + The unique ID of the item. This may be provided by the client or generated by the server. - How the model chooses tools. Options are `auto`, `none`, `required`, or - specify a function. + - `object: optional "realtime.item"` - - `tools: optional array of RealtimeFunctionTool` + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - Tools (functions) available to the model. + - `"realtime.item"` - - `description: optional string` + - `status: optional "completed" or "incomplete" or "in_progress"` - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). + The status of the item. Has no effect on the conversation. - - `name: optional string` + - `"completed"` - The name of the function. + - `"incomplete"` - - `parameters: optional unknown` + - `"in_progress"` - Parameters of the function in JSON Schema. + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` - - `type: optional "function"` + A Realtime item responding to an MCP approval request. - The type of the tool, i.e. `function`. + - `id: string` - - `"function"` + The unique ID of the approval response. - - `tracing: optional "auto" or object { group_id, metadata, workflow_name }` + - `approval_request_id: string` - Configuration options for tracing. Set to null to disable tracing. Once - tracing is enabled for a session, the configuration cannot be modified. + The ID of the approval request being answered. - `auto` will create a trace for the session with default values for the - workflow name, group id, and metadata. + - `approve: boolean` - - `"auto"` + Whether the request was approved. - Default tracing mode for the session. + - `type: "mcp_approval_response"` - - `"auto"` + The type of the item. Always `mcp_approval_response`. - - `TracingConfiguration = object { group_id, metadata, workflow_name }` + - `"mcp_approval_response"` - Granular configuration for tracing. + - `reason: optional string` - - `group_id: optional string` + Optional reason for the decision. - The group id to attach to this trace to enable filtering and - grouping in the traces dashboard. + - `RealtimeMcpListTools object { server_label, tools, type, id }` - - `metadata: optional unknown` + A Realtime item listing tools available on an MCP server. - The arbitrary metadata to attach to this trace to enable - filtering in the traces dashboard. + - `server_label: string` - - `workflow_name: optional string` + The label of the MCP server. - The name of the workflow to attach to this trace. This is used to - name the trace in the traces dashboard. + - `tools: array of object { input_schema, name, annotations, description }` - - `turn_detection: optional object { type, create_response, idle_timeout_ms, 4 more } or object { type, create_response, eagerness, interrupt_response }` + The tools available on the server. - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + - `input_schema: unknown` - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + The JSON schema describing the tool's input. - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + - `name: string` - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + The name of the tool. - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. + - `annotations: optional unknown` - - `type: "server_vad"` + Additional annotations about the tool. - Type of turn detection, `server_vad` to turn on simple Server VAD. + - `description: optional string` - - `"server_vad"` + The description of the tool. - - `create_response: optional boolean` + - `type: "mcp_list_tools"` - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + The type of the item. Always `mcp_list_tools`. - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + - `"mcp_list_tools"` - - `idle_timeout_ms: optional number` + - `id: optional string` - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. + The unique ID of the list. - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. + A Realtime item representing an invocation of a tool on an MCP server. - - `interrupt_response: optional boolean` + - `id: string` - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + The unique ID of the tool call. - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + - `arguments: string` - - `prefix_padding_ms: optional number` + A JSON string of the arguments passed to the tool. - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. + - `name: string` - - `silence_duration_ms: optional number` + The name of the tool that was run. - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. + - `server_label: string` - - `threshold: optional number` + The label of the MCP server running the tool. - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. + - `type: "mcp_call"` - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + The type of the item. Always `mcp_call`. - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. + - `"mcp_call"` - - `type: "semantic_vad"` + - `approval_request_id: optional string` - Type of turn detection, `semantic_vad` to turn on Semantic VAD. + The ID of an associated approval request, if any. - - `"semantic_vad"` + - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` - - `create_response: optional boolean` + The error from the tool call, if any. - Whether or not to automatically generate a response when a VAD stop event occurs. + - `RealtimeMcpProtocolError object { code, message, type }` - - `eagerness: optional "low" or "medium" or "high" or "auto"` + - `code: number` - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + - `message: string` - - `"low"` + - `type: "protocol_error"` - - `"medium"` + - `"protocol_error"` - - `"high"` + - `RealtimeMcpToolExecutionError object { message, type }` - - `"auto"` + - `message: string` - - `interrupt_response: optional boolean` + - `type: "tool_execution_error"` - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + - `"tool_execution_error"` - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more` + - `RealtimeMcphttpError object { code, message, type }` - The voice the model uses to respond. Voice cannot be changed during the - session once the model has responded with audio at least once. Current - voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, - `shimmer`, and `verse`. + - `code: number` - - `string` + - `message: string` - - `"alloy" or "ash" or "ballad" or 7 more` + - `type: "http_error"` - The voice the model uses to respond. Voice cannot be changed during the - session once the model has responded with audio at least once. Current - voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, - `shimmer`, and `verse`. + - `"http_error"` - - `"alloy"` + - `output: optional string` - - `"ash"` + The output from the tool call. - - `"ballad"` + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` - - `"coral"` + A Realtime item requesting human approval of a tool invocation. - - `"echo"` + - `id: string` - - `"sage"` + The unique ID of the approval request. - - `"shimmer"` + - `arguments: string` - - `"verse"` + A JSON string of arguments for the tool. - - `"marin"` + - `name: string` - - `"cedar"` + The name of the tool to run. -### Realtime Session Create Request + - `server_label: string` -- `RealtimeSessionCreateRequest = object { type, audio, include, 9 more }` + The label of the MCP server making the request. - Realtime session object configuration. + - `type: "mcp_approval_request"` - - `type: "realtime"` + The type of the item. Always `mcp_approval_request`. - The type of session to create. Always `realtime` for the Realtime API. + - `"mcp_approval_request"` - - `"realtime"` + - `output_index: number` - - `audio: optional RealtimeAudioConfig` + The index of the output item in the Response. - Configuration for input and output audio. + - `response_id: string` - - `input: optional RealtimeAudioConfigInput` + The ID of the Response to which the item belongs. - - `format: optional RealtimeAudioFormats` + - `type: "response.output_item.added"` - The format of the input audio. + The event type, must be `response.output_item.added`. - - `PCMAudioFormat = object { rate, type }` + - `"response.output_item.added"` - The PCM audio format. Only a 24kHz sample rate is supported. +### Response Output Item Done Event - - `rate: optional 24000` +- `ResponseOutputItemDoneEvent object { event_id, item, output_index, 2 more }` - The sample rate of the audio. Always `24000`. + Returned when an Item is done streaming. Also emitted when a Response is + interrupted, incomplete, or cancelled. - - `24000` + - `event_id: string` - - `type: optional "audio/pcm"` + The unique ID of the server event. - The audio format. Always `audio/pcm`. + - `item: ConversationItem` - - `"audio/pcm"` + A single item within a Realtime conversation. - - `PCMUAudioFormat = object { type }` + - `RealtimeConversationItemSystemMessage object { content, role, type, 3 more }` - The G.711 μ-law format. + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. - - `type: optional "audio/pcmu"` + - `content: array of object { text, type }` - The audio format. Always `audio/pcmu`. + The content of the message. - - `"audio/pcmu"` + - `text: optional string` - - `PCMAAudioFormat = object { type }` + The text content. - The G.711 A-law format. + - `type: optional "input_text"` - - `type: optional "audio/pcma"` + The content type. Always `input_text` for system messages. - The audio format. Always `audio/pcma`. + - `"input_text"` - - `"audio/pcma"` + - `role: "system"` - - `noise_reduction: optional object { type }` + The role of the message sender. Always `system`. - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + - `"system"` - - `type: optional NoiseReductionType` + - `type: "message"` - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + The type of the item. Always `message`. - - `"near_field"` + - `"message"` - - `"far_field"` + - `id: optional string` - - `transcription: optional AudioTranscription` + The unique ID of the item. This may be provided by the client or generated by the server. - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + - `object: optional "realtime.item"` - - `language: optional string` + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. + - `"realtime.item"` - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `status: optional "completed" or "incomplete" or "in_progress"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + The status of the item. Has no effect on the conversation. - - `string` + - `"completed"` - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `"incomplete"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `"in_progress"` - - `"whisper-1"` + - `RealtimeConversationItemUserMessage object { content, role, type, 3 more }` - - `"gpt-4o-mini-transcribe"` + A user message item in a Realtime conversation. - - `"gpt-4o-mini-transcribe-2025-12-15"` + - `content: array of object { audio, detail, image_url, 3 more }` - - `"gpt-4o-transcribe"` + The content of the message. - - `"gpt-4o-transcribe-diarize"` + - `audio: optional string` - - `prompt: optional string` + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + - `detail: optional "auto" or "low" or "high"` - - `turn_detection: optional RealtimeAudioInputTurnDetection` + The detail level of the image (for `input_image`). `auto` will default to `high`. - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + - `"auto"` - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + - `"low"` - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + - `"high"` - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + - `image_url: optional string` - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. - - `type: "server_vad"` + - `text: optional string` - Type of turn detection, `server_vad` to turn on simple Server VAD. + The text content (for `input_text`). - - `"server_vad"` + - `transcript: optional string` - - `create_response: optional boolean` + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + - `type: optional "input_text" or "input_audio" or "input_image"` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + The content type (`input_text`, `input_audio`, or `input_image`). - - `idle_timeout_ms: optional number` + - `"input_text"` - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. + - `"input_audio"` - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. + - `"input_image"` - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. + - `role: "user"` - - `interrupt_response: optional boolean` + The role of the message sender. Always `user`. - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + - `"user"` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + - `type: "message"` - - `prefix_padding_ms: optional number` + The type of the item. Always `message`. - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. + - `"message"` - - `silence_duration_ms: optional number` + - `id: optional string` - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. + The unique ID of the item. This may be provided by the client or generated by the server. - - `threshold: optional number` + - `object: optional "realtime.item"` - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `"realtime.item"` - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. + - `status: optional "completed" or "incomplete" or "in_progress"` - - `type: "semantic_vad"` + The status of the item. Has no effect on the conversation. - Type of turn detection, `semantic_vad` to turn on Semantic VAD. + - `"completed"` - - `"semantic_vad"` + - `"incomplete"` - - `create_response: optional boolean` + - `"in_progress"` - Whether or not to automatically generate a response when a VAD stop event occurs. + - `RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }` - - `eagerness: optional "low" or "medium" or "high" or "auto"` + An assistant message item in a Realtime conversation. - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + - `content: array of object { audio, text, transcript, type }` - - `"low"` + The content of the message. - - `"medium"` + - `audio: optional string` - - `"high"` + Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. - - `"auto"` + - `text: optional string` - - `interrupt_response: optional boolean` + The text content. - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + - `transcript: optional string` - - `output: optional RealtimeAudioConfigOutput` + The transcript of the audio content, this will always be present if the output type is `audio`. - - `format: optional RealtimeAudioFormats` + - `type: optional "output_text" or "output_audio"` - The format of the output audio. + The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. - - `PCMAudioFormat = object { rate, type }` + - `"output_text"` - The PCM audio format. Only a 24kHz sample rate is supported. + - `"output_audio"` - - `rate: optional 24000` + - `role: "assistant"` - The sample rate of the audio. Always `24000`. + The role of the message sender. Always `assistant`. - - `24000` + - `"assistant"` - - `type: optional "audio/pcm"` + - `type: "message"` - The audio format. Always `audio/pcm`. + The type of the item. Always `message`. - - `"audio/pcm"` + - `"message"` - - `PCMUAudioFormat = object { type }` + - `id: optional string` - The G.711 μ-law format. + The unique ID of the item. This may be provided by the client or generated by the server. - - `type: optional "audio/pcmu"` + - `object: optional "realtime.item"` - The audio format. Always `audio/pcmu`. + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - `"audio/pcmu"` + - `"realtime.item"` - - `PCMAAudioFormat = object { type }` + - `status: optional "completed" or "incomplete" or "in_progress"` - The G.711 A-law format. + The status of the item. Has no effect on the conversation. - - `type: optional "audio/pcma"` + - `"completed"` - The audio format. Always `audio/pcma`. + - `"incomplete"` - - `"audio/pcma"` + - `"in_progress"` - - `speed: optional number` + - `RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }` - The speed of the model's spoken response as a multiple of the original speed. - 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. + A function call item in a Realtime conversation. - This parameter is a post-processing adjustment to the audio after it is generated, it's - also possible to prompt the model to speak faster or slower. + - `arguments: string` - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` + The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with - an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed - during the session once the model has responded with audio at least once. - We recommend `marin` and `cedar` for best quality. + - `name: string` - - `string` + The name of the function being called. - - `"alloy" or "ash" or "ballad" or 7 more` + - `type: "function_call"` - - `"alloy"` + The type of the item. Always `function_call`. - - `"ash"` + - `"function_call"` - - `"ballad"` + - `id: optional string` - - `"coral"` + The unique ID of the item. This may be provided by the client or generated by the server. - - `"echo"` + - `call_id: optional string` - - `"sage"` + The ID of the function call. - - `"shimmer"` + - `object: optional "realtime.item"` - - `"verse"` + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - `"marin"` + - `"realtime.item"` - - `"cedar"` + - `status: optional "completed" or "incomplete" or "in_progress"` - - `ID = object { id }` + The status of the item. Has no effect on the conversation. - Custom voice reference. + - `"completed"` - - `id: string` + - `"incomplete"` - The custom voice ID, e.g. `voice_1234`. + - `"in_progress"` - - `include: optional array of "item.input_audio_transcription.logprobs"` + - `RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }` - Additional fields to include in server outputs. + A function call output item in a Realtime conversation. - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + - `call_id: string` - - `"item.input_audio_transcription.logprobs"` + The ID of the function call this output is for. - - `instructions: optional string` + - `output: string` - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. + The output of the function call, this is free text and can contain any information or simply be empty. - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. + - `type: "function_call_output"` - - `max_output_tokens: optional number or "inf"` + The type of the item. Always `function_call_output`. - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. + - `"function_call_output"` - - `number` + - `id: optional string` - - `"inf"` + The unique ID of the item. This may be provided by the client or generated by the server. - - `"inf"` + - `object: optional "realtime.item"` - - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - The Realtime model used for this session. + - `"realtime.item"` - - `string` + - `status: optional "completed" or "incomplete" or "in_progress"` - - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` + The status of the item. Has no effect on the conversation. - The Realtime model used for this session. + - `"completed"` - - `"gpt-realtime"` + - `"incomplete"` - - `"gpt-realtime-1.5"` + - `"in_progress"` - - `"gpt-realtime-2025-08-28"` + - `RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }` - - `"gpt-4o-realtime-preview"` + A Realtime item responding to an MCP approval request. - - `"gpt-4o-realtime-preview-2024-10-01"` + - `id: string` - - `"gpt-4o-realtime-preview-2024-12-17"` + The unique ID of the approval response. - - `"gpt-4o-realtime-preview-2025-06-03"` + - `approval_request_id: string` - - `"gpt-4o-mini-realtime-preview"` + The ID of the approval request being answered. - - `"gpt-4o-mini-realtime-preview-2024-12-17"` + - `approve: boolean` - - `"gpt-realtime-mini"` + Whether the request was approved. - - `"gpt-realtime-mini-2025-10-06"` + - `type: "mcp_approval_response"` - - `"gpt-realtime-mini-2025-12-15"` + The type of the item. Always `mcp_approval_response`. - - `"gpt-audio-1.5"` + - `"mcp_approval_response"` - - `"gpt-audio-mini"` + - `reason: optional string` - - `"gpt-audio-mini-2025-10-06"` + Optional reason for the decision. - - `"gpt-audio-mini-2025-12-15"` + - `RealtimeMcpListTools object { server_label, tools, type, id }` - - `output_modalities: optional array of "text" or "audio"` + A Realtime item listing tools available on an MCP server. - The set of modalities the model can respond with. It defaults to `["audio"]`, indicating - that the model will respond with audio plus a transcript. `["text"]` can be used to make - the model respond with text only. It is not possible to request both `text` and `audio` at the same time. + - `server_label: string` - - `"text"` + The label of the MCP server. - - `"audio"` + - `tools: array of object { input_schema, name, annotations, description }` - - `prompt: optional ResponsePrompt` + The tools available on the server. - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). + - `input_schema: unknown` - - `id: string` + The JSON schema describing the tool's input. - The unique identifier of the prompt template to use. + - `name: string` - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` + The name of the tool. - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. + - `annotations: optional unknown` - - `string` + Additional annotations about the tool. - - `ResponseInputText = object { text, type }` + - `description: optional string` - A text input to the model. + The description of the tool. - - `text: string` + - `type: "mcp_list_tools"` - The text input to the model. + The type of the item. Always `mcp_list_tools`. - - `type: "input_text"` + - `"mcp_list_tools"` - The type of the input item. Always `input_text`. + - `id: optional string` - - `"input_text"` + The unique ID of the list. - - `ResponseInputImage = object { detail, type, file_id, image_url }` + - `RealtimeMcpToolCall object { id, arguments, name, 5 more }` - An image input to the model. Learn about [image inputs](/docs/guides/vision). + A Realtime item representing an invocation of a tool on an MCP server. - - `detail: "low" or "high" or "auto" or "original"` + - `id: string` - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. + The unique ID of the tool call. - - `"low"` + - `arguments: string` - - `"high"` + A JSON string of the arguments passed to the tool. - - `"auto"` + - `name: string` - - `"original"` + The name of the tool that was run. - - `type: "input_image"` + - `server_label: string` - The type of the input item. Always `input_image`. + The label of the MCP server running the tool. - - `"input_image"` + - `type: "mcp_call"` - - `file_id: optional string` + The type of the item. Always `mcp_call`. - The ID of the file to be sent to the model. + - `"mcp_call"` - - `image_url: optional string` + - `approval_request_id: optional string` - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. + The ID of an associated approval request, if any. - - `ResponseInputFile = object { type, detail, file_data, 3 more }` + - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` - A file input to the model. + The error from the tool call, if any. - - `type: "input_file"` + - `RealtimeMcpProtocolError object { code, message, type }` - The type of the input item. Always `input_file`. + - `code: number` - - `"input_file"` + - `message: string` - - `detail: optional "low" or "high"` + - `type: "protocol_error"` - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. + - `"protocol_error"` - - `"low"` + - `RealtimeMcpToolExecutionError object { message, type }` - - `"high"` + - `message: string` - - `file_data: optional string` + - `type: "tool_execution_error"` - The content of the file to be sent to the model. + - `"tool_execution_error"` - - `file_id: optional string` + - `RealtimeMcphttpError object { code, message, type }` - The ID of the file to be sent to the model. + - `code: number` - - `file_url: optional string` + - `message: string` - The URL of the file to be sent to the model. + - `type: "http_error"` - - `filename: optional string` + - `"http_error"` - The name of the file to be sent to the model. + - `output: optional string` - - `version: optional string` + The output from the tool call. - Optional version of the prompt template. + - `RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }` - - `tool_choice: optional RealtimeToolChoiceConfig` + A Realtime item requesting human approval of a tool invocation. - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. + - `id: string` - - `ToolChoiceOptions = "none" or "auto" or "required"` + The unique ID of the approval request. - Controls which (if any) tool is called by the model. + - `arguments: string` - `none` means the model will not call any tool and instead generates a message. + A JSON string of arguments for the tool. - `auto` means the model can pick between generating a message or calling one or - more tools. + - `name: string` - `required` means the model must call one or more tools. + The name of the tool to run. - - `"none"` + - `server_label: string` - - `"auto"` + The label of the MCP server making the request. - - `"required"` + - `type: "mcp_approval_request"` - - `ToolChoiceFunction = object { name, type }` + The type of the item. Always `mcp_approval_request`. - Use this option to force the model to call a specific function. + - `"mcp_approval_request"` - - `name: string` + - `output_index: number` - The name of the function to call. + The index of the output item in the Response. - - `type: "function"` + - `response_id: string` - For function calling, the type is always `function`. + The ID of the Response to which the item belongs. - - `"function"` + - `type: "response.output_item.done"` - - `ToolChoiceMcp = object { server_label, type, name }` + The event type, must be `response.output_item.done`. - Use this option to force the model to call a specific tool on a remote MCP server. + - `"response.output_item.done"` - - `server_label: string` +### Response Text Delta Event - The label of the MCP server to use. +- `ResponseTextDeltaEvent object { content_index, delta, event_id, 4 more }` - - `type: "mcp"` + Returned when the text value of an "output_text" content part is updated. - For MCP tools, the type is always `mcp`. + - `content_index: number` - - `"mcp"` + The index of the content part in the item's content array. - - `name: optional string` + - `delta: string` - The name of the tool to call on the server. + The text delta. - - `tools: optional RealtimeToolsConfig` + - `event_id: string` - Tools available to the model. + The unique ID of the server event. - - `RealtimeFunctionTool = object { description, name, parameters, type }` + - `item_id: string` - - `description: optional string` + The ID of the item. - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). + - `output_index: number` - - `name: optional string` + The index of the output item in the response. - The name of the function. + - `response_id: string` - - `parameters: optional unknown` + The ID of the response. - Parameters of the function in JSON Schema. + - `type: "response.output_text.delta"` - - `type: optional "function"` + The event type, must be `response.output_text.delta`. - The type of the tool, i.e. `function`. + - `"response.output_text.delta"` - - `"function"` +### Response Text Done Event - - `McpTool = object { server_label, type, allowed_tools, 7 more }` +- `ResponseTextDoneEvent object { content_index, event_id, item_id, 4 more }` - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). + Returned when the text value of an "output_text" content part is done streaming. Also + emitted when a Response is interrupted, incomplete, or cancelled. - - `server_label: string` + - `content_index: number` - A label for this MCP server, used to identify it in tool calls. + The index of the content part in the item's content array. - - `type: "mcp"` + - `event_id: string` - The type of the MCP tool. Always `mcp`. + The unique ID of the server event. - - `"mcp"` + - `item_id: string` - - `allowed_tools: optional array of string or object { read_only, tool_names }` + The ID of the item. - List of allowed tool names or a filter object. + - `output_index: number` - - `McpAllowedTools = array of string` + The index of the output item in the response. - A string array of allowed tool names + - `response_id: string` - - `McpToolFilter = object { read_only, tool_names }` + The ID of the response. - A filter object to specify which tools are allowed. + - `text: string` - - `read_only: optional boolean` + The final text content. - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `type: "response.output_text.done"` - - `tool_names: optional array of string` + The event type, must be `response.output_text.done`. - List of allowed tool names. + - `"response.output_text.done"` - - `authorization: optional string` +### Session Created Event - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. +- `SessionCreatedEvent object { event_id, session, type }` - - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` + Returned when a Session is created. Emitted automatically when a new + connection is established as the first server event. This event will contain + the default Session configuration. - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](/docs/guides/tools-remote-mcp#connectors). + - `event_id: string` - Currently supported `connector_id` values are: + The unique ID of the server event. - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` + - `session: RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` - - `"connector_dropbox"` + The session configuration. - - `"connector_gmail"` + - `RealtimeSessionCreateRequest object { type, audio, include, 9 more }` - - `"connector_googlecalendar"` + Realtime session object configuration. - - `"connector_googledrive"` + - `type: "realtime"` - - `"connector_microsoftteams"` + The type of session to create. Always `realtime` for the Realtime API. - - `"connector_outlookcalendar"` + - `"realtime"` - - `"connector_outlookemail"` + - `audio: optional RealtimeAudioConfig` - - `"connector_sharepoint"` + Configuration for input and output audio. - - `defer_loading: optional boolean` + - `input: optional RealtimeAudioConfigInput` - Whether this MCP tool is deferred and discovered via tool search. + - `format: optional RealtimeAudioFormats` - - `headers: optional map[string]` + The format of the input audio. - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. + - `PCMAudioFormat object { rate, type }` - - `require_approval: optional object { always, never } or "always" or "never"` + The PCM audio format. Only a 24kHz sample rate is supported. - Specify which of the MCP server's tools require approval. + - `rate: optional 24000` - - `McpToolApprovalFilter = object { always, never }` + The sample rate of the audio. Always `24000`. - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. + - `24000` - - `always: optional object { read_only, tool_names }` + - `type: optional "audio/pcm"` - A filter object to specify which tools are allowed. + The audio format. Always `audio/pcm`. - - `read_only: optional boolean` + - `"audio/pcm"` - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `PCMUAudioFormat object { type }` - - `tool_names: optional array of string` + The G.711 μ-law format. - List of allowed tool names. + - `type: optional "audio/pcmu"` - - `never: optional object { read_only, tool_names }` + The audio format. Always `audio/pcmu`. - A filter object to specify which tools are allowed. + - `"audio/pcmu"` - - `read_only: optional boolean` + - `PCMAAudioFormat object { type }` - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + The G.711 A-law format. - - `tool_names: optional array of string` + - `type: optional "audio/pcma"` - List of allowed tool names. + The audio format. Always `audio/pcma`. - - `McpToolApprovalSetting = "always" or "never"` + - `"audio/pcma"` - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. + - `noise_reduction: optional object { type }` - - `"always"` + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `"never"` + - `type: optional NoiseReductionType` - - `server_description: optional string` + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - Optional description of the MCP server, used to provide more context. + - `"near_field"` - - `server_url: optional string` + - `"far_field"` - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. + - `transcription: optional AudioTranscription` - - `tracing: optional RealtimeTracingConfig` + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once - tracing is enabled for a session, the configuration cannot be modified. + - `language: optional string` - `auto` will create a trace for the session with default values for the - workflow name, group id, and metadata. + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - - `Auto = "auto"` + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - Enables tracing and sets default values for tracing configuration options. Always `auto`. + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `"auto"` + - `string` - - `TracingConfiguration = object { group_id, metadata, workflow_name }` + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - Granular configuration for tracing. + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `group_id: optional string` + - `"whisper-1"` - The group id to attach to this trace to enable filtering and - grouping in the Traces Dashboard. + - `"gpt-4o-mini-transcribe"` - - `metadata: optional unknown` + - `"gpt-4o-mini-transcribe-2025-12-15"` - The arbitrary metadata to attach to this trace to enable - filtering in the Traces Dashboard. + - `"gpt-4o-transcribe"` - - `workflow_name: optional string` + - `"gpt-4o-transcribe-diarize"` - The name of the workflow to attach to this trace. This is used to - name the trace in the Traces Dashboard. + - `prompt: optional string` - - `truncation: optional RealtimeTruncation` + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. + - `turn_detection: optional RealtimeAudioInputTurnDetection` - Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `"auto" or "disabled"` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - `"auto"` + - `type: "server_vad"` - - `"disabled"` + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` + - `"server_vad"` - Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. + - `create_response: optional boolean` - - `retention_ratio: number` + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `type: "retention_ratio"` + - `idle_timeout_ms: optional number` - Use retention ratio truncation. + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - - `"retention_ratio"` + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - - `token_limits: optional object { post_instructions }` + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. + - `interrupt_response: optional boolean` - - `post_instructions: optional number` + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. -### Realtime Tool Choice Config + - `prefix_padding_ms: optional number` -- `RealtimeToolChoiceConfig = ToolChoiceOptions or ToolChoiceFunction or ToolChoiceMcp` + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. + - `silence_duration_ms: optional number` - - `ToolChoiceOptions = "none" or "auto" or "required"` + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - Controls which (if any) tool is called by the model. + - `threshold: optional number` - `none` means the model will not call any tool and instead generates a message. + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - `auto` means the model can pick between generating a message or calling one or - more tools. + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - `required` means the model must call one or more tools. + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - `"none"` + - `type: "semantic_vad"` - - `"auto"` + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - `"required"` + - `"semantic_vad"` - - `ToolChoiceFunction = object { name, type }` + - `create_response: optional boolean` - Use this option to force the model to call a specific function. + Whether or not to automatically generate a response when a VAD stop event occurs. - - `name: string` + - `eagerness: optional "low" or "medium" or "high" or "auto"` - The name of the function to call. + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - `type: "function"` + - `"low"` - For function calling, the type is always `function`. + - `"medium"` - - `"function"` + - `"high"` - - `ToolChoiceMcp = object { server_label, type, name }` + - `"auto"` - Use this option to force the model to call a specific tool on a remote MCP server. + - `interrupt_response: optional boolean` - - `server_label: string` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - The label of the MCP server to use. + - `output: optional RealtimeAudioConfigOutput` - - `type: "mcp"` + - `format: optional RealtimeAudioFormats` - For MCP tools, the type is always `mcp`. + The format of the output audio. - - `"mcp"` + - `speed: optional number` - - `name: optional string` + The speed of the model's spoken response as a multiple of the original speed. + 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - The name of the tool to call on the server. + This parameter is a post-processing adjustment to the audio after it is generated, it's + also possible to prompt the model to speak faster or slower. -### Realtime Tools Config + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` -- `RealtimeToolsConfig = array of RealtimeToolsConfigUnion` + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. - Tools available to the model. + - `string` - - `RealtimeFunctionTool = object { description, name, parameters, type }` + - `"alloy" or "ash" or "ballad" or 7 more` - - `description: optional string` + - `"alloy"` - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). + - `"ash"` - - `name: optional string` + - `"ballad"` - The name of the function. + - `"coral"` - - `parameters: optional unknown` + - `"echo"` - Parameters of the function in JSON Schema. + - `"sage"` - - `type: optional "function"` + - `"shimmer"` - The type of the tool, i.e. `function`. + - `"verse"` - - `"function"` + - `"marin"` - - `McpTool = object { server_label, type, allowed_tools, 7 more }` + - `"cedar"` - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). + - `ID object { id }` - - `server_label: string` + Custom voice reference. - A label for this MCP server, used to identify it in tool calls. + - `id: string` - - `type: "mcp"` + The custom voice ID, e.g. `voice_1234`. - The type of the MCP tool. Always `mcp`. + - `include: optional array of "item.input_audio_transcription.logprobs"` - - `"mcp"` + Additional fields to include in server outputs. - - `allowed_tools: optional array of string or object { read_only, tool_names }` + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - List of allowed tool names or a filter object. + - `"item.input_audio_transcription.logprobs"` - - `McpAllowedTools = array of string` + - `instructions: optional string` - A string array of allowed tool names + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - - `McpToolFilter = object { read_only, tool_names }` + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - A filter object to specify which tools are allowed. + - `max_output_tokens: optional number or "inf"` - - `read_only: optional boolean` + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `number` - - `tool_names: optional array of string` + - `"inf"` - List of allowed tool names. + - `"inf"` - - `authorization: optional string` + - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. + The Realtime model used for this session. - - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` + - `string` - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](/docs/guides/tools-remote-mcp#connectors). + - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - Currently supported `connector_id` values are: + The Realtime model used for this session. - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` + - `"gpt-realtime"` - - `"connector_dropbox"` + - `"gpt-realtime-1.5"` - - `"connector_gmail"` + - `"gpt-realtime-2025-08-28"` - - `"connector_googlecalendar"` + - `"gpt-4o-realtime-preview"` - - `"connector_googledrive"` + - `"gpt-4o-realtime-preview-2024-10-01"` - - `"connector_microsoftteams"` + - `"gpt-4o-realtime-preview-2024-12-17"` - - `"connector_outlookcalendar"` + - `"gpt-4o-realtime-preview-2025-06-03"` - - `"connector_outlookemail"` + - `"gpt-4o-mini-realtime-preview"` - - `"connector_sharepoint"` + - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - `defer_loading: optional boolean` + - `"gpt-realtime-mini"` - Whether this MCP tool is deferred and discovered via tool search. + - `"gpt-realtime-mini-2025-10-06"` - - `headers: optional map[string]` + - `"gpt-realtime-mini-2025-12-15"` - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. + - `"gpt-audio-1.5"` - - `require_approval: optional object { always, never } or "always" or "never"` + - `"gpt-audio-mini"` - Specify which of the MCP server's tools require approval. + - `"gpt-audio-mini-2025-10-06"` - - `McpToolApprovalFilter = object { always, never }` + - `"gpt-audio-mini-2025-12-15"` - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. + - `output_modalities: optional array of "text" or "audio"` - - `always: optional object { read_only, tool_names }` + The set of modalities the model can respond with. It defaults to `["audio"]`, indicating + that the model will respond with audio plus a transcript. `["text"]` can be used to make + the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - A filter object to specify which tools are allowed. + - `"text"` - - `read_only: optional boolean` + - `"audio"` - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `prompt: optional ResponsePrompt` - - `tool_names: optional array of string` + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - List of allowed tool names. + - `id: string` - - `never: optional object { read_only, tool_names }` + The unique identifier of the prompt template to use. - A filter object to specify which tools are allowed. + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - `read_only: optional boolean` + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `string` - - `tool_names: optional array of string` + - `ResponseInputText object { text, type }` - List of allowed tool names. + A text input to the model. - - `McpToolApprovalSetting = "always" or "never"` + - `text: string` - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. + The text input to the model. - - `"always"` + - `type: "input_text"` - - `"never"` + The type of the input item. Always `input_text`. - - `server_description: optional string` + - `"input_text"` - Optional description of the MCP server, used to provide more context. + - `ResponseInputImage object { detail, type, file_id, image_url }` - - `server_url: optional string` + An image input to the model. Learn about [image inputs](/docs/guides/vision). - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. + - `detail: "low" or "high" or "auto" or "original"` -### Realtime Tools Config Union + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. -- `RealtimeToolsConfigUnion = RealtimeFunctionTool or object { server_label, type, allowed_tools, 7 more }` + - `"low"` - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). + - `"high"` + + - `"auto"` + + - `"original"` + + - `type: "input_image"` + + The type of the input item. Always `input_image`. + + - `"input_image"` + + - `file_id: optional string` + + The ID of the file to be sent to the model. + + - `image_url: optional string` + + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. + + - `ResponseInputFile object { type, detail, file_data, 3 more }` + + A file input to the model. + + - `type: "input_file"` + + The type of the input item. Always `input_file`. + + - `"input_file"` + + - `detail: optional "low" or "high"` + + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. + + - `"low"` + + - `"high"` + + - `file_data: optional string` + + The content of the file to be sent to the model. + + - `file_id: optional string` + + The ID of the file to be sent to the model. + + - `file_url: optional string` + + The URL of the file to be sent to the model. + + - `filename: optional string` + + The name of the file to be sent to the model. + + - `version: optional string` + + Optional version of the prompt template. + + - `tool_choice: optional RealtimeToolChoiceConfig` + + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. + + - `ToolChoiceOptions = "none" or "auto" or "required"` + + Controls which (if any) tool is called by the model. + + `none` means the model will not call any tool and instead generates a message. + + `auto` means the model can pick between generating a message or calling one or + more tools. + + `required` means the model must call one or more tools. + + - `"none"` + + - `"auto"` + + - `"required"` + + - `ToolChoiceFunction object { name, type }` + + Use this option to force the model to call a specific function. + + - `name: string` + + The name of the function to call. + + - `type: "function"` + + For function calling, the type is always `function`. - - `RealtimeFunctionTool = object { description, name, parameters, type }` + - `"function"` + + - `ToolChoiceMcp object { server_label, type, name }` + + Use this option to force the model to call a specific tool on a remote MCP server. + + - `server_label: string` + + The label of the MCP server to use. + + - `type: "mcp"` + + For MCP tools, the type is always `mcp`. + + - `"mcp"` + + - `name: optional string` + + The name of the tool to call on the server. + + - `tools: optional RealtimeToolsConfig` + + Tools available to the model. + + - `RealtimeFunctionTool object { description, name, parameters, type }` - `description: optional string` @@ -17685,7 +17916,7 @@ - `"function"` - - `McpTool = object { server_label, type, allowed_tools, 7 more }` + - `McpTool object { server_label, type, allowed_tools, 7 more }` Give the model access to additional tools via remote Model Context Protocol (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). @@ -17708,7 +17939,7 @@ A string array of allowed tool names - - `McpToolFilter = object { read_only, tool_names }` + - `McpToolFilter object { read_only, tool_names }` A filter object to specify which tools are allowed. @@ -17774,7 +18005,7 @@ Specify which of the MCP server's tools require approval. - - `McpToolApprovalFilter = object { always, never }` + - `McpToolApprovalFilter object { always, never }` Specify which of the MCP server's tools require approval. Can be `always`, `never`, or a filter object associated with tools @@ -17827,9 +18058,7 @@ The URL for the MCP server. One of `server_url` or `connector_id` must be provided. -### Realtime Tracing Config - -- `RealtimeTracingConfig = "auto" or object { group_id, metadata, workflow_name }` + - `tracing: optional RealtimeTracingConfig` Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified. @@ -17843,7 +18072,7 @@ - `"auto"` - - `TracingConfiguration = object { group_id, metadata, workflow_name }` + - `TracingConfiguration object { group_id, metadata, workflow_name }` Granular configuration for tracing. @@ -17862,53 +18091,65 @@ The name of the workflow to attach to this trace. This is used to name the trace in the Traces Dashboard. -### Realtime Transcription Session Audio + - `truncation: optional RealtimeTruncation` -- `RealtimeTranscriptionSessionAudio = object { input }` + When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - Configuration for input and output audio. + Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - - `input: optional RealtimeTranscriptionSessionAudioInput` + Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - `format: optional RealtimeAudioFormats` + Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - The PCM audio format. Only a 24kHz sample rate is supported. + - `"auto" or "disabled"` - - `PCMAudioFormat = object { rate, type }` + The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - The PCM audio format. Only a 24kHz sample rate is supported. + - `"auto"` - - `rate: optional 24000` + - `"disabled"` - The sample rate of the audio. Always `24000`. + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` - - `24000` + Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - - `type: optional "audio/pcm"` + - `retention_ratio: number` - The audio format. Always `audio/pcm`. + Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - `"audio/pcm"` + - `type: "retention_ratio"` - - `PCMUAudioFormat = object { type }` + Use retention ratio truncation. - The G.711 μ-law format. + - `"retention_ratio"` - - `type: optional "audio/pcmu"` + - `token_limits: optional object { post_instructions }` - The audio format. Always `audio/pcmu`. + Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - - `"audio/pcmu"` + - `post_instructions: optional number` - - `PCMAAudioFormat = object { type }` + Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - The G.711 A-law format. + - `RealtimeTranscriptionSessionCreateRequest object { type, audio, include }` - - `type: optional "audio/pcma"` + Realtime transcription session object configuration. - The audio format. Always `audio/pcma`. + - `type: "transcription"` - - `"audio/pcma"` + The type of session to create. Always `transcription` for transcription sessions. + + - `"transcription"` + + - `audio: optional RealtimeTranscriptionSessionAudio` + + Configuration for input and output audio. + + - `input: optional RealtimeTranscriptionSessionAudioInput` + + - `format: optional RealtimeAudioFormats` + + The PCM audio format. Only a 24kHz sample rate is supported. - `noise_reduction: optional object { type }` @@ -17920,47 +18161,10 @@ Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - `"near_field"` - - - `"far_field"` - - `transcription: optional AudioTranscription` Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `string` - - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. @@ -17969,7 +18173,7 @@ Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. @@ -18023,7 +18227,7 @@ higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` Server-side semantic turn detection which uses a model to determine when the user has finished speaking. @@ -18054,15 +18258,60 @@ Whether or not to automatically interrupt any ongoing response with output to the default conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. -### Realtime Transcription Session Audio Input + - `include: optional array of "item.input_audio_transcription.logprobs"` + + Additional fields to include in server outputs. + + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + + - `"item.input_audio_transcription.logprobs"` + + - `type: "session.created"` + + The event type, must be `session.created`. + + - `"session.created"` + +### Session Update Event + +- `SessionUpdateEvent object { session, type, event_id }` + + Send this event to update the session’s configuration. + The client may send this event at any time to update any field + except for `voice` and `model`. `voice` can be updated only if there have been no other audio outputs yet. + + When the server receives a `session.update`, it will respond + with a `session.updated` event showing the full, effective configuration. + Only the fields that are present in the `session.update` are updated. To clear a field like + `instructions`, pass an empty string. To clear a field like `tools`, pass an empty array. + To clear a field like `turn_detection`, pass `null`. + + - `session: RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` + + Update the Realtime session. Choose either a realtime + session or a transcription session. + + - `RealtimeSessionCreateRequest object { type, audio, include, 9 more }` + + Realtime session object configuration. + + - `type: "realtime"` + + The type of session to create. Always `realtime` for the Realtime API. + + - `"realtime"` + + - `audio: optional RealtimeAudioConfig` + + Configuration for input and output audio. -- `RealtimeTranscriptionSessionAudioInput = object { format, noise_reduction, transcription, turn_detection }` + - `input: optional RealtimeAudioConfigInput` - `format: optional RealtimeAudioFormats` - The PCM audio format. Only a 24kHz sample rate is supported. + The format of the input audio. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -18078,7 +18327,7 @@ - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -18088,7 +18337,7 @@ - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -18149,7 +18398,7 @@ For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` + - `turn_detection: optional RealtimeAudioInputTurnDetection` Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. @@ -18157,7 +18406,7 @@ Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. @@ -18211,7 +18460,7 @@ higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` Server-side semantic turn detection which uses a model to determine when the user has finished speaking. @@ -18242,690 +18491,715 @@ Whether or not to automatically interrupt any ongoing response with output to the default conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. -### Realtime Transcription Session Audio Input Turn Detection + - `output: optional RealtimeAudioConfigOutput` -- `RealtimeTranscriptionSessionAudioInputTurnDetection = object { type, create_response, idle_timeout_ms, 4 more } or object { type, create_response, eagerness, interrupt_response }` + - `format: optional RealtimeAudioFormats` - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + The format of the output audio. - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + - `speed: optional number` - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + The speed of the model's spoken response as a multiple of the original speed. + 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + This parameter is a post-processing adjustment to the audio after it is generated, it's + also possible to prompt the model to speak faster or slower. - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - - `type: "server_vad"` + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. - Type of turn detection, `server_vad` to turn on simple Server VAD. + - `string` - - `"server_vad"` + - `"alloy" or "ash" or "ballad" or 7 more` - - `create_response: optional boolean` + - `"alloy"` - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + - `"ash"` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + - `"ballad"` - - `idle_timeout_ms: optional number` + - `"coral"` - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. + - `"echo"` - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. + - `"sage"` - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. + - `"shimmer"` - - `interrupt_response: optional boolean` + - `"verse"` - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + - `"marin"` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + - `"cedar"` - - `prefix_padding_ms: optional number` + - `ID object { id }` - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. + Custom voice reference. - - `silence_duration_ms: optional number` + - `id: string` - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. + The custom voice ID, e.g. `voice_1234`. - - `threshold: optional number` + - `include: optional array of "item.input_audio_transcription.logprobs"` - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. + Additional fields to include in server outputs. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. + - `"item.input_audio_transcription.logprobs"` - - `type: "semantic_vad"` + - `instructions: optional string` - Type of turn detection, `semantic_vad` to turn on Semantic VAD. + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - - `"semantic_vad"` + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - `create_response: optional boolean` + - `max_output_tokens: optional number or "inf"` - Whether or not to automatically generate a response when a VAD stop event occurs. + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - - `eagerness: optional "low" or "medium" or "high" or "auto"` + - `number` - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + - `"inf"` - - `"low"` + - `"inf"` - - `"medium"` + - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - `"high"` + The Realtime model used for this session. - - `"auto"` + - `string` - - `interrupt_response: optional boolean` + - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + The Realtime model used for this session. -### Realtime Transcription Session Create Request + - `"gpt-realtime"` -- `RealtimeTranscriptionSessionCreateRequest = object { type, audio, include }` + - `"gpt-realtime-1.5"` - Realtime transcription session object configuration. + - `"gpt-realtime-2025-08-28"` - - `type: "transcription"` + - `"gpt-4o-realtime-preview"` - The type of session to create. Always `transcription` for transcription sessions. + - `"gpt-4o-realtime-preview-2024-10-01"` - - `"transcription"` + - `"gpt-4o-realtime-preview-2024-12-17"` - - `audio: optional RealtimeTranscriptionSessionAudio` + - `"gpt-4o-realtime-preview-2025-06-03"` - Configuration for input and output audio. + - `"gpt-4o-mini-realtime-preview"` - - `input: optional RealtimeTranscriptionSessionAudioInput` + - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - `format: optional RealtimeAudioFormats` + - `"gpt-realtime-mini"` - The PCM audio format. Only a 24kHz sample rate is supported. + - `"gpt-realtime-mini-2025-10-06"` - - `PCMAudioFormat = object { rate, type }` + - `"gpt-realtime-mini-2025-12-15"` - The PCM audio format. Only a 24kHz sample rate is supported. + - `"gpt-audio-1.5"` - - `rate: optional 24000` + - `"gpt-audio-mini"` - The sample rate of the audio. Always `24000`. + - `"gpt-audio-mini-2025-10-06"` - - `24000` + - `"gpt-audio-mini-2025-12-15"` - - `type: optional "audio/pcm"` + - `output_modalities: optional array of "text" or "audio"` - The audio format. Always `audio/pcm`. + The set of modalities the model can respond with. It defaults to `["audio"]`, indicating + that the model will respond with audio plus a transcript. `["text"]` can be used to make + the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - - `"audio/pcm"` + - `"text"` - - `PCMUAudioFormat = object { type }` + - `"audio"` - The G.711 μ-law format. + - `prompt: optional ResponsePrompt` - - `type: optional "audio/pcmu"` + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - The audio format. Always `audio/pcmu`. + - `id: string` - - `"audio/pcmu"` + The unique identifier of the prompt template to use. - - `PCMAAudioFormat = object { type }` + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - The G.711 A-law format. + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - - `type: optional "audio/pcma"` + - `string` - The audio format. Always `audio/pcma`. + - `ResponseInputText object { text, type }` - - `"audio/pcma"` + A text input to the model. - - `noise_reduction: optional object { type }` + - `text: string` - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + The text input to the model. - - `type: optional NoiseReductionType` + - `type: "input_text"` - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + The type of the input item. Always `input_text`. - - `"near_field"` + - `"input_text"` - - `"far_field"` + - `ResponseInputImage object { detail, type, file_id, image_url }` - - `transcription: optional AudioTranscription` + An image input to the model. Learn about [image inputs](/docs/guides/vision). - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + - `detail: "low" or "high" or "auto" or "original"` - - `language: optional string` + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. + - `"low"` - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `"high"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `"auto"` - - `string` + - `"original"` - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `type: "input_image"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + The type of the input item. Always `input_image`. - - `"whisper-1"` + - `"input_image"` - - `"gpt-4o-mini-transcribe"` + - `file_id: optional string` - - `"gpt-4o-mini-transcribe-2025-12-15"` + The ID of the file to be sent to the model. - - `"gpt-4o-transcribe"` + - `image_url: optional string` - - `"gpt-4o-transcribe-diarize"` + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - `prompt: optional string` + - `ResponseInputFile object { type, detail, file_data, 3 more }` - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + A file input to the model. - - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` + - `type: "input_file"` - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + The type of the input item. Always `input_file`. - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + - `"input_file"` - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + - `detail: optional "low" or "high"` - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. + - `"low"` - - `type: "server_vad"` + - `"high"` - Type of turn detection, `server_vad` to turn on simple Server VAD. + - `file_data: optional string` - - `"server_vad"` + The content of the file to be sent to the model. - - `create_response: optional boolean` + - `file_id: optional string` - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + The ID of the file to be sent to the model. - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + - `file_url: optional string` - - `idle_timeout_ms: optional number` + The URL of the file to be sent to the model. - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. + - `filename: optional string` - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. + The name of the file to be sent to the model. - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. + - `version: optional string` - - `interrupt_response: optional boolean` + Optional version of the prompt template. - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + - `tool_choice: optional RealtimeToolChoiceConfig` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. - - `prefix_padding_ms: optional number` + - `ToolChoiceOptions = "none" or "auto" or "required"` - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. + Controls which (if any) tool is called by the model. - - `silence_duration_ms: optional number` + `none` means the model will not call any tool and instead generates a message. - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. + `auto` means the model can pick between generating a message or calling one or + more tools. - - `threshold: optional number` + `required` means the model must call one or more tools. - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. + - `"none"` - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `"auto"` - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. + - `"required"` - - `type: "semantic_vad"` + - `ToolChoiceFunction object { name, type }` - Type of turn detection, `semantic_vad` to turn on Semantic VAD. + Use this option to force the model to call a specific function. - - `"semantic_vad"` + - `name: string` - - `create_response: optional boolean` + The name of the function to call. - Whether or not to automatically generate a response when a VAD stop event occurs. + - `type: "function"` - - `eagerness: optional "low" or "medium" or "high" or "auto"` + For function calling, the type is always `function`. - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + - `"function"` - - `"low"` + - `ToolChoiceMcp object { server_label, type, name }` - - `"medium"` + Use this option to force the model to call a specific tool on a remote MCP server. - - `"high"` + - `server_label: string` - - `"auto"` + The label of the MCP server to use. - - `interrupt_response: optional boolean` + - `type: "mcp"` - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + For MCP tools, the type is always `mcp`. - - `include: optional array of "item.input_audio_transcription.logprobs"` + - `"mcp"` - Additional fields to include in server outputs. + - `name: optional string` - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + The name of the tool to call on the server. - - `"item.input_audio_transcription.logprobs"` + - `tools: optional RealtimeToolsConfig` -### Realtime Truncation + Tools available to the model. -- `RealtimeTruncation = "auto" or "disabled" or object { retention_ratio, type, token_limits }` + - `RealtimeFunctionTool object { description, name, parameters, type }` - When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. + - `description: optional string` - Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. + - `name: optional string` - Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. + The name of the function. - - `"auto" or "disabled"` + - `parameters: optional unknown` - The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. + Parameters of the function in JSON Schema. - - `"auto"` + - `type: optional "function"` - - `"disabled"` + The type of the tool, i.e. `function`. - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` + - `"function"` - Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. + - `McpTool object { server_label, type, allowed_tools, 7 more }` - - `retention_ratio: number` + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. + - `server_label: string` - - `type: "retention_ratio"` + A label for this MCP server, used to identify it in tool calls. - Use retention ratio truncation. + - `type: "mcp"` - - `"retention_ratio"` + The type of the MCP tool. Always `mcp`. - - `token_limits: optional object { post_instructions }` + - `"mcp"` - Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. + - `allowed_tools: optional array of string or object { read_only, tool_names }` - - `post_instructions: optional number` + List of allowed tool names or a filter object. - Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. + - `McpAllowedTools = array of string` -### Response Audio Delta Event + A string array of allowed tool names -- `ResponseAudioDeltaEvent = object { content_index, delta, event_id, 4 more }` + - `McpToolFilter object { read_only, tool_names }` - Returned when the model-generated audio is updated. + A filter object to specify which tools are allowed. - - `content_index: number` + - `read_only: optional boolean` - The index of the content part in the item's content array. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `delta: string` + - `tool_names: optional array of string` - Base64-encoded audio data delta. + List of allowed tool names. - - `event_id: string` + - `authorization: optional string` - The unique ID of the server event. + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. - - `item_id: string` + - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - The ID of the item. + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](/docs/guides/tools-remote-mcp#connectors). - - `output_index: number` + Currently supported `connector_id` values are: - The index of the output item in the response. + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` - - `response_id: string` + - `"connector_dropbox"` - The ID of the response. + - `"connector_gmail"` - - `type: "response.output_audio.delta"` + - `"connector_googlecalendar"` - The event type, must be `response.output_audio.delta`. + - `"connector_googledrive"` - - `"response.output_audio.delta"` + - `"connector_microsoftteams"` -### Response Audio Done Event + - `"connector_outlookcalendar"` -- `ResponseAudioDoneEvent = object { content_index, event_id, item_id, 3 more }` + - `"connector_outlookemail"` - Returned when the model-generated audio is done. Also emitted when a Response - is interrupted, incomplete, or cancelled. + - `"connector_sharepoint"` - - `content_index: number` + - `defer_loading: optional boolean` - The index of the content part in the item's content array. + Whether this MCP tool is deferred and discovered via tool search. - - `event_id: string` + - `headers: optional map[string]` - The unique ID of the server event. + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. - - `item_id: string` + - `require_approval: optional object { always, never } or "always" or "never"` - The ID of the item. + Specify which of the MCP server's tools require approval. - - `output_index: number` + - `McpToolApprovalFilter object { always, never }` - The index of the output item in the response. + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. - - `response_id: string` + - `always: optional object { read_only, tool_names }` - The ID of the response. + A filter object to specify which tools are allowed. - - `type: "response.output_audio.done"` + - `read_only: optional boolean` - The event type, must be `response.output_audio.done`. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `"response.output_audio.done"` + - `tool_names: optional array of string` -### Response Audio Transcript Delta Event + List of allowed tool names. -- `ResponseAudioTranscriptDeltaEvent = object { content_index, delta, event_id, 4 more }` + - `never: optional object { read_only, tool_names }` - Returned when the model-generated transcription of audio output is updated. + A filter object to specify which tools are allowed. - - `content_index: number` + - `read_only: optional boolean` - The index of the content part in the item's content array. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `delta: string` + - `tool_names: optional array of string` - The transcript delta. + List of allowed tool names. - - `event_id: string` + - `McpToolApprovalSetting = "always" or "never"` - The unique ID of the server event. + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. - - `item_id: string` + - `"always"` - The ID of the item. + - `"never"` - - `output_index: number` + - `server_description: optional string` - The index of the output item in the response. + Optional description of the MCP server, used to provide more context. - - `response_id: string` + - `server_url: optional string` - The ID of the response. + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. - - `type: "response.output_audio_transcript.delta"` + - `tracing: optional RealtimeTracingConfig` - The event type, must be `response.output_audio_transcript.delta`. + Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once + tracing is enabled for a session, the configuration cannot be modified. - - `"response.output_audio_transcript.delta"` + `auto` will create a trace for the session with default values for the + workflow name, group id, and metadata. -### Response Audio Transcript Done Event + - `Auto = "auto"` -- `ResponseAudioTranscriptDoneEvent = object { content_index, event_id, item_id, 4 more }` + Enables tracing and sets default values for tracing configuration options. Always `auto`. - Returned when the model-generated transcription of audio output is done - streaming. Also emitted when a Response is interrupted, incomplete, or - cancelled. + - `"auto"` - - `content_index: number` + - `TracingConfiguration object { group_id, metadata, workflow_name }` - The index of the content part in the item's content array. + Granular configuration for tracing. - - `event_id: string` + - `group_id: optional string` - The unique ID of the server event. + The group id to attach to this trace to enable filtering and + grouping in the Traces Dashboard. - - `item_id: string` + - `metadata: optional unknown` - The ID of the item. + The arbitrary metadata to attach to this trace to enable + filtering in the Traces Dashboard. - - `output_index: number` + - `workflow_name: optional string` - The index of the output item in the response. + The name of the workflow to attach to this trace. This is used to + name the trace in the Traces Dashboard. - - `response_id: string` + - `truncation: optional RealtimeTruncation` - The ID of the response. + When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - - `transcript: string` + Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - The final transcript of the audio. + Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - `type: "response.output_audio_transcript.done"` + Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - The event type, must be `response.output_audio_transcript.done`. + - `"auto" or "disabled"` - - `"response.output_audio_transcript.done"` + The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. -### Response Cancel Event + - `"auto"` -- `ResponseCancelEvent = object { type, event_id, response_id }` + - `"disabled"` - Send this event to cancel an in-progress response. The server will respond - with a `response.done` event with a status of `response.status=cancelled`. If - there is no response to cancel, the server will respond with an error. It's safe - to call `response.cancel` even if no response is in progress, an error will be - returned the session will remain unaffected. + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` - - `type: "response.cancel"` + Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - The event type, must be `response.cancel`. + - `retention_ratio: number` - - `"response.cancel"` + Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - `event_id: optional string` + - `type: "retention_ratio"` - Optional client-generated ID used to identify this event. + Use retention ratio truncation. - - `response_id: optional string` + - `"retention_ratio"` - A specific response ID to cancel - if not provided, will cancel an - in-progress response in the default conversation. + - `token_limits: optional object { post_instructions }` -### Response Content Part Added Event + Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. -- `ResponseContentPartAddedEvent = object { content_index, event_id, item_id, 4 more }` + - `post_instructions: optional number` - Returned when a new content part is added to an assistant message item during - response generation. + Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - - `content_index: number` + - `RealtimeTranscriptionSessionCreateRequest object { type, audio, include }` - The index of the content part in the item's content array. + Realtime transcription session object configuration. - - `event_id: string` + - `type: "transcription"` - The unique ID of the server event. + The type of session to create. Always `transcription` for transcription sessions. - - `item_id: string` + - `"transcription"` - The ID of the item to which the content part was added. + - `audio: optional RealtimeTranscriptionSessionAudio` - - `output_index: number` + Configuration for input and output audio. - The index of the output item in the response. + - `input: optional RealtimeTranscriptionSessionAudioInput` - - `part: object { audio, text, transcript, type }` + - `format: optional RealtimeAudioFormats` - The content part that was added. + The PCM audio format. Only a 24kHz sample rate is supported. - - `audio: optional string` + - `noise_reduction: optional object { type }` - Base64-encoded audio data (if type is "audio"). + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `text: optional string` + - `type: optional NoiseReductionType` - The text content (if type is "text"). + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - `transcript: optional string` + - `transcription: optional AudioTranscription` - The transcript of the audio (if type is "audio"). + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - `type: optional "audio" or "text"` + - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` - The content type ("text", "audio"). + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - `"audio"` + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - `"text"` + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `response_id: string` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - The ID of the response. + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - `type: "response.content_part.added"` + - `type: "server_vad"` - The event type, must be `response.content_part.added`. + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `"response.content_part.added"` + - `"server_vad"` -### Response Content Part Done Event + - `create_response: optional boolean` -- `ResponseContentPartDoneEvent = object { content_index, event_id, item_id, 4 more }` + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - Returned when a content part is done streaming in an assistant message item. - Also emitted when a Response is interrupted, incomplete, or cancelled. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `content_index: number` + - `idle_timeout_ms: optional number` - The index of the content part in the item's content array. + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - - `event_id: string` + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - The unique ID of the server event. + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - - `item_id: string` + - `interrupt_response: optional boolean` - The ID of the item. + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - `output_index: number` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - The index of the output item in the response. + - `prefix_padding_ms: optional number` - - `part: object { audio, text, transcript, type }` + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - The content part that is done. + - `silence_duration_ms: optional number` - - `audio: optional string` + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - Base64-encoded audio data (if type is "audio"). - - - `text: optional string` + - `threshold: optional number` - The text content (if type is "text"). + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - - `transcript: optional string` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - The transcript of the audio (if type is "audio"). + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - `type: optional "audio" or "text"` + - `type: "semantic_vad"` - The content type ("text", "audio"). + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - `"audio"` + - `"semantic_vad"` - - `"text"` + - `create_response: optional boolean` - - `response_id: string` + Whether or not to automatically generate a response when a VAD stop event occurs. - The ID of the response. + - `eagerness: optional "low" or "medium" or "high" or "auto"` - - `type: "response.content_part.done"` + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - The event type, must be `response.content_part.done`. + - `"low"` - - `"response.content_part.done"` + - `"medium"` -### Response Create Event + - `"high"` -- `ResponseCreateEvent = object { type, event_id, response }` + - `"auto"` - This event instructs the server to create a Response, which means triggering - model inference. When in Server VAD mode, the server will create Responses - automatically. + - `interrupt_response: optional boolean` - A Response will include at least one Item, and may have two, in which case - the second will be a function call. These Items will be appended to the - conversation history by default. + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - The server will respond with a `response.created` event, events for Items - and content created, and finally a `response.done` event to indicate the - Response is complete. + - `include: optional array of "item.input_audio_transcription.logprobs"` - The `response.create` event includes inference configuration like - `instructions` and `tools`. If these are set, they will override the Session's - configuration for this Response only. + Additional fields to include in server outputs. - Responses can be created out-of-band of the default Conversation, meaning that they can - have arbitrary input, and it's possible to disable writing the output to the Conversation. - Only one Response can write to the default Conversation at a time, but otherwise multiple - Responses can be created in parallel. The `metadata` field is a good way to disambiguate - multiple simultaneous Responses. + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - Clients can set `conversation` to `none` to create a Response that does not write to the default - Conversation. Arbitrary input can be provided with the `input` field, which is an array accepting - raw Items and references to existing Items. + - `"item.input_audio_transcription.logprobs"` - - `type: "response.create"` + - `type: "session.update"` - The event type, must be `response.create`. + The event type, must be `session.update`. - - `"response.create"` + - `"session.update"` - `event_id: optional string` - Optional client-generated ID used to identify this event. + Optional client-generated ID used to identify this event. This is an arbitrary string that a client may assign. It will be passed back if there is an error with the event, but the corresponding `session.updated` event will not include it. - - `response: optional RealtimeResponseCreateParams` +### Session Updated Event - Create a new Realtime response with these parameters +- `SessionUpdatedEvent object { event_id, session, type }` - - `audio: optional RealtimeResponseCreateAudioOutput` + Returned when a session is updated with a `session.update` event, unless + there is an error. - Configuration for audio input and output. + - `event_id: string` - - `output: optional object { format, voice }` + The unique ID of the server event. + + - `session: RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` + + The session configuration. + + - `RealtimeSessionCreateRequest object { type, audio, include, 9 more }` + + Realtime session object configuration. + + - `type: "realtime"` + + The type of session to create. Always `realtime` for the Realtime API. + + - `"realtime"` + + - `audio: optional RealtimeAudioConfig` + + Configuration for input and output audio. + + - `input: optional RealtimeAudioConfigInput` - `format: optional RealtimeAudioFormats` - The format of the output audio. + The format of the input audio. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -18941,7 +19215,7 @@ - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -18951,7 +19225,7 @@ - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -18961,7721 +19235,2965 @@ - `"audio/pcma"` - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with - an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed - during the session once the model has responded with audio at least once. - We recommend `marin` and `cedar` for best quality. + - `noise_reduction: optional object { type }` - - `string` + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `"alloy" or "ash" or "ballad" or 7 more` + - `type: optional NoiseReductionType` - - `"alloy"` + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - `"ash"` + - `"near_field"` - - `"ballad"` + - `"far_field"` - - `"coral"` + - `transcription: optional AudioTranscription` - - `"echo"` + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - `"sage"` + - `language: optional string` - - `"shimmer"` + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - - `"verse"` + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `"marin"` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `"cedar"` + - `string` - - `ID = object { id }` + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - Custom voice reference. + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `id: string` + - `"whisper-1"` - The custom voice ID, e.g. `voice_1234`. + - `"gpt-4o-mini-transcribe"` - - `conversation: optional string or "auto" or "none"` + - `"gpt-4o-mini-transcribe-2025-12-15"` - Controls which conversation the response is added to. Currently supports - `auto` and `none`, with `auto` as the default value. The `auto` value - means that the contents of the response will be added to the default - conversation. Set this to `none` to create an out-of-band response which - will not add items to default conversation. + - `"gpt-4o-transcribe"` - - `string` + - `"gpt-4o-transcribe-diarize"` - - `"auto" or "none"` + - `prompt: optional string` - Controls which conversation the response is added to. Currently supports - `auto` and `none`, with `auto` as the default value. The `auto` value - means that the contents of the response will be added to the default - conversation. Set this to `none` to create an out-of-band response which - will not add items to default conversation. + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - `"auto"` + - `turn_detection: optional RealtimeAudioInputTurnDetection` - - `"none"` + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - `input: optional array of ConversationItem` + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - Input items to include in the prompt for the model. Using this field - creates a new context for this Response instead of using the default - conversation. An empty array `[]` will clear the context for this Response. - Note that this can include references to items that previously appeared in the session - using their id. + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - `content: array of object { text, type }` + - `type: "server_vad"` - The content of the message. + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `text: optional string` + - `"server_vad"` - The text content. + - `create_response: optional boolean` - - `type: optional "input_text"` + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - The content type. Always `input_text` for system messages. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `"input_text"` + - `idle_timeout_ms: optional number` - - `role: "system"` + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - The role of the message sender. Always `system`. + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - - `"system"` + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - - `type: "message"` + - `interrupt_response: optional boolean` - The type of the item. Always `message`. + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - `"message"` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `id: optional string` + - `prefix_padding_ms: optional number` - The unique ID of the item. This may be provided by the client or generated by the server. + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - - `object: optional "realtime.item"` + - `silence_duration_ms: optional number` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - - `"realtime.item"` + - `threshold: optional number` - - `status: optional "completed" or "incomplete" or "in_progress"` + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - The status of the item. Has no effect on the conversation. + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - - `"completed"` + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - `"incomplete"` + - `type: "semantic_vad"` - - `"in_progress"` + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `"semantic_vad"` - A user message item in a Realtime conversation. + - `create_response: optional boolean` - - `content: array of object { audio, detail, image_url, 3 more }` + Whether or not to automatically generate a response when a VAD stop event occurs. - The content of the message. + - `eagerness: optional "low" or "medium" or "high" or "auto"` - - `audio: optional string` + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `"low"` - - `detail: optional "auto" or "low" or "high"` + - `"medium"` - The detail level of the image (for `input_image`). `auto` will default to `high`. + - `"high"` - `"auto"` - - `"low"` + - `interrupt_response: optional boolean` - - `"high"` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - `image_url: optional string` + - `output: optional RealtimeAudioConfigOutput` - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + - `format: optional RealtimeAudioFormats` - - `text: optional string` + The format of the output audio. - The text content (for `input_text`). + - `speed: optional number` - - `transcript: optional string` + The speed of the model's spoken response as a multiple of the original speed. + 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + This parameter is a post-processing adjustment to the audio after it is generated, it's + also possible to prompt the model to speak faster or slower. - - `type: optional "input_text" or "input_audio" or "input_image"` + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - The content type (`input_text`, `input_audio`, or `input_image`). + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. - - `"input_text"` + - `string` - - `"input_audio"` + - `"alloy" or "ash" or "ballad" or 7 more` - - `"input_image"` + - `"alloy"` - - `role: "user"` + - `"ash"` - The role of the message sender. Always `user`. + - `"ballad"` - - `"user"` + - `"coral"` - - `type: "message"` + - `"echo"` - The type of the item. Always `message`. + - `"sage"` - - `"message"` + - `"shimmer"` - - `id: optional string` + - `"verse"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `"marin"` - - `object: optional "realtime.item"` + - `"cedar"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `ID object { id }` - - `"realtime.item"` + Custom voice reference. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `id: string` - The status of the item. Has no effect on the conversation. + The custom voice ID, e.g. `voice_1234`. - - `"completed"` + - `include: optional array of "item.input_audio_transcription.logprobs"` - - `"incomplete"` + Additional fields to include in server outputs. - - `"in_progress"` + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `"item.input_audio_transcription.logprobs"` - An assistant message item in a Realtime conversation. + - `instructions: optional string` - - `content: array of object { audio, text, transcript, type }` + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - The content of the message. + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - `audio: optional string` + - `max_output_tokens: optional number or "inf"` - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - - `text: optional string` + - `number` - The text content. + - `"inf"` - - `transcript: optional string` + - `"inf"` - The transcript of the audio content, this will always be present if the output type is `audio`. + - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - `type: optional "output_text" or "output_audio"` + The Realtime model used for this session. - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + - `string` - - `"output_text"` + - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - `"output_audio"` + The Realtime model used for this session. - - `role: "assistant"` + - `"gpt-realtime"` - The role of the message sender. Always `assistant`. + - `"gpt-realtime-1.5"` - - `"assistant"` + - `"gpt-realtime-2025-08-28"` - - `type: "message"` + - `"gpt-4o-realtime-preview"` - The type of the item. Always `message`. + - `"gpt-4o-realtime-preview-2024-10-01"` - - `"message"` + - `"gpt-4o-realtime-preview-2024-12-17"` - - `id: optional string` + - `"gpt-4o-realtime-preview-2025-06-03"` - The unique ID of the item. This may be provided by the client or generated by the server. + - `"gpt-4o-mini-realtime-preview"` - - `object: optional "realtime.item"` + - `"gpt-4o-mini-realtime-preview-2024-12-17"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"gpt-realtime-mini"` - - `"realtime.item"` + - `"gpt-realtime-mini-2025-10-06"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"gpt-realtime-mini-2025-12-15"` - The status of the item. Has no effect on the conversation. + - `"gpt-audio-1.5"` - - `"completed"` + - `"gpt-audio-mini"` - - `"incomplete"` + - `"gpt-audio-mini-2025-10-06"` - - `"in_progress"` + - `"gpt-audio-mini-2025-12-15"` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `output_modalities: optional array of "text" or "audio"` - A function call item in a Realtime conversation. + The set of modalities the model can respond with. It defaults to `["audio"]`, indicating + that the model will respond with audio plus a transcript. `["text"]` can be used to make + the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - - `arguments: string` + - `"text"` - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + - `"audio"` - - `name: string` + - `prompt: optional ResponsePrompt` - The name of the function being called. + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - - `type: "function_call"` + - `id: string` - The type of the item. Always `function_call`. + The unique identifier of the prompt template to use. - - `"function_call"` + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - `id: optional string` + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - The unique ID of the item. This may be provided by the client or generated by the server. + - `string` - - `call_id: optional string` + - `ResponseInputText object { text, type }` - The ID of the function call. + A text input to the model. - - `object: optional "realtime.item"` + - `text: string` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + The text input to the model. - - `"realtime.item"` + - `type: "input_text"` - - `status: optional "completed" or "incomplete" or "in_progress"` + The type of the input item. Always `input_text`. - The status of the item. Has no effect on the conversation. + - `"input_text"` - - `"completed"` + - `ResponseInputImage object { detail, type, file_id, image_url }` - - `"incomplete"` + An image input to the model. Learn about [image inputs](/docs/guides/vision). - - `"in_progress"` + - `detail: "low" or "high" or "auto" or "original"` - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - A function call output item in a Realtime conversation. + - `"low"` - - `call_id: string` + - `"high"` - The ID of the function call this output is for. + - `"auto"` - - `output: string` + - `"original"` - The output of the function call, this is free text and can contain any information or simply be empty. + - `type: "input_image"` - - `type: "function_call_output"` + The type of the input item. Always `input_image`. - The type of the item. Always `function_call_output`. + - `"input_image"` - - `"function_call_output"` + - `file_id: optional string` - - `id: optional string` + The ID of the file to be sent to the model. - The unique ID of the item. This may be provided by the client or generated by the server. + - `image_url: optional string` - - `object: optional "realtime.item"` + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `ResponseInputFile object { type, detail, file_data, 3 more }` - - `"realtime.item"` + A file input to the model. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `type: "input_file"` - The status of the item. Has no effect on the conversation. + The type of the input item. Always `input_file`. - - `"completed"` + - `"input_file"` - - `"incomplete"` + - `detail: optional "low" or "high"` - - `"in_progress"` + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `"low"` - A Realtime item responding to an MCP approval request. + - `"high"` - - `id: string` + - `file_data: optional string` - The unique ID of the approval response. + The content of the file to be sent to the model. - - `approval_request_id: string` + - `file_id: optional string` - The ID of the approval request being answered. + The ID of the file to be sent to the model. - - `approve: boolean` + - `file_url: optional string` - Whether the request was approved. + The URL of the file to be sent to the model. - - `type: "mcp_approval_response"` + - `filename: optional string` - The type of the item. Always `mcp_approval_response`. + The name of the file to be sent to the model. - - `"mcp_approval_response"` + - `version: optional string` - - `reason: optional string` + Optional version of the prompt template. - Optional reason for the decision. + - `tool_choice: optional RealtimeToolChoiceConfig` - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. - A Realtime item listing tools available on an MCP server. + - `ToolChoiceOptions = "none" or "auto" or "required"` - - `server_label: string` + Controls which (if any) tool is called by the model. - The label of the MCP server. + `none` means the model will not call any tool and instead generates a message. - - `tools: array of object { input_schema, name, annotations, description }` + `auto` means the model can pick between generating a message or calling one or + more tools. - The tools available on the server. + `required` means the model must call one or more tools. - - `input_schema: unknown` + - `"none"` - The JSON schema describing the tool's input. + - `"auto"` - - `name: string` + - `"required"` - The name of the tool. + - `ToolChoiceFunction object { name, type }` - - `annotations: optional unknown` + Use this option to force the model to call a specific function. - Additional annotations about the tool. + - `name: string` - - `description: optional string` + The name of the function to call. - The description of the tool. + - `type: "function"` - - `type: "mcp_list_tools"` + For function calling, the type is always `function`. - The type of the item. Always `mcp_list_tools`. + - `"function"` - - `"mcp_list_tools"` + - `ToolChoiceMcp object { server_label, type, name }` - - `id: optional string` + Use this option to force the model to call a specific tool on a remote MCP server. - The unique ID of the list. + - `server_label: string` - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + The label of the MCP server to use. - A Realtime item representing an invocation of a tool on an MCP server. + - `type: "mcp"` - - `id: string` + For MCP tools, the type is always `mcp`. - The unique ID of the tool call. + - `"mcp"` - - `arguments: string` + - `name: optional string` - A JSON string of the arguments passed to the tool. + The name of the tool to call on the server. - - `name: string` + - `tools: optional RealtimeToolsConfig` - The name of the tool that was run. + Tools available to the model. - - `server_label: string` + - `RealtimeFunctionTool object { description, name, parameters, type }` - The label of the MCP server running the tool. + - `description: optional string` - - `type: "mcp_call"` + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - The type of the item. Always `mcp_call`. + - `name: optional string` - - `"mcp_call"` + The name of the function. - - `approval_request_id: optional string` + - `parameters: optional unknown` - The ID of an associated approval request, if any. + Parameters of the function in JSON Schema. - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` + - `type: optional "function"` - The error from the tool call, if any. + The type of the tool, i.e. `function`. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `"function"` - - `code: number` + - `McpTool object { server_label, type, allowed_tools, 7 more }` - - `message: string` + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - - `type: "protocol_error"` + - `server_label: string` - - `"protocol_error"` + A label for this MCP server, used to identify it in tool calls. - - `RealtimeMcpToolExecutionError = object { message, type }` + - `type: "mcp"` - - `message: string` + The type of the MCP tool. Always `mcp`. - - `type: "tool_execution_error"` + - `"mcp"` - - `"tool_execution_error"` + - `allowed_tools: optional array of string or object { read_only, tool_names }` - - `RealtimeMcphttpError = object { code, message, type }` + List of allowed tool names or a filter object. - - `code: number` + - `McpAllowedTools = array of string` - - `message: string` + A string array of allowed tool names - - `type: "http_error"` + - `McpToolFilter object { read_only, tool_names }` - - `"http_error"` + A filter object to specify which tools are allowed. - - `output: optional string` + - `read_only: optional boolean` - The output from the tool call. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `tool_names: optional array of string` - A Realtime item requesting human approval of a tool invocation. + List of allowed tool names. - - `id: string` + - `authorization: optional string` - The unique ID of the approval request. + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. - - `arguments: string` + - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - A JSON string of arguments for the tool. + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](/docs/guides/tools-remote-mcp#connectors). - - `name: string` + Currently supported `connector_id` values are: - The name of the tool to run. + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` - - `server_label: string` + - `"connector_dropbox"` - The label of the MCP server making the request. + - `"connector_gmail"` - - `type: "mcp_approval_request"` + - `"connector_googlecalendar"` - The type of the item. Always `mcp_approval_request`. + - `"connector_googledrive"` - - `"mcp_approval_request"` + - `"connector_microsoftteams"` - - `instructions: optional string` + - `"connector_outlookcalendar"` - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. + - `"connector_outlookemail"` - - `max_output_tokens: optional number or "inf"` + - `"connector_sharepoint"` - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. + - `defer_loading: optional boolean` - - `number` + Whether this MCP tool is deferred and discovered via tool search. - - `"inf"` + - `headers: optional map[string]` - - `"inf"` + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. - - `metadata: optional Metadata` + - `require_approval: optional object { always, never } or "always" or "never"` - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. + Specify which of the MCP server's tools require approval. - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. + - `McpToolApprovalFilter object { always, never }` - - `output_modalities: optional array of "text" or "audio"` + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. - The set of modalities the model used to respond, currently the only possible values are - `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the - output to mode `text` will disable audio output from the model. + - `always: optional object { read_only, tool_names }` - - `"text"` + A filter object to specify which tools are allowed. - - `"audio"` + - `read_only: optional boolean` - - `prompt: optional ResponsePrompt` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). + - `tool_names: optional array of string` - - `id: string` + List of allowed tool names. - The unique identifier of the prompt template to use. + - `never: optional object { read_only, tool_names }` - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` + A filter object to specify which tools are allowed. - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. + - `read_only: optional boolean` - - `string` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `ResponseInputText = object { text, type }` + - `tool_names: optional array of string` - A text input to the model. + List of allowed tool names. - - `text: string` + - `McpToolApprovalSetting = "always" or "never"` - The text input to the model. + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. - - `type: "input_text"` + - `"always"` - The type of the input item. Always `input_text`. + - `"never"` - - `"input_text"` + - `server_description: optional string` - - `ResponseInputImage = object { detail, type, file_id, image_url }` + Optional description of the MCP server, used to provide more context. - An image input to the model. Learn about [image inputs](/docs/guides/vision). + - `server_url: optional string` - - `detail: "low" or "high" or "auto" or "original"` + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. + - `tracing: optional RealtimeTracingConfig` - - `"low"` + Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once + tracing is enabled for a session, the configuration cannot be modified. - - `"high"` + `auto` will create a trace for the session with default values for the + workflow name, group id, and metadata. + + - `Auto = "auto"` + + Enables tracing and sets default values for tracing configuration options. Always `auto`. - `"auto"` - - `"original"` + - `TracingConfiguration object { group_id, metadata, workflow_name }` - - `type: "input_image"` + Granular configuration for tracing. - The type of the input item. Always `input_image`. + - `group_id: optional string` - - `"input_image"` + The group id to attach to this trace to enable filtering and + grouping in the Traces Dashboard. - - `file_id: optional string` + - `metadata: optional unknown` - The ID of the file to be sent to the model. + The arbitrary metadata to attach to this trace to enable + filtering in the Traces Dashboard. - - `image_url: optional string` + - `workflow_name: optional string` - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. + The name of the workflow to attach to this trace. This is used to + name the trace in the Traces Dashboard. - - `ResponseInputFile = object { type, detail, file_data, 3 more }` + - `truncation: optional RealtimeTruncation` - A file input to the model. + When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - - `type: "input_file"` + Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - The type of the input item. Always `input_file`. + Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - `"input_file"` + Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - - `detail: optional "low" or "high"` + - `"auto" or "disabled"` - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. + The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - - `"low"` + - `"auto"` - - `"high"` + - `"disabled"` - - `file_data: optional string` + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` - The content of the file to be sent to the model. + Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - - `file_id: optional string` + - `retention_ratio: number` - The ID of the file to be sent to the model. + Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - `file_url: optional string` + - `type: "retention_ratio"` - The URL of the file to be sent to the model. + Use retention ratio truncation. - - `filename: optional string` + - `"retention_ratio"` - The name of the file to be sent to the model. + - `token_limits: optional object { post_instructions }` - - `version: optional string` + Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - Optional version of the prompt template. + - `post_instructions: optional number` - - `tool_choice: optional ToolChoiceOptions or ToolChoiceFunction or ToolChoiceMcp` + Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. + - `RealtimeTranscriptionSessionCreateRequest object { type, audio, include }` - - `ToolChoiceOptions = "none" or "auto" or "required"` - - Controls which (if any) tool is called by the model. + Realtime transcription session object configuration. - `none` means the model will not call any tool and instead generates a message. + - `type: "transcription"` - `auto` means the model can pick between generating a message or calling one or - more tools. + The type of session to create. Always `transcription` for transcription sessions. - `required` means the model must call one or more tools. + - `"transcription"` - - `"none"` + - `audio: optional RealtimeTranscriptionSessionAudio` - - `"auto"` + Configuration for input and output audio. - - `"required"` + - `input: optional RealtimeTranscriptionSessionAudioInput` - - `ToolChoiceFunction = object { name, type }` + - `format: optional RealtimeAudioFormats` - Use this option to force the model to call a specific function. + The PCM audio format. Only a 24kHz sample rate is supported. - - `name: string` + - `noise_reduction: optional object { type }` - The name of the function to call. + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `type: "function"` + - `type: optional NoiseReductionType` - For function calling, the type is always `function`. + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - `"function"` + - `transcription: optional AudioTranscription` - - `ToolChoiceMcp = object { server_label, type, name }` + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - Use this option to force the model to call a specific tool on a remote MCP server. + - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` - - `server_label: string` + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - The label of the MCP server to use. + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - `type: "mcp"` + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - For MCP tools, the type is always `mcp`. + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - - `"mcp"` + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - `name: optional string` + - `type: "server_vad"` - The name of the tool to call on the server. + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `tools: optional array of RealtimeFunctionTool or object { server_label, type, allowed_tools, 7 more }` + - `"server_vad"` - Tools available to the model. + - `create_response: optional boolean` - - `RealtimeFunctionTool = object { description, name, parameters, type }` + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - `description: optional string` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). + - `idle_timeout_ms: optional number` - - `name: optional string` + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - The name of the function. + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - - `parameters: optional unknown` + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - Parameters of the function in JSON Schema. + - `interrupt_response: optional boolean` - - `type: optional "function"` + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - The type of the tool, i.e. `function`. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `"function"` + - `prefix_padding_ms: optional number` - - `McpTool = object { server_label, type, allowed_tools, 7 more }` + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). + - `silence_duration_ms: optional number` - - `server_label: string` + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - A label for this MCP server, used to identify it in tool calls. + - `threshold: optional number` - - `type: "mcp"` + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - The type of the MCP tool. Always `mcp`. + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - - `"mcp"` + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - `allowed_tools: optional array of string or object { read_only, tool_names }` + - `type: "semantic_vad"` - List of allowed tool names or a filter object. + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - `McpAllowedTools = array of string` + - `"semantic_vad"` - A string array of allowed tool names + - `create_response: optional boolean` - - `McpToolFilter = object { read_only, tool_names }` + Whether or not to automatically generate a response when a VAD stop event occurs. - A filter object to specify which tools are allowed. + - `eagerness: optional "low" or "medium" or "high" or "auto"` - - `read_only: optional boolean` + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `"low"` - - `tool_names: optional array of string` + - `"medium"` - List of allowed tool names. + - `"high"` - - `authorization: optional string` + - `"auto"` - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. + - `interrupt_response: optional boolean` - - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](/docs/guides/tools-remote-mcp#connectors). + - `include: optional array of "item.input_audio_transcription.logprobs"` - Currently supported `connector_id` values are: + Additional fields to include in server outputs. - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - `"connector_dropbox"` + - `"item.input_audio_transcription.logprobs"` - - `"connector_gmail"` + - `type: "session.updated"` - - `"connector_googlecalendar"` + The event type, must be `session.updated`. - - `"connector_googledrive"` + - `"session.updated"` - - `"connector_microsoftteams"` +### Transcription Session Update - - `"connector_outlookcalendar"` +- `TranscriptionSessionUpdate object { session, type, event_id }` - - `"connector_outlookemail"` + Send this event to update a transcription session. - - `"connector_sharepoint"` + - `session: object { include, input_audio_format, input_audio_noise_reduction, 2 more }` - - `defer_loading: optional boolean` + Realtime transcription session object configuration. - Whether this MCP tool is deferred and discovered via tool search. + - `include: optional array of "item.input_audio_transcription.logprobs"` - - `headers: optional map[string]` + The set of items to include in the transcription. Current available items are: + `item.input_audio_transcription.logprobs` - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. + - `"item.input_audio_transcription.logprobs"` - - `require_approval: optional object { always, never } or "always" or "never"` + - `input_audio_format: optional "pcm16" or "g711_ulaw" or "g711_alaw"` - Specify which of the MCP server's tools require approval. + The format of input audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. + For `pcm16`, input audio must be 16-bit PCM at a 24kHz sample rate, + single channel (mono), and little-endian byte order. - - `McpToolApprovalFilter = object { always, never }` + - `"pcm16"` - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. + - `"g711_ulaw"` - - `always: optional object { read_only, tool_names }` + - `"g711_alaw"` - A filter object to specify which tools are allowed. + - `input_audio_noise_reduction: optional object { type }` - - `read_only: optional boolean` + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `type: optional NoiseReductionType` - - `tool_names: optional array of string` + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - List of allowed tool names. + - `"near_field"` - - `never: optional object { read_only, tool_names }` + - `"far_field"` - A filter object to specify which tools are allowed. + - `input_audio_transcription: optional AudioTranscription` - - `read_only: optional boolean` + Configuration for input audio transcription. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `language: optional string` - - `tool_names: optional array of string` + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - List of allowed tool names. + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `McpToolApprovalSetting = "always" or "never"` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. + - `string` - - `"always"` + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `"never"` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `server_description: optional string` + - `"whisper-1"` - Optional description of the MCP server, used to provide more context. + - `"gpt-4o-mini-transcribe"` - - `server_url: optional string` + - `"gpt-4o-mini-transcribe-2025-12-15"` - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. + - `"gpt-4o-transcribe"` -### Response Created Event + - `"gpt-4o-transcribe-diarize"` -- `ResponseCreatedEvent = object { event_id, response, type }` + - `prompt: optional string` - Returned when a new Response is created. The first event of response creation, - where the response is in an initial state of `in_progress`. + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - `event_id: string` + - `turn_detection: optional object { prefix_padding_ms, silence_duration_ms, threshold, type }` - The unique ID of the server event. + Configuration for turn detection. Can be set to `null` to turn off. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - `response: RealtimeResponse` + - `prefix_padding_ms: optional number` - The response resource. + Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - - `id: optional string` + - `silence_duration_ms: optional number` - The unique ID of the response, will look like `resp_1234`. + Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - - `audio: optional object { output }` + - `threshold: optional number` - Configuration for audio output. + Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - - `output: optional object { format, voice }` + - `type: optional "server_vad"` - - `format: optional RealtimeAudioFormats` + Type of turn detection. Only `server_vad` is currently supported for transcription sessions. - The format of the output audio. + - `"server_vad"` - - `PCMAudioFormat = object { rate, type }` + - `type: "transcription_session.update"` - The PCM audio format. Only a 24kHz sample rate is supported. + The event type, must be `transcription_session.update`. - - `rate: optional 24000` + - `"transcription_session.update"` - The sample rate of the audio. Always `24000`. + - `event_id: optional string` - - `24000` + Optional client-generated ID used to identify this event. - - `type: optional "audio/pcm"` +### Transcription Session Updated Event - The audio format. Always `audio/pcm`. +- `TranscriptionSessionUpdatedEvent object { event_id, session, type }` - - `"audio/pcm"` + Returned when a transcription session is updated with a `transcription_session.update` event, unless + there is an error. - - `PCMUAudioFormat = object { type }` + - `event_id: string` - The G.711 μ-law format. + The unique ID of the server event. - - `type: optional "audio/pcmu"` + - `session: object { client_secret, input_audio_format, input_audio_transcription, 2 more }` - The audio format. Always `audio/pcmu`. + A new Realtime transcription session configuration. - - `"audio/pcmu"` + When a session is created on the server via REST API, the session object + also contains an ephemeral key. Default TTL for keys is 10 minutes. This + property is not present when a session is updated via the WebSocket API. - - `PCMAAudioFormat = object { type }` + - `client_secret: object { expires_at, value }` - The G.711 A-law format. + Ephemeral key returned by the API. Only present when the session is + created on the server via REST API. - - `type: optional "audio/pcma"` + - `expires_at: number` - The audio format. Always `audio/pcma`. + Timestamp for when the token expires. Currently, all tokens expire + after one minute. - - `"audio/pcma"` + - `value: string` - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more` + Ephemeral key usable in client environments to authenticate connections + to the Realtime API. Use this in client-side environments rather than + a standard API token, which should only be used server-side. - The voice the model uses to respond. Voice cannot be changed during the - session once the model has responded with audio at least once. Current - voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, - `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for - best quality. + - `input_audio_format: optional string` - - `string` + The format of input audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. - - `"alloy" or "ash" or "ballad" or 7 more` + - `input_audio_transcription: optional AudioTranscription` - The voice the model uses to respond. Voice cannot be changed during the - session once the model has responded with audio at least once. Current - voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, - `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for - best quality. + Configuration of the transcription model. - - `"alloy"` + - `language: optional string` - - `"ash"` + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - - `"ballad"` + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `"coral"` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `"echo"` + - `string` - - `"sage"` + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `"shimmer"` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `"verse"` + - `"whisper-1"` - - `"marin"` + - `"gpt-4o-mini-transcribe"` - - `"cedar"` + - `"gpt-4o-mini-transcribe-2025-12-15"` - - `conversation_id: optional string` + - `"gpt-4o-transcribe"` - Which conversation the response is added to, determined by the `conversation` - field in the `response.create` event. If `auto`, the response will be added to - the default conversation and the value of `conversation_id` will be an id like - `conv_1234`. If `none`, the response will not be added to any conversation and - the value of `conversation_id` will be `null`. If responses are being triggered - automatically by VAD the response will be added to the default conversation + - `"gpt-4o-transcribe-diarize"` - - `max_output_tokens: optional number or "inf"` + - `prompt: optional string` - Maximum number of output tokens for a single assistant response, - inclusive of tool calls, that was used in this response. + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - `number` + - `modalities: optional array of "text" or "audio"` - - `"inf"` + The set of modalities the model can respond with. To disable audio, + set this to ["text"]. - - `"inf"` + - `"text"` - - `metadata: optional Metadata` + - `"audio"` - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. + - `turn_detection: optional object { prefix_padding_ms, silence_duration_ms, threshold, type }` - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. + Configuration for turn detection. Can be set to `null` to turn off. Server + VAD means that the model will detect the start and end of speech based on + audio volume and respond at the end of user speech. - - `object: optional "realtime.response"` + - `prefix_padding_ms: optional number` - The object type, must be `realtime.response`. + Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - - `"realtime.response"` + - `silence_duration_ms: optional number` - - `output: optional array of ConversationItem` + Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - The list of output items generated by the response. + - `threshold: optional number` - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + - `type: optional string` - - `content: array of object { text, type }` + Type of turn detection, only `server_vad` is currently supported. - The content of the message. + - `type: "transcription_session.updated"` - - `text: optional string` + The event type, must be `transcription_session.updated`. - The text content. + - `"transcription_session.updated"` - - `type: optional "input_text"` +# Client Secrets - The content type. Always `input_text` for system messages. +## Create client secret - - `"input_text"` +**post** `/realtime/client_secrets` - - `role: "system"` +Create a Realtime client secret with an associated session configuration. - The role of the message sender. Always `system`. +Client secrets are short-lived tokens that can be passed to a client app, +such as a web frontend or mobile client, which grants access to the Realtime API without +leaking your main API key. You can configure a custom TTL for each client secret. - - `"system"` +You can also attach session configuration options to the client secret, which will be +applied to any sessions created using that client secret, but these can also be overridden +by the client connection. - - `type: "message"` +[Learn more about authentication with client secrets over WebRTC](/docs/guides/realtime-webrtc). - The type of the item. Always `message`. +Returns the created client secret and the effective session object. The client secret is a string that looks like `ek_1234`. - - `"message"` +### Body Parameters - - `id: optional string` +- `expires_after: optional object { anchor, seconds }` - The unique ID of the item. This may be provided by the client or generated by the server. + Configuration for the client secret expiration. Expiration refers to the time after which + a client secret will no longer be valid for creating sessions. The session itself may + continue after that time once started. A secret can be used to create multiple sessions + until it expires. - - `object: optional "realtime.item"` + - `anchor: optional "created_at"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + The anchor point for the client secret expiration, meaning that `seconds` will be added to the `created_at` time of the client secret to produce an expiration timestamp. Only `created_at` is currently supported. - - `"realtime.item"` + - `"created_at"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `seconds: optional number` - The status of the item. Has no effect on the conversation. + The number of seconds from the anchor point to the expiration. Select a value between `10` and `7200` (2 hours). This default to 600 seconds (10 minutes) if not specified. - - `"completed"` +- `session: optional RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` - - `"incomplete"` + Session configuration to use for the client secret. Choose either a realtime + session or a transcription session. - - `"in_progress"` + - `RealtimeSessionCreateRequest object { type, audio, include, 9 more }` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + Realtime session object configuration. - A user message item in a Realtime conversation. + - `type: "realtime"` - - `content: array of object { audio, detail, image_url, 3 more }` + The type of session to create. Always `realtime` for the Realtime API. - The content of the message. + - `"realtime"` - - `audio: optional string` + - `audio: optional RealtimeAudioConfig` - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + Configuration for input and output audio. - - `detail: optional "auto" or "low" or "high"` + - `input: optional RealtimeAudioConfigInput` - The detail level of the image (for `input_image`). `auto` will default to `high`. + - `format: optional RealtimeAudioFormats` - - `"auto"` + The format of the input audio. - - `"low"` + - `PCMAudioFormat object { rate, type }` - - `"high"` + The PCM audio format. Only a 24kHz sample rate is supported. - - `image_url: optional string` + - `rate: optional 24000` - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + The sample rate of the audio. Always `24000`. - - `text: optional string` + - `24000` - The text content (for `input_text`). + - `type: optional "audio/pcm"` - - `transcript: optional string` + The audio format. Always `audio/pcm`. - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + - `"audio/pcm"` - - `type: optional "input_text" or "input_audio" or "input_image"` + - `PCMUAudioFormat object { type }` - The content type (`input_text`, `input_audio`, or `input_image`). + The G.711 μ-law format. - - `"input_text"` + - `type: optional "audio/pcmu"` - - `"input_audio"` + The audio format. Always `audio/pcmu`. - - `"input_image"` + - `"audio/pcmu"` - - `role: "user"` + - `PCMAAudioFormat object { type }` - The role of the message sender. Always `user`. + The G.711 A-law format. - - `"user"` + - `type: optional "audio/pcma"` - - `type: "message"` + The audio format. Always `audio/pcma`. - The type of the item. Always `message`. + - `"audio/pcma"` - - `"message"` + - `noise_reduction: optional object { type }` - - `id: optional string` + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - The unique ID of the item. This may be provided by the client or generated by the server. + - `type: optional NoiseReductionType` - - `object: optional "realtime.item"` + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"near_field"` - - `"realtime.item"` + - `"far_field"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `transcription: optional AudioTranscription` - The status of the item. Has no effect on the conversation. + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - `"completed"` + - `language: optional string` - - `"incomplete"` + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - - `"in_progress"` + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - An assistant message item in a Realtime conversation. + - `string` - - `content: array of object { audio, text, transcript, type }` + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - The content of the message. + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `audio: optional string` + - `"whisper-1"` - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + - `"gpt-4o-mini-transcribe"` - - `text: optional string` + - `"gpt-4o-mini-transcribe-2025-12-15"` - The text content. + - `"gpt-4o-transcribe"` - - `transcript: optional string` + - `"gpt-4o-transcribe-diarize"` - The transcript of the audio content, this will always be present if the output type is `audio`. + - `prompt: optional string` - - `type: optional "output_text" or "output_audio"` + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + - `turn_detection: optional RealtimeAudioInputTurnDetection` - - `"output_text"` + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - `"output_audio"` + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - `role: "assistant"` + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - The role of the message sender. Always `assistant`. + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - - `"assistant"` + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - `type: "message"` + - `type: "server_vad"` - The type of the item. Always `message`. + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `"message"` + - `"server_vad"` - - `id: optional string` + - `create_response: optional boolean` - The unique ID of the item. This may be provided by the client or generated by the server. + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - `object: optional "realtime.item"` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `idle_timeout_ms: optional number` - - `"realtime.item"` + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - - `status: optional "completed" or "incomplete" or "in_progress"` + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - The status of the item. Has no effect on the conversation. + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - - `"completed"` + - `interrupt_response: optional boolean` - - `"incomplete"` + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - `"in_progress"` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + - `prefix_padding_ms: optional number` - A function call item in a Realtime conversation. + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - - `arguments: string` + - `silence_duration_ms: optional number` - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - - `name: string` + - `threshold: optional number` - The name of the function being called. + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - - `type: "function_call"` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - The type of the item. Always `function_call`. + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - `"function_call"` + - `type: "semantic_vad"` - - `id: optional string` + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - The unique ID of the item. This may be provided by the client or generated by the server. + - `"semantic_vad"` - - `call_id: optional string` + - `create_response: optional boolean` - The ID of the function call. + Whether or not to automatically generate a response when a VAD stop event occurs. - - `object: optional "realtime.item"` + - `eagerness: optional "low" or "medium" or "high" or "auto"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - `"realtime.item"` + - `"low"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"medium"` - The status of the item. Has no effect on the conversation. + - `"high"` - - `"completed"` + - `"auto"` - - `"incomplete"` + - `interrupt_response: optional boolean` - - `"in_progress"` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `output: optional RealtimeAudioConfigOutput` - A function call output item in a Realtime conversation. + - `format: optional RealtimeAudioFormats` - - `call_id: string` + The format of the output audio. - The ID of the function call this output is for. + - `speed: optional number` - - `output: string` + The speed of the model's spoken response as a multiple of the original speed. + 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - The output of the function call, this is free text and can contain any information or simply be empty. + This parameter is a post-processing adjustment to the audio after it is generated, it's + also possible to prompt the model to speak faster or slower. - - `type: "function_call_output"` + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - The type of the item. Always `function_call_output`. + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. - - `"function_call_output"` + - `string` - - `id: optional string` + - `"alloy" or "ash" or "ballad" or 7 more` - The unique ID of the item. This may be provided by the client or generated by the server. + - `"alloy"` - - `object: optional "realtime.item"` + - `"ash"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `"ballad"` - - `"realtime.item"` + - `"coral"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"echo"` - The status of the item. Has no effect on the conversation. + - `"sage"` - - `"completed"` + - `"shimmer"` - - `"incomplete"` + - `"verse"` - - `"in_progress"` + - `"marin"` + + - `"cedar"` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + - `ID object { id }` - A Realtime item responding to an MCP approval request. + Custom voice reference. - `id: string` - The unique ID of the approval response. + The custom voice ID, e.g. `voice_1234`. - - `approval_request_id: string` + - `include: optional array of "item.input_audio_transcription.logprobs"` - The ID of the approval request being answered. + Additional fields to include in server outputs. - - `approve: boolean` + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - Whether the request was approved. + - `"item.input_audio_transcription.logprobs"` - - `type: "mcp_approval_response"` + - `instructions: optional string` - The type of the item. Always `mcp_approval_response`. + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - - `"mcp_approval_response"` + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - `reason: optional string` + - `max_output_tokens: optional number or "inf"` - Optional reason for the decision. + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `number` - A Realtime item listing tools available on an MCP server. + - `"inf"` - - `server_label: string` + - `"inf"` - The label of the MCP server. + - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - `tools: array of object { input_schema, name, annotations, description }` + The Realtime model used for this session. - The tools available on the server. + - `string` - - `input_schema: unknown` + - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - The JSON schema describing the tool's input. + The Realtime model used for this session. - - `name: string` + - `"gpt-realtime"` - The name of the tool. + - `"gpt-realtime-1.5"` - - `annotations: optional unknown` + - `"gpt-realtime-2025-08-28"` - Additional annotations about the tool. + - `"gpt-4o-realtime-preview"` - - `description: optional string` + - `"gpt-4o-realtime-preview-2024-10-01"` - The description of the tool. + - `"gpt-4o-realtime-preview-2024-12-17"` - - `type: "mcp_list_tools"` + - `"gpt-4o-realtime-preview-2025-06-03"` - The type of the item. Always `mcp_list_tools`. + - `"gpt-4o-mini-realtime-preview"` - - `"mcp_list_tools"` + - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - `id: optional string` + - `"gpt-realtime-mini"` - The unique ID of the list. + - `"gpt-realtime-mini-2025-10-06"` - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + - `"gpt-realtime-mini-2025-12-15"` - A Realtime item representing an invocation of a tool on an MCP server. + - `"gpt-audio-1.5"` - - `id: string` + - `"gpt-audio-mini"` - The unique ID of the tool call. + - `"gpt-audio-mini-2025-10-06"` - - `arguments: string` + - `"gpt-audio-mini-2025-12-15"` - A JSON string of the arguments passed to the tool. + - `output_modalities: optional array of "text" or "audio"` - - `name: string` + The set of modalities the model can respond with. It defaults to `["audio"]`, indicating + that the model will respond with audio plus a transcript. `["text"]` can be used to make + the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - The name of the tool that was run. + - `"text"` - - `server_label: string` + - `"audio"` - The label of the MCP server running the tool. + - `prompt: optional ResponsePrompt` - - `type: "mcp_call"` + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - The type of the item. Always `mcp_call`. + - `id: string` - - `"mcp_call"` + The unique identifier of the prompt template to use. - - `approval_request_id: optional string` + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - The ID of an associated approval request, if any. + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` + - `string` - The error from the tool call, if any. + - `ResponseInputText object { text, type }` - - `RealtimeMcpProtocolError = object { code, message, type }` + A text input to the model. - - `code: number` + - `text: string` - - `message: string` + The text input to the model. - - `type: "protocol_error"` + - `type: "input_text"` - - `"protocol_error"` + The type of the input item. Always `input_text`. - - `RealtimeMcpToolExecutionError = object { message, type }` + - `"input_text"` - - `message: string` + - `ResponseInputImage object { detail, type, file_id, image_url }` - - `type: "tool_execution_error"` + An image input to the model. Learn about [image inputs](/docs/guides/vision). - - `"tool_execution_error"` + - `detail: "low" or "high" or "auto" or "original"` - - `RealtimeMcphttpError = object { code, message, type }` + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - `code: number` + - `"low"` - - `message: string` + - `"high"` - - `type: "http_error"` + - `"auto"` - - `"http_error"` + - `"original"` - - `output: optional string` + - `type: "input_image"` - The output from the tool call. + The type of the input item. Always `input_image`. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `"input_image"` - A Realtime item requesting human approval of a tool invocation. + - `file_id: optional string` - - `id: string` + The ID of the file to be sent to the model. - The unique ID of the approval request. + - `image_url: optional string` - - `arguments: string` + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - A JSON string of arguments for the tool. + - `ResponseInputFile object { type, detail, file_data, 3 more }` - - `name: string` + A file input to the model. - The name of the tool to run. + - `type: "input_file"` - - `server_label: string` + The type of the input item. Always `input_file`. - The label of the MCP server making the request. + - `"input_file"` - - `type: "mcp_approval_request"` + - `detail: optional "low" or "high"` - The type of the item. Always `mcp_approval_request`. + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - `"mcp_approval_request"` + - `"low"` - - `output_modalities: optional array of "text" or "audio"` + - `"high"` - The set of modalities the model used to respond, currently the only possible values are - `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the - output to mode `text` will disable audio output from the model. + - `file_data: optional string` - - `"text"` + The content of the file to be sent to the model. - - `"audio"` + - `file_id: optional string` - - `status: optional "completed" or "cancelled" or "failed" or 2 more` + The ID of the file to be sent to the model. - The final status of the response (`completed`, `cancelled`, `failed`, or - `incomplete`, `in_progress`). + - `file_url: optional string` - - `"completed"` + The URL of the file to be sent to the model. - - `"cancelled"` + - `filename: optional string` - - `"failed"` + The name of the file to be sent to the model. - - `"incomplete"` + - `version: optional string` - - `"in_progress"` + Optional version of the prompt template. - - `status_details: optional RealtimeResponseStatus` + - `tool_choice: optional RealtimeToolChoiceConfig` - Additional details about the status. + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. - - `error: optional object { code, type }` + - `ToolChoiceOptions = "none" or "auto" or "required"` - A description of the error that caused the response to fail, - populated when the `status` is `failed`. + Controls which (if any) tool is called by the model. - - `code: optional string` + `none` means the model will not call any tool and instead generates a message. - Error code, if any. + `auto` means the model can pick between generating a message or calling one or + more tools. - - `type: optional string` + `required` means the model must call one or more tools. - The type of error. + - `"none"` - - `reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"` + - `"auto"` - The reason the Response did not complete. For a `cancelled` Response, one of `turn_detected` (the server VAD detected a new start of speech) or `client_cancelled` (the client sent a cancel event). For an `incomplete` Response, one of `max_output_tokens` or `content_filter` (the server-side safety filter activated and cut off the response). + - `"required"` - - `"turn_detected"` + - `ToolChoiceFunction object { name, type }` - - `"client_cancelled"` + Use this option to force the model to call a specific function. - - `"max_output_tokens"` + - `name: string` - - `"content_filter"` + The name of the function to call. - - `type: optional "completed" or "cancelled" or "failed" or "incomplete"` + - `type: "function"` - The type of error that caused the response to fail, corresponding - with the `status` field (`completed`, `cancelled`, `incomplete`, - `failed`). + For function calling, the type is always `function`. - - `"completed"` + - `"function"` - - `"cancelled"` + - `ToolChoiceMcp object { server_label, type, name }` - - `"failed"` + Use this option to force the model to call a specific tool on a remote MCP server. - - `"incomplete"` + - `server_label: string` - - `usage: optional RealtimeResponseUsage` + The label of the MCP server to use. - Usage statistics for the Response, this will correspond to billing. A - Realtime API session will maintain a conversation context and append new - Items to the Conversation, thus output from previous turns (text and - audio tokens) will become the input for later turns. + - `type: "mcp"` - - `input_token_details: optional RealtimeResponseUsageInputTokenDetails` + For MCP tools, the type is always `mcp`. - Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. + - `"mcp"` - - `audio_tokens: optional number` + - `name: optional string` - The number of audio tokens used as input for the Response. + The name of the tool to call on the server. - - `cached_tokens: optional number` + - `tools: optional RealtimeToolsConfig` - The number of cached tokens used as input for the Response. + Tools available to the model. - - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` + - `RealtimeFunctionTool object { description, name, parameters, type }` - Details about the cached tokens used as input for the Response. + - `description: optional string` - - `audio_tokens: optional number` + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - The number of cached audio tokens used as input for the Response. + - `name: optional string` - - `image_tokens: optional number` + The name of the function. - The number of cached image tokens used as input for the Response. + - `parameters: optional unknown` - - `text_tokens: optional number` + Parameters of the function in JSON Schema. - The number of cached text tokens used as input for the Response. + - `type: optional "function"` - - `image_tokens: optional number` + The type of the tool, i.e. `function`. - The number of image tokens used as input for the Response. + - `"function"` - - `text_tokens: optional number` + - `McpTool object { server_label, type, allowed_tools, 7 more }` - The number of text tokens used as input for the Response. + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - - `input_tokens: optional number` + - `server_label: string` - The number of input tokens used in the Response, including text and - audio tokens. + A label for this MCP server, used to identify it in tool calls. - - `output_token_details: optional RealtimeResponseUsageOutputTokenDetails` + - `type: "mcp"` - Details about the output tokens used in the Response. + The type of the MCP tool. Always `mcp`. - - `audio_tokens: optional number` + - `"mcp"` - The number of audio tokens used in the Response. + - `allowed_tools: optional array of string or object { read_only, tool_names }` - - `text_tokens: optional number` + List of allowed tool names or a filter object. - The number of text tokens used in the Response. + - `McpAllowedTools = array of string` - - `output_tokens: optional number` + A string array of allowed tool names - The number of output tokens sent in the Response, including text and - audio tokens. + - `McpToolFilter object { read_only, tool_names }` - - `total_tokens: optional number` + A filter object to specify which tools are allowed. - The total number of tokens in the Response including input and output - text and audio tokens. + - `read_only: optional boolean` - - `type: "response.created"` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - The event type, must be `response.created`. + - `tool_names: optional array of string` - - `"response.created"` + List of allowed tool names. -### Response Done Event + - `authorization: optional string` -- `ResponseDoneEvent = object { event_id, response, type }` + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. - Returned when a Response is done streaming. Always emitted, no matter the - final state. The Response object included in the `response.done` event will - include all output Items in the Response but will omit the raw audio data. + - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - Clients should check the `status` field of the Response to determine if it was successful - (`completed`) or if there was another outcome: `cancelled`, `failed`, or `incomplete`. + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](/docs/guides/tools-remote-mcp#connectors). - A response will contain all output items that were generated during the response, excluding - any audio content. + Currently supported `connector_id` values are: - - `event_id: string` + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` - The unique ID of the server event. + - `"connector_dropbox"` - - `response: RealtimeResponse` + - `"connector_gmail"` - The response resource. + - `"connector_googlecalendar"` - - `id: optional string` + - `"connector_googledrive"` - The unique ID of the response, will look like `resp_1234`. + - `"connector_microsoftteams"` - - `audio: optional object { output }` + - `"connector_outlookcalendar"` - Configuration for audio output. + - `"connector_outlookemail"` - - `output: optional object { format, voice }` + - `"connector_sharepoint"` - - `format: optional RealtimeAudioFormats` + - `defer_loading: optional boolean` - The format of the output audio. + Whether this MCP tool is deferred and discovered via tool search. - - `PCMAudioFormat = object { rate, type }` + - `headers: optional map[string]` - The PCM audio format. Only a 24kHz sample rate is supported. + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. - - `rate: optional 24000` + - `require_approval: optional object { always, never } or "always" or "never"` - The sample rate of the audio. Always `24000`. + Specify which of the MCP server's tools require approval. - - `24000` + - `McpToolApprovalFilter object { always, never }` - - `type: optional "audio/pcm"` + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. - The audio format. Always `audio/pcm`. + - `always: optional object { read_only, tool_names }` - - `"audio/pcm"` + A filter object to specify which tools are allowed. - - `PCMUAudioFormat = object { type }` + - `read_only: optional boolean` - The G.711 μ-law format. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `type: optional "audio/pcmu"` + - `tool_names: optional array of string` - The audio format. Always `audio/pcmu`. + List of allowed tool names. - - `"audio/pcmu"` + - `never: optional object { read_only, tool_names }` - - `PCMAAudioFormat = object { type }` + A filter object to specify which tools are allowed. - The G.711 A-law format. + - `read_only: optional boolean` - - `type: optional "audio/pcma"` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - The audio format. Always `audio/pcma`. + - `tool_names: optional array of string` - - `"audio/pcma"` + List of allowed tool names. - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more` + - `McpToolApprovalSetting = "always" or "never"` - The voice the model uses to respond. Voice cannot be changed during the - session once the model has responded with audio at least once. Current - voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, - `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for - best quality. + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. - - `string` + - `"always"` - - `"alloy" or "ash" or "ballad" or 7 more` + - `"never"` - The voice the model uses to respond. Voice cannot be changed during the - session once the model has responded with audio at least once. Current - voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, - `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for - best quality. + - `server_description: optional string` - - `"alloy"` + Optional description of the MCP server, used to provide more context. - - `"ash"` + - `server_url: optional string` - - `"ballad"` + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. - - `"coral"` + - `tracing: optional RealtimeTracingConfig` - - `"echo"` + Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once + tracing is enabled for a session, the configuration cannot be modified. - - `"sage"` + `auto` will create a trace for the session with default values for the + workflow name, group id, and metadata. - - `"shimmer"` + - `Auto = "auto"` - - `"verse"` + Enables tracing and sets default values for tracing configuration options. Always `auto`. - - `"marin"` + - `"auto"` - - `"cedar"` + - `TracingConfiguration object { group_id, metadata, workflow_name }` - - `conversation_id: optional string` + Granular configuration for tracing. - Which conversation the response is added to, determined by the `conversation` - field in the `response.create` event. If `auto`, the response will be added to - the default conversation and the value of `conversation_id` will be an id like - `conv_1234`. If `none`, the response will not be added to any conversation and - the value of `conversation_id` will be `null`. If responses are being triggered - automatically by VAD the response will be added to the default conversation + - `group_id: optional string` - - `max_output_tokens: optional number or "inf"` + The group id to attach to this trace to enable filtering and + grouping in the Traces Dashboard. - Maximum number of output tokens for a single assistant response, - inclusive of tool calls, that was used in this response. + - `metadata: optional unknown` - - `number` + The arbitrary metadata to attach to this trace to enable + filtering in the Traces Dashboard. - - `"inf"` + - `workflow_name: optional string` - - `"inf"` + The name of the workflow to attach to this trace. This is used to + name the trace in the Traces Dashboard. - - `metadata: optional Metadata` + - `truncation: optional RealtimeTruncation` - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. + When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. + Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - - `object: optional "realtime.response"` + Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - The object type, must be `realtime.response`. + Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - - `"realtime.response"` + - `"auto" or "disabled"` - - `output: optional array of ConversationItem` + The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - The list of output items generated by the response. + - `"auto"` - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + - `"disabled"` - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` - - `content: array of object { text, type }` + Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - The content of the message. + - `retention_ratio: number` - - `text: optional string` + Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - The text content. + - `type: "retention_ratio"` - - `type: optional "input_text"` + Use retention ratio truncation. - The content type. Always `input_text` for system messages. + - `"retention_ratio"` - - `"input_text"` + - `token_limits: optional object { post_instructions }` - - `role: "system"` + Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - The role of the message sender. Always `system`. + - `post_instructions: optional number` - - `"system"` + Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - - `type: "message"` + - `RealtimeTranscriptionSessionCreateRequest object { type, audio, include }` - The type of the item. Always `message`. + Realtime transcription session object configuration. - - `"message"` + - `type: "transcription"` - - `id: optional string` + The type of session to create. Always `transcription` for transcription sessions. - The unique ID of the item. This may be provided by the client or generated by the server. + - `"transcription"` - - `object: optional "realtime.item"` + - `audio: optional RealtimeTranscriptionSessionAudio` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Configuration for input and output audio. - - `"realtime.item"` + - `input: optional RealtimeTranscriptionSessionAudioInput` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `format: optional RealtimeAudioFormats` - The status of the item. Has no effect on the conversation. + The PCM audio format. Only a 24kHz sample rate is supported. - - `"completed"` + - `noise_reduction: optional object { type }` - - `"incomplete"` + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `"in_progress"` + - `type: optional NoiseReductionType` - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - A user message item in a Realtime conversation. + - `transcription: optional AudioTranscription` - - `content: array of object { audio, detail, image_url, 3 more }` + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - The content of the message. + - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` - - `audio: optional string` + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - `detail: optional "auto" or "low" or "high"` + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - The detail level of the image (for `input_image`). `auto` will default to `high`. + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - - `"auto"` + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - `"low"` + - `type: "server_vad"` - - `"high"` + Type of turn detection, `server_vad` to turn on simple Server VAD. - - `image_url: optional string` + - `"server_vad"` - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + - `create_response: optional boolean` - - `text: optional string` + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - The text content (for `input_text`). + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `transcript: optional string` + - `idle_timeout_ms: optional number` - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - - `type: optional "input_text" or "input_audio" or "input_image"` + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - The content type (`input_text`, `input_audio`, or `input_image`). + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - - `"input_text"` + - `interrupt_response: optional boolean` - - `"input_audio"` + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - `"input_image"` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `role: "user"` + - `prefix_padding_ms: optional number` - The role of the message sender. Always `user`. + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - - `"user"` + - `silence_duration_ms: optional number` - - `type: "message"` + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - The type of the item. Always `message`. + - `threshold: optional number` - - `"message"` + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - - `id: optional string` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - The unique ID of the item. This may be provided by the client or generated by the server. + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - `object: optional "realtime.item"` + - `type: "semantic_vad"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - `"realtime.item"` + - `"semantic_vad"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `create_response: optional boolean` - The status of the item. Has no effect on the conversation. - - - `"completed"` + Whether or not to automatically generate a response when a VAD stop event occurs. - - `"incomplete"` + - `eagerness: optional "low" or "medium" or "high" or "auto"` - - `"in_progress"` + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` + - `"low"` - An assistant message item in a Realtime conversation. + - `"medium"` - - `content: array of object { audio, text, transcript, type }` + - `"high"` - The content of the message. + - `"auto"` - - `audio: optional string` + - `interrupt_response: optional boolean` - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - `text: optional string` + - `include: optional array of "item.input_audio_transcription.logprobs"` - The text content. + Additional fields to include in server outputs. - - `transcript: optional string` + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - The transcript of the audio content, this will always be present if the output type is `audio`. + - `"item.input_audio_transcription.logprobs"` - - `type: optional "output_text" or "output_audio"` +### Returns - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. +- `expires_at: number` - - `"output_text"` + Expiration timestamp for the client secret, in seconds since epoch. - - `"output_audio"` +- `session: RealtimeSessionCreateResponse or RealtimeTranscriptionSessionCreateResponse` - - `role: "assistant"` + The session configuration for either a realtime or transcription session. - The role of the message sender. Always `assistant`. + - `RealtimeSessionCreateResponse object { client_secret, type, audio, 10 more }` - - `"assistant"` + A new Realtime session configuration, with an ephemeral key. Default TTL + for keys is one minute. - - `type: "message"` + - `client_secret: RealtimeSessionClientSecret` - The type of the item. Always `message`. + Ephemeral key returned by the API. - - `"message"` + - `expires_at: number` - - `id: optional string` + Timestamp for when the token expires. Currently, all tokens expire + after one minute. - The unique ID of the item. This may be provided by the client or generated by the server. + - `value: string` - - `object: optional "realtime.item"` + Ephemeral key usable in client environments to authenticate connections to the Realtime API. Use this in client-side environments rather than a standard API token, which should only be used server-side. - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `type: "realtime"` - - `"realtime.item"` + The type of session to create. Always `realtime` for the Realtime API. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"realtime"` - The status of the item. Has no effect on the conversation. + - `audio: optional object { input, output }` - - `"completed"` + Configuration for input and output audio. - - `"incomplete"` + - `input: optional object { format, noise_reduction, transcription, turn_detection }` - - `"in_progress"` + - `format: optional RealtimeAudioFormats` - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` + The format of the input audio. - A function call item in a Realtime conversation. + - `PCMAudioFormat object { rate, type }` - - `arguments: string` + The PCM audio format. Only a 24kHz sample rate is supported. - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + - `rate: optional 24000` - - `name: string` + The sample rate of the audio. Always `24000`. - The name of the function being called. + - `24000` - - `type: "function_call"` + - `type: optional "audio/pcm"` - The type of the item. Always `function_call`. + The audio format. Always `audio/pcm`. - - `"function_call"` + - `"audio/pcm"` - - `id: optional string` + - `PCMUAudioFormat object { type }` - The unique ID of the item. This may be provided by the client or generated by the server. + The G.711 μ-law format. - - `call_id: optional string` + - `type: optional "audio/pcmu"` - The ID of the function call. + The audio format. Always `audio/pcmu`. - - `object: optional "realtime.item"` + - `"audio/pcmu"` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + - `PCMAAudioFormat object { type }` - - `"realtime.item"` + The G.711 A-law format. - - `status: optional "completed" or "incomplete" or "in_progress"` + - `type: optional "audio/pcma"` - The status of the item. Has no effect on the conversation. + The audio format. Always `audio/pcma`. - - `"completed"` + - `"audio/pcma"` - - `"incomplete"` + - `noise_reduction: optional object { type }` - - `"in_progress"` + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` + - `type: optional NoiseReductionType` - A function call output item in a Realtime conversation. + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - `call_id: string` + - `"near_field"` - The ID of the function call this output is for. + - `"far_field"` - - `output: string` + - `transcription: optional AudioTranscription` - The output of the function call, this is free text and can contain any information or simply be empty. + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - `type: "function_call_output"` + - `language: optional string` - The type of the item. Always `function_call_output`. + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. - - `"function_call_output"` + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - `id: optional string` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - The unique ID of the item. This may be provided by the client or generated by the server. + - `string` - - `object: optional "realtime.item"` + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `"realtime.item"` + - `"whisper-1"` - - `status: optional "completed" or "incomplete" or "in_progress"` + - `"gpt-4o-mini-transcribe"` - The status of the item. Has no effect on the conversation. + - `"gpt-4o-mini-transcribe-2025-12-15"` - - `"completed"` + - `"gpt-4o-transcribe"` - - `"incomplete"` + - `"gpt-4o-transcribe-diarize"` - - `"in_progress"` + - `prompt: optional string` - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - A Realtime item responding to an MCP approval request. + - `turn_detection: optional object { type, create_response, idle_timeout_ms, 4 more } or object { type, create_response, eagerness, interrupt_response }` - - `id: string` + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - The unique ID of the approval response. + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - `approval_request_id: string` + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - The ID of the approval request being answered. + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` - - `approve: boolean` + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - Whether the request was approved. + - `type: "server_vad"` - - `type: "mcp_approval_response"` + Type of turn detection, `server_vad` to turn on simple Server VAD. - The type of the item. Always `mcp_approval_response`. + - `"server_vad"` - - `"mcp_approval_response"` + - `create_response: optional boolean` - - `reason: optional string` + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - Optional reason for the decision. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `RealtimeMcpListTools = object { server_label, tools, type, id }` + - `idle_timeout_ms: optional number` - A Realtime item listing tools available on an MCP server. + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. - - `server_label: string` + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. - The label of the MCP server. + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. - - `tools: array of object { input_schema, name, annotations, description }` + - `interrupt_response: optional boolean` - The tools available on the server. + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - `input_schema: unknown` + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - The JSON schema describing the tool's input. + - `prefix_padding_ms: optional number` - - `name: string` + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - The name of the tool. + - `silence_duration_ms: optional number` - - `annotations: optional unknown` + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - Additional annotations about the tool. + - `threshold: optional number` - - `description: optional string` + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - The description of the tool. + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - - `type: "mcp_list_tools"` + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - The type of the item. Always `mcp_list_tools`. + - `type: "semantic_vad"` - - `"mcp_list_tools"` + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - `id: optional string` + - `"semantic_vad"` - The unique ID of the list. + - `create_response: optional boolean` - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` + Whether or not to automatically generate a response when a VAD stop event occurs. - A Realtime item representing an invocation of a tool on an MCP server. + - `eagerness: optional "low" or "medium" or "high" or "auto"` - - `id: string` + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - The unique ID of the tool call. + - `"low"` - - `arguments: string` + - `"medium"` - A JSON string of the arguments passed to the tool. + - `"high"` - - `name: string` + - `"auto"` - The name of the tool that was run. + - `interrupt_response: optional boolean` - - `server_label: string` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - The label of the MCP server running the tool. + - `output: optional object { format, speed, voice }` - - `type: "mcp_call"` + - `format: optional RealtimeAudioFormats` - The type of the item. Always `mcp_call`. + The format of the output audio. - - `"mcp_call"` + - `speed: optional number` - - `approval_request_id: optional string` + The speed of the model's spoken response as a multiple of the original speed. + 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - The ID of an associated approval request, if any. + This parameter is a post-processing adjustment to the audio after it is generated, it's + also possible to prompt the model to speak faster or slower. - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more` - The error from the tool call, if any. + The voice the model uses to respond. Voice cannot be changed during the + session once the model has responded with audio at least once. Current + voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, + `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for + best quality. - - `RealtimeMcpProtocolError = object { code, message, type }` + - `string` - - `code: number` + - `"alloy" or "ash" or "ballad" or 7 more` - - `message: string` + The voice the model uses to respond. Voice cannot be changed during the + session once the model has responded with audio at least once. Current + voice options are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, + `shimmer`, `verse`, `marin`, and `cedar`. We recommend `marin` and `cedar` for + best quality. - - `type: "protocol_error"` + - `"alloy"` - - `"protocol_error"` + - `"ash"` - - `RealtimeMcpToolExecutionError = object { message, type }` + - `"ballad"` - - `message: string` + - `"coral"` - - `type: "tool_execution_error"` + - `"echo"` - - `"tool_execution_error"` + - `"sage"` - - `RealtimeMcphttpError = object { code, message, type }` + - `"shimmer"` - - `code: number` + - `"verse"` - - `message: string` + - `"marin"` - - `type: "http_error"` + - `"cedar"` - - `"http_error"` + - `include: optional array of "item.input_audio_transcription.logprobs"` - - `output: optional string` + Additional fields to include in server outputs. - The output from the tool call. + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` + - `"item.input_audio_transcription.logprobs"` - A Realtime item requesting human approval of a tool invocation. + - `instructions: optional string` - - `id: string` + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - The unique ID of the approval request. + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - `arguments: string` + - `max_output_tokens: optional number or "inf"` - A JSON string of arguments for the tool. + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - - `name: string` + - `number` - The name of the tool to run. + - `"inf"` - - `server_label: string` + - `"inf"` - The label of the MCP server making the request. + - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - `type: "mcp_approval_request"` + The Realtime model used for this session. - The type of the item. Always `mcp_approval_request`. + - `string` - - `"mcp_approval_request"` + - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - `output_modalities: optional array of "text" or "audio"` + The Realtime model used for this session. - The set of modalities the model used to respond, currently the only possible values are - `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the - output to mode `text` will disable audio output from the model. + - `"gpt-realtime"` - - `"text"` + - `"gpt-realtime-1.5"` - - `"audio"` + - `"gpt-realtime-2025-08-28"` - - `status: optional "completed" or "cancelled" or "failed" or 2 more` + - `"gpt-4o-realtime-preview"` - The final status of the response (`completed`, `cancelled`, `failed`, or - `incomplete`, `in_progress`). + - `"gpt-4o-realtime-preview-2024-10-01"` - - `"completed"` + - `"gpt-4o-realtime-preview-2024-12-17"` - - `"cancelled"` + - `"gpt-4o-realtime-preview-2025-06-03"` - - `"failed"` + - `"gpt-4o-mini-realtime-preview"` - - `"incomplete"` + - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - `"in_progress"` + - `"gpt-realtime-mini"` - - `status_details: optional RealtimeResponseStatus` + - `"gpt-realtime-mini-2025-10-06"` - Additional details about the status. + - `"gpt-realtime-mini-2025-12-15"` - - `error: optional object { code, type }` + - `"gpt-audio-1.5"` - A description of the error that caused the response to fail, - populated when the `status` is `failed`. + - `"gpt-audio-mini"` - - `code: optional string` + - `"gpt-audio-mini-2025-10-06"` - Error code, if any. + - `"gpt-audio-mini-2025-12-15"` - - `type: optional string` + - `output_modalities: optional array of "text" or "audio"` - The type of error. + The set of modalities the model can respond with. It defaults to `["audio"]`, indicating + that the model will respond with audio plus a transcript. `["text"]` can be used to make + the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - - `reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"` + - `"text"` - The reason the Response did not complete. For a `cancelled` Response, one of `turn_detected` (the server VAD detected a new start of speech) or `client_cancelled` (the client sent a cancel event). For an `incomplete` Response, one of `max_output_tokens` or `content_filter` (the server-side safety filter activated and cut off the response). + - `"audio"` - - `"turn_detected"` + - `prompt: optional ResponsePrompt` - - `"client_cancelled"` + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - - `"max_output_tokens"` + - `id: string` - - `"content_filter"` + The unique identifier of the prompt template to use. - - `type: optional "completed" or "cancelled" or "failed" or "incomplete"` + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - The type of error that caused the response to fail, corresponding - with the `status` field (`completed`, `cancelled`, `incomplete`, - `failed`). + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - - `"completed"` + - `string` - - `"cancelled"` + - `ResponseInputText object { text, type }` - - `"failed"` + A text input to the model. - - `"incomplete"` + - `text: string` - - `usage: optional RealtimeResponseUsage` + The text input to the model. - Usage statistics for the Response, this will correspond to billing. A - Realtime API session will maintain a conversation context and append new - Items to the Conversation, thus output from previous turns (text and - audio tokens) will become the input for later turns. + - `type: "input_text"` - - `input_token_details: optional RealtimeResponseUsageInputTokenDetails` + The type of the input item. Always `input_text`. - Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens. + - `"input_text"` - - `audio_tokens: optional number` + - `ResponseInputImage object { detail, type, file_id, image_url }` - The number of audio tokens used as input for the Response. + An image input to the model. Learn about [image inputs](/docs/guides/vision). - - `cached_tokens: optional number` + - `detail: "low" or "high" or "auto" or "original"` - The number of cached tokens used as input for the Response. + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - `cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }` + - `"low"` - Details about the cached tokens used as input for the Response. + - `"high"` - - `audio_tokens: optional number` + - `"auto"` - The number of cached audio tokens used as input for the Response. + - `"original"` - - `image_tokens: optional number` + - `type: "input_image"` - The number of cached image tokens used as input for the Response. + The type of the input item. Always `input_image`. - - `text_tokens: optional number` + - `"input_image"` - The number of cached text tokens used as input for the Response. + - `file_id: optional string` - - `image_tokens: optional number` + The ID of the file to be sent to the model. - The number of image tokens used as input for the Response. + - `image_url: optional string` - - `text_tokens: optional number` + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - The number of text tokens used as input for the Response. + - `ResponseInputFile object { type, detail, file_data, 3 more }` - - `input_tokens: optional number` + A file input to the model. - The number of input tokens used in the Response, including text and - audio tokens. + - `type: "input_file"` - - `output_token_details: optional RealtimeResponseUsageOutputTokenDetails` + The type of the input item. Always `input_file`. - Details about the output tokens used in the Response. + - `"input_file"` - - `audio_tokens: optional number` + - `detail: optional "low" or "high"` - The number of audio tokens used in the Response. + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - `text_tokens: optional number` + - `"low"` - The number of text tokens used in the Response. + - `"high"` - - `output_tokens: optional number` + - `file_data: optional string` - The number of output tokens sent in the Response, including text and - audio tokens. + The content of the file to be sent to the model. - - `total_tokens: optional number` + - `file_id: optional string` - The total number of tokens in the Response including input and output - text and audio tokens. + The ID of the file to be sent to the model. - - `type: "response.done"` + - `file_url: optional string` - The event type, must be `response.done`. + The URL of the file to be sent to the model. - - `"response.done"` + - `filename: optional string` -### Response Function Call Arguments Delta Event + The name of the file to be sent to the model. -- `ResponseFunctionCallArgumentsDeltaEvent = object { call_id, delta, event_id, 4 more }` + - `version: optional string` - Returned when the model-generated function call arguments are updated. + Optional version of the prompt template. - - `call_id: string` + - `tool_choice: optional ToolChoiceOptions or ToolChoiceFunction or ToolChoiceMcp` - The ID of the function call. + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. - - `delta: string` + - `ToolChoiceOptions = "none" or "auto" or "required"` - The arguments delta as a JSON string. + Controls which (if any) tool is called by the model. - - `event_id: string` + `none` means the model will not call any tool and instead generates a message. - The unique ID of the server event. + `auto` means the model can pick between generating a message or calling one or + more tools. - - `item_id: string` + `required` means the model must call one or more tools. - The ID of the function call item. + - `"none"` - - `output_index: number` + - `"auto"` - The index of the output item in the response. + - `"required"` - - `response_id: string` + - `ToolChoiceFunction object { name, type }` - The ID of the response. + Use this option to force the model to call a specific function. - - `type: "response.function_call_arguments.delta"` + - `name: string` - The event type, must be `response.function_call_arguments.delta`. + The name of the function to call. - - `"response.function_call_arguments.delta"` + - `type: "function"` -### Response Function Call Arguments Done Event + For function calling, the type is always `function`. -- `ResponseFunctionCallArgumentsDoneEvent = object { arguments, call_id, event_id, 5 more }` + - `"function"` - Returned when the model-generated function call arguments are done streaming. - Also emitted when a Response is interrupted, incomplete, or cancelled. + - `ToolChoiceMcp object { server_label, type, name }` - - `arguments: string` + Use this option to force the model to call a specific tool on a remote MCP server. - The final arguments as a JSON string. + - `server_label: string` - - `call_id: string` + The label of the MCP server to use. - The ID of the function call. + - `type: "mcp"` - - `event_id: string` + For MCP tools, the type is always `mcp`. - The unique ID of the server event. + - `"mcp"` - - `item_id: string` + - `name: optional string` - The ID of the function call item. + The name of the tool to call on the server. - - `name: string` + - `tools: optional array of RealtimeFunctionTool or object { server_label, type, allowed_tools, 7 more }` - The name of the function that was called. + Tools available to the model. - - `output_index: number` + - `RealtimeFunctionTool object { description, name, parameters, type }` - The index of the output item in the response. + - `description: optional string` - - `response_id: string` + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - The ID of the response. + - `name: optional string` - - `type: "response.function_call_arguments.done"` + The name of the function. - The event type, must be `response.function_call_arguments.done`. + - `parameters: optional unknown` - - `"response.function_call_arguments.done"` + Parameters of the function in JSON Schema. -### Response Mcp Call Arguments Delta + - `type: optional "function"` -- `ResponseMcpCallArgumentsDelta = object { delta, event_id, item_id, 4 more }` + The type of the tool, i.e. `function`. - Returned when MCP tool call arguments are updated during response generation. + - `"function"` - - `delta: string` + - `McpTool object { server_label, type, allowed_tools, 7 more }` - The JSON-encoded arguments delta. + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - - `event_id: string` + - `server_label: string` - The unique ID of the server event. + A label for this MCP server, used to identify it in tool calls. - - `item_id: string` + - `type: "mcp"` - The ID of the MCP tool call item. + The type of the MCP tool. Always `mcp`. - - `output_index: number` + - `"mcp"` - The index of the output item in the response. + - `allowed_tools: optional array of string or object { read_only, tool_names }` - - `response_id: string` + List of allowed tool names or a filter object. - The ID of the response. + - `McpAllowedTools = array of string` - - `type: "response.mcp_call_arguments.delta"` + A string array of allowed tool names - The event type, must be `response.mcp_call_arguments.delta`. + - `McpToolFilter object { read_only, tool_names }` - - `"response.mcp_call_arguments.delta"` + A filter object to specify which tools are allowed. - - `obfuscation: optional string` + - `read_only: optional boolean` - If present, indicates the delta text was obfuscated. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. -### Response Mcp Call Arguments Done + - `tool_names: optional array of string` -- `ResponseMcpCallArgumentsDone = object { arguments, event_id, item_id, 3 more }` + List of allowed tool names. - Returned when MCP tool call arguments are finalized during response generation. + - `authorization: optional string` - - `arguments: string` + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. - The final JSON-encoded arguments string. + - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - - `event_id: string` + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](/docs/guides/tools-remote-mcp#connectors). - The unique ID of the server event. + Currently supported `connector_id` values are: - - `item_id: string` + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` - The ID of the MCP tool call item. + - `"connector_dropbox"` - - `output_index: number` + - `"connector_gmail"` - The index of the output item in the response. + - `"connector_googlecalendar"` - - `response_id: string` + - `"connector_googledrive"` - The ID of the response. + - `"connector_microsoftteams"` - - `type: "response.mcp_call_arguments.done"` + - `"connector_outlookcalendar"` - The event type, must be `response.mcp_call_arguments.done`. + - `"connector_outlookemail"` - - `"response.mcp_call_arguments.done"` + - `"connector_sharepoint"` -### Response Mcp Call Completed + - `defer_loading: optional boolean` -- `ResponseMcpCallCompleted = object { event_id, item_id, output_index, type }` + Whether this MCP tool is deferred and discovered via tool search. - Returned when an MCP tool call has completed successfully. + - `headers: optional map[string]` - - `event_id: string` + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. - The unique ID of the server event. + - `require_approval: optional object { always, never } or "always" or "never"` - - `item_id: string` + Specify which of the MCP server's tools require approval. - The ID of the MCP tool call item. + - `McpToolApprovalFilter object { always, never }` - - `output_index: number` + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. - The index of the output item in the response. + - `always: optional object { read_only, tool_names }` - - `type: "response.mcp_call.completed"` + A filter object to specify which tools are allowed. - The event type, must be `response.mcp_call.completed`. + - `read_only: optional boolean` - - `"response.mcp_call.completed"` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. -### Response Mcp Call Failed + - `tool_names: optional array of string` -- `ResponseMcpCallFailed = object { event_id, item_id, output_index, type }` + List of allowed tool names. - Returned when an MCP tool call has failed. + - `never: optional object { read_only, tool_names }` - - `event_id: string` + A filter object to specify which tools are allowed. - The unique ID of the server event. + - `read_only: optional boolean` - - `item_id: string` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - The ID of the MCP tool call item. + - `tool_names: optional array of string` - - `output_index: number` + List of allowed tool names. - The index of the output item in the response. + - `McpToolApprovalSetting = "always" or "never"` - - `type: "response.mcp_call.failed"` + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. - The event type, must be `response.mcp_call.failed`. + - `"always"` - - `"response.mcp_call.failed"` + - `"never"` -### Response Mcp Call In Progress + - `server_description: optional string` -- `ResponseMcpCallInProgress = object { event_id, item_id, output_index, type }` + Optional description of the MCP server, used to provide more context. - Returned when an MCP tool call has started and is in progress. + - `server_url: optional string` - - `event_id: string` + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. - The unique ID of the server event. + - `tracing: optional "auto" or object { group_id, metadata, workflow_name }` - - `item_id: string` + Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once + tracing is enabled for a session, the configuration cannot be modified. - The ID of the MCP tool call item. + `auto` will create a trace for the session with default values for the + workflow name, group id, and metadata. - - `output_index: number` + - `Auto = "auto"` - The index of the output item in the response. + Enables tracing and sets default values for tracing configuration options. Always `auto`. - - `type: "response.mcp_call.in_progress"` + - `"auto"` - The event type, must be `response.mcp_call.in_progress`. + - `TracingConfiguration object { group_id, metadata, workflow_name }` - - `"response.mcp_call.in_progress"` + Granular configuration for tracing. -### Response Output Item Added Event + - `group_id: optional string` -- `ResponseOutputItemAddedEvent = object { event_id, item, output_index, 2 more }` + The group id to attach to this trace to enable filtering and + grouping in the Traces Dashboard. - Returned when a new Item is created during Response generation. + - `metadata: optional unknown` - - `event_id: string` + The arbitrary metadata to attach to this trace to enable + filtering in the Traces Dashboard. - The unique ID of the server event. + - `workflow_name: optional string` - - `item: ConversationItem` + The name of the workflow to attach to this trace. This is used to + name the trace in the Traces Dashboard. - A single item within a Realtime conversation. + - `truncation: optional RealtimeTruncation` - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` + When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - - `content: array of object { text, type }` + Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - The content of the message. + Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - - `text: optional string` + - `"auto" or "disabled"` - The text content. + The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - - `type: optional "input_text"` + - `"auto"` - The content type. Always `input_text` for system messages. + - `"disabled"` - - `"input_text"` + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` - - `role: "system"` + Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - The role of the message sender. Always `system`. + - `retention_ratio: number` - - `"system"` + Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - `type: "message"` + - `type: "retention_ratio"` - The type of the item. Always `message`. + Use retention ratio truncation. - - `"message"` + - `"retention_ratio"` - - `id: optional string` + - `token_limits: optional object { post_instructions }` - The unique ID of the item. This may be provided by the client or generated by the server. + Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - - `object: optional "realtime.item"` + - `post_instructions: optional number` - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - - `"realtime.item"` + - `RealtimeTranscriptionSessionCreateResponse object { id, object, type, 3 more }` - - `status: optional "completed" or "incomplete" or "in_progress"` + A Realtime transcription session configuration object. - The status of the item. Has no effect on the conversation. + - `id: string` - - `"completed"` + Unique identifier for the session that looks like `sess_1234567890abcdef`. - - `"incomplete"` + - `object: string` - - `"in_progress"` + The object type. Always `realtime.transcription_session`. - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` + - `type: "transcription"` - A user message item in a Realtime conversation. + The type of session. Always `transcription` for transcription sessions. - - `content: array of object { audio, detail, image_url, 3 more }` - - The content of the message. - - - `audio: optional string` - - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. - - - `detail: optional "auto" or "low" or "high"` - - The detail level of the image (for `input_image`). `auto` will default to `high`. - - - `"auto"` - - - `"low"` - - - `"high"` - - - `image_url: optional string` - - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. - - - `text: optional string` - - The text content (for `input_text`). - - - `transcript: optional string` - - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. - - - `type: optional "input_text" or "input_audio" or "input_image"` - - The content type (`input_text`, `input_audio`, or `input_image`). - - - `"input_text"` - - - `"input_audio"` - - - `"input_image"` - - - `role: "user"` - - The role of the message sender. Always `user`. - - - `"user"` - - - `type: "message"` - - The type of the item. Always `message`. - - - `"message"` - - - `id: optional string` - - The unique ID of the item. This may be provided by the client or generated by the server. - - - `object: optional "realtime.item"` - - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - - `"realtime.item"` - - - `status: optional "completed" or "incomplete" or "in_progress"` - - The status of the item. Has no effect on the conversation. - - - `"completed"` - - - `"incomplete"` - - - `"in_progress"` - - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` - - An assistant message item in a Realtime conversation. - - - `content: array of object { audio, text, transcript, type }` - - The content of the message. - - - `audio: optional string` - - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. - - - `text: optional string` - - The text content. - - - `transcript: optional string` - - The transcript of the audio content, this will always be present if the output type is `audio`. - - - `type: optional "output_text" or "output_audio"` - - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. - - - `"output_text"` - - - `"output_audio"` - - - `role: "assistant"` - - The role of the message sender. Always `assistant`. - - - `"assistant"` - - - `type: "message"` - - The type of the item. Always `message`. - - - `"message"` - - - `id: optional string` - - The unique ID of the item. This may be provided by the client or generated by the server. - - - `object: optional "realtime.item"` - - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - - `"realtime.item"` - - - `status: optional "completed" or "incomplete" or "in_progress"` - - The status of the item. Has no effect on the conversation. - - - `"completed"` - - - `"incomplete"` - - - `"in_progress"` - - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` - - A function call item in a Realtime conversation. - - - `arguments: string` - - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. - - - `name: string` - - The name of the function being called. - - - `type: "function_call"` - - The type of the item. Always `function_call`. - - - `"function_call"` - - - `id: optional string` - - The unique ID of the item. This may be provided by the client or generated by the server. - - - `call_id: optional string` - - The ID of the function call. - - - `object: optional "realtime.item"` - - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - - `"realtime.item"` - - - `status: optional "completed" or "incomplete" or "in_progress"` - - The status of the item. Has no effect on the conversation. - - - `"completed"` - - - `"incomplete"` - - - `"in_progress"` - - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` - - A function call output item in a Realtime conversation. - - - `call_id: string` - - The ID of the function call this output is for. - - - `output: string` - - The output of the function call, this is free text and can contain any information or simply be empty. - - - `type: "function_call_output"` - - The type of the item. Always `function_call_output`. - - - `"function_call_output"` - - - `id: optional string` - - The unique ID of the item. This may be provided by the client or generated by the server. - - - `object: optional "realtime.item"` - - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - - `"realtime.item"` - - - `status: optional "completed" or "incomplete" or "in_progress"` - - The status of the item. Has no effect on the conversation. - - - `"completed"` - - - `"incomplete"` - - - `"in_progress"` - - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` - - A Realtime item responding to an MCP approval request. - - - `id: string` - - The unique ID of the approval response. - - - `approval_request_id: string` - - The ID of the approval request being answered. - - - `approve: boolean` - - Whether the request was approved. - - - `type: "mcp_approval_response"` - - The type of the item. Always `mcp_approval_response`. - - - `"mcp_approval_response"` - - - `reason: optional string` - - Optional reason for the decision. - - - `RealtimeMcpListTools = object { server_label, tools, type, id }` - - A Realtime item listing tools available on an MCP server. - - - `server_label: string` - - The label of the MCP server. - - - `tools: array of object { input_schema, name, annotations, description }` - - The tools available on the server. - - - `input_schema: unknown` - - The JSON schema describing the tool's input. - - - `name: string` - - The name of the tool. - - - `annotations: optional unknown` - - Additional annotations about the tool. - - - `description: optional string` - - The description of the tool. - - - `type: "mcp_list_tools"` - - The type of the item. Always `mcp_list_tools`. - - - `"mcp_list_tools"` - - - `id: optional string` - - The unique ID of the list. - - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` - - A Realtime item representing an invocation of a tool on an MCP server. - - - `id: string` - - The unique ID of the tool call. - - - `arguments: string` - - A JSON string of the arguments passed to the tool. - - - `name: string` - - The name of the tool that was run. - - - `server_label: string` - - The label of the MCP server running the tool. - - - `type: "mcp_call"` - - The type of the item. Always `mcp_call`. - - - `"mcp_call"` - - - `approval_request_id: optional string` - - The ID of an associated approval request, if any. - - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` - - The error from the tool call, if any. - - - `RealtimeMcpProtocolError = object { code, message, type }` - - - `code: number` - - - `message: string` - - - `type: "protocol_error"` - - - `"protocol_error"` - - - `RealtimeMcpToolExecutionError = object { message, type }` - - - `message: string` - - - `type: "tool_execution_error"` - - - `"tool_execution_error"` - - - `RealtimeMcphttpError = object { code, message, type }` - - - `code: number` - - - `message: string` - - - `type: "http_error"` - - - `"http_error"` - - - `output: optional string` - - The output from the tool call. - - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` - - A Realtime item requesting human approval of a tool invocation. - - - `id: string` - - The unique ID of the approval request. - - - `arguments: string` - - A JSON string of arguments for the tool. - - - `name: string` - - The name of the tool to run. - - - `server_label: string` - - The label of the MCP server making the request. - - - `type: "mcp_approval_request"` - - The type of the item. Always `mcp_approval_request`. - - - `"mcp_approval_request"` - - - `output_index: number` - - The index of the output item in the Response. - - - `response_id: string` - - The ID of the Response to which the item belongs. - - - `type: "response.output_item.added"` - - The event type, must be `response.output_item.added`. - - - `"response.output_item.added"` - -### Response Output Item Done Event - -- `ResponseOutputItemDoneEvent = object { event_id, item, output_index, 2 more }` - - Returned when an Item is done streaming. Also emitted when a Response is - interrupted, incomplete, or cancelled. - - - `event_id: string` - - The unique ID of the server event. - - - `item: ConversationItem` - - A single item within a Realtime conversation. - - - `RealtimeConversationItemSystemMessage = object { content, role, type, 3 more }` - - A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. - - - `content: array of object { text, type }` - - The content of the message. - - - `text: optional string` - - The text content. - - - `type: optional "input_text"` - - The content type. Always `input_text` for system messages. - - - `"input_text"` - - - `role: "system"` - - The role of the message sender. Always `system`. - - - `"system"` - - - `type: "message"` - - The type of the item. Always `message`. - - - `"message"` - - - `id: optional string` - - The unique ID of the item. This may be provided by the client or generated by the server. - - - `object: optional "realtime.item"` - - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - - `"realtime.item"` - - - `status: optional "completed" or "incomplete" or "in_progress"` - - The status of the item. Has no effect on the conversation. - - - `"completed"` - - - `"incomplete"` - - - `"in_progress"` - - - `RealtimeConversationItemUserMessage = object { content, role, type, 3 more }` - - A user message item in a Realtime conversation. - - - `content: array of object { audio, detail, image_url, 3 more }` - - The content of the message. - - - `audio: optional string` - - Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. - - - `detail: optional "auto" or "low" or "high"` - - The detail level of the image (for `input_image`). `auto` will default to `high`. - - - `"auto"` - - - `"low"` - - - `"high"` - - - `image_url: optional string` - - Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. - - - `text: optional string` - - The text content (for `input_text`). - - - `transcript: optional string` - - Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. - - - `type: optional "input_text" or "input_audio" or "input_image"` - - The content type (`input_text`, `input_audio`, or `input_image`). - - - `"input_text"` - - - `"input_audio"` - - - `"input_image"` - - - `role: "user"` - - The role of the message sender. Always `user`. - - - `"user"` - - - `type: "message"` - - The type of the item. Always `message`. - - - `"message"` - - - `id: optional string` - - The unique ID of the item. This may be provided by the client or generated by the server. - - - `object: optional "realtime.item"` - - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - - `"realtime.item"` - - - `status: optional "completed" or "incomplete" or "in_progress"` - - The status of the item. Has no effect on the conversation. - - - `"completed"` - - - `"incomplete"` - - - `"in_progress"` - - - `RealtimeConversationItemAssistantMessage = object { content, role, type, 3 more }` - - An assistant message item in a Realtime conversation. - - - `content: array of object { audio, text, transcript, type }` - - The content of the message. - - - `audio: optional string` - - Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. - - - `text: optional string` - - The text content. - - - `transcript: optional string` - - The transcript of the audio content, this will always be present if the output type is `audio`. - - - `type: optional "output_text" or "output_audio"` - - The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. - - - `"output_text"` - - - `"output_audio"` - - - `role: "assistant"` - - The role of the message sender. Always `assistant`. - - - `"assistant"` - - - `type: "message"` - - The type of the item. Always `message`. - - - `"message"` - - - `id: optional string` - - The unique ID of the item. This may be provided by the client or generated by the server. - - - `object: optional "realtime.item"` - - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - - `"realtime.item"` - - - `status: optional "completed" or "incomplete" or "in_progress"` - - The status of the item. Has no effect on the conversation. - - - `"completed"` - - - `"incomplete"` - - - `"in_progress"` - - - `RealtimeConversationItemFunctionCall = object { arguments, name, type, 4 more }` - - A function call item in a Realtime conversation. - - - `arguments: string` - - The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. - - - `name: string` - - The name of the function being called. - - - `type: "function_call"` - - The type of the item. Always `function_call`. - - - `"function_call"` - - - `id: optional string` - - The unique ID of the item. This may be provided by the client or generated by the server. - - - `call_id: optional string` - - The ID of the function call. - - - `object: optional "realtime.item"` - - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - - `"realtime.item"` - - - `status: optional "completed" or "incomplete" or "in_progress"` - - The status of the item. Has no effect on the conversation. - - - `"completed"` - - - `"incomplete"` - - - `"in_progress"` - - - `RealtimeConversationItemFunctionCallOutput = object { call_id, output, type, 3 more }` - - A function call output item in a Realtime conversation. - - - `call_id: string` - - The ID of the function call this output is for. - - - `output: string` - - The output of the function call, this is free text and can contain any information or simply be empty. - - - `type: "function_call_output"` - - The type of the item. Always `function_call_output`. - - - `"function_call_output"` - - - `id: optional string` - - The unique ID of the item. This may be provided by the client or generated by the server. - - - `object: optional "realtime.item"` - - Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. - - - `"realtime.item"` - - - `status: optional "completed" or "incomplete" or "in_progress"` - - The status of the item. Has no effect on the conversation. - - - `"completed"` - - - `"incomplete"` - - - `"in_progress"` - - - `RealtimeMcpApprovalResponse = object { id, approval_request_id, approve, 2 more }` - - A Realtime item responding to an MCP approval request. - - - `id: string` - - The unique ID of the approval response. - - - `approval_request_id: string` - - The ID of the approval request being answered. - - - `approve: boolean` - - Whether the request was approved. - - - `type: "mcp_approval_response"` - - The type of the item. Always `mcp_approval_response`. - - - `"mcp_approval_response"` - - - `reason: optional string` - - Optional reason for the decision. - - - `RealtimeMcpListTools = object { server_label, tools, type, id }` - - A Realtime item listing tools available on an MCP server. - - - `server_label: string` - - The label of the MCP server. - - - `tools: array of object { input_schema, name, annotations, description }` - - The tools available on the server. - - - `input_schema: unknown` - - The JSON schema describing the tool's input. - - - `name: string` - - The name of the tool. - - - `annotations: optional unknown` - - Additional annotations about the tool. - - - `description: optional string` - - The description of the tool. - - - `type: "mcp_list_tools"` - - The type of the item. Always `mcp_list_tools`. - - - `"mcp_list_tools"` - - - `id: optional string` - - The unique ID of the list. - - - `RealtimeMcpToolCall = object { id, arguments, name, 5 more }` - - A Realtime item representing an invocation of a tool on an MCP server. - - - `id: string` - - The unique ID of the tool call. - - - `arguments: string` - - A JSON string of the arguments passed to the tool. - - - `name: string` - - The name of the tool that was run. - - - `server_label: string` - - The label of the MCP server running the tool. - - - `type: "mcp_call"` - - The type of the item. Always `mcp_call`. - - - `"mcp_call"` - - - `approval_request_id: optional string` - - The ID of an associated approval request, if any. - - - `error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError` - - The error from the tool call, if any. - - - `RealtimeMcpProtocolError = object { code, message, type }` - - - `code: number` - - - `message: string` - - - `type: "protocol_error"` - - - `"protocol_error"` - - - `RealtimeMcpToolExecutionError = object { message, type }` - - - `message: string` - - - `type: "tool_execution_error"` - - - `"tool_execution_error"` - - - `RealtimeMcphttpError = object { code, message, type }` - - - `code: number` - - - `message: string` - - - `type: "http_error"` - - - `"http_error"` - - - `output: optional string` - - The output from the tool call. - - - `RealtimeMcpApprovalRequest = object { id, arguments, name, 2 more }` - - A Realtime item requesting human approval of a tool invocation. - - - `id: string` - - The unique ID of the approval request. - - - `arguments: string` - - A JSON string of arguments for the tool. - - - `name: string` - - The name of the tool to run. - - - `server_label: string` - - The label of the MCP server making the request. - - - `type: "mcp_approval_request"` - - The type of the item. Always `mcp_approval_request`. - - - `"mcp_approval_request"` - - - `output_index: number` - - The index of the output item in the Response. - - - `response_id: string` - - The ID of the Response to which the item belongs. - - - `type: "response.output_item.done"` - - The event type, must be `response.output_item.done`. - - - `"response.output_item.done"` - -### Response Text Delta Event - -- `ResponseTextDeltaEvent = object { content_index, delta, event_id, 4 more }` - - Returned when the text value of an "output_text" content part is updated. - - - `content_index: number` - - The index of the content part in the item's content array. - - - `delta: string` - - The text delta. - - - `event_id: string` - - The unique ID of the server event. - - - `item_id: string` - - The ID of the item. - - - `output_index: number` - - The index of the output item in the response. - - - `response_id: string` - - The ID of the response. - - - `type: "response.output_text.delta"` - - The event type, must be `response.output_text.delta`. - - - `"response.output_text.delta"` - -### Response Text Done Event - -- `ResponseTextDoneEvent = object { content_index, event_id, item_id, 4 more }` - - Returned when the text value of an "output_text" content part is done streaming. Also - emitted when a Response is interrupted, incomplete, or cancelled. - - - `content_index: number` - - The index of the content part in the item's content array. - - - `event_id: string` - - The unique ID of the server event. - - - `item_id: string` - - The ID of the item. - - - `output_index: number` - - The index of the output item in the response. - - - `response_id: string` - - The ID of the response. - - - `text: string` - - The final text content. - - - `type: "response.output_text.done"` - - The event type, must be `response.output_text.done`. - - - `"response.output_text.done"` - -### Session Created Event - -- `SessionCreatedEvent = object { event_id, session, type }` - - Returned when a Session is created. Emitted automatically when a new - connection is established as the first server event. This event will contain - the default Session configuration. - - - `event_id: string` - - The unique ID of the server event. - - - `session: RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` - - The session configuration. - - - `RealtimeSessionCreateRequest = object { type, audio, include, 9 more }` - - Realtime session object configuration. - - - `type: "realtime"` - - The type of session to create. Always `realtime` for the Realtime API. - - - `"realtime"` - - - `audio: optional RealtimeAudioConfig` - - Configuration for input and output audio. - - - `input: optional RealtimeAudioConfigInput` - - - `format: optional RealtimeAudioFormats` - - The format of the input audio. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `noise_reduction: optional object { type }` - - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - - `type: optional NoiseReductionType` - - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - - `"near_field"` - - - `"far_field"` - - - `transcription: optional AudioTranscription` - - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `string` - - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - - `turn_detection: optional RealtimeAudioInputTurnDetection` - - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` - - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - - `type: "server_vad"` - - Type of turn detection, `server_vad` to turn on simple Server VAD. - - - `"server_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `idle_timeout_ms: optional number` - - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. - - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. - - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `prefix_padding_ms: optional number` - - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. - - - `silence_duration_ms: optional number` - - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. - - - `threshold: optional number` - - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. - - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` - - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - - `type: "semantic_vad"` - - Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - - `"semantic_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. - - - `eagerness: optional "low" or "medium" or "high" or "auto"` - - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - - `"low"` - - - `"medium"` - - - `"high"` - - - `"auto"` - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - - `output: optional RealtimeAudioConfigOutput` - - - `format: optional RealtimeAudioFormats` - - The format of the output audio. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `speed: optional number` - - The speed of the model's spoken response as a multiple of the original speed. - 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - - This parameter is a post-processing adjustment to the audio after it is generated, it's - also possible to prompt the model to speak faster or slower. - - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with - an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed - during the session once the model has responded with audio at least once. - We recommend `marin` and `cedar` for best quality. - - - `string` - - - `"alloy" or "ash" or "ballad" or 7 more` - - - `"alloy"` - - - `"ash"` - - - `"ballad"` - - - `"coral"` - - - `"echo"` - - - `"sage"` - - - `"shimmer"` - - - `"verse"` - - - `"marin"` - - - `"cedar"` - - - `ID = object { id }` - - Custom voice reference. - - - `id: string` - - The custom voice ID, e.g. `voice_1234`. - - - `include: optional array of "item.input_audio_transcription.logprobs"` - - Additional fields to include in server outputs. - - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - - `"item.input_audio_transcription.logprobs"` - - - `instructions: optional string` - - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - - `max_output_tokens: optional number or "inf"` - - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. - - - `number` - - - `"inf"` - - - `"inf"` - - - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - The Realtime model used for this session. - - - `string` - - - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - The Realtime model used for this session. - - - `"gpt-realtime"` - - - `"gpt-realtime-1.5"` - - - `"gpt-realtime-2025-08-28"` - - - `"gpt-4o-realtime-preview"` - - - `"gpt-4o-realtime-preview-2024-10-01"` - - - `"gpt-4o-realtime-preview-2024-12-17"` - - - `"gpt-4o-realtime-preview-2025-06-03"` - - - `"gpt-4o-mini-realtime-preview"` - - - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - - `"gpt-realtime-mini"` - - - `"gpt-realtime-mini-2025-10-06"` - - - `"gpt-realtime-mini-2025-12-15"` - - - `"gpt-audio-1.5"` - - - `"gpt-audio-mini"` - - - `"gpt-audio-mini-2025-10-06"` - - - `"gpt-audio-mini-2025-12-15"` - - - `output_modalities: optional array of "text" or "audio"` - - The set of modalities the model can respond with. It defaults to `["audio"]`, indicating - that the model will respond with audio plus a transcript. `["text"]` can be used to make - the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - - - `"text"` - - - `"audio"` - - - `prompt: optional ResponsePrompt` - - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - - - `id: string` - - The unique identifier of the prompt template to use. - - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. - - - `string` - - - `ResponseInputText = object { text, type }` - - A text input to the model. - - - `text: string` - - The text input to the model. - - - `type: "input_text"` - - The type of the input item. Always `input_text`. - - - `"input_text"` - - - `ResponseInputImage = object { detail, type, file_id, image_url }` - - An image input to the model. Learn about [image inputs](/docs/guides/vision). - - - `detail: "low" or "high" or "auto" or "original"` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `"low"` - - - `"high"` - - - `"auto"` - - - `"original"` - - - `type: "input_image"` - - The type of the input item. Always `input_image`. - - - `"input_image"` - - - `file_id: optional string` - - The ID of the file to be sent to the model. - - - `image_url: optional string` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `ResponseInputFile = object { type, detail, file_data, 3 more }` - - A file input to the model. - - - `type: "input_file"` - - The type of the input item. Always `input_file`. - - - `"input_file"` - - - `detail: optional "low" or "high"` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `"low"` - - - `"high"` - - - `file_data: optional string` - - The content of the file to be sent to the model. - - - `file_id: optional string` - - The ID of the file to be sent to the model. - - - `file_url: optional string` - - The URL of the file to be sent to the model. - - - `filename: optional string` - - The name of the file to be sent to the model. - - - `version: optional string` - - Optional version of the prompt template. - - - `tool_choice: optional RealtimeToolChoiceConfig` - - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. - - - `ToolChoiceOptions = "none" or "auto" or "required"` - - Controls which (if any) tool is called by the model. - - `none` means the model will not call any tool and instead generates a message. - - `auto` means the model can pick between generating a message or calling one or - more tools. - - `required` means the model must call one or more tools. - - - `"none"` - - - `"auto"` - - - `"required"` - - - `ToolChoiceFunction = object { name, type }` - - Use this option to force the model to call a specific function. - - - `name: string` - - The name of the function to call. - - - `type: "function"` - - For function calling, the type is always `function`. - - - `"function"` - - - `ToolChoiceMcp = object { server_label, type, name }` - - Use this option to force the model to call a specific tool on a remote MCP server. - - - `server_label: string` - - The label of the MCP server to use. - - - `type: "mcp"` - - For MCP tools, the type is always `mcp`. - - - `"mcp"` - - - `name: optional string` - - The name of the tool to call on the server. - - - `tools: optional RealtimeToolsConfig` - - Tools available to the model. - - - `RealtimeFunctionTool = object { description, name, parameters, type }` - - - `description: optional string` - - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). - - - `name: optional string` - - The name of the function. - - - `parameters: optional unknown` - - Parameters of the function in JSON Schema. - - - `type: optional "function"` - - The type of the tool, i.e. `function`. - - - `"function"` - - - `McpTool = object { server_label, type, allowed_tools, 7 more }` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - - - `server_label: string` - - A label for this MCP server, used to identify it in tool calls. - - - `type: "mcp"` - - The type of the MCP tool. Always `mcp`. - - - `"mcp"` - - - `allowed_tools: optional array of string or object { read_only, tool_names }` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = array of string` - - A string array of allowed tool names - - - `McpToolFilter = object { read_only, tool_names }` - - A filter object to specify which tools are allowed. - - - `read_only: optional boolean` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: optional array of string` - - List of allowed tool names. - - - `authorization: optional string` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `"connector_dropbox"` - - - `"connector_gmail"` - - - `"connector_googlecalendar"` - - - `"connector_googledrive"` - - - `"connector_microsoftteams"` - - - `"connector_outlookcalendar"` - - - `"connector_outlookemail"` - - - `"connector_sharepoint"` - - - `defer_loading: optional boolean` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: optional map[string]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: optional object { always, never } or "always" or "never"` - - Specify which of the MCP server's tools require approval. - - - `McpToolApprovalFilter = object { always, never }` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: optional object { read_only, tool_names }` - - A filter object to specify which tools are allowed. - - - `read_only: optional boolean` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: optional array of string` - - List of allowed tool names. - - - `never: optional object { read_only, tool_names }` - - A filter object to specify which tools are allowed. - - - `read_only: optional boolean` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: optional array of string` - - List of allowed tool names. - - - `McpToolApprovalSetting = "always" or "never"` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `"always"` - - - `"never"` - - - `server_description: optional string` - - Optional description of the MCP server, used to provide more context. - - - `server_url: optional string` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `tracing: optional RealtimeTracingConfig` - - Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once - tracing is enabled for a session, the configuration cannot be modified. - - `auto` will create a trace for the session with default values for the - workflow name, group id, and metadata. - - - `Auto = "auto"` - - Enables tracing and sets default values for tracing configuration options. Always `auto`. - - - `"auto"` - - - `TracingConfiguration = object { group_id, metadata, workflow_name }` - - Granular configuration for tracing. - - - `group_id: optional string` - - The group id to attach to this trace to enable filtering and - grouping in the Traces Dashboard. - - - `metadata: optional unknown` - - The arbitrary metadata to attach to this trace to enable - filtering in the Traces Dashboard. - - - `workflow_name: optional string` - - The name of the workflow to attach to this trace. This is used to - name the trace in the Traces Dashboard. - - - `truncation: optional RealtimeTruncation` - - When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - - Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - - Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - - - `"auto" or "disabled"` - - The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - - - `"auto"` - - - `"disabled"` - - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` - - Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - - - `retention_ratio: number` - - Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - - `type: "retention_ratio"` - - Use retention ratio truncation. - - - `"retention_ratio"` - - - `token_limits: optional object { post_instructions }` - - Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - - - `post_instructions: optional number` - - Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - - - `RealtimeTranscriptionSessionCreateRequest = object { type, audio, include }` - - Realtime transcription session object configuration. - - - `type: "transcription"` - - The type of session to create. Always `transcription` for transcription sessions. - - - `"transcription"` - - - `audio: optional RealtimeTranscriptionSessionAudio` - - Configuration for input and output audio. - - - `input: optional RealtimeTranscriptionSessionAudioInput` - - - `format: optional RealtimeAudioFormats` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `noise_reduction: optional object { type }` - - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - - `type: optional NoiseReductionType` - - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - - `"near_field"` - - - `"far_field"` - - - `transcription: optional AudioTranscription` - - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `string` - - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` - - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` - - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - - `type: "server_vad"` - - Type of turn detection, `server_vad` to turn on simple Server VAD. - - - `"server_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `idle_timeout_ms: optional number` - - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. - - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. - - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `prefix_padding_ms: optional number` - - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. - - - `silence_duration_ms: optional number` - - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. - - - `threshold: optional number` - - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. - - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` - - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - - `type: "semantic_vad"` - - Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - - `"semantic_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. - - - `eagerness: optional "low" or "medium" or "high" or "auto"` - - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - - `"low"` - - - `"medium"` - - - `"high"` - - - `"auto"` - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - - `include: optional array of "item.input_audio_transcription.logprobs"` - - Additional fields to include in server outputs. - - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - - `"item.input_audio_transcription.logprobs"` - - - `type: "session.created"` - - The event type, must be `session.created`. - - - `"session.created"` - -### Session Update Event - -- `SessionUpdateEvent = object { session, type, event_id }` - - Send this event to update the session’s configuration. - The client may send this event at any time to update any field - except for `voice` and `model`. `voice` can be updated only if there have been no other audio outputs yet. - - When the server receives a `session.update`, it will respond - with a `session.updated` event showing the full, effective configuration. - Only the fields that are present in the `session.update` are updated. To clear a field like - `instructions`, pass an empty string. To clear a field like `tools`, pass an empty array. - To clear a field like `turn_detection`, pass `null`. - - - `session: RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` - - Update the Realtime session. Choose either a realtime - session or a transcription session. - - - `RealtimeSessionCreateRequest = object { type, audio, include, 9 more }` - - Realtime session object configuration. - - - `type: "realtime"` - - The type of session to create. Always `realtime` for the Realtime API. - - - `"realtime"` - - - `audio: optional RealtimeAudioConfig` - - Configuration for input and output audio. - - - `input: optional RealtimeAudioConfigInput` - - - `format: optional RealtimeAudioFormats` - - The format of the input audio. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `noise_reduction: optional object { type }` - - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - - `type: optional NoiseReductionType` - - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - - `"near_field"` - - - `"far_field"` - - - `transcription: optional AudioTranscription` - - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `string` - - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - - `turn_detection: optional RealtimeAudioInputTurnDetection` - - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` - - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - - `type: "server_vad"` - - Type of turn detection, `server_vad` to turn on simple Server VAD. - - - `"server_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `idle_timeout_ms: optional number` - - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. - - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. - - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `prefix_padding_ms: optional number` - - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. - - - `silence_duration_ms: optional number` - - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. - - - `threshold: optional number` - - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. - - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` - - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - - `type: "semantic_vad"` - - Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - - `"semantic_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. - - - `eagerness: optional "low" or "medium" or "high" or "auto"` - - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - - `"low"` - - - `"medium"` - - - `"high"` - - - `"auto"` - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - - `output: optional RealtimeAudioConfigOutput` - - - `format: optional RealtimeAudioFormats` - - The format of the output audio. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `speed: optional number` - - The speed of the model's spoken response as a multiple of the original speed. - 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - - This parameter is a post-processing adjustment to the audio after it is generated, it's - also possible to prompt the model to speak faster or slower. - - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with - an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed - during the session once the model has responded with audio at least once. - We recommend `marin` and `cedar` for best quality. - - - `string` - - - `"alloy" or "ash" or "ballad" or 7 more` - - - `"alloy"` - - - `"ash"` - - - `"ballad"` - - - `"coral"` - - - `"echo"` - - - `"sage"` - - - `"shimmer"` - - - `"verse"` - - - `"marin"` - - - `"cedar"` - - - `ID = object { id }` - - Custom voice reference. - - - `id: string` - - The custom voice ID, e.g. `voice_1234`. - - - `include: optional array of "item.input_audio_transcription.logprobs"` - - Additional fields to include in server outputs. - - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - - `"item.input_audio_transcription.logprobs"` - - - `instructions: optional string` - - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - - `max_output_tokens: optional number or "inf"` - - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. - - - `number` - - - `"inf"` - - - `"inf"` - - - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - The Realtime model used for this session. - - - `string` - - - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - The Realtime model used for this session. - - - `"gpt-realtime"` - - - `"gpt-realtime-1.5"` - - - `"gpt-realtime-2025-08-28"` - - - `"gpt-4o-realtime-preview"` - - - `"gpt-4o-realtime-preview-2024-10-01"` - - - `"gpt-4o-realtime-preview-2024-12-17"` - - - `"gpt-4o-realtime-preview-2025-06-03"` - - - `"gpt-4o-mini-realtime-preview"` - - - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - - `"gpt-realtime-mini"` - - - `"gpt-realtime-mini-2025-10-06"` - - - `"gpt-realtime-mini-2025-12-15"` - - - `"gpt-audio-1.5"` - - - `"gpt-audio-mini"` - - - `"gpt-audio-mini-2025-10-06"` - - - `"gpt-audio-mini-2025-12-15"` - - - `output_modalities: optional array of "text" or "audio"` - - The set of modalities the model can respond with. It defaults to `["audio"]`, indicating - that the model will respond with audio plus a transcript. `["text"]` can be used to make - the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - - - `"text"` - - - `"audio"` - - - `prompt: optional ResponsePrompt` - - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - - - `id: string` - - The unique identifier of the prompt template to use. - - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. - - - `string` - - - `ResponseInputText = object { text, type }` - - A text input to the model. - - - `text: string` - - The text input to the model. - - - `type: "input_text"` - - The type of the input item. Always `input_text`. - - - `"input_text"` - - - `ResponseInputImage = object { detail, type, file_id, image_url }` - - An image input to the model. Learn about [image inputs](/docs/guides/vision). - - - `detail: "low" or "high" or "auto" or "original"` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `"low"` - - - `"high"` - - - `"auto"` - - - `"original"` - - - `type: "input_image"` - - The type of the input item. Always `input_image`. - - - `"input_image"` - - - `file_id: optional string` - - The ID of the file to be sent to the model. - - - `image_url: optional string` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `ResponseInputFile = object { type, detail, file_data, 3 more }` - - A file input to the model. - - - `type: "input_file"` - - The type of the input item. Always `input_file`. - - - `"input_file"` - - - `detail: optional "low" or "high"` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `"low"` - - - `"high"` - - - `file_data: optional string` - - The content of the file to be sent to the model. - - - `file_id: optional string` - - The ID of the file to be sent to the model. - - - `file_url: optional string` - - The URL of the file to be sent to the model. - - - `filename: optional string` - - The name of the file to be sent to the model. - - - `version: optional string` - - Optional version of the prompt template. - - - `tool_choice: optional RealtimeToolChoiceConfig` - - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. - - - `ToolChoiceOptions = "none" or "auto" or "required"` - - Controls which (if any) tool is called by the model. - - `none` means the model will not call any tool and instead generates a message. - - `auto` means the model can pick between generating a message or calling one or - more tools. - - `required` means the model must call one or more tools. - - - `"none"` - - - `"auto"` - - - `"required"` - - - `ToolChoiceFunction = object { name, type }` - - Use this option to force the model to call a specific function. - - - `name: string` - - The name of the function to call. - - - `type: "function"` - - For function calling, the type is always `function`. - - - `"function"` - - - `ToolChoiceMcp = object { server_label, type, name }` - - Use this option to force the model to call a specific tool on a remote MCP server. - - - `server_label: string` - - The label of the MCP server to use. - - - `type: "mcp"` - - For MCP tools, the type is always `mcp`. - - - `"mcp"` - - - `name: optional string` - - The name of the tool to call on the server. - - - `tools: optional RealtimeToolsConfig` - - Tools available to the model. - - - `RealtimeFunctionTool = object { description, name, parameters, type }` - - - `description: optional string` - - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). - - - `name: optional string` - - The name of the function. - - - `parameters: optional unknown` - - Parameters of the function in JSON Schema. - - - `type: optional "function"` - - The type of the tool, i.e. `function`. - - - `"function"` - - - `McpTool = object { server_label, type, allowed_tools, 7 more }` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - - - `server_label: string` - - A label for this MCP server, used to identify it in tool calls. - - - `type: "mcp"` - - The type of the MCP tool. Always `mcp`. - - - `"mcp"` - - - `allowed_tools: optional array of string or object { read_only, tool_names }` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = array of string` - - A string array of allowed tool names - - - `McpToolFilter = object { read_only, tool_names }` - - A filter object to specify which tools are allowed. - - - `read_only: optional boolean` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: optional array of string` - - List of allowed tool names. - - - `authorization: optional string` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `"connector_dropbox"` - - - `"connector_gmail"` - - - `"connector_googlecalendar"` - - - `"connector_googledrive"` - - - `"connector_microsoftteams"` - - - `"connector_outlookcalendar"` - - - `"connector_outlookemail"` - - - `"connector_sharepoint"` - - - `defer_loading: optional boolean` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: optional map[string]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: optional object { always, never } or "always" or "never"` - - Specify which of the MCP server's tools require approval. - - - `McpToolApprovalFilter = object { always, never }` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: optional object { read_only, tool_names }` - - A filter object to specify which tools are allowed. - - - `read_only: optional boolean` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: optional array of string` - - List of allowed tool names. - - - `never: optional object { read_only, tool_names }` - - A filter object to specify which tools are allowed. - - - `read_only: optional boolean` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: optional array of string` - - List of allowed tool names. - - - `McpToolApprovalSetting = "always" or "never"` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `"always"` - - - `"never"` - - - `server_description: optional string` - - Optional description of the MCP server, used to provide more context. - - - `server_url: optional string` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `tracing: optional RealtimeTracingConfig` - - Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once - tracing is enabled for a session, the configuration cannot be modified. - - `auto` will create a trace for the session with default values for the - workflow name, group id, and metadata. - - - `Auto = "auto"` - - Enables tracing and sets default values for tracing configuration options. Always `auto`. - - - `"auto"` - - - `TracingConfiguration = object { group_id, metadata, workflow_name }` - - Granular configuration for tracing. - - - `group_id: optional string` - - The group id to attach to this trace to enable filtering and - grouping in the Traces Dashboard. - - - `metadata: optional unknown` - - The arbitrary metadata to attach to this trace to enable - filtering in the Traces Dashboard. - - - `workflow_name: optional string` - - The name of the workflow to attach to this trace. This is used to - name the trace in the Traces Dashboard. - - - `truncation: optional RealtimeTruncation` - - When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - - Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - - Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - - - `"auto" or "disabled"` - - The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - - - `"auto"` - - - `"disabled"` - - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` - - Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - - - `retention_ratio: number` - - Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - - `type: "retention_ratio"` - - Use retention ratio truncation. - - - `"retention_ratio"` - - - `token_limits: optional object { post_instructions }` - - Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - - - `post_instructions: optional number` - - Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - - - `RealtimeTranscriptionSessionCreateRequest = object { type, audio, include }` - - Realtime transcription session object configuration. - - - `type: "transcription"` - - The type of session to create. Always `transcription` for transcription sessions. - - - `"transcription"` - - - `audio: optional RealtimeTranscriptionSessionAudio` - - Configuration for input and output audio. - - - `input: optional RealtimeTranscriptionSessionAudioInput` - - - `format: optional RealtimeAudioFormats` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `noise_reduction: optional object { type }` - - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - - `type: optional NoiseReductionType` - - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - - `"near_field"` - - - `"far_field"` - - - `transcription: optional AudioTranscription` - - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `string` - - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` - - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` - - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - - `type: "server_vad"` - - Type of turn detection, `server_vad` to turn on simple Server VAD. - - - `"server_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `idle_timeout_ms: optional number` - - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. - - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. - - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `prefix_padding_ms: optional number` - - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. - - - `silence_duration_ms: optional number` - - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. - - - `threshold: optional number` - - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. - - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` - - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - - `type: "semantic_vad"` - - Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - - `"semantic_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. - - - `eagerness: optional "low" or "medium" or "high" or "auto"` - - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - - `"low"` - - - `"medium"` - - - `"high"` - - - `"auto"` - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - - `include: optional array of "item.input_audio_transcription.logprobs"` - - Additional fields to include in server outputs. - - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - - `"item.input_audio_transcription.logprobs"` - - - `type: "session.update"` - - The event type, must be `session.update`. - - - `"session.update"` - - - `event_id: optional string` - - Optional client-generated ID used to identify this event. This is an arbitrary string that a client may assign. It will be passed back if there is an error with the event, but the corresponding `session.updated` event will not include it. - -### Session Updated Event - -- `SessionUpdatedEvent = object { event_id, session, type }` - - Returned when a session is updated with a `session.update` event, unless - there is an error. - - - `event_id: string` - - The unique ID of the server event. - - - `session: RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` - - The session configuration. - - - `RealtimeSessionCreateRequest = object { type, audio, include, 9 more }` - - Realtime session object configuration. - - - `type: "realtime"` - - The type of session to create. Always `realtime` for the Realtime API. - - - `"realtime"` - - - `audio: optional RealtimeAudioConfig` - - Configuration for input and output audio. - - - `input: optional RealtimeAudioConfigInput` - - - `format: optional RealtimeAudioFormats` - - The format of the input audio. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `noise_reduction: optional object { type }` - - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - - `type: optional NoiseReductionType` - - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - - `"near_field"` - - - `"far_field"` - - - `transcription: optional AudioTranscription` - - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `string` - - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - - `turn_detection: optional RealtimeAudioInputTurnDetection` - - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` - - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - - `type: "server_vad"` - - Type of turn detection, `server_vad` to turn on simple Server VAD. - - - `"server_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `idle_timeout_ms: optional number` - - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. - - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. - - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `prefix_padding_ms: optional number` - - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. - - - `silence_duration_ms: optional number` - - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. - - - `threshold: optional number` - - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. - - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` - - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - - `type: "semantic_vad"` - - Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - - `"semantic_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. - - - `eagerness: optional "low" or "medium" or "high" or "auto"` - - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - - `"low"` - - - `"medium"` - - - `"high"` - - - `"auto"` - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - - `output: optional RealtimeAudioConfigOutput` - - - `format: optional RealtimeAudioFormats` - - The format of the output audio. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `speed: optional number` - - The speed of the model's spoken response as a multiple of the original speed. - 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - - This parameter is a post-processing adjustment to the audio after it is generated, it's - also possible to prompt the model to speak faster or slower. - - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with - an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed - during the session once the model has responded with audio at least once. - We recommend `marin` and `cedar` for best quality. - - - `string` - - - `"alloy" or "ash" or "ballad" or 7 more` - - - `"alloy"` - - - `"ash"` - - - `"ballad"` - - - `"coral"` - - - `"echo"` - - - `"sage"` - - - `"shimmer"` - - - `"verse"` - - - `"marin"` - - - `"cedar"` - - - `ID = object { id }` - - Custom voice reference. - - - `id: string` - - The custom voice ID, e.g. `voice_1234`. - - - `include: optional array of "item.input_audio_transcription.logprobs"` - - Additional fields to include in server outputs. - - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - - `"item.input_audio_transcription.logprobs"` - - - `instructions: optional string` - - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - - `max_output_tokens: optional number or "inf"` - - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. - - - `number` - - - `"inf"` - - - `"inf"` - - - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - The Realtime model used for this session. - - - `string` - - - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - The Realtime model used for this session. - - - `"gpt-realtime"` - - - `"gpt-realtime-1.5"` - - - `"gpt-realtime-2025-08-28"` - - - `"gpt-4o-realtime-preview"` - - - `"gpt-4o-realtime-preview-2024-10-01"` - - - `"gpt-4o-realtime-preview-2024-12-17"` - - - `"gpt-4o-realtime-preview-2025-06-03"` - - - `"gpt-4o-mini-realtime-preview"` - - - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - - `"gpt-realtime-mini"` - - - `"gpt-realtime-mini-2025-10-06"` - - - `"gpt-realtime-mini-2025-12-15"` - - - `"gpt-audio-1.5"` - - - `"gpt-audio-mini"` - - - `"gpt-audio-mini-2025-10-06"` - - - `"gpt-audio-mini-2025-12-15"` - - - `output_modalities: optional array of "text" or "audio"` - - The set of modalities the model can respond with. It defaults to `["audio"]`, indicating - that the model will respond with audio plus a transcript. `["text"]` can be used to make - the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - - - `"text"` - - - `"audio"` - - - `prompt: optional ResponsePrompt` - - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - - - `id: string` - - The unique identifier of the prompt template to use. - - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. - - - `string` - - - `ResponseInputText = object { text, type }` - - A text input to the model. - - - `text: string` - - The text input to the model. - - - `type: "input_text"` - - The type of the input item. Always `input_text`. - - - `"input_text"` - - - `ResponseInputImage = object { detail, type, file_id, image_url }` - - An image input to the model. Learn about [image inputs](/docs/guides/vision). - - - `detail: "low" or "high" or "auto" or "original"` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `"low"` - - - `"high"` - - - `"auto"` - - - `"original"` - - - `type: "input_image"` - - The type of the input item. Always `input_image`. - - - `"input_image"` - - - `file_id: optional string` - - The ID of the file to be sent to the model. - - - `image_url: optional string` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `ResponseInputFile = object { type, detail, file_data, 3 more }` - - A file input to the model. - - - `type: "input_file"` - - The type of the input item. Always `input_file`. - - - `"input_file"` - - - `detail: optional "low" or "high"` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `"low"` - - - `"high"` - - - `file_data: optional string` - - The content of the file to be sent to the model. - - - `file_id: optional string` - - The ID of the file to be sent to the model. - - - `file_url: optional string` - - The URL of the file to be sent to the model. - - - `filename: optional string` - - The name of the file to be sent to the model. - - - `version: optional string` - - Optional version of the prompt template. - - - `tool_choice: optional RealtimeToolChoiceConfig` - - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. - - - `ToolChoiceOptions = "none" or "auto" or "required"` - - Controls which (if any) tool is called by the model. - - `none` means the model will not call any tool and instead generates a message. - - `auto` means the model can pick between generating a message or calling one or - more tools. - - `required` means the model must call one or more tools. - - - `"none"` - - - `"auto"` - - - `"required"` - - - `ToolChoiceFunction = object { name, type }` - - Use this option to force the model to call a specific function. - - - `name: string` - - The name of the function to call. - - - `type: "function"` - - For function calling, the type is always `function`. - - - `"function"` - - - `ToolChoiceMcp = object { server_label, type, name }` - - Use this option to force the model to call a specific tool on a remote MCP server. - - - `server_label: string` - - The label of the MCP server to use. - - - `type: "mcp"` - - For MCP tools, the type is always `mcp`. - - - `"mcp"` - - - `name: optional string` - - The name of the tool to call on the server. - - - `tools: optional RealtimeToolsConfig` - - Tools available to the model. - - - `RealtimeFunctionTool = object { description, name, parameters, type }` - - - `description: optional string` - - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). - - - `name: optional string` - - The name of the function. - - - `parameters: optional unknown` - - Parameters of the function in JSON Schema. - - - `type: optional "function"` - - The type of the tool, i.e. `function`. - - - `"function"` - - - `McpTool = object { server_label, type, allowed_tools, 7 more }` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - - - `server_label: string` - - A label for this MCP server, used to identify it in tool calls. - - - `type: "mcp"` - - The type of the MCP tool. Always `mcp`. - - - `"mcp"` - - - `allowed_tools: optional array of string or object { read_only, tool_names }` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = array of string` - - A string array of allowed tool names - - - `McpToolFilter = object { read_only, tool_names }` - - A filter object to specify which tools are allowed. - - - `read_only: optional boolean` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: optional array of string` - - List of allowed tool names. - - - `authorization: optional string` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `"connector_dropbox"` - - - `"connector_gmail"` - - - `"connector_googlecalendar"` - - - `"connector_googledrive"` - - - `"connector_microsoftteams"` - - - `"connector_outlookcalendar"` - - - `"connector_outlookemail"` - - - `"connector_sharepoint"` - - - `defer_loading: optional boolean` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: optional map[string]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: optional object { always, never } or "always" or "never"` - - Specify which of the MCP server's tools require approval. - - - `McpToolApprovalFilter = object { always, never }` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: optional object { read_only, tool_names }` - - A filter object to specify which tools are allowed. - - - `read_only: optional boolean` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: optional array of string` - - List of allowed tool names. - - - `never: optional object { read_only, tool_names }` - - A filter object to specify which tools are allowed. - - - `read_only: optional boolean` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: optional array of string` - - List of allowed tool names. - - - `McpToolApprovalSetting = "always" or "never"` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `"always"` - - - `"never"` - - - `server_description: optional string` - - Optional description of the MCP server, used to provide more context. - - - `server_url: optional string` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `tracing: optional RealtimeTracingConfig` - - Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once - tracing is enabled for a session, the configuration cannot be modified. - - `auto` will create a trace for the session with default values for the - workflow name, group id, and metadata. - - - `Auto = "auto"` - - Enables tracing and sets default values for tracing configuration options. Always `auto`. - - - `"auto"` - - - `TracingConfiguration = object { group_id, metadata, workflow_name }` - - Granular configuration for tracing. - - - `group_id: optional string` - - The group id to attach to this trace to enable filtering and - grouping in the Traces Dashboard. - - - `metadata: optional unknown` - - The arbitrary metadata to attach to this trace to enable - filtering in the Traces Dashboard. - - - `workflow_name: optional string` - - The name of the workflow to attach to this trace. This is used to - name the trace in the Traces Dashboard. - - - `truncation: optional RealtimeTruncation` - - When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - - Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - - Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - - - `"auto" or "disabled"` - - The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - - - `"auto"` - - - `"disabled"` - - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` - - Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - - - `retention_ratio: number` - - Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - - `type: "retention_ratio"` - - Use retention ratio truncation. - - - `"retention_ratio"` - - - `token_limits: optional object { post_instructions }` - - Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - - - `post_instructions: optional number` - - Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - - - `RealtimeTranscriptionSessionCreateRequest = object { type, audio, include }` - - Realtime transcription session object configuration. - - - `type: "transcription"` - - The type of session to create. Always `transcription` for transcription sessions. - - - `"transcription"` - - - `audio: optional RealtimeTranscriptionSessionAudio` - - Configuration for input and output audio. - - - `input: optional RealtimeTranscriptionSessionAudioInput` - - - `format: optional RealtimeAudioFormats` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `noise_reduction: optional object { type }` - - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - - `type: optional NoiseReductionType` - - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - - `"near_field"` - - - `"far_field"` - - - `transcription: optional AudioTranscription` - - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `string` - - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` - - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` - - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - - `type: "server_vad"` - - Type of turn detection, `server_vad` to turn on simple Server VAD. - - - `"server_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `idle_timeout_ms: optional number` - - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. - - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. - - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `prefix_padding_ms: optional number` - - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. - - - `silence_duration_ms: optional number` - - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. - - - `threshold: optional number` - - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. - - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` - - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - - `type: "semantic_vad"` - - Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - - `"semantic_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. - - - `eagerness: optional "low" or "medium" or "high" or "auto"` - - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - - `"low"` - - - `"medium"` - - - `"high"` - - - `"auto"` - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - - `include: optional array of "item.input_audio_transcription.logprobs"` - - Additional fields to include in server outputs. - - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - - `"item.input_audio_transcription.logprobs"` - - - `type: "session.updated"` - - The event type, must be `session.updated`. - - - `"session.updated"` - -### Transcription Session Update - -- `TranscriptionSessionUpdate = object { session, type, event_id }` - - Send this event to update a transcription session. - - - `session: object { include, input_audio_format, input_audio_noise_reduction, 2 more }` - - Realtime transcription session object configuration. - - - `include: optional array of "item.input_audio_transcription.logprobs"` - - The set of items to include in the transcription. Current available items are: - `item.input_audio_transcription.logprobs` - - - `"item.input_audio_transcription.logprobs"` - - - `input_audio_format: optional "pcm16" or "g711_ulaw" or "g711_alaw"` - - The format of input audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. - For `pcm16`, input audio must be 16-bit PCM at a 24kHz sample rate, - single channel (mono), and little-endian byte order. - - - `"pcm16"` - - - `"g711_ulaw"` - - - `"g711_alaw"` - - - `input_audio_noise_reduction: optional object { type }` - - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - - `type: optional NoiseReductionType` - - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - - `"near_field"` - - - `"far_field"` - - - `input_audio_transcription: optional AudioTranscription` - - Configuration for input audio transcription. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `string` - - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - - `turn_detection: optional object { prefix_padding_ms, silence_duration_ms, threshold, type }` - - Configuration for turn detection. Can be set to `null` to turn off. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - - `prefix_padding_ms: optional number` - - Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. - - - `silence_duration_ms: optional number` - - Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. - - - `threshold: optional number` - - Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. - - - `type: optional "server_vad"` - - Type of turn detection. Only `server_vad` is currently supported for transcription sessions. - - - `"server_vad"` - - - `type: "transcription_session.update"` - - The event type, must be `transcription_session.update`. - - - `"transcription_session.update"` - - - `event_id: optional string` - - Optional client-generated ID used to identify this event. - -### Transcription Session Updated Event - -- `TranscriptionSessionUpdatedEvent = object { event_id, session, type }` - - Returned when a transcription session is updated with a `transcription_session.update` event, unless - there is an error. - - - `event_id: string` - - The unique ID of the server event. - - - `session: object { client_secret, input_audio_format, input_audio_transcription, 2 more }` - - A new Realtime transcription session configuration. - - When a session is created on the server via REST API, the session object - also contains an ephemeral key. Default TTL for keys is 10 minutes. This - property is not present when a session is updated via the WebSocket API. - - - `client_secret: object { expires_at, value }` - - Ephemeral key returned by the API. Only present when the session is - created on the server via REST API. - - - `expires_at: number` - - Timestamp for when the token expires. Currently, all tokens expire - after one minute. - - - `value: string` - - Ephemeral key usable in client environments to authenticate connections - to the Realtime API. Use this in client-side environments rather than - a standard API token, which should only be used server-side. - - - `input_audio_format: optional string` - - The format of input audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. - - - `input_audio_transcription: optional AudioTranscription` - - Configuration of the transcription model. - - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `string` - - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - - `modalities: optional array of "text" or "audio"` - - The set of modalities the model can respond with. To disable audio, - set this to ["text"]. - - - `"text"` - - - `"audio"` - - - `turn_detection: optional object { prefix_padding_ms, silence_duration_ms, threshold, type }` - - Configuration for turn detection. Can be set to `null` to turn off. Server - VAD means that the model will detect the start and end of speech based on - audio volume and respond at the end of user speech. - - - `prefix_padding_ms: optional number` - - Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. - - - `silence_duration_ms: optional number` - - Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. - - - `threshold: optional number` - - Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. - - - `type: optional string` - - Type of turn detection, only `server_vad` is currently supported. - - - `type: "transcription_session.updated"` - - The event type, must be `transcription_session.updated`. - - - `"transcription_session.updated"` - -# Client Secrets - -## Create client secret - -**post** `/realtime/client_secrets` - -Create a Realtime client secret with an associated session configuration. - -Client secrets are short-lived tokens that can be passed to a client app, -such as a web frontend or mobile client, which grants access to the Realtime API without -leaking your main API key. You can configure a custom TTL for each client secret. - -You can also attach session configuration options to the client secret, which will be -applied to any sessions created using that client secret, but these can also be overridden -by the client connection. - -[Learn more about authentication with client secrets over WebRTC](/docs/guides/realtime-webrtc). - -Returns the created client secret and the effective session object. The client secret is a string that looks like `ek_1234`. - -### Body Parameters - -- `expires_after: optional object { anchor, seconds }` - - Configuration for the client secret expiration. Expiration refers to the time after which - a client secret will no longer be valid for creating sessions. The session itself may - continue after that time once started. A secret can be used to create multiple sessions - until it expires. - - - `anchor: optional "created_at"` - - The anchor point for the client secret expiration, meaning that `seconds` will be added to the `created_at` time of the client secret to produce an expiration timestamp. Only `created_at` is currently supported. - - - `"created_at"` - - - `seconds: optional number` - - The number of seconds from the anchor point to the expiration. Select a value between `10` and `7200` (2 hours). This default to 600 seconds (10 minutes) if not specified. - -- `session: optional RealtimeSessionCreateRequest or RealtimeTranscriptionSessionCreateRequest` - - Session configuration to use for the client secret. Choose either a realtime - session or a transcription session. - - - `RealtimeSessionCreateRequest = object { type, audio, include, 9 more }` - - Realtime session object configuration. - - - `type: "realtime"` - - The type of session to create. Always `realtime` for the Realtime API. - - - `"realtime"` - - - `audio: optional RealtimeAudioConfig` - - Configuration for input and output audio. - - - `input: optional RealtimeAudioConfigInput` - - - `format: optional RealtimeAudioFormats` - - The format of the input audio. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `noise_reduction: optional object { type }` - - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - - `type: optional NoiseReductionType` - - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - - `"near_field"` - - - `"far_field"` - - - `transcription: optional AudioTranscription` - - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `string` - - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - - `turn_detection: optional RealtimeAudioInputTurnDetection` - - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` - - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. - - - `type: "server_vad"` - - Type of turn detection, `server_vad` to turn on simple Server VAD. - - - `"server_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `idle_timeout_ms: optional number` - - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. - - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. - - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - - `prefix_padding_ms: optional number` - - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. - - - `silence_duration_ms: optional number` - - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. - - - `threshold: optional number` - - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. - - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` - - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - - `type: "semantic_vad"` - - Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - - `"semantic_vad"` - - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. - - - `eagerness: optional "low" or "medium" or "high" or "auto"` - - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - - `"low"` - - - `"medium"` - - - `"high"` - - - `"auto"` - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - - `output: optional RealtimeAudioConfigOutput` - - - `format: optional RealtimeAudioFormats` - - The format of the output audio. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `speed: optional number` - - The speed of the model's spoken response as a multiple of the original speed. - 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - - This parameter is a post-processing adjustment to the audio after it is generated, it's - also possible to prompt the model to speak faster or slower. - - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with - an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed - during the session once the model has responded with audio at least once. - We recommend `marin` and `cedar` for best quality. - - - `string` - - - `"alloy" or "ash" or "ballad" or 7 more` - - - `"alloy"` - - - `"ash"` - - - `"ballad"` - - - `"coral"` - - - `"echo"` - - - `"sage"` - - - `"shimmer"` - - - `"verse"` - - - `"marin"` - - - `"cedar"` - - - `ID = object { id }` - - Custom voice reference. - - - `id: string` - - The custom voice ID, e.g. `voice_1234`. - - - `include: optional array of "item.input_audio_transcription.logprobs"` - - Additional fields to include in server outputs. - - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - - `"item.input_audio_transcription.logprobs"` - - - `instructions: optional string` - - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - - `max_output_tokens: optional number or "inf"` - - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. - - - `number` - - - `"inf"` - - - `"inf"` - - - `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - The Realtime model used for this session. - - - `string` - - - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - The Realtime model used for this session. - - - `"gpt-realtime"` - - - `"gpt-realtime-1.5"` - - - `"gpt-realtime-2025-08-28"` - - - `"gpt-4o-realtime-preview"` - - - `"gpt-4o-realtime-preview-2024-10-01"` - - - `"gpt-4o-realtime-preview-2024-12-17"` - - - `"gpt-4o-realtime-preview-2025-06-03"` - - - `"gpt-4o-mini-realtime-preview"` - - - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - - `"gpt-realtime-mini"` - - - `"gpt-realtime-mini-2025-10-06"` - - - `"gpt-realtime-mini-2025-12-15"` - - - `"gpt-audio-1.5"` - - - `"gpt-audio-mini"` - - - `"gpt-audio-mini-2025-10-06"` - - - `"gpt-audio-mini-2025-12-15"` - - - `output_modalities: optional array of "text" or "audio"` - - The set of modalities the model can respond with. It defaults to `["audio"]`, indicating - that the model will respond with audio plus a transcript. `["text"]` can be used to make - the model respond with text only. It is not possible to request both `text` and `audio` at the same time. - - - `"text"` - - - `"audio"` - - - `prompt: optional ResponsePrompt` - - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - - - `id: string` - - The unique identifier of the prompt template to use. - - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. - - - `string` - - - `ResponseInputText = object { text, type }` - - A text input to the model. - - - `text: string` - - The text input to the model. - - - `type: "input_text"` - - The type of the input item. Always `input_text`. - - - `"input_text"` - - - `ResponseInputImage = object { detail, type, file_id, image_url }` - - An image input to the model. Learn about [image inputs](/docs/guides/vision). - - - `detail: "low" or "high" or "auto" or "original"` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `"low"` - - - `"high"` - - - `"auto"` - - - `"original"` - - - `type: "input_image"` - - The type of the input item. Always `input_image`. - - - `"input_image"` - - - `file_id: optional string` - - The ID of the file to be sent to the model. - - - `image_url: optional string` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `ResponseInputFile = object { type, detail, file_data, 3 more }` - - A file input to the model. - - - `type: "input_file"` - - The type of the input item. Always `input_file`. - - - `"input_file"` - - - `detail: optional "low" or "high"` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `"low"` - - - `"high"` - - - `file_data: optional string` - - The content of the file to be sent to the model. - - - `file_id: optional string` - - The ID of the file to be sent to the model. - - - `file_url: optional string` - - The URL of the file to be sent to the model. - - - `filename: optional string` - - The name of the file to be sent to the model. - - - `version: optional string` - - Optional version of the prompt template. - - - `tool_choice: optional RealtimeToolChoiceConfig` - - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. - - - `ToolChoiceOptions = "none" or "auto" or "required"` - - Controls which (if any) tool is called by the model. - - `none` means the model will not call any tool and instead generates a message. - - `auto` means the model can pick between generating a message or calling one or - more tools. - - `required` means the model must call one or more tools. - - - `"none"` - - - `"auto"` - - - `"required"` - - - `ToolChoiceFunction = object { name, type }` - - Use this option to force the model to call a specific function. - - - `name: string` - - The name of the function to call. - - - `type: "function"` - - For function calling, the type is always `function`. - - - `"function"` - - - `ToolChoiceMcp = object { server_label, type, name }` - - Use this option to force the model to call a specific tool on a remote MCP server. - - - `server_label: string` - - The label of the MCP server to use. - - - `type: "mcp"` - - For MCP tools, the type is always `mcp`. - - - `"mcp"` - - - `name: optional string` - - The name of the tool to call on the server. - - - `tools: optional RealtimeToolsConfig` - - Tools available to the model. - - - `RealtimeFunctionTool = object { description, name, parameters, type }` - - - `description: optional string` - - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). - - - `name: optional string` - - The name of the function. - - - `parameters: optional unknown` - - Parameters of the function in JSON Schema. - - - `type: optional "function"` - - The type of the tool, i.e. `function`. - - - `"function"` - - - `McpTool = object { server_label, type, allowed_tools, 7 more }` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - - - `server_label: string` - - A label for this MCP server, used to identify it in tool calls. - - - `type: "mcp"` - - The type of the MCP tool. Always `mcp`. - - - `"mcp"` - - - `allowed_tools: optional array of string or object { read_only, tool_names }` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = array of string` - - A string array of allowed tool names - - - `McpToolFilter = object { read_only, tool_names }` - - A filter object to specify which tools are allowed. - - - `read_only: optional boolean` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: optional array of string` - - List of allowed tool names. - - - `authorization: optional string` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `"connector_dropbox"` - - - `"connector_gmail"` - - - `"connector_googlecalendar"` - - - `"connector_googledrive"` - - - `"connector_microsoftteams"` - - - `"connector_outlookcalendar"` - - - `"connector_outlookemail"` - - - `"connector_sharepoint"` - - - `defer_loading: optional boolean` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: optional map[string]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: optional object { always, never } or "always" or "never"` - - Specify which of the MCP server's tools require approval. - - - `McpToolApprovalFilter = object { always, never }` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: optional object { read_only, tool_names }` - - A filter object to specify which tools are allowed. - - - `read_only: optional boolean` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: optional array of string` - - List of allowed tool names. - - - `never: optional object { read_only, tool_names }` - - A filter object to specify which tools are allowed. - - - `read_only: optional boolean` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: optional array of string` - - List of allowed tool names. - - - `McpToolApprovalSetting = "always" or "never"` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `"always"` - - - `"never"` - - - `server_description: optional string` - - Optional description of the MCP server, used to provide more context. - - - `server_url: optional string` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `tracing: optional RealtimeTracingConfig` - - Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once - tracing is enabled for a session, the configuration cannot be modified. - - `auto` will create a trace for the session with default values for the - workflow name, group id, and metadata. - - - `Auto = "auto"` - - Enables tracing and sets default values for tracing configuration options. Always `auto`. - - - `"auto"` - - - `TracingConfiguration = object { group_id, metadata, workflow_name }` - - Granular configuration for tracing. - - - `group_id: optional string` - - The group id to attach to this trace to enable filtering and - grouping in the Traces Dashboard. - - - `metadata: optional unknown` - - The arbitrary metadata to attach to this trace to enable - filtering in the Traces Dashboard. - - - `workflow_name: optional string` - - The name of the workflow to attach to this trace. This is used to - name the trace in the Traces Dashboard. - - - `truncation: optional RealtimeTruncation` - - When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - - Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - - Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - - - `"auto" or "disabled"` - - The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - - - `"auto"` - - - `"disabled"` - - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` - - Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - - - `retention_ratio: number` - - Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - - `type: "retention_ratio"` - - Use retention ratio truncation. - - - `"retention_ratio"` - - - `token_limits: optional object { post_instructions }` - - Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - - - `post_instructions: optional number` - - Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - - - `RealtimeTranscriptionSessionCreateRequest = object { type, audio, include }` - - Realtime transcription session object configuration. - - - `type: "transcription"` - - The type of session to create. Always `transcription` for transcription sessions. - - - `"transcription"` - - - `audio: optional RealtimeTranscriptionSessionAudio` - - Configuration for input and output audio. - - - `input: optional RealtimeTranscriptionSessionAudioInput` - - - `format: optional RealtimeAudioFormats` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - - `noise_reduction: optional object { type }` - - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. - - - `type: optional NoiseReductionType` - - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - - - `"near_field"` - - - `"far_field"` - - - `transcription: optional AudioTranscription` - - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. - - - `language: optional string` - - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. - - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `string` - - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - - `"whisper-1"` - - - `"gpt-4o-mini-transcribe"` - - - `"gpt-4o-mini-transcribe-2025-12-15"` - - - `"gpt-4o-transcribe"` - - - `"gpt-4o-transcribe-diarize"` - - - `prompt: optional string` - - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - - `turn_detection: optional RealtimeTranscriptionSessionAudioInputTurnDetection` - - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. - - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. - - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` - - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. + - `"transcription"` - - `type: "server_vad"` + - `audio: optional object { input }` - Type of turn detection, `server_vad` to turn on simple Server VAD. + Configuration for input audio for the session. - - `"server_vad"` + - `input: optional object { format, noise_reduction, transcription, turn_detection }` - - `create_response: optional boolean` + - `format: optional RealtimeAudioFormats` - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + The PCM audio format. Only a 24kHz sample rate is supported. - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + - `noise_reduction: optional object { type }` - - `idle_timeout_ms: optional number` + Configuration for input audio noise reduction. - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. + - `type: optional NoiseReductionType` - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. + - `transcription: optional AudioTranscription` - - `interrupt_response: optional boolean` + Configuration of the transcription model. - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + - `turn_detection: optional RealtimeTranscriptionSessionTurnDetection` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + Configuration for turn detection. Can be set to `null` to turn off. Server + VAD means that the model will detect the start and end of speech based on + audio volume and respond at the end of user speech. - `prefix_padding_ms: optional number` - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms. - `silence_duration_ms: optional number` - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user. - `threshold: optional number` - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `type: optional string` - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. + Type of turn detection, only `server_vad` is currently supported. - - `type: "semantic_vad"` + - `expires_at: optional number` - Type of turn detection, `semantic_vad` to turn on Semantic VAD. + Expiration timestamp for the session, in seconds since epoch. - - `"semantic_vad"` + - `include: optional array of "item.input_audio_transcription.logprobs"` - - `create_response: optional boolean` + Additional fields to include in server outputs. - Whether or not to automatically generate a response when a VAD stop event occurs. + - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - `eagerness: optional "low" or "medium" or "high" or "auto"` + - `"item.input_audio_transcription.logprobs"` - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. +- `value: string` - - `"low"` + The generated client secret value. - - `"medium"` +### Example - - `"high"` +```http +curl https://api.openai.com/v1/realtime/client_secrets \ + -H 'Content-Type: application/json' \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -d '{}' +``` - - `"auto"` +#### Response - - `interrupt_response: optional boolean` +```json +{ + "expires_at": 0, + "session": { + "client_secret": { + "expires_at": 0, + "value": "value" + }, + "type": "realtime", + "audio": { + "input": { + "format": { + "rate": 24000, + "type": "audio/pcm" + }, + "noise_reduction": { + "type": "near_field" + }, + "transcription": { + "language": "language", + "model": "string", + "prompt": "prompt" + }, + "turn_detection": { + "type": "server_vad", + "create_response": true, + "idle_timeout_ms": 5000, + "interrupt_response": true, + "prefix_padding_ms": 0, + "silence_duration_ms": 0, + "threshold": 0 + } + }, + "output": { + "format": { + "rate": 24000, + "type": "audio/pcm" + }, + "speed": 0.25, + "voice": "ash" + } + }, + "include": [ + "item.input_audio_transcription.logprobs" + ], + "instructions": "instructions", + "max_output_tokens": 0, + "model": "string", + "output_modalities": [ + "text" + ], + "prompt": { + "id": "id", + "variables": { + "foo": "string" + }, + "version": "version" + }, + "tool_choice": "none", + "tools": [ + { + "description": "description", + "name": "name", + "parameters": {}, + "type": "function" + } + ], + "tracing": "auto", + "truncation": "auto" + }, + "value": "value" +} +``` - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. +### Example - - `include: optional array of "item.input_audio_transcription.logprobs"` +```http +curl -X POST https://api.openai.com/v1/realtime/client_secrets \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "expires_after": { + "anchor": "created_at", + "seconds": 600 + }, + "session": { + "type": "realtime", + "model": "gpt-realtime", + "instructions": "You are a friendly assistant." + } + }' +``` - Additional fields to include in server outputs. +#### Response - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. +```json +{ + "value": "ek_68af296e8e408191a1120ab6383263c2", + "expires_at": 1756310470, + "session": { + "type": "realtime", + "object": "realtime.session", + "id": "sess_C9CiUVUzUzYIssh3ELY1d", + "model": "gpt-realtime", + "output_modalities": [ + "audio" + ], + "instructions": "You are a friendly assistant.", + "tools": [], + "tool_choice": "auto", + "max_output_tokens": "inf", + "tracing": null, + "truncation": "auto", + "prompt": null, + "expires_at": 0, + "audio": { + "input": { + "format": { + "type": "audio/pcm", + "rate": 24000 + }, + "transcription": null, + "noise_reduction": null, + "turn_detection": { + "type": "server_vad", + } + }, + "output": { + "format": { + "type": "audio/pcm", + "rate": 24000 + }, + "voice": "alloy", + "speed": 1.0 + } + }, + "include": null + } +} +``` - - `"item.input_audio_transcription.logprobs"` +## Domain Types + +### Realtime Session Client Secret + +- `RealtimeSessionClientSecret object { expires_at, value }` + + Ephemeral key returned by the API. -### Returns + - `expires_at: number` -- `expires_at: number` + Timestamp for when the token expires. Currently, all tokens expire + after one minute. - Expiration timestamp for the client secret, in seconds since epoch. + - `value: string` -- `session: RealtimeSessionCreateResponse or RealtimeTranscriptionSessionCreateResponse` + Ephemeral key usable in client environments to authenticate connections to the Realtime API. Use this in client-side environments rather than a standard API token, which should only be used server-side. - The session configuration for either a realtime or transcription session. +### Realtime Session Create Response - - `RealtimeSessionCreateResponse = object { client_secret, type, audio, 10 more }` +- `RealtimeSessionCreateResponse object { client_secret, type, audio, 10 more }` A new Realtime session configuration, with an ephemeral key. Default TTL for keys is one minute. @@ -26709,7 +22227,7 @@ Returns the created client secret and the effective session object. The client s The format of the input audio. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -26725,7 +22243,7 @@ Returns the created client secret and the effective session object. The client s - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -26735,7 +22253,7 @@ Returns the created client secret and the effective session object. The client s - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -26804,7 +22322,7 @@ Returns the created client secret and the effective session object. The client s Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. @@ -26858,7 +22376,7 @@ Returns the created client secret and the effective session object. The client s higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` Server-side semantic turn detection which uses a model to determine when the user has finished speaking. @@ -26895,42 +22413,6 @@ Returns the created client secret and the effective session object. The client s The format of the output audio. - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` - - - `PCMAAudioFormat = object { type }` - - The G.711 A-law format. - - - `type: optional "audio/pcma"` - - The audio format. Always `audio/pcma`. - - - `"audio/pcma"` - - `speed: optional number` The speed of the model's spoken response as a multiple of the original speed. @@ -27073,7 +22555,7 @@ Returns the created client secret and the effective session object. The client s - `string` - - `ResponseInputText = object { text, type }` + - `ResponseInputText object { text, type }` A text input to the model. @@ -27087,7 +22569,7 @@ Returns the created client secret and the effective session object. The client s - `"input_text"` - - `ResponseInputImage = object { detail, type, file_id, image_url }` + - `ResponseInputImage object { detail, type, file_id, image_url }` An image input to the model. Learn about [image inputs](/docs/guides/vision). @@ -27117,7 +22599,7 @@ Returns the created client secret and the effective session object. The client s The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - `ResponseInputFile = object { type, detail, file_data, 3 more }` + - `ResponseInputFile object { type, detail, file_data, 3 more }` A file input to the model. @@ -27177,7 +22659,7 @@ Returns the created client secret and the effective session object. The client s - `"required"` - - `ToolChoiceFunction = object { name, type }` + - `ToolChoiceFunction object { name, type }` Use this option to force the model to call a specific function. @@ -27191,7 +22673,7 @@ Returns the created client secret and the effective session object. The client s - `"function"` - - `ToolChoiceMcp = object { server_label, type, name }` + - `ToolChoiceMcp object { server_label, type, name }` Use this option to force the model to call a specific tool on a remote MCP server. @@ -27213,7 +22695,7 @@ Returns the created client secret and the effective session object. The client s Tools available to the model. - - `RealtimeFunctionTool = object { description, name, parameters, type }` + - `RealtimeFunctionTool object { description, name, parameters, type }` - `description: optional string` @@ -27235,7 +22717,7 @@ Returns the created client secret and the effective session object. The client s - `"function"` - - `McpTool = object { server_label, type, allowed_tools, 7 more }` + - `McpTool object { server_label, type, allowed_tools, 7 more }` Give the model access to additional tools via remote Model Context Protocol (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). @@ -27258,7 +22740,7 @@ Returns the created client secret and the effective session object. The client s A string array of allowed tool names - - `McpToolFilter = object { read_only, tool_names }` + - `McpToolFilter object { read_only, tool_names }` A filter object to specify which tools are allowed. @@ -27324,7 +22806,7 @@ Returns the created client secret and the effective session object. The client s Specify which of the MCP server's tools require approval. - - `McpToolApprovalFilter = object { always, never }` + - `McpToolApprovalFilter object { always, never }` Specify which of the MCP server's tools require approval. Can be `always`, `never`, or a filter object associated with tools @@ -27391,7 +22873,7 @@ Returns the created client secret and the effective session object. The client s - `"auto"` - - `TracingConfiguration = object { group_id, metadata, workflow_name }` + - `TracingConfiguration object { group_id, metadata, workflow_name }` Granular configuration for tracing. @@ -27428,7 +22910,7 @@ Returns the created client secret and the effective session object. The client s - `"disabled"` - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. @@ -27450,7 +22932,9 @@ Returns the created client secret and the effective session object. The client s Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - - `RealtimeTranscriptionSessionCreateResponse = object { id, object, type, 3 more }` +### Realtime Transcription Session Create Response + +- `RealtimeTranscriptionSessionCreateResponse object { id, object, type, 3 more }` A Realtime transcription session configuration object. @@ -27478,7 +22962,7 @@ Returns the created client secret and the effective session object. The client s The PCM audio format. Only a 24kHz sample rate is supported. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -27494,7 +22978,7 @@ Returns the created client secret and the effective session object. The client s - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -27504,7 +22988,7 @@ Returns the created client secret and the effective session object. The client s - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -27602,182 +23086,50 @@ Returns the created client secret and the effective session object. The client s - `"item.input_audio_transcription.logprobs"` -- `value: string` +### Realtime Transcription Session Turn Detection - The generated client secret value. +- `RealtimeTranscriptionSessionTurnDetection object { prefix_padding_ms, silence_duration_ms, threshold, type }` -### Example + Configuration for turn detection. Can be set to `null` to turn off. Server + VAD means that the model will detect the start and end of speech based on + audio volume and respond at the end of user speech. -```http -curl https://api.openai.com/v1/realtime/client_secrets \ - -H 'Content-Type: application/json' \ - -H "Authorization: Bearer $OPENAI_API_KEY" \ - -d '{}' -``` + - `prefix_padding_ms: optional number` -#### Response + Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. -```json -{ - "expires_at": 0, - "session": { - "client_secret": { - "expires_at": 0, - "value": "value" - }, - "type": "realtime", - "audio": { - "input": { - "format": { - "rate": 24000, - "type": "audio/pcm" - }, - "noise_reduction": { - "type": "near_field" - }, - "transcription": { - "language": "language", - "model": "string", - "prompt": "prompt" - }, - "turn_detection": { - "type": "server_vad", - "create_response": true, - "idle_timeout_ms": 5000, - "interrupt_response": true, - "prefix_padding_ms": 0, - "silence_duration_ms": 0, - "threshold": 0 - } - }, - "output": { - "format": { - "rate": 24000, - "type": "audio/pcm" - }, - "speed": 0.25, - "voice": "ash" - } - }, - "include": [ - "item.input_audio_transcription.logprobs" - ], - "instructions": "instructions", - "max_output_tokens": 0, - "model": "string", - "output_modalities": [ - "text" - ], - "prompt": { - "id": "id", - "variables": { - "foo": "string" - }, - "version": "version" - }, - "tool_choice": "none", - "tools": [ - { - "description": "description", - "name": "name", - "parameters": {}, - "type": "function" - } - ], - "tracing": "auto", - "truncation": "auto" - }, - "value": "value" -} -``` + - `silence_duration_ms: optional number` -### Example + Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. -```http -curl -X POST https://api.openai.com/v1/realtime/client_secrets \ - -H "Authorization: Bearer $OPENAI_API_KEY" \ - -H "Content-Type: application/json" \ - -d '{ - "expires_after": { - "anchor": "created_at", - "seconds": 600 - }, - "session": { - "type": "realtime", - "model": "gpt-realtime", - "instructions": "You are a friendly assistant." - } - }' -``` + - `threshold: optional number` -#### Response + Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. -```json -{ - "value": "ek_68af296e8e408191a1120ab6383263c2", - "expires_at": 1756310470, - "session": { - "type": "realtime", - "object": "realtime.session", - "id": "sess_C9CiUVUzUzYIssh3ELY1d", - "model": "gpt-realtime", - "output_modalities": [ - "audio" - ], - "instructions": "You are a friendly assistant.", - "tools": [], - "tool_choice": "auto", - "max_output_tokens": "inf", - "tracing": null, - "truncation": "auto", - "prompt": null, - "expires_at": 0, - "audio": { - "input": { - "format": { - "type": "audio/pcm", - "rate": 24000 - }, - "transcription": null, - "noise_reduction": null, - "turn_detection": { - "type": "server_vad", - } - }, - "output": { - "format": { - "type": "audio/pcm", - "rate": 24000 - }, - "voice": "alloy", - "speed": 1.0 - } - }, - "include": null - } -} -``` + - `type: optional string` -## Domain Types + Type of turn detection, only `server_vad` is currently supported. -### Realtime Session Client Secret +### Client Secret Create Response -- `RealtimeSessionClientSecret = object { expires_at, value }` +- `ClientSecretCreateResponse object { expires_at, session, value }` - Ephemeral key returned by the API. + Response from creating a session and client secret for the Realtime API. - `expires_at: number` - Timestamp for when the token expires. Currently, all tokens expire - after one minute. - - - `value: string` + Expiration timestamp for the client secret, in seconds since epoch. - Ephemeral key usable in client environments to authenticate connections to the Realtime API. Use this in client-side environments rather than a standard API token, which should only be used server-side. + - `session: RealtimeSessionCreateResponse or RealtimeTranscriptionSessionCreateResponse` -### Realtime Session Create Response + The session configuration for either a realtime or transcription session. -- `RealtimeSessionCreateResponse = object { client_secret, type, audio, 10 more }` + - `RealtimeSessionCreateResponse object { client_secret, type, audio, 10 more }` A new Realtime session configuration, with an ephemeral key. Default TTL for keys is one minute. @@ -27811,7 +23163,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ The format of the input audio. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -27827,7 +23179,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -27837,7 +23189,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -27906,7 +23258,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. @@ -27960,7 +23312,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` Server-side semantic turn detection which uses a model to determine when the user has finished speaking. @@ -27970,68 +23322,32 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ - `"semantic_vad"` - - `create_response: optional boolean` - - Whether or not to automatically generate a response when a VAD stop event occurs. - - - `eagerness: optional "low" or "medium" or "high" or "auto"` - - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - - `"low"` - - - `"medium"` - - - `"high"` - - - `"auto"` - - - `interrupt_response: optional boolean` - - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - - `output: optional object { format, speed, voice }` - - - `format: optional RealtimeAudioFormats` - - The format of the output audio. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` + - `create_response: optional boolean` - The audio format. Always `audio/pcm`. + Whether or not to automatically generate a response when a VAD stop event occurs. - - `"audio/pcm"` + - `eagerness: optional "low" or "medium" or "high" or "auto"` - - `PCMUAudioFormat = object { type }` + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - The G.711 μ-law format. + - `"low"` - - `type: optional "audio/pcmu"` + - `"medium"` - The audio format. Always `audio/pcmu`. + - `"high"` - - `"audio/pcmu"` + - `"auto"` - - `PCMAAudioFormat = object { type }` + - `interrupt_response: optional boolean` - The G.711 A-law format. + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - `type: optional "audio/pcma"` + - `output: optional object { format, speed, voice }` - The audio format. Always `audio/pcma`. + - `format: optional RealtimeAudioFormats` - - `"audio/pcma"` + The format of the output audio. - `speed: optional number` @@ -28175,7 +23491,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ - `string` - - `ResponseInputText = object { text, type }` + - `ResponseInputText object { text, type }` A text input to the model. @@ -28189,7 +23505,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ - `"input_text"` - - `ResponseInputImage = object { detail, type, file_id, image_url }` + - `ResponseInputImage object { detail, type, file_id, image_url }` An image input to the model. Learn about [image inputs](/docs/guides/vision). @@ -28219,7 +23535,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - `ResponseInputFile = object { type, detail, file_data, 3 more }` + - `ResponseInputFile object { type, detail, file_data, 3 more }` A file input to the model. @@ -28279,7 +23595,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ - `"required"` - - `ToolChoiceFunction = object { name, type }` + - `ToolChoiceFunction object { name, type }` Use this option to force the model to call a specific function. @@ -28293,7 +23609,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ - `"function"` - - `ToolChoiceMcp = object { server_label, type, name }` + - `ToolChoiceMcp object { server_label, type, name }` Use this option to force the model to call a specific tool on a remote MCP server. @@ -28315,7 +23631,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ Tools available to the model. - - `RealtimeFunctionTool = object { description, name, parameters, type }` + - `RealtimeFunctionTool object { description, name, parameters, type }` - `description: optional string` @@ -28337,7 +23653,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ - `"function"` - - `McpTool = object { server_label, type, allowed_tools, 7 more }` + - `McpTool object { server_label, type, allowed_tools, 7 more }` Give the model access to additional tools via remote Model Context Protocol (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). @@ -28360,7 +23676,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ A string array of allowed tool names - - `McpToolFilter = object { read_only, tool_names }` + - `McpToolFilter object { read_only, tool_names }` A filter object to specify which tools are allowed. @@ -28426,7 +23742,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ Specify which of the MCP server's tools require approval. - - `McpToolApprovalFilter = object { always, never }` + - `McpToolApprovalFilter object { always, never }` Specify which of the MCP server's tools require approval. Can be `always`, `never`, or a filter object associated with tools @@ -28493,7 +23809,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ - `"auto"` - - `TracingConfiguration = object { group_id, metadata, workflow_name }` + - `TracingConfiguration object { group_id, metadata, workflow_name }` Granular configuration for tracing. @@ -28530,7 +23846,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ - `"disabled"` - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. @@ -28552,9 +23868,7 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. -### Realtime Transcription Session Create Response - -- `RealtimeTranscriptionSessionCreateResponse = object { id, object, type, 3 more }` + - `RealtimeTranscriptionSessionCreateResponse object { id, object, type, 3 more }` A Realtime transcription session configuration object. @@ -28562,913 +23876,1191 @@ curl -X POST https://api.openai.com/v1/realtime/client_secrets \ Unique identifier for the session that looks like `sess_1234567890abcdef`. - - `object: string` + - `object: string` + + The object type. Always `realtime.transcription_session`. + + - `type: "transcription"` + + The type of session. Always `transcription` for transcription sessions. + + - `"transcription"` + + - `audio: optional object { input }` + + Configuration for input audio for the session. + + - `input: optional object { format, noise_reduction, transcription, turn_detection }` + + - `format: optional RealtimeAudioFormats` + + The PCM audio format. Only a 24kHz sample rate is supported. + + - `noise_reduction: optional object { type }` + + Configuration for input audio noise reduction. + + - `type: optional NoiseReductionType` + + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + + - `transcription: optional AudioTranscription` + + Configuration of the transcription model. + + - `turn_detection: optional RealtimeTranscriptionSessionTurnDetection` + + Configuration for turn detection. Can be set to `null` to turn off. Server + VAD means that the model will detect the start and end of speech based on + audio volume and respond at the end of user speech. + + - `prefix_padding_ms: optional number` + + Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. + + - `silence_duration_ms: optional number` + + Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. + + - `threshold: optional number` + + Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. + + - `type: optional string` + + Type of turn detection, only `server_vad` is currently supported. + + - `expires_at: optional number` + + Expiration timestamp for the session, in seconds since epoch. + + - `include: optional array of "item.input_audio_transcription.logprobs"` + + Additional fields to include in server outputs. + + - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + + - `"item.input_audio_transcription.logprobs"` + + - `value: string` + + The generated client secret value. + +# Calls + +## Accept call + +**post** `/realtime/calls/{call_id}/accept` + +Accept an incoming SIP call and configure the realtime session that will +handle it. + +### Path Parameters + +- `call_id: string` + +### Body Parameters + +- `type: "realtime"` + + The type of session to create. Always `realtime` for the Realtime API. + + - `"realtime"` + +- `audio: optional RealtimeAudioConfig` + + Configuration for input and output audio. + + - `input: optional RealtimeAudioConfigInput` + + - `format: optional RealtimeAudioFormats` + + The format of the input audio. + + - `PCMAudioFormat object { rate, type }` + + The PCM audio format. Only a 24kHz sample rate is supported. + + - `rate: optional 24000` + + The sample rate of the audio. Always `24000`. + + - `24000` + + - `type: optional "audio/pcm"` + + The audio format. Always `audio/pcm`. + + - `"audio/pcm"` + + - `PCMUAudioFormat object { type }` + + The G.711 μ-law format. + + - `type: optional "audio/pcmu"` + + The audio format. Always `audio/pcmu`. + + - `"audio/pcmu"` + + - `PCMAAudioFormat object { type }` + + The G.711 A-law format. + + - `type: optional "audio/pcma"` + + The audio format. Always `audio/pcma`. + + - `"audio/pcma"` + + - `noise_reduction: optional object { type }` + + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + + - `type: optional NoiseReductionType` + + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + + - `"near_field"` + + - `"far_field"` + + - `transcription: optional AudioTranscription` + + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + + - `language: optional string` + + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. + + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + + - `string` + + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + + - `"whisper-1"` + + - `"gpt-4o-mini-transcribe"` + + - `"gpt-4o-mini-transcribe-2025-12-15"` + + - `"gpt-4o-transcribe"` + + - `"gpt-4o-transcribe-diarize"` + + - `prompt: optional string` + + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + + - `turn_detection: optional RealtimeAudioInputTurnDetection` + + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + + - `ServerVad object { type, create_response, idle_timeout_ms, 4 more }` + + Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. + + - `type: "server_vad"` + + Type of turn detection, `server_vad` to turn on simple Server VAD. + + - `"server_vad"` + + - `create_response: optional boolean` + + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + + - `idle_timeout_ms: optional number` + + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. + + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. + + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. + + - `interrupt_response: optional boolean` + + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. - The object type. Always `realtime.transcription_session`. + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. - - `type: "transcription"` + - `prefix_padding_ms: optional number` - The type of session. Always `transcription` for transcription sessions. + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. - - `"transcription"` + - `silence_duration_ms: optional number` - - `audio: optional object { input }` + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. - Configuration for input audio for the session. + - `threshold: optional number` - - `input: optional object { format, noise_reduction, transcription, turn_detection }` + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. - - `format: optional RealtimeAudioFormats` + - `SemanticVad object { type, create_response, eagerness, interrupt_response }` - The PCM audio format. Only a 24kHz sample rate is supported. + Server-side semantic turn detection which uses a model to determine when the user has finished speaking. - - `PCMAudioFormat = object { rate, type }` + - `type: "semantic_vad"` - The PCM audio format. Only a 24kHz sample rate is supported. + Type of turn detection, `semantic_vad` to turn on Semantic VAD. - - `rate: optional 24000` + - `"semantic_vad"` - The sample rate of the audio. Always `24000`. + - `create_response: optional boolean` - - `24000` + Whether or not to automatically generate a response when a VAD stop event occurs. - - `type: optional "audio/pcm"` + - `eagerness: optional "low" or "medium" or "high" or "auto"` - The audio format. Always `audio/pcm`. + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. - - `"audio/pcm"` + - `"low"` - - `PCMUAudioFormat = object { type }` + - `"medium"` - The G.711 μ-law format. + - `"high"` - - `type: optional "audio/pcmu"` + - `"auto"` - The audio format. Always `audio/pcmu`. + - `interrupt_response: optional boolean` - - `"audio/pcmu"` + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. - - `PCMAAudioFormat = object { type }` + - `output: optional RealtimeAudioConfigOutput` - The G.711 A-law format. + - `format: optional RealtimeAudioFormats` - - `type: optional "audio/pcma"` + The format of the output audio. - The audio format. Always `audio/pcma`. + - `speed: optional number` - - `"audio/pcma"` + The speed of the model's spoken response as a multiple of the original speed. + 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. - - `noise_reduction: optional object { type }` + This parameter is a post-processing adjustment to the audio after it is generated, it's + also possible to prompt the model to speak faster or slower. - Configuration for input audio noise reduction. + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - - `type: optional NoiseReductionType` + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + - `string` - - `"near_field"` + - `"alloy" or "ash" or "ballad" or 7 more` - - `"far_field"` + - `"alloy"` - - `transcription: optional AudioTranscription` + - `"ash"` - Configuration of the transcription model. + - `"ballad"` - - `language: optional string` + - `"coral"` - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. + - `"echo"` - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `"sage"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `"shimmer"` - - `string` + - `"verse"` - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `"marin"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `"cedar"` - - `"whisper-1"` + - `ID object { id }` - - `"gpt-4o-mini-transcribe"` + Custom voice reference. - - `"gpt-4o-mini-transcribe-2025-12-15"` + - `id: string` - - `"gpt-4o-transcribe"` + The custom voice ID, e.g. `voice_1234`. - - `"gpt-4o-transcribe-diarize"` +- `include: optional array of "item.input_audio_transcription.logprobs"` - - `prompt: optional string` + Additional fields to include in server outputs. - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - - `turn_detection: optional RealtimeTranscriptionSessionTurnDetection` + - `"item.input_audio_transcription.logprobs"` - Configuration for turn detection. Can be set to `null` to turn off. Server - VAD means that the model will detect the start and end of speech based on - audio volume and respond at the end of user speech. +- `instructions: optional string` - - `prefix_padding_ms: optional number` + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - `silence_duration_ms: optional number` +- `max_output_tokens: optional number or "inf"` - Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - - `threshold: optional number` + - `number` - Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. + - `"inf"` - - `type: optional string` + - `"inf"` - Type of turn detection, only `server_vad` is currently supported. +- `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - - `expires_at: optional number` + The Realtime model used for this session. - Expiration timestamp for the session, in seconds since epoch. + - `string` - - `include: optional array of "item.input_audio_transcription.logprobs"` + - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` - Additional fields to include in server outputs. + The Realtime model used for this session. - - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + - `"gpt-realtime"` - - `"item.input_audio_transcription.logprobs"` + - `"gpt-realtime-1.5"` -### Realtime Transcription Session Turn Detection + - `"gpt-realtime-2025-08-28"` -- `RealtimeTranscriptionSessionTurnDetection = object { prefix_padding_ms, silence_duration_ms, threshold, type }` + - `"gpt-4o-realtime-preview"` - Configuration for turn detection. Can be set to `null` to turn off. Server - VAD means that the model will detect the start and end of speech based on - audio volume and respond at the end of user speech. + - `"gpt-4o-realtime-preview-2024-10-01"` - - `prefix_padding_ms: optional number` + - `"gpt-4o-realtime-preview-2024-12-17"` - Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. + - `"gpt-4o-realtime-preview-2025-06-03"` - - `silence_duration_ms: optional number` + - `"gpt-4o-mini-realtime-preview"` - Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. + - `"gpt-4o-mini-realtime-preview-2024-12-17"` - - `threshold: optional number` + - `"gpt-realtime-mini"` - Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. + - `"gpt-realtime-mini-2025-10-06"` - - `type: optional string` + - `"gpt-realtime-mini-2025-12-15"` - Type of turn detection, only `server_vad` is currently supported. + - `"gpt-audio-1.5"` -# Calls + - `"gpt-audio-mini"` -## Accept call + - `"gpt-audio-mini-2025-10-06"` -**post** `/realtime/calls/{call_id}/accept` + - `"gpt-audio-mini-2025-12-15"` -Accept an incoming SIP call and configure the realtime session that will -handle it. +- `output_modalities: optional array of "text" or "audio"` -### Path Parameters + The set of modalities the model can respond with. It defaults to `["audio"]`, indicating + that the model will respond with audio plus a transcript. `["text"]` can be used to make + the model respond with text only. It is not possible to request both `text` and `audio` at the same time. -- `call_id: string` + - `"text"` -### Body Parameters + - `"audio"` -- `type: "realtime"` +- `prompt: optional ResponsePrompt` - The type of session to create. Always `realtime` for the Realtime API. + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - - `"realtime"` + - `id: string` -- `audio: optional RealtimeAudioConfig` + The unique identifier of the prompt template to use. - Configuration for input and output audio. + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - `input: optional RealtimeAudioConfigInput` + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - - `format: optional RealtimeAudioFormats` + - `string` - The format of the input audio. + - `ResponseInputText object { text, type }` - - `PCMAudioFormat = object { rate, type }` + A text input to the model. - The PCM audio format. Only a 24kHz sample rate is supported. + - `text: string` - - `rate: optional 24000` + The text input to the model. - The sample rate of the audio. Always `24000`. + - `type: "input_text"` - - `24000` + The type of the input item. Always `input_text`. - - `type: optional "audio/pcm"` + - `"input_text"` - The audio format. Always `audio/pcm`. + - `ResponseInputImage object { detail, type, file_id, image_url }` - - `"audio/pcm"` + An image input to the model. Learn about [image inputs](/docs/guides/vision). - - `PCMUAudioFormat = object { type }` + - `detail: "low" or "high" or "auto" or "original"` - The G.711 μ-law format. + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - `type: optional "audio/pcmu"` + - `"low"` - The audio format. Always `audio/pcmu`. + - `"high"` - - `"audio/pcmu"` + - `"auto"` - - `PCMAAudioFormat = object { type }` + - `"original"` - The G.711 A-law format. + - `type: "input_image"` - - `type: optional "audio/pcma"` + The type of the input item. Always `input_image`. - The audio format. Always `audio/pcma`. + - `"input_image"` - - `"audio/pcma"` + - `file_id: optional string` - - `noise_reduction: optional object { type }` + The ID of the file to be sent to the model. - Configuration for input audio noise reduction. This can be set to `null` to turn off. - Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. - Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + - `image_url: optional string` - - `type: optional NoiseReductionType` + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + - `ResponseInputFile object { type, detail, file_data, 3 more }` - - `"near_field"` + A file input to the model. - - `"far_field"` + - `type: "input_file"` - - `transcription: optional AudioTranscription` + The type of the input item. Always `input_file`. - Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + - `"input_file"` - - `language: optional string` + - `detail: optional "low" or "high"` - The language of the input audio. Supplying the input language in - [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format - will improve accuracy and latency. + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + - `"low"` - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `"high"` - - `string` + - `file_data: optional string` - - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + The content of the file to be sent to the model. - The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + - `file_id: optional string` - - `"whisper-1"` + The ID of the file to be sent to the model. - - `"gpt-4o-mini-transcribe"` + - `file_url: optional string` - - `"gpt-4o-mini-transcribe-2025-12-15"` + The URL of the file to be sent to the model. - - `"gpt-4o-transcribe"` + - `filename: optional string` - - `"gpt-4o-transcribe-diarize"` + The name of the file to be sent to the model. - - `prompt: optional string` + - `version: optional string` - An optional text to guide the model's style or continue a previous audio - segment. - For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). - For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + Optional version of the prompt template. - - `turn_detection: optional RealtimeAudioInputTurnDetection` +- `tool_choice: optional RealtimeToolChoiceConfig` - Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. - Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + - `ToolChoiceOptions = "none" or "auto" or "required"` - Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + Controls which (if any) tool is called by the model. - - `ServerVad = object { type, create_response, idle_timeout_ms, 4 more }` + `none` means the model will not call any tool and instead generates a message. - Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence. + `auto` means the model can pick between generating a message or calling one or + more tools. - - `type: "server_vad"` + `required` means the model must call one or more tools. - Type of turn detection, `server_vad` to turn on simple Server VAD. + - `"none"` - - `"server_vad"` + - `"auto"` - - `create_response: optional boolean` + - `"required"` - Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + - `ToolChoiceFunction object { name, type }` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + Use this option to force the model to call a specific function. - - `idle_timeout_ms: optional number` + - `name: string` - Optional timeout after which a model response will be triggered automatically. This is - useful for situations in which a long pause from the user is unexpected, such as a phone - call. The model will effectively prompt the user to continue the conversation based - on the current context. + The name of the function to call. - The timeout value will be applied after the last model response's audio has finished playing, - i.e. it's set to the `response.done` time plus audio playback duration. + - `type: "function"` - An `input_audio_buffer.timeout_triggered` event (plus events - associated with the Response) will be emitted when the timeout is reached. - Idle timeout is currently only supported for `server_vad` mode. + For function calling, the type is always `function`. - - `interrupt_response: optional boolean` + - `"function"` - Whether or not to automatically interrupt (cancel) any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + - `ToolChoiceMcp object { server_label, type, name }` - If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + Use this option to force the model to call a specific tool on a remote MCP server. - - `prefix_padding_ms: optional number` + - `server_label: string` - Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in - milliseconds). Defaults to 300ms. + The label of the MCP server to use. - - `silence_duration_ms: optional number` + - `type: "mcp"` - Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults - to 500ms. With shorter values the model will respond more quickly, - but may jump in on short pauses from the user. + For MCP tools, the type is always `mcp`. - - `threshold: optional number` + - `"mcp"` - Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A - higher threshold will require louder audio to activate the model, and - thus might perform better in noisy environments. + - `name: optional string` - - `SemanticVad = object { type, create_response, eagerness, interrupt_response }` + The name of the tool to call on the server. - Server-side semantic turn detection which uses a model to determine when the user has finished speaking. +- `tools: optional RealtimeToolsConfig` - - `type: "semantic_vad"` + Tools available to the model. - Type of turn detection, `semantic_vad` to turn on Semantic VAD. + - `RealtimeFunctionTool object { description, name, parameters, type }` - - `"semantic_vad"` + - `description: optional string` - - `create_response: optional boolean` + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - Whether or not to automatically generate a response when a VAD stop event occurs. + - `name: optional string` - - `eagerness: optional "low" or "medium" or "high" or "auto"` + The name of the function. - Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + - `parameters: optional unknown` - - `"low"` + Parameters of the function in JSON Schema. - - `"medium"` + - `type: optional "function"` - - `"high"` + The type of the tool, i.e. `function`. - - `"auto"` + - `"function"` - - `interrupt_response: optional boolean` + - `McpTool object { server_label, type, allowed_tools, 7 more }` - Whether or not to automatically interrupt any ongoing response with output to the default - conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). - - `output: optional RealtimeAudioConfigOutput` + - `server_label: string` - - `format: optional RealtimeAudioFormats` + A label for this MCP server, used to identify it in tool calls. - The format of the output audio. + - `type: "mcp"` - - `PCMAudioFormat = object { rate, type }` + The type of the MCP tool. Always `mcp`. - The PCM audio format. Only a 24kHz sample rate is supported. + - `"mcp"` - - `rate: optional 24000` + - `allowed_tools: optional array of string or object { read_only, tool_names }` - The sample rate of the audio. Always `24000`. + List of allowed tool names or a filter object. - - `24000` + - `McpAllowedTools = array of string` - - `type: optional "audio/pcm"` + A string array of allowed tool names - The audio format. Always `audio/pcm`. + - `McpToolFilter object { read_only, tool_names }` - - `"audio/pcm"` + A filter object to specify which tools are allowed. - - `PCMUAudioFormat = object { type }` + - `read_only: optional boolean` - The G.711 μ-law format. + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - - `type: optional "audio/pcmu"` + - `tool_names: optional array of string` - The audio format. Always `audio/pcmu`. + List of allowed tool names. - - `"audio/pcmu"` + - `authorization: optional string` - - `PCMAAudioFormat = object { type }` + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. - The G.711 A-law format. + - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` - - `type: optional "audio/pcma"` + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](/docs/guides/tools-remote-mcp#connectors). - The audio format. Always `audio/pcma`. + Currently supported `connector_id` values are: - - `"audio/pcma"` + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` - - `speed: optional number` + - `"connector_dropbox"` - The speed of the model's spoken response as a multiple of the original speed. - 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. + - `"connector_gmail"` - This parameter is a post-processing adjustment to the audio after it is generated, it's - also possible to prompt the model to speak faster or slower. + - `"connector_googlecalendar"` - - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` + - `"connector_googledrive"` - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with - an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed - during the session once the model has responded with audio at least once. - We recommend `marin` and `cedar` for best quality. + - `"connector_microsoftteams"` - - `string` + - `"connector_outlookcalendar"` - - `"alloy" or "ash" or "ballad" or 7 more` + - `"connector_outlookemail"` - - `"alloy"` + - `"connector_sharepoint"` - - `"ash"` + - `defer_loading: optional boolean` - - `"ballad"` + Whether this MCP tool is deferred and discovered via tool search. - - `"coral"` + - `headers: optional map[string]` - - `"echo"` + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. - - `"sage"` + - `require_approval: optional object { always, never } or "always" or "never"` - - `"shimmer"` + Specify which of the MCP server's tools require approval. - - `"verse"` + - `McpToolApprovalFilter object { always, never }` - - `"marin"` + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. - - `"cedar"` + - `always: optional object { read_only, tool_names }` - - `ID = object { id }` + A filter object to specify which tools are allowed. - Custom voice reference. + - `read_only: optional boolean` - - `id: string` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - The custom voice ID, e.g. `voice_1234`. + - `tool_names: optional array of string` -- `include: optional array of "item.input_audio_transcription.logprobs"` + List of allowed tool names. - Additional fields to include in server outputs. + - `never: optional object { read_only, tool_names }` - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + A filter object to specify which tools are allowed. - - `"item.input_audio_transcription.logprobs"` + - `read_only: optional boolean` -- `instructions: optional string` + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. + - `tool_names: optional array of string` + + List of allowed tool names. - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. + - `McpToolApprovalSetting = "always" or "never"` -- `max_output_tokens: optional number or "inf"` + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. + - `"always"` - - `number` + - `"never"` - - `"inf"` + - `server_description: optional string` - - `"inf"` + Optional description of the MCP server, used to provide more context. -- `model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` + - `server_url: optional string` - The Realtime model used for this session. + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. - - `string` +- `tracing: optional RealtimeTracingConfig` - - `"gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2025-08-28" or 13 more` + Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once + tracing is enabled for a session, the configuration cannot be modified. - The Realtime model used for this session. + `auto` will create a trace for the session with default values for the + workflow name, group id, and metadata. - - `"gpt-realtime"` + - `Auto = "auto"` - - `"gpt-realtime-1.5"` + Enables tracing and sets default values for tracing configuration options. Always `auto`. - - `"gpt-realtime-2025-08-28"` + - `"auto"` - - `"gpt-4o-realtime-preview"` + - `TracingConfiguration object { group_id, metadata, workflow_name }` - - `"gpt-4o-realtime-preview-2024-10-01"` + Granular configuration for tracing. - - `"gpt-4o-realtime-preview-2024-12-17"` + - `group_id: optional string` - - `"gpt-4o-realtime-preview-2025-06-03"` + The group id to attach to this trace to enable filtering and + grouping in the Traces Dashboard. - - `"gpt-4o-mini-realtime-preview"` + - `metadata: optional unknown` - - `"gpt-4o-mini-realtime-preview-2024-12-17"` + The arbitrary metadata to attach to this trace to enable + filtering in the Traces Dashboard. - - `"gpt-realtime-mini"` + - `workflow_name: optional string` - - `"gpt-realtime-mini-2025-10-06"` + The name of the workflow to attach to this trace. This is used to + name the trace in the Traces Dashboard. - - `"gpt-realtime-mini-2025-12-15"` +- `truncation: optional RealtimeTruncation` - - `"gpt-audio-1.5"` + When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - - `"gpt-audio-mini"` + Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - - `"gpt-audio-mini-2025-10-06"` + Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - `"gpt-audio-mini-2025-12-15"` + Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. -- `output_modalities: optional array of "text" or "audio"` + - `"auto" or "disabled"` - The set of modalities the model can respond with. It defaults to `["audio"]`, indicating - that the model will respond with audio plus a transcript. `["text"]` can be used to make - the model respond with text only. It is not possible to request both `text` and `audio` at the same time. + The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - - `"text"` + - `"auto"` - - `"audio"` + - `"disabled"` -- `prompt: optional ResponsePrompt` + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). + Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - - `id: string` + - `retention_ratio: number` - The unique identifier of the prompt template to use. + Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` + - `type: "retention_ratio"` - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. + Use retention ratio truncation. - - `string` + - `"retention_ratio"` - - `ResponseInputText = object { text, type }` + - `token_limits: optional object { post_instructions }` - A text input to the model. + Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - - `text: string` + - `post_instructions: optional number` - The text input to the model. + Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - - `type: "input_text"` +### Example - The type of the input item. Always `input_text`. +```http +curl https://api.openai.com/v1/realtime/calls/$CALL_ID/accept \ + -H 'Content-Type: application/json' \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -d '{ + "type": "realtime" + }' +``` - - `"input_text"` +### Example - - `ResponseInputImage = object { detail, type, file_id, image_url }` +```http +curl -X POST https://api.openai.com/v1/realtime/calls/$CALL_ID/accept \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "type": "realtime", + "model": "gpt-realtime", + "instructions": "You are Alex, a friendly concierge for Example Corp.", + }' +``` - An image input to the model. Learn about [image inputs](/docs/guides/vision). +## Hang up call - - `detail: "low" or "high" or "auto" or "original"` +**post** `/realtime/calls/{call_id}/hangup` - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. +End an active Realtime API call, whether it was initiated over SIP or +WebRTC. - - `"low"` +### Path Parameters - - `"high"` +- `call_id: string` - - `"auto"` +### Example - - `"original"` +```http +curl https://api.openai.com/v1/realtime/calls/$CALL_ID/hangup \ + -X POST \ + -H "Authorization: Bearer $OPENAI_API_KEY" +``` - - `type: "input_image"` +### Example - The type of the input item. Always `input_image`. +```http +curl -X POST https://api.openai.com/v1/realtime/calls/$CALL_ID/hangup \ + -H "Authorization: Bearer $OPENAI_API_KEY" +``` - - `"input_image"` +## Refer call - - `file_id: optional string` +**post** `/realtime/calls/{call_id}/refer` - The ID of the file to be sent to the model. +Transfer an active SIP call to a new destination using the SIP REFER verb. - - `image_url: optional string` +### Path Parameters - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. +- `call_id: string` - - `ResponseInputFile = object { type, detail, file_data, 3 more }` +### Body Parameters - A file input to the model. +- `target_uri: string` - - `type: "input_file"` + URI that should appear in the SIP Refer-To header. Supports values like + `tel:+14155550123` or `sip:agent@example.com`. - The type of the input item. Always `input_file`. +### Example - - `"input_file"` +```http +curl https://api.openai.com/v1/realtime/calls/$CALL_ID/refer \ + -H 'Content-Type: application/json' \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -d '{ + "target_uri": "tel:+14155550123" + }' +``` - - `detail: optional "low" or "high"` +### Example - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. +```http +curl -X POST https://api.openai.com/v1/realtime/calls/$CALL_ID/refer \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"target_uri": "tel:+14155550123"}' +``` - - `"low"` +## Reject call - - `"high"` +**post** `/realtime/calls/{call_id}/reject` - - `file_data: optional string` +Decline an incoming SIP call by returning a SIP status code to the caller. - The content of the file to be sent to the model. +### Path Parameters - - `file_id: optional string` +- `call_id: string` - The ID of the file to be sent to the model. +### Body Parameters - - `file_url: optional string` +- `status_code: optional number` - The URL of the file to be sent to the model. + SIP response code to send back to the caller. Defaults to `603` (Decline) + when omitted. - - `filename: optional string` +### Example - The name of the file to be sent to the model. +```http +curl https://api.openai.com/v1/realtime/calls/$CALL_ID/reject \ + -X POST \ + -H "Authorization: Bearer $OPENAI_API_KEY" +``` - - `version: optional string` +### Example - Optional version of the prompt template. +```http +curl -X POST https://api.openai.com/v1/realtime/calls/$CALL_ID/reject \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"status_code": 486}' +``` -- `tool_choice: optional RealtimeToolChoiceConfig` +# Sessions - How the model chooses tools. Provide one of the string modes or force a specific - function/MCP tool. +## Create session - - `ToolChoiceOptions = "none" or "auto" or "required"` +**post** `/realtime/sessions` - Controls which (if any) tool is called by the model. +Create an ephemeral API token for use in client-side applications with the +Realtime API. Can be configured with the same session parameters as the +`session.update` client event. - `none` means the model will not call any tool and instead generates a message. +It responds with a session object, plus a `client_secret` key which contains +a usable ephemeral API token that can be used to authenticate browser clients +for the Realtime API. - `auto` means the model can pick between generating a message or calling one or - more tools. +Returns the created Realtime session object, plus an ephemeral key. - `required` means the model must call one or more tools. +### Body Parameters - - `"none"` +- `client_secret: object { expires_at, value }` - - `"auto"` + Ephemeral key returned by the API. - - `"required"` + - `expires_at: number` - - `ToolChoiceFunction = object { name, type }` + Timestamp for when the token expires. Currently, all tokens expire + after one minute. - Use this option to force the model to call a specific function. + - `value: string` - - `name: string` + Ephemeral key usable in client environments to authenticate connections + to the Realtime API. Use this in client-side environments rather than + a standard API token, which should only be used server-side. - The name of the function to call. +- `input_audio_format: optional string` - - `type: "function"` + The format of input audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. - For function calling, the type is always `function`. +- `input_audio_transcription: optional object { model }` - - `"function"` + Configuration for input audio transcription, defaults to off and can be + set to `null` to turn off once on. Input audio transcription is not native + to the model, since the model consumes audio directly. Transcription runs + asynchronously and should be treated as rough guidance + rather than the representation understood by the model. - - `ToolChoiceMcp = object { server_label, type, name }` + - `model: optional string` - Use this option to force the model to call a specific tool on a remote MCP server. + The model to use for transcription. - - `server_label: string` +- `instructions: optional string` - The label of the MCP server to use. + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. - - `type: "mcp"` +- `max_response_output_tokens: optional number or "inf"` - For MCP tools, the type is always `mcp`. + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - - `"mcp"` + - `number` - - `name: optional string` + - `"inf"` - The name of the tool to call on the server. + - `"inf"` -- `tools: optional RealtimeToolsConfig` +- `modalities: optional array of "text" or "audio"` - Tools available to the model. + The set of modalities the model can respond with. To disable audio, + set this to ["text"]. - - `RealtimeFunctionTool = object { description, name, parameters, type }` + - `"text"` - - `description: optional string` + - `"audio"` - The description of the function, including guidance on when and how - to call it, and guidance about what to tell the user when calling - (if anything). +- `output_audio_format: optional string` - - `name: optional string` + The format of output audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. - The name of the function. +- `prompt: optional ResponsePrompt` - - `parameters: optional unknown` + Reference to a prompt template and its variables. + [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). - Parameters of the function in JSON Schema. + - `id: string` - - `type: optional "function"` + The unique identifier of the prompt template to use. - The type of the tool, i.e. `function`. + - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` - - `"function"` + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. - - `McpTool = object { server_label, type, allowed_tools, 7 more }` + - `string` - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp). + - `ResponseInputText object { text, type }` - - `server_label: string` + A text input to the model. - A label for this MCP server, used to identify it in tool calls. + - `text: string` - - `type: "mcp"` + The text input to the model. - The type of the MCP tool. Always `mcp`. + - `type: "input_text"` - - `"mcp"` + The type of the input item. Always `input_text`. - - `allowed_tools: optional array of string or object { read_only, tool_names }` + - `"input_text"` - List of allowed tool names or a filter object. + - `ResponseInputImage object { detail, type, file_id, image_url }` - - `McpAllowedTools = array of string` + An image input to the model. Learn about [image inputs](/docs/guides/vision). - A string array of allowed tool names + - `detail: "low" or "high" or "auto" or "original"` - - `McpToolFilter = object { read_only, tool_names }` + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - A filter object to specify which tools are allowed. + - `"low"` - - `read_only: optional boolean` + - `"high"` - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. + - `"auto"` - - `tool_names: optional array of string` + - `"original"` - List of allowed tool names. + - `type: "input_image"` - - `authorization: optional string` + The type of the input item. Always `input_image`. - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. + - `"input_image"` - - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more` + - `file_id: optional string` - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](/docs/guides/tools-remote-mcp#connectors). + The ID of the file to be sent to the model. - Currently supported `connector_id` values are: + - `image_url: optional string` - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - `"connector_dropbox"` + - `ResponseInputFile object { type, detail, file_data, 3 more }` - - `"connector_gmail"` + A file input to the model. - - `"connector_googlecalendar"` + - `type: "input_file"` - - `"connector_googledrive"` + The type of the input item. Always `input_file`. - - `"connector_microsoftteams"` + - `"input_file"` - - `"connector_outlookcalendar"` + - `detail: optional "low" or "high"` - - `"connector_outlookemail"` + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - `"connector_sharepoint"` + - `"low"` - - `defer_loading: optional boolean` + - `"high"` - Whether this MCP tool is deferred and discovered via tool search. + - `file_data: optional string` - - `headers: optional map[string]` + The content of the file to be sent to the model. - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. + - `file_id: optional string` - - `require_approval: optional object { always, never } or "always" or "never"` + The ID of the file to be sent to the model. - Specify which of the MCP server's tools require approval. + - `file_url: optional string` - - `McpToolApprovalFilter = object { always, never }` + The URL of the file to be sent to the model. - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. + - `filename: optional string` - - `always: optional object { read_only, tool_names }` + The name of the file to be sent to the model. - A filter object to specify which tools are allowed. + - `version: optional string` - - `read_only: optional boolean` + Optional version of the prompt template. - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. +- `speed: optional number` - - `tool_names: optional array of string` + The speed of the model's spoken response. 1.0 is the default speed. 0.25 is + the minimum speed. 1.5 is the maximum speed. This value can only be changed + in between model turns, not while a response is in progress. - List of allowed tool names. +- `temperature: optional number` - - `never: optional object { read_only, tool_names }` + Sampling temperature for the model, limited to [0.6, 1.2]. Defaults to 0.8. - A filter object to specify which tools are allowed. +- `tool_choice: optional string` - - `read_only: optional boolean` + How the model chooses tools. Options are `auto`, `none`, `required`, or + specify a function. - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. +- `tools: optional array of object { description, name, parameters, type }` - - `tool_names: optional array of string` + Tools (functions) available to the model. - List of allowed tool names. + - `description: optional string` - - `McpToolApprovalSetting = "always" or "never"` + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. + - `name: optional string` - - `"always"` + The name of the function. - - `"never"` + - `parameters: optional unknown` - - `server_description: optional string` + Parameters of the function in JSON Schema. - Optional description of the MCP server, used to provide more context. + - `type: optional "function"` - - `server_url: optional string` + The type of the tool, i.e. `function`. - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. + - `"function"` -- `tracing: optional RealtimeTracingConfig` +- `tracing: optional "auto" or object { group_id, metadata, workflow_name }` - Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once + Configuration options for tracing. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified. `auto` will create a trace for the session with default values for the workflow name, group id, and metadata. - - `Auto = "auto"` + - `"auto"` - Enables tracing and sets default values for tracing configuration options. Always `auto`. + Default tracing mode for the session. - `"auto"` - - `TracingConfiguration = object { group_id, metadata, workflow_name }` + - `TracingConfiguration object { group_id, metadata, workflow_name }` Granular configuration for tracing. - `group_id: optional string` The group id to attach to this trace to enable filtering and - grouping in the Traces Dashboard. + grouping in the traces dashboard. - `metadata: optional unknown` The arbitrary metadata to attach to this trace to enable - filtering in the Traces Dashboard. + filtering in the traces dashboard. - `workflow_name: optional string` The name of the workflow to attach to this trace. This is used to - name the trace in the Traces Dashboard. + name the trace in the traces dashboard. - `truncation: optional RealtimeTruncation` @@ -29488,7 +25080,7 @@ handle it. - `"disabled"` - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` + - `RetentionRatioTruncation object { retention_ratio, type, token_limits }` Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. @@ -29510,322 +25102,285 @@ handle it. Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. -### Example +- `turn_detection: optional object { prefix_padding_ms, silence_duration_ms, threshold, type }` -```http -curl https://api.openai.com/v1/realtime/calls/$CALL_ID/accept \ - -H 'Content-Type: application/json' \ - -H "Authorization: Bearer $OPENAI_API_KEY" \ - -d '{ - "type": "realtime" - }' -``` + Configuration for turn detection. Can be set to `null` to turn off. Server + VAD means that the model will detect the start and end of speech based on + audio volume and respond at the end of user speech. -### Example + - `prefix_padding_ms: optional number` -```http -curl -X POST https://api.openai.com/v1/realtime/calls/$CALL_ID/accept \ - -H "Authorization: Bearer $OPENAI_API_KEY" \ - -H "Content-Type: application/json" \ - -d '{ - "type": "realtime", - "model": "gpt-realtime", - "instructions": "You are Alex, a friendly concierge for Example Corp.", - }' -``` + Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. -## Hang up call + - `silence_duration_ms: optional number` -**post** `/realtime/calls/{call_id}/hangup` + Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. -End an active Realtime API call, whether it was initiated over SIP or -WebRTC. + - `threshold: optional number` -### Path Parameters + Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. -- `call_id: string` + - `type: optional string` -### Example + Type of turn detection, only `server_vad` is currently supported. + +- `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` -```http -curl https://api.openai.com/v1/realtime/calls/$CALL_ID/hangup \ - -X POST \ - -H "Authorization: Bearer $OPENAI_API_KEY" -``` + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with an + `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed during + the session once the model has responded with audio at least once. -### Example + - `string` -```http -curl -X POST https://api.openai.com/v1/realtime/calls/$CALL_ID/hangup \ - -H "Authorization: Bearer $OPENAI_API_KEY" -``` + - `"alloy" or "ash" or "ballad" or 7 more` -## Refer call + - `"alloy"` -**post** `/realtime/calls/{call_id}/refer` + - `"ash"` -Transfer an active SIP call to a new destination using the SIP REFER verb. + - `"ballad"` -### Path Parameters + - `"coral"` -- `call_id: string` + - `"echo"` -### Body Parameters + - `"sage"` -- `target_uri: string` + - `"shimmer"` - URI that should appear in the SIP Refer-To header. Supports values like - `tel:+14155550123` or `sip:agent@example.com`. + - `"verse"` -### Example + - `"marin"` -```http -curl https://api.openai.com/v1/realtime/calls/$CALL_ID/refer \ - -H 'Content-Type: application/json' \ - -H "Authorization: Bearer $OPENAI_API_KEY" \ - -d '{ - "target_uri": "tel:+14155550123" - }' -``` + - `"cedar"` -### Example + - `ID object { id }` -```http -curl -X POST https://api.openai.com/v1/realtime/calls/$CALL_ID/refer \ - -H "Authorization: Bearer $OPENAI_API_KEY" \ - -H "Content-Type: application/json" \ - -d '{"target_uri": "tel:+14155550123"}' -``` + Custom voice reference. -## Reject call + - `id: string` -**post** `/realtime/calls/{call_id}/reject` + The custom voice ID, e.g. `voice_1234`. -Decline an incoming SIP call by returning a SIP status code to the caller. +### Returns -### Path Parameters +- `id: optional string` -- `call_id: string` + Unique identifier for the session that looks like `sess_1234567890abcdef`. -### Body Parameters +- `audio: optional object { input, output }` -- `status_code: optional number` + Configuration for input and output audio for the session. - SIP response code to send back to the caller. Defaults to `603` (Decline) - when omitted. + - `input: optional object { format, noise_reduction, transcription, turn_detection }` -### Example + - `format: optional RealtimeAudioFormats` -```http -curl https://api.openai.com/v1/realtime/calls/$CALL_ID/reject \ - -X POST \ - -H "Authorization: Bearer $OPENAI_API_KEY" -``` + The PCM audio format. Only a 24kHz sample rate is supported. -### Example + - `PCMAudioFormat object { rate, type }` -```http -curl -X POST https://api.openai.com/v1/realtime/calls/$CALL_ID/reject \ - -H "Authorization: Bearer $OPENAI_API_KEY" \ - -H "Content-Type: application/json" \ - -d '{"status_code": 486}' -``` + The PCM audio format. Only a 24kHz sample rate is supported. -# Sessions + - `rate: optional 24000` -## Create session + The sample rate of the audio. Always `24000`. -**post** `/realtime/sessions` + - `24000` -Create an ephemeral API token for use in client-side applications with the -Realtime API. Can be configured with the same session parameters as the -`session.update` client event. + - `type: optional "audio/pcm"` -It responds with a session object, plus a `client_secret` key which contains -a usable ephemeral API token that can be used to authenticate browser clients -for the Realtime API. + The audio format. Always `audio/pcm`. -Returns the created Realtime session object, plus an ephemeral key. + - `"audio/pcm"` -### Body Parameters + - `PCMUAudioFormat object { type }` -- `client_secret: object { expires_at, value }` + The G.711 μ-law format. - Ephemeral key returned by the API. + - `type: optional "audio/pcmu"` - - `expires_at: number` + The audio format. Always `audio/pcmu`. - Timestamp for when the token expires. Currently, all tokens expire - after one minute. + - `"audio/pcmu"` - - `value: string` + - `PCMAAudioFormat object { type }` - Ephemeral key usable in client environments to authenticate connections - to the Realtime API. Use this in client-side environments rather than - a standard API token, which should only be used server-side. + The G.711 A-law format. -- `input_audio_format: optional string` + - `type: optional "audio/pcma"` - The format of input audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. + The audio format. Always `audio/pcma`. -- `input_audio_transcription: optional object { model }` + - `"audio/pcma"` - Configuration for input audio transcription, defaults to off and can be - set to `null` to turn off once on. Input audio transcription is not native - to the model, since the model consumes audio directly. Transcription runs - asynchronously and should be treated as rough guidance - rather than the representation understood by the model. + - `noise_reduction: optional object { type }` - - `model: optional string` + Configuration for input audio noise reduction. - The model to use for transcription. + - `type: optional NoiseReductionType` -- `instructions: optional string` + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. - The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. - Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. + - `"near_field"` -- `max_response_output_tokens: optional number or "inf"` + - `"far_field"` - Maximum number of output tokens for a single assistant response, - inclusive of tool calls. Provide an integer between 1 and 4096 to - limit output tokens, or `inf` for the maximum available tokens for a - given model. Defaults to `inf`. + - `transcription: optional AudioTranscription` - - `number` + Configuration for input audio transcription. - - `"inf"` + - `language: optional string` - - `"inf"` + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. -- `modalities: optional array of "text" or "audio"` + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` - The set of modalities the model can respond with. To disable audio, - set this to ["text"]. + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - - `"text"` + - `string` - - `"audio"` + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` -- `output_audio_format: optional string` + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. - The format of output audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. + - `"whisper-1"` -- `prompt: optional ResponsePrompt` + - `"gpt-4o-mini-transcribe"` - Reference to a prompt template and its variables. - [Learn more](/docs/guides/text?api-mode=responses#reusable-prompts). + - `"gpt-4o-mini-transcribe-2025-12-15"` - - `id: string` + - `"gpt-4o-transcribe"` - The unique identifier of the prompt template to use. + - `"gpt-4o-transcribe-diarize"` - - `variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile]` + - `prompt: optional string` - Optional map of values to substitute in for variables in your - prompt. The substitution values can either be strings, or other - Response input types like images or files. + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". - - `string` + - `turn_detection: optional object { prefix_padding_ms, silence_duration_ms, threshold, type }` - - `ResponseInputText = object { text, type }` + Configuration for turn detection. - A text input to the model. + - `prefix_padding_ms: optional number` - - `text: string` + - `silence_duration_ms: optional number` - The text input to the model. + - `threshold: optional number` - - `type: "input_text"` + - `type: optional string` - The type of the input item. Always `input_text`. + Type of turn detection, only `server_vad` is currently supported. - - `"input_text"` + - `output: optional object { format, speed, voice }` - - `ResponseInputImage = object { detail, type, file_id, image_url }` + - `format: optional RealtimeAudioFormats` - An image input to the model. Learn about [image inputs](/docs/guides/vision). + The PCM audio format. Only a 24kHz sample rate is supported. - - `detail: "low" or "high" or "auto" or "original"` + - `speed: optional number` - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. + - `voice: optional string or "alloy" or "ash" or "ballad" or 7 more` - - `"low"` + - `string` - - `"high"` + - `"alloy" or "ash" or "ballad" or 7 more` - - `"auto"` + - `"alloy"` - - `"original"` + - `"ash"` - - `type: "input_image"` + - `"ballad"` - The type of the input item. Always `input_image`. + - `"coral"` - - `"input_image"` + - `"echo"` - - `file_id: optional string` + - `"sage"` - The ID of the file to be sent to the model. + - `"shimmer"` - - `image_url: optional string` + - `"verse"` - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. + - `"marin"` - - `ResponseInputFile = object { type, detail, file_data, 3 more }` + - `"cedar"` - A file input to the model. +- `expires_at: optional number` - - `type: "input_file"` + Expiration timestamp for the session, in seconds since epoch. - The type of the input item. Always `input_file`. +- `include: optional array of "item.input_audio_transcription.logprobs"` - - `"input_file"` + Additional fields to include in server outputs. - - `detail: optional "low" or "high"` + - `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. + - `"item.input_audio_transcription.logprobs"` - - `"low"` +- `instructions: optional string` - - `"high"` + The default system instructions (i.e. system message) prepended to model + calls. This field allows the client to guide the model on desired + responses. The model can be instructed on response content and format, + (e.g. "be extremely succinct", "act friendly", "here are examples of good + responses") and on audio behavior (e.g. "talk quickly", "inject emotion + into your voice", "laugh frequently"). The instructions are not guaranteed + to be followed by the model, but they provide guidance to the model on the + desired behavior. - - `file_data: optional string` + Note that the server sets default instructions which will be used if this + field is not set and are visible in the `session.created` event at the + start of the session. - The content of the file to be sent to the model. +- `max_output_tokens: optional number or "inf"` - - `file_id: optional string` + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. - The ID of the file to be sent to the model. + - `number` - - `file_url: optional string` + - `"inf"` - The URL of the file to be sent to the model. + - `"inf"` - - `filename: optional string` +- `model: optional string` - The name of the file to be sent to the model. + The Realtime model used for this session. - - `version: optional string` +- `object: optional string` - Optional version of the prompt template. + The object type. Always `realtime.session`. -- `speed: optional number` +- `output_modalities: optional array of "text" or "audio"` - The speed of the model's spoken response. 1.0 is the default speed. 0.25 is - the minimum speed. 1.5 is the maximum speed. This value can only be changed - in between model turns, not while a response is in progress. + The set of modalities the model can respond with. To disable audio, + set this to ["text"]. -- `temperature: optional number` + - `"text"` - Sampling temperature for the model, limited to [0.6, 1.2]. Defaults to 0.8. + - `"audio"` - `tool_choice: optional string` How the model chooses tools. Options are `auto`, `none`, `required`, or specify a function. -- `tools: optional array of object { description, name, parameters, type }` +- `tools: optional array of RealtimeFunctionTool` Tools (functions) available to the model. @@ -29863,7 +25418,7 @@ Returns the created Realtime session object, plus an ephemeral key. - `"auto"` - - `TracingConfiguration = object { group_id, metadata, workflow_name }` + - `TracingConfiguration object { group_id, metadata, workflow_name }` Granular configuration for tracing. @@ -29882,46 +25437,6 @@ Returns the created Realtime session object, plus an ephemeral key. The name of the workflow to attach to this trace. This is used to name the trace in the traces dashboard. -- `truncation: optional RealtimeTruncation` - - When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. - - Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. - - Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. - - Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. - - - `"auto" or "disabled"` - - The truncation strategy to use for the session. `auto` is the default truncation strategy. `disabled` will disable truncation and emit errors when the conversation exceeds the input token limit. - - - `"auto"` - - - `"disabled"` - - - `RetentionRatioTruncation = object { retention_ratio, type, token_limits }` - - Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. - - - `retention_ratio: number` - - Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. - - - `type: "retention_ratio"` - - Use retention ratio truncation. - - - `"retention_ratio"` - - - `token_limits: optional object { post_instructions }` - - Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. - - - `post_instructions: optional number` - - Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. - - `turn_detection: optional object { prefix_padding_ms, silence_duration_ms, threshold, type }` Configuration for turn detection. Can be set to `null` to turn off. Server @@ -29949,53 +25464,140 @@ Returns the created Realtime session object, plus an ephemeral key. Type of turn detection, only `server_vad` is currently supported. -- `voice: optional string or "alloy" or "ash" or "ballad" or 7 more or object { id }` - - The voice the model uses to respond. Supported built-in voices are - `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, - `marin`, and `cedar`. You may also provide a custom voice object with an - `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed during - the session once the model has responded with audio at least once. - - - `string` - - - `"alloy" or "ash" or "ballad" or 7 more` - - - `"alloy"` - - - `"ash"` - - - `"ballad"` - - - `"coral"` +### Example - - `"echo"` +```http +curl https://api.openai.com/v1/realtime/sessions \ + -H 'Content-Type: application/json' \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -d '{ + "client_secret": { + "expires_at": 0, + "value": "value" + } + }' +``` - - `"sage"` +#### Response - - `"shimmer"` +```json +{ + "id": "id", + "audio": { + "input": { + "format": { + "rate": 24000, + "type": "audio/pcm" + }, + "noise_reduction": { + "type": "near_field" + }, + "transcription": { + "language": "language", + "model": "string", + "prompt": "prompt" + }, + "turn_detection": { + "prefix_padding_ms": 0, + "silence_duration_ms": 0, + "threshold": 0, + "type": "type" + } + }, + "output": { + "format": { + "rate": 24000, + "type": "audio/pcm" + }, + "speed": 0, + "voice": "ash" + } + }, + "expires_at": 0, + "include": [ + "item.input_audio_transcription.logprobs" + ], + "instructions": "instructions", + "max_output_tokens": 0, + "model": "model", + "object": "object", + "output_modalities": [ + "text" + ], + "tool_choice": "tool_choice", + "tools": [ + { + "description": "description", + "name": "name", + "parameters": {}, + "type": "function" + } + ], + "tracing": "auto", + "turn_detection": { + "prefix_padding_ms": 0, + "silence_duration_ms": 0, + "threshold": 0, + "type": "type" + } +} +``` - - `"verse"` +### Example - - `"marin"` +```http +curl -X POST https://api.openai.com/v1/realtime/sessions \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "gpt-realtime", + "modalities": ["audio", "text"], + "instructions": "You are a friendly assistant." + }' +``` - - `"cedar"` +#### Response - - `ID = object { id }` +```json +{ + "id": "sess_001", + "object": "realtime.session", + "model": "gpt-realtime-2025-08-25", + "modalities": ["audio", "text"], + "instructions": "You are a friendly assistant.", + "voice": "alloy", + "input_audio_format": "pcm16", + "output_audio_format": "pcm16", + "input_audio_transcription": { + "model": "whisper-1" + }, + "turn_detection": null, + "tools": [], + "tool_choice": "none", + "temperature": 0.7, + "max_response_output_tokens": 200, + "speed": 1.1, + "tracing": "auto", + "client_secret": { + "value": "ek_abc123", + "expires_at": 1234567890 + } +} +``` - Custom voice reference. +## Domain Types - - `id: string` +### Session Create Response - The custom voice ID, e.g. `voice_1234`. +- `SessionCreateResponse object { id, audio, expires_at, 10 more }` -### Returns + A Realtime session configuration object. -- `id: optional string` + - `id: optional string` Unique identifier for the session that looks like `sess_1234567890abcdef`. -- `audio: optional object { input, output }` + - `audio: optional object { input, output }` Configuration for input and output audio for the session. @@ -30005,7 +25607,7 @@ Returns the created Realtime session object, plus an ephemeral key. The PCM audio format. Only a 24kHz sample rate is supported. - - `PCMAudioFormat = object { rate, type }` + - `PCMAudioFormat object { rate, type }` The PCM audio format. Only a 24kHz sample rate is supported. @@ -30021,7 +25623,7 @@ Returns the created Realtime session object, plus an ephemeral key. - `"audio/pcm"` - - `PCMUAudioFormat = object { type }` + - `PCMUAudioFormat object { type }` The G.711 μ-law format. @@ -30031,7 +25633,7 @@ Returns the created Realtime session object, plus an ephemeral key. - `"audio/pcmu"` - - `PCMAAudioFormat = object { type }` + - `PCMAAudioFormat object { type }` The G.711 A-law format. @@ -30098,53 +25700,17 @@ Returns the created Realtime session object, plus an ephemeral key. - `silence_duration_ms: optional number` - - `threshold: optional number` - - - `type: optional string` - - Type of turn detection, only `server_vad` is currently supported. - - - `output: optional object { format, speed, voice }` - - - `format: optional RealtimeAudioFormats` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `PCMAudioFormat = object { rate, type }` - - The PCM audio format. Only a 24kHz sample rate is supported. - - - `rate: optional 24000` - - The sample rate of the audio. Always `24000`. - - - `24000` - - - `type: optional "audio/pcm"` - - The audio format. Always `audio/pcm`. - - - `"audio/pcm"` - - - `PCMUAudioFormat = object { type }` - - The G.711 μ-law format. - - - `type: optional "audio/pcmu"` - - The audio format. Always `audio/pcmu`. - - - `"audio/pcmu"` + - `threshold: optional number` - - `PCMAAudioFormat = object { type }` + - `type: optional string` - The G.711 A-law format. + Type of turn detection, only `server_vad` is currently supported. - - `type: optional "audio/pcma"` + - `output: optional object { format, speed, voice }` - The audio format. Always `audio/pcma`. + - `format: optional RealtimeAudioFormats` - - `"audio/pcma"` + The PCM audio format. Only a 24kHz sample rate is supported. - `speed: optional number` @@ -30174,11 +25740,11 @@ Returns the created Realtime session object, plus an ephemeral key. - `"cedar"` -- `expires_at: optional number` + - `expires_at: optional number` Expiration timestamp for the session, in seconds since epoch. -- `include: optional array of "item.input_audio_transcription.logprobs"` + - `include: optional array of "item.input_audio_transcription.logprobs"` Additional fields to include in server outputs. @@ -30186,7 +25752,7 @@ Returns the created Realtime session object, plus an ephemeral key. - `"item.input_audio_transcription.logprobs"` -- `instructions: optional string` + - `instructions: optional string` The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired @@ -30201,7 +25767,7 @@ Returns the created Realtime session object, plus an ephemeral key. field is not set and are visible in the `session.created` event at the start of the session. -- `max_output_tokens: optional number or "inf"` + - `max_output_tokens: optional number or "inf"` Maximum number of output tokens for a single assistant response, inclusive of tool calls. Provide an integer between 1 and 4096 to @@ -30214,15 +25780,15 @@ Returns the created Realtime session object, plus an ephemeral key. - `"inf"` -- `model: optional string` + - `model: optional string` The Realtime model used for this session. -- `object: optional string` + - `object: optional string` The object type. Always `realtime.session`. -- `output_modalities: optional array of "text" or "audio"` + - `output_modalities: optional array of "text" or "audio"` The set of modalities the model can respond with. To disable audio, set this to ["text"]. @@ -30231,12 +25797,12 @@ Returns the created Realtime session object, plus an ephemeral key. - `"audio"` -- `tool_choice: optional string` + - `tool_choice: optional string` How the model chooses tools. Options are `auto`, `none`, `required`, or specify a function. -- `tools: optional array of RealtimeFunctionTool` + - `tools: optional array of RealtimeFunctionTool` Tools (functions) available to the model. @@ -30260,7 +25826,7 @@ Returns the created Realtime session object, plus an ephemeral key. - `"function"` -- `tracing: optional "auto" or object { group_id, metadata, workflow_name }` + - `tracing: optional "auto" or object { group_id, metadata, workflow_name }` Configuration options for tracing. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified. @@ -30274,7 +25840,7 @@ Returns the created Realtime session object, plus an ephemeral key. - `"auto"` - - `TracingConfiguration = object { group_id, metadata, workflow_name }` + - `TracingConfiguration object { group_id, metadata, workflow_name }` Granular configuration for tracing. @@ -30293,7 +25859,7 @@ Returns the created Realtime session object, plus an ephemeral key. The name of the workflow to attach to this trace. This is used to name the trace in the traces dashboard. -- `turn_detection: optional object { prefix_padding_ms, silence_duration_ms, threshold, type }` + - `turn_detection: optional object { prefix_padding_ms, silence_duration_ms, threshold, type }` Configuration for turn detection. Can be set to `null` to turn off. Server VAD means that the model will detect the start and end of speech based on @@ -30320,127 +25886,6 @@ Returns the created Realtime session object, plus an ephemeral key. Type of turn detection, only `server_vad` is currently supported. -### Example - -```http -curl https://api.openai.com/v1/realtime/sessions \ - -H 'Content-Type: application/json' \ - -H "Authorization: Bearer $OPENAI_API_KEY" \ - -d '{ - "client_secret": { - "expires_at": 0, - "value": "value" - } - }' -``` - -#### Response - -```json -{ - "id": "id", - "audio": { - "input": { - "format": { - "rate": 24000, - "type": "audio/pcm" - }, - "noise_reduction": { - "type": "near_field" - }, - "transcription": { - "language": "language", - "model": "string", - "prompt": "prompt" - }, - "turn_detection": { - "prefix_padding_ms": 0, - "silence_duration_ms": 0, - "threshold": 0, - "type": "type" - } - }, - "output": { - "format": { - "rate": 24000, - "type": "audio/pcm" - }, - "speed": 0, - "voice": "ash" - } - }, - "expires_at": 0, - "include": [ - "item.input_audio_transcription.logprobs" - ], - "instructions": "instructions", - "max_output_tokens": 0, - "model": "model", - "object": "object", - "output_modalities": [ - "text" - ], - "tool_choice": "tool_choice", - "tools": [ - { - "description": "description", - "name": "name", - "parameters": {}, - "type": "function" - } - ], - "tracing": "auto", - "turn_detection": { - "prefix_padding_ms": 0, - "silence_duration_ms": 0, - "threshold": 0, - "type": "type" - } -} -``` - -### Example - -```http -curl -X POST https://api.openai.com/v1/realtime/sessions \ - -H "Authorization: Bearer $OPENAI_API_KEY" \ - -H "Content-Type: application/json" \ - -d '{ - "model": "gpt-realtime", - "modalities": ["audio", "text"], - "instructions": "You are a friendly assistant." - }' -``` - -#### Response - -```json -{ - "id": "sess_001", - "object": "realtime.session", - "model": "gpt-realtime-2025-08-25", - "modalities": ["audio", "text"], - "instructions": "You are a friendly assistant.", - "voice": "alloy", - "input_audio_format": "pcm16", - "output_audio_format": "pcm16", - "input_audio_transcription": { - "model": "whisper-1" - }, - "turn_detection": null, - "tools": [], - "tool_choice": "none", - "temperature": 0.7, - "max_response_output_tokens": 200, - "speed": 1.1, - "tracing": "auto", - "client_secret": { - "value": "ek_abc123", - "expires_at": 1234567890 - } -} -``` - # Transcription Sessions ## Create transcription session @@ -30717,3 +26162,108 @@ curl -X POST https://api.openai.com/v1/realtime/transcription_sessions \ "client_secret": null } ``` + +## Domain Types + +### Transcription Session Create Response + +- `TranscriptionSessionCreateResponse object { client_secret, input_audio_format, input_audio_transcription, 2 more }` + + A new Realtime transcription session configuration. + + When a session is created on the server via REST API, the session object + also contains an ephemeral key. Default TTL for keys is 10 minutes. This + property is not present when a session is updated via the WebSocket API. + + - `client_secret: object { expires_at, value }` + + Ephemeral key returned by the API. Only present when the session is + created on the server via REST API. + + - `expires_at: number` + + Timestamp for when the token expires. Currently, all tokens expire + after one minute. + + - `value: string` + + Ephemeral key usable in client environments to authenticate connections + to the Realtime API. Use this in client-side environments rather than + a standard API token, which should only be used server-side. + + - `input_audio_format: optional string` + + The format of input audio. Options are `pcm16`, `g711_ulaw`, or `g711_alaw`. + + - `input_audio_transcription: optional AudioTranscription` + + Configuration of the transcription model. + + - `language: optional string` + + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. + + - `model: optional string or "whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + + - `string` + + - `"whisper-1" or "gpt-4o-mini-transcribe" or "gpt-4o-mini-transcribe-2025-12-15" or 2 more` + + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + + - `"whisper-1"` + + - `"gpt-4o-mini-transcribe"` + + - `"gpt-4o-mini-transcribe-2025-12-15"` + + - `"gpt-4o-transcribe"` + + - `"gpt-4o-transcribe-diarize"` + + - `prompt: optional string` + + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + + - `modalities: optional array of "text" or "audio"` + + The set of modalities the model can respond with. To disable audio, + set this to ["text"]. + + - `"text"` + + - `"audio"` + + - `turn_detection: optional object { prefix_padding_ms, silence_duration_ms, threshold, type }` + + Configuration for turn detection. Can be set to `null` to turn off. Server + VAD means that the model will detect the start and end of speech based on + audio volume and respond at the end of user speech. + + - `prefix_padding_ms: optional number` + + Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. + + - `silence_duration_ms: optional number` + + Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. + + - `threshold: optional number` + + Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. + + - `type: optional string` + + Type of turn detection, only `server_vad` is currently supported.