diff --git a/en/java/resources/realtime/index.md b/en/java/resources/realtime/index.md new file mode 100644 index 0000000..7e5f7a3 --- /dev/null +++ b/en/java/resources/realtime/index.md @@ -0,0 +1,22621 @@ +# Realtime + +## Domain Types + +### Audio Transcription + +- `class AudioTranscription:` + + - `Optional language` + + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. + + - `Optional model` + + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + + - `WHISPER_1("whisper-1")` + + - `GPT_4O_MINI_TRANSCRIBE("gpt-4o-mini-transcribe")` + + - `GPT_4O_MINI_TRANSCRIBE_2025_12_15("gpt-4o-mini-transcribe-2025-12-15")` + + - `GPT_4O_TRANSCRIBE("gpt-4o-transcribe")` + + - `GPT_4O_TRANSCRIBE_DIARIZE("gpt-4o-transcribe-diarize")` + + - `Optional prompt` + + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](https://platform.openai.com/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + +### Conversation Created Event + +- `class ConversationCreatedEvent:` + + Returned when a conversation is created. Emitted right after session creation. + + - `Conversation conversation` + + The conversation resource. + + - `Optional id` + + The unique ID of the conversation. + + - `Optional object_` + + The object type, must be `realtime.conversation`. + + - `REALTIME_CONVERSATION("realtime.conversation")` + + - `String eventId` + + The unique ID of the server event. + + - `JsonValue; type "conversation.created"constant` + + The event type, must be `conversation.created`. + + - `CONVERSATION_CREATED("conversation.created")` + +### Conversation Item + +- `class ConversationItem: A class that can be one of several variants.union` + + A single item within a Realtime conversation. + + - `class RealtimeConversationItemSystemMessage:` + + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + + - `List content` + + The content of the message. + + - `Optional text` + + The text content. + + - `Optional type` + + The content type. Always `input_text` for system messages. + + - `INPUT_TEXT("input_text")` + + - `JsonValue; role "system"constant` + + The role of the message sender. Always `system`. + + - `SYSTEM("system")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemUserMessage:` + + A user message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional detail` + + The detail level of the image (for `input_image`). `auto` will default to `high`. + + - `AUTO("auto")` + + - `LOW("low")` + + - `HIGH("high")` + + - `Optional imageUrl` + + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + + - `Optional text` + + The text content (for `input_text`). + + - `Optional transcript` + + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + + - `Optional type` + + The content type (`input_text`, `input_audio`, or `input_image`). + + - `INPUT_TEXT("input_text")` + + - `INPUT_AUDIO("input_audio")` + + - `INPUT_IMAGE("input_image")` + + - `JsonValue; role "user"constant` + + The role of the message sender. Always `user`. + + - `USER("user")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemAssistantMessage:` + + An assistant message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional text` + + The text content. + + - `Optional transcript` + + The transcript of the audio content, this will always be present if the output type is `audio`. + + - `Optional type` + + The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + + - `OUTPUT_TEXT("output_text")` + + - `OUTPUT_AUDIO("output_audio")` + + - `JsonValue; role "assistant"constant` + + The role of the message sender. Always `assistant`. + + - `ASSISTANT("assistant")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemFunctionCall:` + + A function call item in a Realtime conversation. + + - `String arguments` + + The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + + - `String name` + + The name of the function being called. + + - `JsonValue; type "function_call"constant` + + The type of the item. Always `function_call`. + + - `FUNCTION_CALL("function_call")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional callId` + + The ID of the function call. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemFunctionCallOutput:` + + A function call output item in a Realtime conversation. + + - `String callId` + + The ID of the function call this output is for. + + - `String output` + + The output of the function call, this is free text and can contain any information or simply be empty. + + - `JsonValue; type "function_call_output"constant` + + The type of the item. Always `function_call_output`. + + - `FUNCTION_CALL_OUTPUT("function_call_output")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeMcpApprovalResponse:` + + A Realtime item responding to an MCP approval request. + + - `String id` + + The unique ID of the approval response. + + - `String approvalRequestId` + + The ID of the approval request being answered. + + - `boolean approve` + + Whether the request was approved. + + - `JsonValue; type "mcp_approval_response"constant` + + The type of the item. Always `mcp_approval_response`. + + - `MCP_APPROVAL_RESPONSE("mcp_approval_response")` + + - `Optional reason` + + Optional reason for the decision. + + - `class RealtimeMcpListTools:` + + A Realtime item listing tools available on an MCP server. + + - `String serverLabel` + + The label of the MCP server. + + - `List tools` + + The tools available on the server. + + - `JsonValue inputSchema` + + The JSON schema describing the tool's input. + + - `String name` + + The name of the tool. + + - `Optional annotations` + + Additional annotations about the tool. + + - `Optional description` + + The description of the tool. + + - `JsonValue; type "mcp_list_tools"constant` + + The type of the item. Always `mcp_list_tools`. + + - `MCP_LIST_TOOLS("mcp_list_tools")` + + - `Optional id` + + The unique ID of the list. + + - `class RealtimeMcpToolCall:` + + A Realtime item representing an invocation of a tool on an MCP server. + + - `String id` + + The unique ID of the tool call. + + - `String arguments` + + A JSON string of the arguments passed to the tool. + + - `String name` + + The name of the tool that was run. + + - `String serverLabel` + + The label of the MCP server running the tool. + + - `JsonValue; type "mcp_call"constant` + + The type of the item. Always `mcp_call`. + + - `MCP_CALL("mcp_call")` + + - `Optional approvalRequestId` + + The ID of an associated approval request, if any. + + - `Optional error` + + The error from the tool call, if any. + + - `class RealtimeMcpProtocolError:` + + - `long code` + + - `String message` + + - `JsonValue; type "protocol_error"constant` + + - `PROTOCOL_ERROR("protocol_error")` + + - `class RealtimeMcpToolExecutionError:` + + - `String message` + + - `JsonValue; type "tool_execution_error"constant` + + - `TOOL_EXECUTION_ERROR("tool_execution_error")` + + - `class RealtimeMcphttpError:` + + - `long code` + + - `String message` + + - `JsonValue; type "http_error"constant` + + - `HTTP_ERROR("http_error")` + + - `Optional output` + + The output from the tool call. + + - `class RealtimeMcpApprovalRequest:` + + A Realtime item requesting human approval of a tool invocation. + + - `String id` + + The unique ID of the approval request. + + - `String arguments` + + A JSON string of arguments for the tool. + + - `String name` + + The name of the tool to run. + + - `String serverLabel` + + The label of the MCP server making the request. + + - `JsonValue; type "mcp_approval_request"constant` + + The type of the item. Always `mcp_approval_request`. + + - `MCP_APPROVAL_REQUEST("mcp_approval_request")` + +### Conversation Item Added + +- `class ConversationItemAdded:` + + Sent by the server when an Item is added to the default Conversation. This can happen in several cases: + + - When the client sends a `conversation.item.create` event. + - When the input audio buffer is committed. In this case the item will be a user message containing the audio from the buffer. + - When the model is generating a Response. In this case the `conversation.item.added` event will be sent when the model starts generating a specific Item, and thus it will not yet have any content (and `status` will be `in_progress`). + + The event will include the full content of the Item (except when model is generating a Response) except for audio data, which can be retrieved separately with a `conversation.item.retrieve` event if necessary. + + - `String eventId` + + The unique ID of the server event. + + - `ConversationItem item` + + A single item within a Realtime conversation. + + - `class RealtimeConversationItemSystemMessage:` + + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + + - `List content` + + The content of the message. + + - `Optional text` + + The text content. + + - `Optional type` + + The content type. Always `input_text` for system messages. + + - `INPUT_TEXT("input_text")` + + - `JsonValue; role "system"constant` + + The role of the message sender. Always `system`. + + - `SYSTEM("system")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemUserMessage:` + + A user message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional detail` + + The detail level of the image (for `input_image`). `auto` will default to `high`. + + - `AUTO("auto")` + + - `LOW("low")` + + - `HIGH("high")` + + - `Optional imageUrl` + + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + + - `Optional text` + + The text content (for `input_text`). + + - `Optional transcript` + + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + + - `Optional type` + + The content type (`input_text`, `input_audio`, or `input_image`). + + - `INPUT_TEXT("input_text")` + + - `INPUT_AUDIO("input_audio")` + + - `INPUT_IMAGE("input_image")` + + - `JsonValue; role "user"constant` + + The role of the message sender. Always `user`. + + - `USER("user")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemAssistantMessage:` + + An assistant message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional text` + + The text content. + + - `Optional transcript` + + The transcript of the audio content, this will always be present if the output type is `audio`. + + - `Optional type` + + The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + + - `OUTPUT_TEXT("output_text")` + + - `OUTPUT_AUDIO("output_audio")` + + - `JsonValue; role "assistant"constant` + + The role of the message sender. Always `assistant`. + + - `ASSISTANT("assistant")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemFunctionCall:` + + A function call item in a Realtime conversation. + + - `String arguments` + + The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + + - `String name` + + The name of the function being called. + + - `JsonValue; type "function_call"constant` + + The type of the item. Always `function_call`. + + - `FUNCTION_CALL("function_call")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional callId` + + The ID of the function call. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemFunctionCallOutput:` + + A function call output item in a Realtime conversation. + + - `String callId` + + The ID of the function call this output is for. + + - `String output` + + The output of the function call, this is free text and can contain any information or simply be empty. + + - `JsonValue; type "function_call_output"constant` + + The type of the item. Always `function_call_output`. + + - `FUNCTION_CALL_OUTPUT("function_call_output")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeMcpApprovalResponse:` + + A Realtime item responding to an MCP approval request. + + - `String id` + + The unique ID of the approval response. + + - `String approvalRequestId` + + The ID of the approval request being answered. + + - `boolean approve` + + Whether the request was approved. + + - `JsonValue; type "mcp_approval_response"constant` + + The type of the item. Always `mcp_approval_response`. + + - `MCP_APPROVAL_RESPONSE("mcp_approval_response")` + + - `Optional reason` + + Optional reason for the decision. + + - `class RealtimeMcpListTools:` + + A Realtime item listing tools available on an MCP server. + + - `String serverLabel` + + The label of the MCP server. + + - `List tools` + + The tools available on the server. + + - `JsonValue inputSchema` + + The JSON schema describing the tool's input. + + - `String name` + + The name of the tool. + + - `Optional annotations` + + Additional annotations about the tool. + + - `Optional description` + + The description of the tool. + + - `JsonValue; type "mcp_list_tools"constant` + + The type of the item. Always `mcp_list_tools`. + + - `MCP_LIST_TOOLS("mcp_list_tools")` + + - `Optional id` + + The unique ID of the list. + + - `class RealtimeMcpToolCall:` + + A Realtime item representing an invocation of a tool on an MCP server. + + - `String id` + + The unique ID of the tool call. + + - `String arguments` + + A JSON string of the arguments passed to the tool. + + - `String name` + + The name of the tool that was run. + + - `String serverLabel` + + The label of the MCP server running the tool. + + - `JsonValue; type "mcp_call"constant` + + The type of the item. Always `mcp_call`. + + - `MCP_CALL("mcp_call")` + + - `Optional approvalRequestId` + + The ID of an associated approval request, if any. + + - `Optional error` + + The error from the tool call, if any. + + - `class RealtimeMcpProtocolError:` + + - `long code` + + - `String message` + + - `JsonValue; type "protocol_error"constant` + + - `PROTOCOL_ERROR("protocol_error")` + + - `class RealtimeMcpToolExecutionError:` + + - `String message` + + - `JsonValue; type "tool_execution_error"constant` + + - `TOOL_EXECUTION_ERROR("tool_execution_error")` + + - `class RealtimeMcphttpError:` + + - `long code` + + - `String message` + + - `JsonValue; type "http_error"constant` + + - `HTTP_ERROR("http_error")` + + - `Optional output` + + The output from the tool call. + + - `class RealtimeMcpApprovalRequest:` + + A Realtime item requesting human approval of a tool invocation. + + - `String id` + + The unique ID of the approval request. + + - `String arguments` + + A JSON string of arguments for the tool. + + - `String name` + + The name of the tool to run. + + - `String serverLabel` + + The label of the MCP server making the request. + + - `JsonValue; type "mcp_approval_request"constant` + + The type of the item. Always `mcp_approval_request`. + + - `MCP_APPROVAL_REQUEST("mcp_approval_request")` + + - `JsonValue; type "conversation.item.added"constant` + + The event type, must be `conversation.item.added`. + + - `CONVERSATION_ITEM_ADDED("conversation.item.added")` + + - `Optional previousItemId` + + The ID of the item that precedes this one, if any. This is used to + maintain ordering when items are inserted. + +### Conversation Item Create Event + +- `class ConversationItemCreateEvent:` + + Add a new Item to the Conversation's context, including messages, function + calls, and function call responses. This event can be used both to populate a + "history" of the conversation and to add new items mid-stream, but has the + current limitation that it cannot populate assistant audio messages. + + If successful, the server will respond with a `conversation.item.created` + event, otherwise an `error` event will be sent. + + - `ConversationItem item` + + A single item within a Realtime conversation. + + - `class RealtimeConversationItemSystemMessage:` + + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + + - `List content` + + The content of the message. + + - `Optional text` + + The text content. + + - `Optional type` + + The content type. Always `input_text` for system messages. + + - `INPUT_TEXT("input_text")` + + - `JsonValue; role "system"constant` + + The role of the message sender. Always `system`. + + - `SYSTEM("system")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemUserMessage:` + + A user message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional detail` + + The detail level of the image (for `input_image`). `auto` will default to `high`. + + - `AUTO("auto")` + + - `LOW("low")` + + - `HIGH("high")` + + - `Optional imageUrl` + + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + + - `Optional text` + + The text content (for `input_text`). + + - `Optional transcript` + + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + + - `Optional type` + + The content type (`input_text`, `input_audio`, or `input_image`). + + - `INPUT_TEXT("input_text")` + + - `INPUT_AUDIO("input_audio")` + + - `INPUT_IMAGE("input_image")` + + - `JsonValue; role "user"constant` + + The role of the message sender. Always `user`. + + - `USER("user")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemAssistantMessage:` + + An assistant message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional text` + + The text content. + + - `Optional transcript` + + The transcript of the audio content, this will always be present if the output type is `audio`. + + - `Optional type` + + The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + + - `OUTPUT_TEXT("output_text")` + + - `OUTPUT_AUDIO("output_audio")` + + - `JsonValue; role "assistant"constant` + + The role of the message sender. Always `assistant`. + + - `ASSISTANT("assistant")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemFunctionCall:` + + A function call item in a Realtime conversation. + + - `String arguments` + + The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + + - `String name` + + The name of the function being called. + + - `JsonValue; type "function_call"constant` + + The type of the item. Always `function_call`. + + - `FUNCTION_CALL("function_call")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional callId` + + The ID of the function call. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemFunctionCallOutput:` + + A function call output item in a Realtime conversation. + + - `String callId` + + The ID of the function call this output is for. + + - `String output` + + The output of the function call, this is free text and can contain any information or simply be empty. + + - `JsonValue; type "function_call_output"constant` + + The type of the item. Always `function_call_output`. + + - `FUNCTION_CALL_OUTPUT("function_call_output")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeMcpApprovalResponse:` + + A Realtime item responding to an MCP approval request. + + - `String id` + + The unique ID of the approval response. + + - `String approvalRequestId` + + The ID of the approval request being answered. + + - `boolean approve` + + Whether the request was approved. + + - `JsonValue; type "mcp_approval_response"constant` + + The type of the item. Always `mcp_approval_response`. + + - `MCP_APPROVAL_RESPONSE("mcp_approval_response")` + + - `Optional reason` + + Optional reason for the decision. + + - `class RealtimeMcpListTools:` + + A Realtime item listing tools available on an MCP server. + + - `String serverLabel` + + The label of the MCP server. + + - `List tools` + + The tools available on the server. + + - `JsonValue inputSchema` + + The JSON schema describing the tool's input. + + - `String name` + + The name of the tool. + + - `Optional annotations` + + Additional annotations about the tool. + + - `Optional description` + + The description of the tool. + + - `JsonValue; type "mcp_list_tools"constant` + + The type of the item. Always `mcp_list_tools`. + + - `MCP_LIST_TOOLS("mcp_list_tools")` + + - `Optional id` + + The unique ID of the list. + + - `class RealtimeMcpToolCall:` + + A Realtime item representing an invocation of a tool on an MCP server. + + - `String id` + + The unique ID of the tool call. + + - `String arguments` + + A JSON string of the arguments passed to the tool. + + - `String name` + + The name of the tool that was run. + + - `String serverLabel` + + The label of the MCP server running the tool. + + - `JsonValue; type "mcp_call"constant` + + The type of the item. Always `mcp_call`. + + - `MCP_CALL("mcp_call")` + + - `Optional approvalRequestId` + + The ID of an associated approval request, if any. + + - `Optional error` + + The error from the tool call, if any. + + - `class RealtimeMcpProtocolError:` + + - `long code` + + - `String message` + + - `JsonValue; type "protocol_error"constant` + + - `PROTOCOL_ERROR("protocol_error")` + + - `class RealtimeMcpToolExecutionError:` + + - `String message` + + - `JsonValue; type "tool_execution_error"constant` + + - `TOOL_EXECUTION_ERROR("tool_execution_error")` + + - `class RealtimeMcphttpError:` + + - `long code` + + - `String message` + + - `JsonValue; type "http_error"constant` + + - `HTTP_ERROR("http_error")` + + - `Optional output` + + The output from the tool call. + + - `class RealtimeMcpApprovalRequest:` + + A Realtime item requesting human approval of a tool invocation. + + - `String id` + + The unique ID of the approval request. + + - `String arguments` + + A JSON string of arguments for the tool. + + - `String name` + + The name of the tool to run. + + - `String serverLabel` + + The label of the MCP server making the request. + + - `JsonValue; type "mcp_approval_request"constant` + + The type of the item. Always `mcp_approval_request`. + + - `MCP_APPROVAL_REQUEST("mcp_approval_request")` + + - `JsonValue; type "conversation.item.create"constant` + + The event type, must be `conversation.item.create`. + + - `CONVERSATION_ITEM_CREATE("conversation.item.create")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + + - `Optional previousItemId` + + The ID of the preceding item after which the new item will be inserted. If not set, the new item will be appended to the end of the conversation. + + If set to `root`, the new item will be added to the beginning of the conversation. + + If set to an existing ID, it allows an item to be inserted mid-conversation. If the ID cannot be found, an error will be returned and the item will not be added. + +### Conversation Item Created Event + +- `class ConversationItemCreatedEvent:` + + Returned when a conversation item is created. There are several scenarios that produce this event: + + - The server is generating a Response, which if successful will produce + either one or two Items, which will be of type `message` + (role `assistant`) or type `function_call`. + - The input audio buffer has been committed, either by the client or the + server (in `server_vad` mode). The server will take the content of the + input audio buffer and add it to a new user message Item. + - The client has sent a `conversation.item.create` event to add a new Item + to the Conversation. + + - `String eventId` + + The unique ID of the server event. + + - `ConversationItem item` + + A single item within a Realtime conversation. + + - `class RealtimeConversationItemSystemMessage:` + + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + + - `List content` + + The content of the message. + + - `Optional text` + + The text content. + + - `Optional type` + + The content type. Always `input_text` for system messages. + + - `INPUT_TEXT("input_text")` + + - `JsonValue; role "system"constant` + + The role of the message sender. Always `system`. + + - `SYSTEM("system")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemUserMessage:` + + A user message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional detail` + + The detail level of the image (for `input_image`). `auto` will default to `high`. + + - `AUTO("auto")` + + - `LOW("low")` + + - `HIGH("high")` + + - `Optional imageUrl` + + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + + - `Optional text` + + The text content (for `input_text`). + + - `Optional transcript` + + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + + - `Optional type` + + The content type (`input_text`, `input_audio`, or `input_image`). + + - `INPUT_TEXT("input_text")` + + - `INPUT_AUDIO("input_audio")` + + - `INPUT_IMAGE("input_image")` + + - `JsonValue; role "user"constant` + + The role of the message sender. Always `user`. + + - `USER("user")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemAssistantMessage:` + + An assistant message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional text` + + The text content. + + - `Optional transcript` + + The transcript of the audio content, this will always be present if the output type is `audio`. + + - `Optional type` + + The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + + - `OUTPUT_TEXT("output_text")` + + - `OUTPUT_AUDIO("output_audio")` + + - `JsonValue; role "assistant"constant` + + The role of the message sender. Always `assistant`. + + - `ASSISTANT("assistant")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemFunctionCall:` + + A function call item in a Realtime conversation. + + - `String arguments` + + The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + + - `String name` + + The name of the function being called. + + - `JsonValue; type "function_call"constant` + + The type of the item. Always `function_call`. + + - `FUNCTION_CALL("function_call")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional callId` + + The ID of the function call. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemFunctionCallOutput:` + + A function call output item in a Realtime conversation. + + - `String callId` + + The ID of the function call this output is for. + + - `String output` + + The output of the function call, this is free text and can contain any information or simply be empty. + + - `JsonValue; type "function_call_output"constant` + + The type of the item. Always `function_call_output`. + + - `FUNCTION_CALL_OUTPUT("function_call_output")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeMcpApprovalResponse:` + + A Realtime item responding to an MCP approval request. + + - `String id` + + The unique ID of the approval response. + + - `String approvalRequestId` + + The ID of the approval request being answered. + + - `boolean approve` + + Whether the request was approved. + + - `JsonValue; type "mcp_approval_response"constant` + + The type of the item. Always `mcp_approval_response`. + + - `MCP_APPROVAL_RESPONSE("mcp_approval_response")` + + - `Optional reason` + + Optional reason for the decision. + + - `class RealtimeMcpListTools:` + + A Realtime item listing tools available on an MCP server. + + - `String serverLabel` + + The label of the MCP server. + + - `List tools` + + The tools available on the server. + + - `JsonValue inputSchema` + + The JSON schema describing the tool's input. + + - `String name` + + The name of the tool. + + - `Optional annotations` + + Additional annotations about the tool. + + - `Optional description` + + The description of the tool. + + - `JsonValue; type "mcp_list_tools"constant` + + The type of the item. Always `mcp_list_tools`. + + - `MCP_LIST_TOOLS("mcp_list_tools")` + + - `Optional id` + + The unique ID of the list. + + - `class RealtimeMcpToolCall:` + + A Realtime item representing an invocation of a tool on an MCP server. + + - `String id` + + The unique ID of the tool call. + + - `String arguments` + + A JSON string of the arguments passed to the tool. + + - `String name` + + The name of the tool that was run. + + - `String serverLabel` + + The label of the MCP server running the tool. + + - `JsonValue; type "mcp_call"constant` + + The type of the item. Always `mcp_call`. + + - `MCP_CALL("mcp_call")` + + - `Optional approvalRequestId` + + The ID of an associated approval request, if any. + + - `Optional error` + + The error from the tool call, if any. + + - `class RealtimeMcpProtocolError:` + + - `long code` + + - `String message` + + - `JsonValue; type "protocol_error"constant` + + - `PROTOCOL_ERROR("protocol_error")` + + - `class RealtimeMcpToolExecutionError:` + + - `String message` + + - `JsonValue; type "tool_execution_error"constant` + + - `TOOL_EXECUTION_ERROR("tool_execution_error")` + + - `class RealtimeMcphttpError:` + + - `long code` + + - `String message` + + - `JsonValue; type "http_error"constant` + + - `HTTP_ERROR("http_error")` + + - `Optional output` + + The output from the tool call. + + - `class RealtimeMcpApprovalRequest:` + + A Realtime item requesting human approval of a tool invocation. + + - `String id` + + The unique ID of the approval request. + + - `String arguments` + + A JSON string of arguments for the tool. + + - `String name` + + The name of the tool to run. + + - `String serverLabel` + + The label of the MCP server making the request. + + - `JsonValue; type "mcp_approval_request"constant` + + The type of the item. Always `mcp_approval_request`. + + - `MCP_APPROVAL_REQUEST("mcp_approval_request")` + + - `JsonValue; type "conversation.item.created"constant` + + The event type, must be `conversation.item.created`. + + - `CONVERSATION_ITEM_CREATED("conversation.item.created")` + + - `Optional previousItemId` + + The ID of the preceding item in the Conversation context, allows the + client to understand the order of the conversation. Can be `null` if the + item has no predecessor. + +### Conversation Item Delete Event + +- `class ConversationItemDeleteEvent:` + + Send this event when you want to remove any item from the conversation + history. The server will respond with a `conversation.item.deleted` event, + unless the item does not exist in the conversation history, in which case the + server will respond with an error. + + - `String itemId` + + The ID of the item to delete. + + - `JsonValue; type "conversation.item.delete"constant` + + The event type, must be `conversation.item.delete`. + + - `CONVERSATION_ITEM_DELETE("conversation.item.delete")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + +### Conversation Item Deleted Event + +- `class ConversationItemDeletedEvent:` + + Returned when an item in the conversation is deleted by the client with a + `conversation.item.delete` event. This event is used to synchronize the + server's understanding of the conversation history with the client's view. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the item that was deleted. + + - `JsonValue; type "conversation.item.deleted"constant` + + The event type, must be `conversation.item.deleted`. + + - `CONVERSATION_ITEM_DELETED("conversation.item.deleted")` + +### Conversation Item Done + +- `class ConversationItemDone:` + + Returned when a conversation item is finalized. + + The event will include the full content of the Item except for audio data, which can be retrieved separately with a `conversation.item.retrieve` event if needed. + + - `String eventId` + + The unique ID of the server event. + + - `ConversationItem item` + + A single item within a Realtime conversation. + + - `class RealtimeConversationItemSystemMessage:` + + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + + - `List content` + + The content of the message. + + - `Optional text` + + The text content. + + - `Optional type` + + The content type. Always `input_text` for system messages. + + - `INPUT_TEXT("input_text")` + + - `JsonValue; role "system"constant` + + The role of the message sender. Always `system`. + + - `SYSTEM("system")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemUserMessage:` + + A user message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional detail` + + The detail level of the image (for `input_image`). `auto` will default to `high`. + + - `AUTO("auto")` + + - `LOW("low")` + + - `HIGH("high")` + + - `Optional imageUrl` + + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + + - `Optional text` + + The text content (for `input_text`). + + - `Optional transcript` + + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + + - `Optional type` + + The content type (`input_text`, `input_audio`, or `input_image`). + + - `INPUT_TEXT("input_text")` + + - `INPUT_AUDIO("input_audio")` + + - `INPUT_IMAGE("input_image")` + + - `JsonValue; role "user"constant` + + The role of the message sender. Always `user`. + + - `USER("user")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemAssistantMessage:` + + An assistant message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional text` + + The text content. + + - `Optional transcript` + + The transcript of the audio content, this will always be present if the output type is `audio`. + + - `Optional type` + + The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + + - `OUTPUT_TEXT("output_text")` + + - `OUTPUT_AUDIO("output_audio")` + + - `JsonValue; role "assistant"constant` + + The role of the message sender. Always `assistant`. + + - `ASSISTANT("assistant")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemFunctionCall:` + + A function call item in a Realtime conversation. + + - `String arguments` + + The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + + - `String name` + + The name of the function being called. + + - `JsonValue; type "function_call"constant` + + The type of the item. Always `function_call`. + + - `FUNCTION_CALL("function_call")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional callId` + + The ID of the function call. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemFunctionCallOutput:` + + A function call output item in a Realtime conversation. + + - `String callId` + + The ID of the function call this output is for. + + - `String output` + + The output of the function call, this is free text and can contain any information or simply be empty. + + - `JsonValue; type "function_call_output"constant` + + The type of the item. Always `function_call_output`. + + - `FUNCTION_CALL_OUTPUT("function_call_output")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeMcpApprovalResponse:` + + A Realtime item responding to an MCP approval request. + + - `String id` + + The unique ID of the approval response. + + - `String approvalRequestId` + + The ID of the approval request being answered. + + - `boolean approve` + + Whether the request was approved. + + - `JsonValue; type "mcp_approval_response"constant` + + The type of the item. Always `mcp_approval_response`. + + - `MCP_APPROVAL_RESPONSE("mcp_approval_response")` + + - `Optional reason` + + Optional reason for the decision. + + - `class RealtimeMcpListTools:` + + A Realtime item listing tools available on an MCP server. + + - `String serverLabel` + + The label of the MCP server. + + - `List tools` + + The tools available on the server. + + - `JsonValue inputSchema` + + The JSON schema describing the tool's input. + + - `String name` + + The name of the tool. + + - `Optional annotations` + + Additional annotations about the tool. + + - `Optional description` + + The description of the tool. + + - `JsonValue; type "mcp_list_tools"constant` + + The type of the item. Always `mcp_list_tools`. + + - `MCP_LIST_TOOLS("mcp_list_tools")` + + - `Optional id` + + The unique ID of the list. + + - `class RealtimeMcpToolCall:` + + A Realtime item representing an invocation of a tool on an MCP server. + + - `String id` + + The unique ID of the tool call. + + - `String arguments` + + A JSON string of the arguments passed to the tool. + + - `String name` + + The name of the tool that was run. + + - `String serverLabel` + + The label of the MCP server running the tool. + + - `JsonValue; type "mcp_call"constant` + + The type of the item. Always `mcp_call`. + + - `MCP_CALL("mcp_call")` + + - `Optional approvalRequestId` + + The ID of an associated approval request, if any. + + - `Optional error` + + The error from the tool call, if any. + + - `class RealtimeMcpProtocolError:` + + - `long code` + + - `String message` + + - `JsonValue; type "protocol_error"constant` + + - `PROTOCOL_ERROR("protocol_error")` + + - `class RealtimeMcpToolExecutionError:` + + - `String message` + + - `JsonValue; type "tool_execution_error"constant` + + - `TOOL_EXECUTION_ERROR("tool_execution_error")` + + - `class RealtimeMcphttpError:` + + - `long code` + + - `String message` + + - `JsonValue; type "http_error"constant` + + - `HTTP_ERROR("http_error")` + + - `Optional output` + + The output from the tool call. + + - `class RealtimeMcpApprovalRequest:` + + A Realtime item requesting human approval of a tool invocation. + + - `String id` + + The unique ID of the approval request. + + - `String arguments` + + A JSON string of arguments for the tool. + + - `String name` + + The name of the tool to run. + + - `String serverLabel` + + The label of the MCP server making the request. + + - `JsonValue; type "mcp_approval_request"constant` + + The type of the item. Always `mcp_approval_request`. + + - `MCP_APPROVAL_REQUEST("mcp_approval_request")` + + - `JsonValue; type "conversation.item.done"constant` + + The event type, must be `conversation.item.done`. + + - `CONVERSATION_ITEM_DONE("conversation.item.done")` + + - `Optional previousItemId` + + The ID of the item that precedes this one, if any. This is used to + maintain ordering when items are inserted. + +### Conversation Item Input Audio Transcription Completed Event + +- `class ConversationItemInputAudioTranscriptionCompletedEvent:` + + This event is the output of audio transcription for user audio written to the + user audio buffer. Transcription begins when the input audio buffer is + committed by the client or server (when VAD is enabled). Transcription runs + asynchronously with Response creation, so this event may come before or after + the Response events. + + Realtime API models accept audio natively, and thus input transcription is a + separate process run on a separate ASR (Automatic Speech Recognition) model. + The transcript may diverge somewhat from the model's interpretation, and + should be treated as a rough guide. + + - `long contentIndex` + + The index of the content part containing the audio. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the item containing the audio that is being transcribed. + + - `String transcript` + + The transcribed text. + + - `JsonValue; type "conversation.item.input_audio_transcription.completed"constant` + + The event type, must be + `conversation.item.input_audio_transcription.completed`. + + - `CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_COMPLETED("conversation.item.input_audio_transcription.completed")` + + - `Usage usage` + + Usage statistics for the transcription, this is billed according to the ASR model's pricing rather than the realtime model's pricing. + + - `class TranscriptTextUsageTokens:` + + Usage statistics for models billed by token usage. + + - `long inputTokens` + + Number of input tokens billed for this request. + + - `long outputTokens` + + Number of output tokens generated. + + - `long totalTokens` + + Total number of tokens used (input + output). + + - `JsonValue; type "tokens"constant` + + The type of the usage object. Always `tokens` for this variant. + + - `TOKENS("tokens")` + + - `Optional inputTokenDetails` + + Details about the input tokens billed for this request. + + - `Optional audioTokens` + + Number of audio tokens billed for this request. + + - `Optional textTokens` + + Number of text tokens billed for this request. + + - `class TranscriptTextUsageDuration:` + + Usage statistics for models billed by audio input duration. + + - `double seconds` + + Duration of the input audio in seconds. + + - `JsonValue; type "duration"constant` + + The type of the usage object. Always `duration` for this variant. + + - `DURATION("duration")` + + - `Optional> logprobs` + + The log probabilities of the transcription. + + - `String token` + + The token that was used to generate the log probability. + + - `List bytes` + + The bytes that were used to generate the log probability. + + - `double logprob` + + The log probability of the token. + +### Conversation Item Input Audio Transcription Delta Event + +- `class ConversationItemInputAudioTranscriptionDeltaEvent:` + + Returned when the text value of an input audio transcription content part is updated with incremental transcription results. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the item containing the audio that is being transcribed. + + - `JsonValue; type "conversation.item.input_audio_transcription.delta"constant` + + The event type, must be `conversation.item.input_audio_transcription.delta`. + + - `CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_DELTA("conversation.item.input_audio_transcription.delta")` + + - `Optional contentIndex` + + The index of the content part in the item's content array. + + - `Optional delta` + + The text delta. + + - `Optional> logprobs` + + The log probabilities of the transcription. These can be enabled by configurating the session with `"include": ["item.input_audio_transcription.logprobs"]`. Each entry in the array corresponds a log probability of which token would be selected for this chunk of transcription. This can help to identify if it was possible there were multiple valid options for a given chunk of transcription. + + - `String token` + + The token that was used to generate the log probability. + + - `List bytes` + + The bytes that were used to generate the log probability. + + - `double logprob` + + The log probability of the token. + +### Conversation Item Input Audio Transcription Failed Event + +- `class ConversationItemInputAudioTranscriptionFailedEvent:` + + Returned when input audio transcription is configured, and a transcription + request for a user message failed. These events are separate from other + `error` events so that the client can identify the related Item. + + - `long contentIndex` + + The index of the content part containing the audio. + + - `Error error` + + Details of the transcription error. + + - `Optional code` + + Error code, if any. + + - `Optional message` + + A human-readable error message. + + - `Optional param` + + Parameter related to the error, if any. + + - `Optional type` + + The type of error. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the user message item. + + - `JsonValue; type "conversation.item.input_audio_transcription.failed"constant` + + The event type, must be + `conversation.item.input_audio_transcription.failed`. + + - `CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_FAILED("conversation.item.input_audio_transcription.failed")` + +### Conversation Item Input Audio Transcription Segment + +- `class ConversationItemInputAudioTranscriptionSegment:` + + Returned when an input audio transcription segment is identified for an item. + + - `String id` + + The segment identifier. + + - `long contentIndex` + + The index of the input audio content part within the item. + + - `double end` + + End time of the segment in seconds. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the item containing the input audio content. + + - `String speaker` + + The detected speaker label for this segment. + + - `double start` + + Start time of the segment in seconds. + + - `String text` + + The text for this segment. + + - `JsonValue; type "conversation.item.input_audio_transcription.segment"constant` + + The event type, must be `conversation.item.input_audio_transcription.segment`. + + - `CONVERSATION_ITEM_INPUT_AUDIO_TRANSCRIPTION_SEGMENT("conversation.item.input_audio_transcription.segment")` + +### Conversation Item Retrieve Event + +- `class ConversationItemRetrieveEvent:` + + Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD. + The server will respond with a `conversation.item.retrieved` event, + unless the item does not exist in the conversation history, in which case the + server will respond with an error. + + - `String itemId` + + The ID of the item to retrieve. + + - `JsonValue; type "conversation.item.retrieve"constant` + + The event type, must be `conversation.item.retrieve`. + + - `CONVERSATION_ITEM_RETRIEVE("conversation.item.retrieve")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + +### Conversation Item Truncate Event + +- `class ConversationItemTruncateEvent:` + + Send this event to truncate a previous assistant message’s audio. The server + will produce audio faster than realtime, so this event is useful when the user + interrupts to truncate audio that has already been sent to the client but not + yet played. This will synchronize the server's understanding of the audio with + the client's playback. + + Truncating audio will delete the server-side text transcript to ensure there + is not text in the context that hasn't been heard by the user. + + If successful, the server will respond with a `conversation.item.truncated` + event. + + - `long audioEndMs` + + Inclusive duration up to which audio is truncated, in milliseconds. If + the audio_end_ms is greater than the actual audio duration, the server + will respond with an error. + + - `long contentIndex` + + The index of the content part to truncate. Set this to `0`. + + - `String itemId` + + The ID of the assistant message item to truncate. Only assistant message + items can be truncated. + + - `JsonValue; type "conversation.item.truncate"constant` + + The event type, must be `conversation.item.truncate`. + + - `CONVERSATION_ITEM_TRUNCATE("conversation.item.truncate")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + +### Conversation Item Truncated Event + +- `class ConversationItemTruncatedEvent:` + + Returned when an earlier assistant audio message item is truncated by the + client with a `conversation.item.truncate` event. This event is used to + synchronize the server's understanding of the audio with the client's playback. + + This action will truncate the audio and remove the server-side text transcript + to ensure there is no text in the context that hasn't been heard by the user. + + - `long audioEndMs` + + The duration up to which the audio was truncated, in milliseconds. + + - `long contentIndex` + + The index of the content part that was truncated. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the assistant message item that was truncated. + + - `JsonValue; type "conversation.item.truncated"constant` + + The event type, must be `conversation.item.truncated`. + + - `CONVERSATION_ITEM_TRUNCATED("conversation.item.truncated")` + +### Conversation Item With Reference + +- `class ConversationItemWithReference:` + + The item to add to the conversation. + + - `Optional id` + + For an item of type (`message` | `function_call` | `function_call_output`) + this field allows the client to assign the unique ID of the item. It is + not required because the server will generate one if not provided. + + For an item of type `item_reference`, this field is required and is a + reference to any item that has previously existed in the conversation. + + - `Optional arguments` + + The arguments of the function call (for `function_call` items). + + - `Optional callId` + + The ID of the function call (for `function_call` and + `function_call_output` items). If passed on a `function_call_output` + item, the server will check that a `function_call` item with the same + ID exists in the conversation history. + + - `Optional> content` + + The content of the message, applicable for `message` items. + + - Message items of role `system` support only `input_text` content + - Message items of role `user` support `input_text` and `input_audio` + content + - Message items of role `assistant` support `text` content. + + - `Optional id` + + ID of a previous conversation item to reference (for `item_reference` + content types in `response.create` events). These can reference both + client and server created items. + + - `Optional audio` + + Base64-encoded audio bytes, used for `input_audio` content type. + + - `Optional text` + + The text content, used for `input_text` and `text` content types. + + - `Optional transcript` + + The transcript of the audio, used for `input_audio` content type. + + - `Optional type` + + The content type (`input_text`, `input_audio`, `item_reference`, `text`). + + - `INPUT_TEXT("input_text")` + + - `INPUT_AUDIO("input_audio")` + + - `ITEM_REFERENCE("item_reference")` + + - `TEXT("text")` + + - `Optional name` + + The name of the function being called (for `function_call` items). + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional output` + + The output of the function call (for `function_call_output` items). + + - `Optional role` + + The role of the message sender (`user`, `assistant`, `system`), only + applicable for `message` items. + + - `USER("user")` + + - `ASSISTANT("assistant")` + + - `SYSTEM("system")` + + - `Optional status` + + The status of the item (`completed`, `incomplete`, `in_progress`). These have no effect + on the conversation, but are accepted for consistency with the + `conversation.item.created` event. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `Optional type` + + The type of the item (`message`, `function_call`, `function_call_output`, `item_reference`). + + - `MESSAGE("message")` + + - `FUNCTION_CALL("function_call")` + + - `FUNCTION_CALL_OUTPUT("function_call_output")` + + - `ITEM_REFERENCE("item_reference")` + +### Input Audio Buffer Append Event + +- `class InputAudioBufferAppendEvent:` + + Send this event to append audio bytes to the input audio buffer. The audio + buffer is temporary storage you can write to and later commit. A "commit" will create a new + user message item in the conversation history from the buffer content and clear the buffer. + Input audio transcription (if enabled) will be generated when the buffer is committed. + + If VAD is enabled the audio buffer is used to detect speech and the server will decide + when to commit. When Server VAD is disabled, you must commit the audio buffer + manually. Input audio noise reduction operates on writes to the audio buffer. + + The client may choose how much audio to place in each event up to a maximum + of 15 MiB, for example streaming smaller chunks from the client may allow the + VAD to be more responsive. Unlike most other client events, the server will + not send a confirmation response to this event. + + - `String audio` + + Base64-encoded audio bytes. This must be in the format specified by the + `input_audio_format` field in the session configuration. + + - `JsonValue; type "input_audio_buffer.append"constant` + + The event type, must be `input_audio_buffer.append`. + + - `INPUT_AUDIO_BUFFER_APPEND("input_audio_buffer.append")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + +### Input Audio Buffer Clear Event + +- `class InputAudioBufferClearEvent:` + + Send this event to clear the audio bytes in the buffer. The server will + respond with an `input_audio_buffer.cleared` event. + + - `JsonValue; type "input_audio_buffer.clear"constant` + + The event type, must be `input_audio_buffer.clear`. + + - `INPUT_AUDIO_BUFFER_CLEAR("input_audio_buffer.clear")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + +### Input Audio Buffer Cleared Event + +- `class InputAudioBufferClearedEvent:` + + Returned when the input audio buffer is cleared by the client with a + `input_audio_buffer.clear` event. + + - `String eventId` + + The unique ID of the server event. + + - `JsonValue; type "input_audio_buffer.cleared"constant` + + The event type, must be `input_audio_buffer.cleared`. + + - `INPUT_AUDIO_BUFFER_CLEARED("input_audio_buffer.cleared")` + +### Input Audio Buffer Commit Event + +- `class InputAudioBufferCommitEvent:` + + Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically. + + Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an `input_audio_buffer.committed` event. + + - `JsonValue; type "input_audio_buffer.commit"constant` + + The event type, must be `input_audio_buffer.commit`. + + - `INPUT_AUDIO_BUFFER_COMMIT("input_audio_buffer.commit")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + +### Input Audio Buffer Committed Event + +- `class InputAudioBufferCommittedEvent:` + + Returned when an input audio buffer is committed, either by the client or + automatically in server VAD mode. The `item_id` property is the ID of the user + message item that will be created, thus a `conversation.item.created` event + will also be sent to the client. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the user message item that will be created. + + - `JsonValue; type "input_audio_buffer.committed"constant` + + The event type, must be `input_audio_buffer.committed`. + + - `INPUT_AUDIO_BUFFER_COMMITTED("input_audio_buffer.committed")` + + - `Optional previousItemId` + + The ID of the preceding item after which the new item will be inserted. + Can be `null` if the item has no predecessor. + +### Input Audio Buffer Dtmf Event Received Event + +- `class InputAudioBufferDtmfEventReceivedEvent:` + + **SIP Only:** Returned when an DTMF event is received. A DTMF event is a message that + represents a telephone keypad press (0–9, *, #, A–D). The `event` property + is the keypad that the user press. The `received_at` is the UTC Unix Timestamp + that the server received the event. + + - `String event` + + The telephone keypad that was pressed by the user. + + - `long receivedAt` + + UTC Unix Timestamp when DTMF Event was received by server. + + - `JsonValue; type "input_audio_buffer.dtmf_event_received"constant` + + The event type, must be `input_audio_buffer.dtmf_event_received`. + + - `INPUT_AUDIO_BUFFER_DTMF_EVENT_RECEIVED("input_audio_buffer.dtmf_event_received")` + +### Input Audio Buffer Speech Started Event + +- `class InputAudioBufferSpeechStartedEvent:` + + Sent by the server when in `server_vad` mode to indicate that speech has been + detected in the audio buffer. This can happen any time audio is added to the + buffer (unless speech is already detected). The client may want to use this + event to interrupt audio playback or provide visual feedback to the user. + + The client should expect to receive a `input_audio_buffer.speech_stopped` event + when speech stops. The `item_id` property is the ID of the user message item + that will be created when speech stops and will also be included in the + `input_audio_buffer.speech_stopped` event (unless the client manually commits + the audio buffer during VAD activation). + + - `long audioStartMs` + + Milliseconds from the start of all audio written to the buffer during the + session when speech was first detected. This will correspond to the + beginning of audio sent to the model, and thus includes the + `prefix_padding_ms` configured in the Session. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the user message item that will be created when speech stops. + + - `JsonValue; type "input_audio_buffer.speech_started"constant` + + The event type, must be `input_audio_buffer.speech_started`. + + - `INPUT_AUDIO_BUFFER_SPEECH_STARTED("input_audio_buffer.speech_started")` + +### Input Audio Buffer Speech Stopped Event + +- `class InputAudioBufferSpeechStoppedEvent:` + + Returned in `server_vad` mode when the server detects the end of speech in + the audio buffer. The server will also send an `conversation.item.created` + event with the user message item that is created from the audio buffer. + + - `long audioEndMs` + + Milliseconds since the session started when speech stopped. This will + correspond to the end of audio sent to the model, and thus includes the + `min_silence_duration_ms` configured in the Session. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the user message item that will be created. + + - `JsonValue; type "input_audio_buffer.speech_stopped"constant` + + The event type, must be `input_audio_buffer.speech_stopped`. + + - `INPUT_AUDIO_BUFFER_SPEECH_STOPPED("input_audio_buffer.speech_stopped")` + +### Input Audio Buffer Timeout Triggered + +- `class InputAudioBufferTimeoutTriggered:` + + Returned when the Server VAD timeout is triggered for the input audio buffer. This is configured + with `idle_timeout_ms` in the `turn_detection` settings of the session, and it indicates that + there hasn't been any speech detected for the configured duration. + + The `audio_start_ms` and `audio_end_ms` fields indicate the segment of audio after the last + model response up to the triggering time, as an offset from the beginning of audio written + to the input audio buffer. This means it demarcates the segment of audio that was silent and + the difference between the start and end values will roughly match the configured timeout. + + The empty audio will be committed to the conversation as an `input_audio` item (there will be a + `input_audio_buffer.committed` event) and a model response will be generated. There may be speech + that didn't trigger VAD but is still detected by the model, so the model may respond with + something relevant to the conversation or a prompt to continue speaking. + + - `long audioEndMs` + + Millisecond offset of audio written to the input audio buffer at the time the timeout was triggered. + + - `long audioStartMs` + + Millisecond offset of audio written to the input audio buffer that was after the playback time of the last model response. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the item associated with this segment. + + - `JsonValue; type "input_audio_buffer.timeout_triggered"constant` + + The event type, must be `input_audio_buffer.timeout_triggered`. + + - `INPUT_AUDIO_BUFFER_TIMEOUT_TRIGGERED("input_audio_buffer.timeout_triggered")` + +### Log Prob Properties + +- `class LogProbProperties:` + + A log probability object. + + - `String token` + + The token that was used to generate the log probability. + + - `List bytes` + + The bytes that were used to generate the log probability. + + - `double logprob` + + The log probability of the token. + +### Mcp List Tools Completed + +- `class McpListToolsCompleted:` + + Returned when listing MCP tools has completed for an item. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the MCP list tools item. + + - `JsonValue; type "mcp_list_tools.completed"constant` + + The event type, must be `mcp_list_tools.completed`. + + - `MCP_LIST_TOOLS_COMPLETED("mcp_list_tools.completed")` + +### Mcp List Tools Failed + +- `class McpListToolsFailed:` + + Returned when listing MCP tools has failed for an item. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the MCP list tools item. + + - `JsonValue; type "mcp_list_tools.failed"constant` + + The event type, must be `mcp_list_tools.failed`. + + - `MCP_LIST_TOOLS_FAILED("mcp_list_tools.failed")` + +### Mcp List Tools In Progress + +- `class McpListToolsInProgress:` + + Returned when listing MCP tools is in progress for an item. + + - `String eventId` + + The unique ID of the server event. + + - `String itemId` + + The ID of the MCP list tools item. + + - `JsonValue; type "mcp_list_tools.in_progress"constant` + + The event type, must be `mcp_list_tools.in_progress`. + + - `MCP_LIST_TOOLS_IN_PROGRESS("mcp_list_tools.in_progress")` + +### Noise Reduction Type + +- `enum NoiseReductionType:` + + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + + - `NEAR_FIELD("near_field")` + + - `FAR_FIELD("far_field")` + +### Output Audio Buffer Clear Event + +- `class OutputAudioBufferClearEvent:` + + **WebRTC/SIP Only:** Emit to cut off the current audio response. This will trigger the server to + stop generating audio and emit a `output_audio_buffer.cleared` event. This + event should be preceded by a `response.cancel` client event to stop the + generation of the current response. + [Learn more](https://platform.openai.com/docs/guides/realtime-conversations#client-and-server-events-for-audio-in-webrtc). + + - `JsonValue; type "output_audio_buffer.clear"constant` + + The event type, must be `output_audio_buffer.clear`. + + - `OUTPUT_AUDIO_BUFFER_CLEAR("output_audio_buffer.clear")` + + - `Optional eventId` + + The unique ID of the client event used for error handling. + +### Rate Limits Updated Event + +- `class RateLimitsUpdatedEvent:` + + Emitted at the beginning of a Response to indicate the updated rate limits. + When a Response is created some tokens will be "reserved" for the output + tokens, the rate limits shown here reflect that reservation, which is then + adjusted accordingly once the Response is completed. + + - `String eventId` + + The unique ID of the server event. + + - `List rateLimits` + + List of rate limit information. + + - `Optional limit` + + The maximum allowed value for the rate limit. + + - `Optional name` + + The name of the rate limit (`requests`, `tokens`). + + - `REQUESTS("requests")` + + - `TOKENS("tokens")` + + - `Optional remaining` + + The remaining value before the limit is reached. + + - `Optional resetSeconds` + + Seconds until the rate limit resets. + + - `JsonValue; type "rate_limits.updated"constant` + + The event type, must be `rate_limits.updated`. + + - `RATE_LIMITS_UPDATED("rate_limits.updated")` + +### Realtime Audio Config + +- `class RealtimeAudioConfig:` + + Configuration for input and output audio. + + - `Optional input` + + - `Optional format` + + The format of the input audio. + + - `AudioPcm` + + - `Optional rate` + + The sample rate of the audio. Always `24000`. + + - `_24000(24000)` + + - `Optional type` + + The audio format. Always `audio/pcm`. + + - `AUDIO_PCM("audio/pcm")` + + - `AudioPcmu` + + - `Optional type` + + The audio format. Always `audio/pcmu`. + + - `AUDIO_PCMU("audio/pcmu")` + + - `AudioPcma` + + - `Optional type` + + The audio format. Always `audio/pcma`. + + - `AUDIO_PCMA("audio/pcma")` + + - `Optional noiseReduction` + + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + + - `Optional type` + + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + + - `NEAR_FIELD("near_field")` + + - `FAR_FIELD("far_field")` + + - `Optional transcription` + + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](https://platform.openai.com/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + + - `Optional language` + + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. + + - `Optional model` + + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + + - `WHISPER_1("whisper-1")` + + - `GPT_4O_MINI_TRANSCRIBE("gpt-4o-mini-transcribe")` + + - `GPT_4O_MINI_TRANSCRIBE_2025_12_15("gpt-4o-mini-transcribe-2025-12-15")` + + - `GPT_4O_TRANSCRIBE("gpt-4o-transcribe")` + + - `GPT_4O_TRANSCRIBE_DIARIZE("gpt-4o-transcribe-diarize")` + + - `Optional prompt` + + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](https://platform.openai.com/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + + - `Optional turnDetection` + + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + + - `ServerVad` + + - `JsonValue; type "server_vad"constant` + + Type of turn detection, `server_vad` to turn on simple Server VAD. + + - `SERVER_VAD("server_vad")` + + - `Optional createResponse` + + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + + - `Optional idleTimeoutMs` + + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. + + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. + + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. + + - `Optional interruptResponse` + + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + + - `Optional prefixPaddingMs` + + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. + + - `Optional silenceDurationMs` + + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. + + - `Optional threshold` + + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. + + - `SemanticVad` + + - `JsonValue; type "semantic_vad"constant` + + Type of turn detection, `semantic_vad` to turn on Semantic VAD. + + - `SEMANTIC_VAD("semantic_vad")` + + - `Optional createResponse` + + Whether or not to automatically generate a response when a VAD stop event occurs. + + - `Optional eagerness` + + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + + - `LOW("low")` + + - `MEDIUM("medium")` + + - `HIGH("high")` + + - `AUTO("auto")` + + - `Optional interruptResponse` + + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + + - `Optional output` + + - `Optional format` + + The format of the output audio. + + - `Optional speed` + + The speed of the model's spoken response as a multiple of the original speed. + 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. + + This parameter is a post-processing adjustment to the audio after it is generated, it's + also possible to prompt the model to speak faster or slower. + + - `Optional voice` + + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. + + - `String` + + - `enum UnionMember1:` + + - `ALLOY("alloy")` + + - `ASH("ash")` + + - `BALLAD("ballad")` + + - `CORAL("coral")` + + - `ECHO("echo")` + + - `SAGE("sage")` + + - `SHIMMER("shimmer")` + + - `VERSE("verse")` + + - `MARIN("marin")` + + - `CEDAR("cedar")` + + - `class Id:` + + Custom voice reference. + + - `String id` + + The custom voice ID, e.g. `voice_1234`. + +### Realtime Audio Config Input + +- `class RealtimeAudioConfigInput:` + + - `Optional format` + + The format of the input audio. + + - `AudioPcm` + + - `Optional rate` + + The sample rate of the audio. Always `24000`. + + - `_24000(24000)` + + - `Optional type` + + The audio format. Always `audio/pcm`. + + - `AUDIO_PCM("audio/pcm")` + + - `AudioPcmu` + + - `Optional type` + + The audio format. Always `audio/pcmu`. + + - `AUDIO_PCMU("audio/pcmu")` + + - `AudioPcma` + + - `Optional type` + + The audio format. Always `audio/pcma`. + + - `AUDIO_PCMA("audio/pcma")` + + - `Optional noiseReduction` + + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + + - `Optional type` + + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + + - `NEAR_FIELD("near_field")` + + - `FAR_FIELD("far_field")` + + - `Optional transcription` + + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](https://platform.openai.com/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + + - `Optional language` + + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. + + - `Optional model` + + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + + - `WHISPER_1("whisper-1")` + + - `GPT_4O_MINI_TRANSCRIBE("gpt-4o-mini-transcribe")` + + - `GPT_4O_MINI_TRANSCRIBE_2025_12_15("gpt-4o-mini-transcribe-2025-12-15")` + + - `GPT_4O_TRANSCRIBE("gpt-4o-transcribe")` + + - `GPT_4O_TRANSCRIBE_DIARIZE("gpt-4o-transcribe-diarize")` + + - `Optional prompt` + + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](https://platform.openai.com/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + + - `Optional turnDetection` + + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + + - `ServerVad` + + - `JsonValue; type "server_vad"constant` + + Type of turn detection, `server_vad` to turn on simple Server VAD. + + - `SERVER_VAD("server_vad")` + + - `Optional createResponse` + + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + + - `Optional idleTimeoutMs` + + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. + + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. + + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. + + - `Optional interruptResponse` + + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + + - `Optional prefixPaddingMs` + + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. + + - `Optional silenceDurationMs` + + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. + + - `Optional threshold` + + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. + + - `SemanticVad` + + - `JsonValue; type "semantic_vad"constant` + + Type of turn detection, `semantic_vad` to turn on Semantic VAD. + + - `SEMANTIC_VAD("semantic_vad")` + + - `Optional createResponse` + + Whether or not to automatically generate a response when a VAD stop event occurs. + + - `Optional eagerness` + + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + + - `LOW("low")` + + - `MEDIUM("medium")` + + - `HIGH("high")` + + - `AUTO("auto")` + + - `Optional interruptResponse` + + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + +### Realtime Audio Config Output + +- `class RealtimeAudioConfigOutput:` + + - `Optional format` + + The format of the output audio. + + - `AudioPcm` + + - `Optional rate` + + The sample rate of the audio. Always `24000`. + + - `_24000(24000)` + + - `Optional type` + + The audio format. Always `audio/pcm`. + + - `AUDIO_PCM("audio/pcm")` + + - `AudioPcmu` + + - `Optional type` + + The audio format. Always `audio/pcmu`. + + - `AUDIO_PCMU("audio/pcmu")` + + - `AudioPcma` + + - `Optional type` + + The audio format. Always `audio/pcma`. + + - `AUDIO_PCMA("audio/pcma")` + + - `Optional speed` + + The speed of the model's spoken response as a multiple of the original speed. + 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. + + This parameter is a post-processing adjustment to the audio after it is generated, it's + also possible to prompt the model to speak faster or slower. + + - `Optional voice` + + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. + + - `String` + + - `enum UnionMember1:` + + - `ALLOY("alloy")` + + - `ASH("ash")` + + - `BALLAD("ballad")` + + - `CORAL("coral")` + + - `ECHO("echo")` + + - `SAGE("sage")` + + - `SHIMMER("shimmer")` + + - `VERSE("verse")` + + - `MARIN("marin")` + + - `CEDAR("cedar")` + + - `class Id:` + + Custom voice reference. + + - `String id` + + The custom voice ID, e.g. `voice_1234`. + +### Realtime Audio Formats + +- `class RealtimeAudioFormats: A class that can be one of several variants.union` + + The PCM audio format. Only a 24kHz sample rate is supported. + + - `AudioPcm` + + - `Optional rate` + + The sample rate of the audio. Always `24000`. + + - `_24000(24000)` + + - `Optional type` + + The audio format. Always `audio/pcm`. + + - `AUDIO_PCM("audio/pcm")` + + - `AudioPcmu` + + - `Optional type` + + The audio format. Always `audio/pcmu`. + + - `AUDIO_PCMU("audio/pcmu")` + + - `AudioPcma` + + - `Optional type` + + The audio format. Always `audio/pcma`. + + - `AUDIO_PCMA("audio/pcma")` + +### Realtime Audio Input Turn Detection + +- `class RealtimeAudioInputTurnDetection: A class that can be one of several variants.union` + + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + + - `ServerVad` + + - `JsonValue; type "server_vad"constant` + + Type of turn detection, `server_vad` to turn on simple Server VAD. + + - `SERVER_VAD("server_vad")` + + - `Optional createResponse` + + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + + - `Optional idleTimeoutMs` + + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. + + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. + + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. + + - `Optional interruptResponse` + + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + + - `Optional prefixPaddingMs` + + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. + + - `Optional silenceDurationMs` + + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. + + - `Optional threshold` + + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. + + - `SemanticVad` + + - `JsonValue; type "semantic_vad"constant` + + Type of turn detection, `semantic_vad` to turn on Semantic VAD. + + - `SEMANTIC_VAD("semantic_vad")` + + - `Optional createResponse` + + Whether or not to automatically generate a response when a VAD stop event occurs. + + - `Optional eagerness` + + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + + - `LOW("low")` + + - `MEDIUM("medium")` + + - `HIGH("high")` + + - `AUTO("auto")` + + - `Optional interruptResponse` + + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + +### Realtime Client Event + +- `class RealtimeClientEvent: A class that can be one of several variants.union` + + A realtime client event. + + - `class ConversationItemCreateEvent:` + + Add a new Item to the Conversation's context, including messages, function + calls, and function call responses. This event can be used both to populate a + "history" of the conversation and to add new items mid-stream, but has the + current limitation that it cannot populate assistant audio messages. + + If successful, the server will respond with a `conversation.item.created` + event, otherwise an `error` event will be sent. + + - `ConversationItem item` + + A single item within a Realtime conversation. + + - `class RealtimeConversationItemSystemMessage:` + + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + + - `List content` + + The content of the message. + + - `Optional text` + + The text content. + + - `Optional type` + + The content type. Always `input_text` for system messages. + + - `INPUT_TEXT("input_text")` + + - `JsonValue; role "system"constant` + + The role of the message sender. Always `system`. + + - `SYSTEM("system")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemUserMessage:` + + A user message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional detail` + + The detail level of the image (for `input_image`). `auto` will default to `high`. + + - `AUTO("auto")` + + - `LOW("low")` + + - `HIGH("high")` + + - `Optional imageUrl` + + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + + - `Optional text` + + The text content (for `input_text`). + + - `Optional transcript` + + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + + - `Optional type` + + The content type (`input_text`, `input_audio`, or `input_image`). + + - `INPUT_TEXT("input_text")` + + - `INPUT_AUDIO("input_audio")` + + - `INPUT_IMAGE("input_image")` + + - `JsonValue; role "user"constant` + + The role of the message sender. Always `user`. + + - `USER("user")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemAssistantMessage:` + + An assistant message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional text` + + The text content. + + - `Optional transcript` + + The transcript of the audio content, this will always be present if the output type is `audio`. + + - `Optional type` + + The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + + - `OUTPUT_TEXT("output_text")` + + - `OUTPUT_AUDIO("output_audio")` + + - `JsonValue; role "assistant"constant` + + The role of the message sender. Always `assistant`. + + - `ASSISTANT("assistant")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemFunctionCall:` + + A function call item in a Realtime conversation. + + - `String arguments` + + The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + + - `String name` + + The name of the function being called. + + - `JsonValue; type "function_call"constant` + + The type of the item. Always `function_call`. + + - `FUNCTION_CALL("function_call")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional callId` + + The ID of the function call. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeConversationItemFunctionCallOutput:` + + A function call output item in a Realtime conversation. + + - `String callId` + + The ID of the function call this output is for. + + - `String output` + + The output of the function call, this is free text and can contain any information or simply be empty. + + - `JsonValue; type "function_call_output"constant` + + The type of the item. Always `function_call_output`. + + - `FUNCTION_CALL_OUTPUT("function_call_output")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + + - `class RealtimeMcpApprovalResponse:` + + A Realtime item responding to an MCP approval request. + + - `String id` + + The unique ID of the approval response. + + - `String approvalRequestId` + + The ID of the approval request being answered. + + - `boolean approve` + + Whether the request was approved. + + - `JsonValue; type "mcp_approval_response"constant` + + The type of the item. Always `mcp_approval_response`. + + - `MCP_APPROVAL_RESPONSE("mcp_approval_response")` + + - `Optional reason` + + Optional reason for the decision. + + - `class RealtimeMcpListTools:` + + A Realtime item listing tools available on an MCP server. + + - `String serverLabel` + + The label of the MCP server. + + - `List tools` + + The tools available on the server. + + - `JsonValue inputSchema` + + The JSON schema describing the tool's input. + + - `String name` + + The name of the tool. + + - `Optional annotations` + + Additional annotations about the tool. + + - `Optional description` + + The description of the tool. + + - `JsonValue; type "mcp_list_tools"constant` + + The type of the item. Always `mcp_list_tools`. + + - `MCP_LIST_TOOLS("mcp_list_tools")` + + - `Optional id` + + The unique ID of the list. + + - `class RealtimeMcpToolCall:` + + A Realtime item representing an invocation of a tool on an MCP server. + + - `String id` + + The unique ID of the tool call. + + - `String arguments` + + A JSON string of the arguments passed to the tool. + + - `String name` + + The name of the tool that was run. + + - `String serverLabel` + + The label of the MCP server running the tool. + + - `JsonValue; type "mcp_call"constant` + + The type of the item. Always `mcp_call`. + + - `MCP_CALL("mcp_call")` + + - `Optional approvalRequestId` + + The ID of an associated approval request, if any. + + - `Optional error` + + The error from the tool call, if any. + + - `class RealtimeMcpProtocolError:` + + - `long code` + + - `String message` + + - `JsonValue; type "protocol_error"constant` + + - `PROTOCOL_ERROR("protocol_error")` + + - `class RealtimeMcpToolExecutionError:` + + - `String message` + + - `JsonValue; type "tool_execution_error"constant` + + - `TOOL_EXECUTION_ERROR("tool_execution_error")` + + - `class RealtimeMcphttpError:` + + - `long code` + + - `String message` + + - `JsonValue; type "http_error"constant` + + - `HTTP_ERROR("http_error")` + + - `Optional output` + + The output from the tool call. + + - `class RealtimeMcpApprovalRequest:` + + A Realtime item requesting human approval of a tool invocation. + + - `String id` + + The unique ID of the approval request. + + - `String arguments` + + A JSON string of arguments for the tool. + + - `String name` + + The name of the tool to run. + + - `String serverLabel` + + The label of the MCP server making the request. + + - `JsonValue; type "mcp_approval_request"constant` + + The type of the item. Always `mcp_approval_request`. + + - `MCP_APPROVAL_REQUEST("mcp_approval_request")` + + - `JsonValue; type "conversation.item.create"constant` + + The event type, must be `conversation.item.create`. + + - `CONVERSATION_ITEM_CREATE("conversation.item.create")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + + - `Optional previousItemId` + + The ID of the preceding item after which the new item will be inserted. If not set, the new item will be appended to the end of the conversation. + + If set to `root`, the new item will be added to the beginning of the conversation. + + If set to an existing ID, it allows an item to be inserted mid-conversation. If the ID cannot be found, an error will be returned and the item will not be added. + + - `class ConversationItemDeleteEvent:` + + Send this event when you want to remove any item from the conversation + history. The server will respond with a `conversation.item.deleted` event, + unless the item does not exist in the conversation history, in which case the + server will respond with an error. + + - `String itemId` + + The ID of the item to delete. + + - `JsonValue; type "conversation.item.delete"constant` + + The event type, must be `conversation.item.delete`. + + - `CONVERSATION_ITEM_DELETE("conversation.item.delete")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + + - `class ConversationItemRetrieveEvent:` + + Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD. + The server will respond with a `conversation.item.retrieved` event, + unless the item does not exist in the conversation history, in which case the + server will respond with an error. + + - `String itemId` + + The ID of the item to retrieve. + + - `JsonValue; type "conversation.item.retrieve"constant` + + The event type, must be `conversation.item.retrieve`. + + - `CONVERSATION_ITEM_RETRIEVE("conversation.item.retrieve")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + + - `class ConversationItemTruncateEvent:` + + Send this event to truncate a previous assistant message’s audio. The server + will produce audio faster than realtime, so this event is useful when the user + interrupts to truncate audio that has already been sent to the client but not + yet played. This will synchronize the server's understanding of the audio with + the client's playback. + + Truncating audio will delete the server-side text transcript to ensure there + is not text in the context that hasn't been heard by the user. + + If successful, the server will respond with a `conversation.item.truncated` + event. + + - `long audioEndMs` + + Inclusive duration up to which audio is truncated, in milliseconds. If + the audio_end_ms is greater than the actual audio duration, the server + will respond with an error. + + - `long contentIndex` + + The index of the content part to truncate. Set this to `0`. + + - `String itemId` + + The ID of the assistant message item to truncate. Only assistant message + items can be truncated. + + - `JsonValue; type "conversation.item.truncate"constant` + + The event type, must be `conversation.item.truncate`. + + - `CONVERSATION_ITEM_TRUNCATE("conversation.item.truncate")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + + - `class InputAudioBufferAppendEvent:` + + Send this event to append audio bytes to the input audio buffer. The audio + buffer is temporary storage you can write to and later commit. A "commit" will create a new + user message item in the conversation history from the buffer content and clear the buffer. + Input audio transcription (if enabled) will be generated when the buffer is committed. + + If VAD is enabled the audio buffer is used to detect speech and the server will decide + when to commit. When Server VAD is disabled, you must commit the audio buffer + manually. Input audio noise reduction operates on writes to the audio buffer. + + The client may choose how much audio to place in each event up to a maximum + of 15 MiB, for example streaming smaller chunks from the client may allow the + VAD to be more responsive. Unlike most other client events, the server will + not send a confirmation response to this event. + + - `String audio` + + Base64-encoded audio bytes. This must be in the format specified by the + `input_audio_format` field in the session configuration. + + - `JsonValue; type "input_audio_buffer.append"constant` + + The event type, must be `input_audio_buffer.append`. + + - `INPUT_AUDIO_BUFFER_APPEND("input_audio_buffer.append")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + + - `class InputAudioBufferClearEvent:` + + Send this event to clear the audio bytes in the buffer. The server will + respond with an `input_audio_buffer.cleared` event. + + - `JsonValue; type "input_audio_buffer.clear"constant` + + The event type, must be `input_audio_buffer.clear`. + + - `INPUT_AUDIO_BUFFER_CLEAR("input_audio_buffer.clear")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + + - `class OutputAudioBufferClearEvent:` + + **WebRTC/SIP Only:** Emit to cut off the current audio response. This will trigger the server to + stop generating audio and emit a `output_audio_buffer.cleared` event. This + event should be preceded by a `response.cancel` client event to stop the + generation of the current response. + [Learn more](https://platform.openai.com/docs/guides/realtime-conversations#client-and-server-events-for-audio-in-webrtc). + + - `JsonValue; type "output_audio_buffer.clear"constant` + + The event type, must be `output_audio_buffer.clear`. + + - `OUTPUT_AUDIO_BUFFER_CLEAR("output_audio_buffer.clear")` + + - `Optional eventId` + + The unique ID of the client event used for error handling. + + - `class InputAudioBufferCommitEvent:` + + Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically. + + Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an `input_audio_buffer.committed` event. + + - `JsonValue; type "input_audio_buffer.commit"constant` + + The event type, must be `input_audio_buffer.commit`. + + - `INPUT_AUDIO_BUFFER_COMMIT("input_audio_buffer.commit")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + + - `class ResponseCancelEvent:` + + Send this event to cancel an in-progress response. The server will respond + with a `response.done` event with a status of `response.status=cancelled`. If + there is no response to cancel, the server will respond with an error. It's safe + to call `response.cancel` even if no response is in progress, an error will be + returned the session will remain unaffected. + + - `JsonValue; type "response.cancel"constant` + + The event type, must be `response.cancel`. + + - `RESPONSE_CANCEL("response.cancel")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + + - `Optional responseId` + + A specific response ID to cancel - if not provided, will cancel an + in-progress response in the default conversation. + + - `class ResponseCreateEvent:` + + This event instructs the server to create a Response, which means triggering + model inference. When in Server VAD mode, the server will create Responses + automatically. + + A Response will include at least one Item, and may have two, in which case + the second will be a function call. These Items will be appended to the + conversation history by default. + + The server will respond with a `response.created` event, events for Items + and content created, and finally a `response.done` event to indicate the + Response is complete. + + The `response.create` event includes inference configuration like + `instructions` and `tools`. If these are set, they will override the Session's + configuration for this Response only. + + Responses can be created out-of-band of the default Conversation, meaning that they can + have arbitrary input, and it's possible to disable writing the output to the Conversation. + Only one Response can write to the default Conversation at a time, but otherwise multiple + Responses can be created in parallel. The `metadata` field is a good way to disambiguate + multiple simultaneous Responses. + + Clients can set `conversation` to `none` to create a Response that does not write to the default + Conversation. Arbitrary input can be provided with the `input` field, which is an array accepting + raw Items and references to existing Items. + + - `JsonValue; type "response.create"constant` + + The event type, must be `response.create`. + + - `RESPONSE_CREATE("response.create")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. + + - `Optional response` + + Create a new Realtime response with these parameters + + - `Optional audio` + + Configuration for audio input and output. + + - `Optional output` + + - `Optional format` + + The format of the output audio. + + - `AudioPcm` + + - `Optional rate` + + The sample rate of the audio. Always `24000`. + + - `_24000(24000)` + + - `Optional type` + + The audio format. Always `audio/pcm`. + + - `AUDIO_PCM("audio/pcm")` + + - `AudioPcmu` + + - `Optional type` + + The audio format. Always `audio/pcmu`. + + - `AUDIO_PCMU("audio/pcmu")` + + - `AudioPcma` + + - `Optional type` + + The audio format. Always `audio/pcma`. + + - `AUDIO_PCMA("audio/pcma")` + + - `Optional voice` + + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. + + - `String` + + - `enum UnionMember1:` + + - `ALLOY("alloy")` + + - `ASH("ash")` + + - `BALLAD("ballad")` + + - `CORAL("coral")` + + - `ECHO("echo")` + + - `SAGE("sage")` + + - `SHIMMER("shimmer")` + + - `VERSE("verse")` + + - `MARIN("marin")` + + - `CEDAR("cedar")` + + - `class Id:` + + Custom voice reference. + + - `String id` + + The custom voice ID, e.g. `voice_1234`. + + - `Optional conversation` + + Controls which conversation the response is added to. Currently supports + `auto` and `none`, with `auto` as the default value. The `auto` value + means that the contents of the response will be added to the default + conversation. Set this to `none` to create an out-of-band response which + will not add items to default conversation. + + - `AUTO("auto")` + + - `NONE("none")` + + - `Optional> input` + + Input items to include in the prompt for the model. Using this field + creates a new context for this Response instead of using the default + conversation. An empty array `[]` will clear the context for this Response. + Note that this can include references to items that previously appeared in the session + using their id. + + - `class RealtimeConversationItemSystemMessage:` + + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + + - `class RealtimeConversationItemUserMessage:` + + A user message item in a Realtime conversation. + + - `class RealtimeConversationItemAssistantMessage:` + + An assistant message item in a Realtime conversation. + + - `class RealtimeConversationItemFunctionCall:` + + A function call item in a Realtime conversation. + + - `class RealtimeConversationItemFunctionCallOutput:` + + A function call output item in a Realtime conversation. + + - `class RealtimeMcpApprovalResponse:` + + A Realtime item responding to an MCP approval request. + + - `class RealtimeMcpListTools:` + + A Realtime item listing tools available on an MCP server. + + - `class RealtimeMcpToolCall:` + + A Realtime item representing an invocation of a tool on an MCP server. + + - `class RealtimeMcpApprovalRequest:` + + A Realtime item requesting human approval of a tool invocation. + + - `Optional instructions` + + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. + + - `Optional maxOutputTokens` + + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. + + - `long` + + - `JsonValue;` + + - `INF("inf")` + + - `Optional metadata` + + Set of 16 key-value pairs that can be attached to an object. This can be + useful for storing additional information about the object in a structured + format, and querying for objects via API or the dashboard. + + Keys are strings with a maximum length of 64 characters. Values are strings + with a maximum length of 512 characters. + + - `Optional> outputModalities` + + The set of modalities the model used to respond, currently the only possible values are + `[\"audio\"]`, `[\"text\"]`. Audio output always include a text transcript. Setting the + output to mode `text` will disable audio output from the model. + + - `TEXT("text")` + + - `AUDIO("audio")` + + - `Optional prompt` + + Reference to a prompt template and its variables. + [Learn more](https://platform.openai.com/docs/guides/text?api-mode=responses#reusable-prompts). + + - `String id` + + The unique identifier of the prompt template to use. + + - `Optional variables` + + Optional map of values to substitute in for variables in your + prompt. The substitution values can either be strings, or other + Response input types like images or files. + + - `String` + + - `class ResponseInputText:` + + A text input to the model. + + - `String text` + + The text input to the model. + + - `JsonValue; type "input_text"constant` + + The type of the input item. Always `input_text`. + + - `INPUT_TEXT("input_text")` + + - `class ResponseInputImage:` + + An image input to the model. Learn about [image inputs](https://platform.openai.com/docs/guides/vision). + + - `Detail detail` + + The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. + + - `LOW("low")` + + - `HIGH("high")` + + - `AUTO("auto")` + + - `ORIGINAL("original")` + + - `JsonValue; type "input_image"constant` + + The type of the input item. Always `input_image`. + + - `INPUT_IMAGE("input_image")` + + - `Optional fileId` + + The ID of the file to be sent to the model. + + - `Optional imageUrl` + + The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. + + - `class ResponseInputFile:` + + A file input to the model. + + - `JsonValue; type "input_file"constant` + + The type of the input item. Always `input_file`. + + - `INPUT_FILE("input_file")` + + - `Optional detail` + + The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. + + - `LOW("low")` + + - `HIGH("high")` + + - `Optional fileData` + + The content of the file to be sent to the model. + + - `Optional fileId` + + The ID of the file to be sent to the model. + + - `Optional fileUrl` + + The URL of the file to be sent to the model. + + - `Optional filename` + + The name of the file to be sent to the model. + + - `Optional version` + + Optional version of the prompt template. + + - `Optional toolChoice` + + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. + + - `enum ToolChoiceOptions:` + + Controls which (if any) tool is called by the model. + + `none` means the model will not call any tool and instead generates a message. + + `auto` means the model can pick between generating a message or calling one or + more tools. + + `required` means the model must call one or more tools. + + - `NONE("none")` + + - `AUTO("auto")` + + - `REQUIRED("required")` + + - `class ToolChoiceFunction:` + + Use this option to force the model to call a specific function. + + - `String name` + + The name of the function to call. + + - `JsonValue; type "function"constant` + + For function calling, the type is always `function`. + + - `FUNCTION("function")` + + - `class ToolChoiceMcp:` + + Use this option to force the model to call a specific tool on a remote MCP server. + + - `String serverLabel` + + The label of the MCP server to use. + + - `JsonValue; type "mcp"constant` + + For MCP tools, the type is always `mcp`. + + - `MCP("mcp")` + + - `Optional name` + + The name of the tool to call on the server. + + - `Optional> tools` + + Tools available to the model. + + - `class RealtimeFunctionTool:` + + - `Optional description` + + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). + + - `Optional name` + + The name of the function. + + - `Optional parameters` + + Parameters of the function in JSON Schema. + + - `Optional type` + + The type of the tool, i.e. `function`. + + - `FUNCTION("function")` + + - `class RealtimeResponseCreateMcpTool:` + + Give the model access to additional tools via remote Model Context Protocol + (MCP) servers. [Learn more about MCP](https://platform.openai.com/docs/guides/tools-remote-mcp). + + - `String serverLabel` + + A label for this MCP server, used to identify it in tool calls. + + - `JsonValue; type "mcp"constant` + + The type of the MCP tool. Always `mcp`. + + - `MCP("mcp")` + + - `Optional allowedTools` + + List of allowed tool names or a filter object. + + - `List` + + - `class McpToolFilter:` + + A filter object to specify which tools are allowed. + + - `Optional readOnly` + + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. + + - `Optional> toolNames` + + List of allowed tool names. + + - `Optional authorization` + + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. + + - `Optional connectorId` + + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](https://platform.openai.com/docs/guides/tools-remote-mcp#connectors). + + Currently supported `connector_id` values are: + + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` + + - `CONNECTOR_DROPBOX("connector_dropbox")` + + - `CONNECTOR_GMAIL("connector_gmail")` + + - `CONNECTOR_GOOGLECALENDAR("connector_googlecalendar")` + + - `CONNECTOR_GOOGLEDRIVE("connector_googledrive")` + + - `CONNECTOR_MICROSOFTTEAMS("connector_microsoftteams")` + + - `CONNECTOR_OUTLOOKCALENDAR("connector_outlookcalendar")` + + - `CONNECTOR_OUTLOOKEMAIL("connector_outlookemail")` + + - `CONNECTOR_SHAREPOINT("connector_sharepoint")` + + - `Optional deferLoading` + + Whether this MCP tool is deferred and discovered via tool search. + + - `Optional headers` + + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. + + - `Optional requireApproval` + + Specify which of the MCP server's tools require approval. + + - `class McpToolApprovalFilter:` + + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. + + - `Optional always` + + A filter object to specify which tools are allowed. + + - `Optional readOnly` + + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. + + - `Optional> toolNames` + + List of allowed tool names. + + - `Optional never` + + A filter object to specify which tools are allowed. + + - `Optional readOnly` + + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. + + - `Optional> toolNames` + + List of allowed tool names. + + - `enum McpToolApprovalSetting:` + + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. + + - `ALWAYS("always")` + + - `NEVER("never")` + + - `Optional serverDescription` + + Optional description of the MCP server, used to provide more context. + + - `Optional serverUrl` + + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. + + - `class SessionUpdateEvent:` + + Send this event to update the session’s configuration. + The client may send this event at any time to update any field + except for `voice` and `model`. `voice` can be updated only if there have been no other audio outputs yet. + + When the server receives a `session.update`, it will respond + with a `session.updated` event showing the full, effective configuration. + Only the fields that are present in the `session.update` are updated. To clear a field like + `instructions`, pass an empty string. To clear a field like `tools`, pass an empty array. + To clear a field like `turn_detection`, pass `null`. + + - `Session session` + + Update the Realtime session. Choose either a realtime + session or a transcription session. + + - `class RealtimeSessionCreateRequest:` + + Realtime session object configuration. + + - `JsonValue; type "realtime"constant` + + The type of session to create. Always `realtime` for the Realtime API. + + - `REALTIME("realtime")` + + - `Optional audio` + + Configuration for input and output audio. + + - `Optional input` + + - `Optional format` + + The format of the input audio. + + - `Optional noiseReduction` + + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + + - `Optional type` + + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + + - `NEAR_FIELD("near_field")` + + - `FAR_FIELD("far_field")` + + - `Optional transcription` + + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](https://platform.openai.com/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + + - `Optional language` + + The language of the input audio. Supplying the input language in + [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format + will improve accuracy and latency. + + - `Optional model` + + The model to use for transcription. Current options are `whisper-1`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-transcribe`, and `gpt-4o-transcribe-diarize`. Use `gpt-4o-transcribe-diarize` when you need diarization with speaker labels. + + - `WHISPER_1("whisper-1")` + + - `GPT_4O_MINI_TRANSCRIBE("gpt-4o-mini-transcribe")` + + - `GPT_4O_MINI_TRANSCRIBE_2025_12_15("gpt-4o-mini-transcribe-2025-12-15")` + + - `GPT_4O_TRANSCRIBE("gpt-4o-transcribe")` + + - `GPT_4O_TRANSCRIBE_DIARIZE("gpt-4o-transcribe-diarize")` + + - `Optional prompt` + + An optional text to guide the model's style or continue a previous audio + segment. + For `whisper-1`, the [prompt is a list of keywords](https://platform.openai.com/docs/guides/speech-to-text#prompting). + For `gpt-4o-transcribe` models (excluding `gpt-4o-transcribe-diarize`), the prompt is a free text string, for example "expect words related to technology". + + - `Optional turnDetection` + + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + + - `ServerVad` + + - `JsonValue; type "server_vad"constant` + + Type of turn detection, `server_vad` to turn on simple Server VAD. + + - `SERVER_VAD("server_vad")` + + - `Optional createResponse` + + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + + - `Optional idleTimeoutMs` + + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. + + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. + + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. + + - `Optional interruptResponse` + + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + + - `Optional prefixPaddingMs` + + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. + + - `Optional silenceDurationMs` + + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. + + - `Optional threshold` + + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. + + - `SemanticVad` + + - `JsonValue; type "semantic_vad"constant` + + Type of turn detection, `semantic_vad` to turn on Semantic VAD. + + - `SEMANTIC_VAD("semantic_vad")` + + - `Optional createResponse` + + Whether or not to automatically generate a response when a VAD stop event occurs. + + - `Optional eagerness` + + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + + - `LOW("low")` + + - `MEDIUM("medium")` + + - `HIGH("high")` + + - `AUTO("auto")` + + - `Optional interruptResponse` + + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + + - `Optional output` + + - `Optional format` + + The format of the output audio. + + - `Optional speed` + + The speed of the model's spoken response as a multiple of the original speed. + 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress. + + This parameter is a post-processing adjustment to the audio after it is generated, it's + also possible to prompt the model to speak faster or slower. + + - `Optional voice` + + The voice the model uses to respond. Supported built-in voices are + `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, + `marin`, and `cedar`. You may also provide a custom voice object with + an `id`, for example `{ "id": "voice_1234" }`. Voice cannot be changed + during the session once the model has responded with audio at least once. + We recommend `marin` and `cedar` for best quality. + + - `String` + + - `enum UnionMember1:` + + - `ALLOY("alloy")` + + - `ASH("ash")` + + - `BALLAD("ballad")` + + - `CORAL("coral")` + + - `ECHO("echo")` + + - `SAGE("sage")` + + - `SHIMMER("shimmer")` + + - `VERSE("verse")` + + - `MARIN("marin")` + + - `CEDAR("cedar")` + + - `class Id:` + + Custom voice reference. + + - `String id` + + The custom voice ID, e.g. `voice_1234`. + + - `Optional> include` + + Additional fields to include in server outputs. + + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + + - `ITEM_INPUT_AUDIO_TRANSCRIPTION_LOGPROBS("item.input_audio_transcription.logprobs")` + + - `Optional instructions` + + The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. + + Note that the server sets default instructions which will be used if this field is not set and are visible in the `session.created` event at the start of the session. + + - `Optional maxOutputTokens` + + Maximum number of output tokens for a single assistant response, + inclusive of tool calls. Provide an integer between 1 and 4096 to + limit output tokens, or `inf` for the maximum available tokens for a + given model. Defaults to `inf`. + + - `long` + + - `JsonValue;` + + - `INF("inf")` + + - `Optional model` + + The Realtime model used for this session. + + - `GPT_REALTIME("gpt-realtime")` + + - `GPT_REALTIME_1_5("gpt-realtime-1.5")` + + - `GPT_REALTIME_2025_08_28("gpt-realtime-2025-08-28")` + + - `GPT_4O_REALTIME_PREVIEW("gpt-4o-realtime-preview")` + + - `GPT_4O_REALTIME_PREVIEW_2024_10_01("gpt-4o-realtime-preview-2024-10-01")` + + - `GPT_4O_REALTIME_PREVIEW_2024_12_17("gpt-4o-realtime-preview-2024-12-17")` + + - `GPT_4O_REALTIME_PREVIEW_2025_06_03("gpt-4o-realtime-preview-2025-06-03")` + + - `GPT_4O_MINI_REALTIME_PREVIEW("gpt-4o-mini-realtime-preview")` + + - `GPT_4O_MINI_REALTIME_PREVIEW_2024_12_17("gpt-4o-mini-realtime-preview-2024-12-17")` + + - `GPT_REALTIME_MINI("gpt-realtime-mini")` + + - `GPT_REALTIME_MINI_2025_10_06("gpt-realtime-mini-2025-10-06")` + + - `GPT_REALTIME_MINI_2025_12_15("gpt-realtime-mini-2025-12-15")` + + - `GPT_AUDIO_1_5("gpt-audio-1.5")` + + - `GPT_AUDIO_MINI("gpt-audio-mini")` + + - `GPT_AUDIO_MINI_2025_10_06("gpt-audio-mini-2025-10-06")` + + - `GPT_AUDIO_MINI_2025_12_15("gpt-audio-mini-2025-12-15")` + + - `Optional> outputModalities` + + The set of modalities the model can respond with. It defaults to `["audio"]`, indicating + that the model will respond with audio plus a transcript. `["text"]` can be used to make + the model respond with text only. It is not possible to request both `text` and `audio` at the same time. + + - `TEXT("text")` + + - `AUDIO("audio")` + + - `Optional prompt` + + Reference to a prompt template and its variables. + [Learn more](https://platform.openai.com/docs/guides/text?api-mode=responses#reusable-prompts). + + - `Optional toolChoice` + + How the model chooses tools. Provide one of the string modes or force a specific + function/MCP tool. + + - `enum ToolChoiceOptions:` + + Controls which (if any) tool is called by the model. + + `none` means the model will not call any tool and instead generates a message. + + `auto` means the model can pick between generating a message or calling one or + more tools. + + `required` means the model must call one or more tools. + + - `class ToolChoiceFunction:` + + Use this option to force the model to call a specific function. + + - `class ToolChoiceMcp:` + + Use this option to force the model to call a specific tool on a remote MCP server. + + - `Optional> tools` + + Tools available to the model. + + - `class RealtimeFunctionTool:` + + - `Mcp` + + - `String serverLabel` + + A label for this MCP server, used to identify it in tool calls. + + - `JsonValue; type "mcp"constant` + + The type of the MCP tool. Always `mcp`. + + - `MCP("mcp")` + + - `Optional allowedTools` + + List of allowed tool names or a filter object. + + - `List` + + - `class McpToolFilter:` + + A filter object to specify which tools are allowed. + + - `Optional readOnly` + + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. + + - `Optional> toolNames` + + List of allowed tool names. + + - `Optional authorization` + + An OAuth access token that can be used with a remote MCP server, either + with a custom MCP server URL or a service connector. Your application + must handle the OAuth authorization flow and provide the token here. + + - `Optional connectorId` + + Identifier for service connectors, like those available in ChatGPT. One of + `server_url` or `connector_id` must be provided. Learn more about service + connectors [here](https://platform.openai.com/docs/guides/tools-remote-mcp#connectors). + + Currently supported `connector_id` values are: + + - Dropbox: `connector_dropbox` + - Gmail: `connector_gmail` + - Google Calendar: `connector_googlecalendar` + - Google Drive: `connector_googledrive` + - Microsoft Teams: `connector_microsoftteams` + - Outlook Calendar: `connector_outlookcalendar` + - Outlook Email: `connector_outlookemail` + - SharePoint: `connector_sharepoint` + + - `CONNECTOR_DROPBOX("connector_dropbox")` + + - `CONNECTOR_GMAIL("connector_gmail")` + + - `CONNECTOR_GOOGLECALENDAR("connector_googlecalendar")` + + - `CONNECTOR_GOOGLEDRIVE("connector_googledrive")` + + - `CONNECTOR_MICROSOFTTEAMS("connector_microsoftteams")` + + - `CONNECTOR_OUTLOOKCALENDAR("connector_outlookcalendar")` + + - `CONNECTOR_OUTLOOKEMAIL("connector_outlookemail")` + + - `CONNECTOR_SHAREPOINT("connector_sharepoint")` + + - `Optional deferLoading` + + Whether this MCP tool is deferred and discovered via tool search. + + - `Optional headers` + + Optional HTTP headers to send to the MCP server. Use for authentication + or other purposes. + + - `Optional requireApproval` + + Specify which of the MCP server's tools require approval. + + - `class McpToolApprovalFilter:` + + Specify which of the MCP server's tools require approval. Can be + `always`, `never`, or a filter object associated with tools + that require approval. + + - `Optional always` + + A filter object to specify which tools are allowed. + + - `Optional readOnly` + + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. + + - `Optional> toolNames` + + List of allowed tool names. + + - `Optional never` + + A filter object to specify which tools are allowed. + + - `Optional readOnly` + + Indicates whether or not a tool modifies data or is read-only. If an + MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), + it will match this filter. + + - `Optional> toolNames` + + List of allowed tool names. + + - `enum McpToolApprovalSetting:` + + Specify a single approval policy for all tools. One of `always` or + `never`. When set to `always`, all tools will require approval. When + set to `never`, all tools will not require approval. + + - `ALWAYS("always")` + + - `NEVER("never")` + + - `Optional serverDescription` + + Optional description of the MCP server, used to provide more context. + + - `Optional serverUrl` + + The URL for the MCP server. One of `server_url` or `connector_id` must be + provided. + + - `Optional tracing` + + Realtime API can write session traces to the [Traces Dashboard](https://platform.openai.com/logs?api=traces). Set to null to disable tracing. Once + tracing is enabled for a session, the configuration cannot be modified. + + `auto` will create a trace for the session with default values for the + workflow name, group id, and metadata. + + - `JsonValue;` + + - `AUTO("auto")` + + - `TracingConfiguration` + + - `Optional groupId` + + The group id to attach to this trace to enable filtering and + grouping in the Traces Dashboard. + + - `Optional metadata` + + The arbitrary metadata to attach to this trace to enable + filtering in the Traces Dashboard. + + - `Optional workflowName` + + The name of the workflow to attach to this trace. This is used to + name the trace in the Traces Dashboard. + + - `Optional truncation` + + When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. + + Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost. + + Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate. + + Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit. + + - `RealtimeTruncationStrategy` + + - `AUTO("auto")` + + - `DISABLED("disabled")` + + - `class RealtimeTruncationRetentionRatio:` + + Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage. + + - `double retentionRatio` + + Fraction of post-instruction conversation tokens to retain (`0.0` - `1.0`) when the conversation exceeds the input token limit. Setting this to `0.8` means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates. + + - `JsonValue; type "retention_ratio"constant` + + Use retention ratio truncation. + + - `RETENTION_RATIO("retention_ratio")` + + - `Optional tokenLimits` + + Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used. + + - `Optional postInstructions` + + Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens. + + - `class RealtimeTranscriptionSessionCreateRequest:` + + Realtime transcription session object configuration. + + - `JsonValue; type "transcription"constant` + + The type of session to create. Always `transcription` for transcription sessions. + + - `TRANSCRIPTION("transcription")` + + - `Optional audio` + + Configuration for input and output audio. + + - `Optional input` + + - `Optional format` + + The PCM audio format. Only a 24kHz sample rate is supported. + + - `Optional noiseReduction` + + Configuration for input audio noise reduction. This can be set to `null` to turn off. + Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. + Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio. + + - `Optional type` + + Type of noise reduction. `near_field` is for close-talking microphones such as headphones, `far_field` is for far-field microphones such as laptop or conference room microphones. + + - `Optional transcription` + + Configuration for input audio transcription, defaults to off and can be set to `null` to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through [the /audio/transcriptions endpoint](https://platform.openai.com/docs/api-reference/audio/createTranscription) and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service. + + - `Optional turnDetection` + + Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to `null` to turn off, in which case the client must manually trigger model response. + + Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. + + Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency. + + - `ServerVad` + + - `JsonValue; type "server_vad"constant` + + Type of turn detection, `server_vad` to turn on simple Server VAD. + + - `SERVER_VAD("server_vad")` + + - `Optional createResponse` + + Whether or not to automatically generate a response when a VAD stop event occurs. If `interrupt_response` is set to `false` this may fail to create a response if the model is already responding. + + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + + - `Optional idleTimeoutMs` + + Optional timeout after which a model response will be triggered automatically. This is + useful for situations in which a long pause from the user is unexpected, such as a phone + call. The model will effectively prompt the user to continue the conversation based + on the current context. + + The timeout value will be applied after the last model response's audio has finished playing, + i.e. it's set to the `response.done` time plus audio playback duration. + + An `input_audio_buffer.timeout_triggered` event (plus events + associated with the Response) will be emitted when the timeout is reached. + Idle timeout is currently only supported for `server_vad` mode. + + - `Optional interruptResponse` + + Whether or not to automatically interrupt (cancel) any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. If `true` then the response will be cancelled, otherwise it will continue until complete. + + If both `create_response` and `interrupt_response` are set to `false`, the model will never respond automatically but VAD events will still be emitted. + + - `Optional prefixPaddingMs` + + Used only for `server_vad` mode. Amount of audio to include before the VAD detected speech (in + milliseconds). Defaults to 300ms. + + - `Optional silenceDurationMs` + + Used only for `server_vad` mode. Duration of silence to detect speech stop (in milliseconds). Defaults + to 500ms. With shorter values the model will respond more quickly, + but may jump in on short pauses from the user. + + - `Optional threshold` + + Used only for `server_vad` mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A + higher threshold will require louder audio to activate the model, and + thus might perform better in noisy environments. + + - `SemanticVad` + + - `JsonValue; type "semantic_vad"constant` + + Type of turn detection, `semantic_vad` to turn on Semantic VAD. + + - `SEMANTIC_VAD("semantic_vad")` + + - `Optional createResponse` + + Whether or not to automatically generate a response when a VAD stop event occurs. + + - `Optional eagerness` + + Used only for `semantic_vad` mode. The eagerness of the model to respond. `low` will wait longer for the user to continue speaking, `high` will respond more quickly. `auto` is the default and is equivalent to `medium`. `low`, `medium`, and `high` have max timeouts of 8s, 4s, and 2s respectively. + + - `LOW("low")` + + - `MEDIUM("medium")` + + - `HIGH("high")` + + - `AUTO("auto")` + + - `Optional interruptResponse` + + Whether or not to automatically interrupt any ongoing response with output to the default + conversation (i.e. `conversation` of `auto`) when a VAD start event occurs. + + - `Optional> include` + + Additional fields to include in server outputs. + + `item.input_audio_transcription.logprobs`: Include logprobs for input audio transcription. + + - `ITEM_INPUT_AUDIO_TRANSCRIPTION_LOGPROBS("item.input_audio_transcription.logprobs")` + + - `JsonValue; type "session.update"constant` + + The event type, must be `session.update`. + + - `SESSION_UPDATE("session.update")` + + - `Optional eventId` + + Optional client-generated ID used to identify this event. This is an arbitrary string that a client may assign. It will be passed back if there is an error with the event, but the corresponding `session.updated` event will not include it. + +### Realtime Conversation Item Assistant Message + +- `class RealtimeConversationItemAssistantMessage:` + + An assistant message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional text` + + The text content. + + - `Optional transcript` + + The transcript of the audio content, this will always be present if the output type is `audio`. + + - `Optional type` + + The content type, `output_text` or `output_audio` depending on the session `output_modalities` configuration. + + - `OUTPUT_TEXT("output_text")` + + - `OUTPUT_AUDIO("output_audio")` + + - `JsonValue; role "assistant"constant` + + The role of the message sender. Always `assistant`. + + - `ASSISTANT("assistant")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + +### Realtime Conversation Item Function Call + +- `class RealtimeConversationItemFunctionCall:` + + A function call item in a Realtime conversation. + + - `String arguments` + + The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example `{"arg1": "value1", "arg2": 42}`. + + - `String name` + + The name of the function being called. + + - `JsonValue; type "function_call"constant` + + The type of the item. Always `function_call`. + + - `FUNCTION_CALL("function_call")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional callId` + + The ID of the function call. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + +### Realtime Conversation Item Function Call Output + +- `class RealtimeConversationItemFunctionCallOutput:` + + A function call output item in a Realtime conversation. + + - `String callId` + + The ID of the function call this output is for. + + - `String output` + + The output of the function call, this is free text and can contain any information or simply be empty. + + - `JsonValue; type "function_call_output"constant` + + The type of the item. Always `function_call_output`. + + - `FUNCTION_CALL_OUTPUT("function_call_output")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + +### Realtime Conversation Item System Message + +- `class RealtimeConversationItemSystemMessage:` + + A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages. + + - `List content` + + The content of the message. + + - `Optional text` + + The text content. + + - `Optional type` + + The content type. Always `input_text` for system messages. + + - `INPUT_TEXT("input_text")` + + - `JsonValue; role "system"constant` + + The role of the message sender. Always `system`. + + - `SYSTEM("system")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + +### Realtime Conversation Item User Message + +- `class RealtimeConversationItemUserMessage:` + + A user message item in a Realtime conversation. + + - `List content` + + The content of the message. + + - `Optional audio` + + Base64-encoded audio bytes (for `input_audio`), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified. + + - `Optional detail` + + The detail level of the image (for `input_image`). `auto` will default to `high`. + + - `AUTO("auto")` + + - `LOW("low")` + + - `HIGH("high")` + + - `Optional imageUrl` + + Base64-encoded image bytes (for `input_image`) as a data URI. For example `data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...`. Supported formats are PNG and JPEG. + + - `Optional text` + + The text content (for `input_text`). + + - `Optional transcript` + + Transcript of the audio (for `input_audio`). This is not sent to the model, but will be attached to the message item for reference. + + - `Optional type` + + The content type (`input_text`, `input_audio`, or `input_image`). + + - `INPUT_TEXT("input_text")` + + - `INPUT_AUDIO("input_audio")` + + - `INPUT_IMAGE("input_image")` + + - `JsonValue; role "user"constant` + + The role of the message sender. Always `user`. + + - `USER("user")` + + - `JsonValue; type "message"constant` + + The type of the item. Always `message`. + + - `MESSAGE("message")` + + - `Optional id` + + The unique ID of the item. This may be provided by the client or generated by the server. + + - `Optional object_` + + Identifier for the API object being returned - always `realtime.item`. Optional when creating a new item. + + - `REALTIME_ITEM("realtime.item")` + + - `Optional status` + + The status of the item. Has no effect on the conversation. + + - `COMPLETED("completed")` + + - `INCOMPLETE("incomplete")` + + - `IN_PROGRESS("in_progress")` + +### Realtime Error + +- `class RealtimeError:` + + Details of the error. + + - `String message` + + A human-readable error message. + + - `String type` + + The type of error (e.g., "invalid_request_error", "server_error"). + + - `Optional code` + + Error code, if any. + + - `Optional eventId` + + The event_id of the client event that caused the error, if applicable. + + - `Optional param` + + Parameter related to the error, if any. + +### Realtime Error Event + +- `class RealtimeErrorEvent:` + + Returned when an error occurs, which could be a client problem or a server + problem. Most errors are recoverable and the session will stay open, we + recommend to implementors to monitor and log error messages by default. + + - `RealtimeError error` + + Details of the error. + + - `String message` + + A human-readable error message. + + - `String type` + + The type of error (e.g., "invalid_request_error", "server_error"). + + - `Optional code` + + Error code, if any. + + - `Optional eventId` + + The event_id of the client event that caused the error, if applicable. + + - `Optional param` + + Parameter related to the error, if any. + + - `String eventId` + + The unique ID of the server event. + + - `JsonValue; type "error"constant` + + The event type, must be `error`. + + - `ERROR("error")` + +### Realtime Function Tool + +- `class RealtimeFunctionTool:` + + - `Optional description` + + The description of the function, including guidance on when and how + to call it, and guidance about what to tell the user when calling + (if anything). + + - `Optional name` + + The name of the function. + + - `Optional parameters` + + Parameters of the function in JSON Schema. + + - `Optional type` + + The type of the tool, i.e. `function`. + + - `FUNCTION("function")` + +### Realtime Mcp Approval Request + +- `class RealtimeMcpApprovalRequest:` + + A Realtime item requesting human approval of a tool invocation. + + - `String id` + + The unique ID of the approval request. + + - `String arguments` + + A JSON string of arguments for the tool. + + - `String name` + + The name of the tool to run. + + - `String serverLabel` + + The label of the MCP server making the request. + + - `JsonValue; type "mcp_approval_request"constant` + + The type of the item. Always `mcp_approval_request`. + + - `MCP_APPROVAL_REQUEST("mcp_approval_request")` + +### Realtime Mcp Approval Response + +- `class RealtimeMcpApprovalResponse:` + + A Realtime item responding to an MCP approval request. + + - `String id` + + The unique ID of the approval response. + + - `String approvalRequestId` + + The ID of the approval request being answered. + + - `boolean approve` + + Whether the request was approved. + + - `JsonValue; type "mcp_approval_response"constant` + + The type of the item. Always `mcp_approval_response`. + + - `MCP_APPROVAL_RESPONSE("mcp_approval_response")` + + - `Optional reason` + + Optional reason for the decision. + +### Realtime Mcp List Tools + +- `class RealtimeMcpListTools:` + + A Realtime item listing tools available on an MCP server. + + - `String serverLabel` + + The label of the MCP server. + + - `List tools` + + The tools available on the server. + + - `JsonValue inputSchema` + + The JSON schema describing the tool's input. + + - `String name` + + The name of the tool. + + - `Optional annotations` + + Additional annotations about the tool. + + - `Optional description` + + The description of the tool. + + - `JsonValue; type "mcp_list_tools"constant` + + The type of the item. Always `mcp_list_tools`. + + - `MCP_LIST_TOOLS("mcp_list_tools")` + + - `Optional id` + + The unique ID of the list. + +### Realtime Mcp Protocol Error + +- `class RealtimeMcpProtocolError:` + + - `long code` + + - `String message` + + - `JsonValue; type "protocol_error"constant` + + - `PROTOCOL_ERROR("protocol_error")` + +### Realtime Mcp Tool Call + +- `class RealtimeMcpToolCall:` + + A Realtime item representing an invocation of a tool on an MCP server. + + - `String id` + + The unique ID of the tool call. + + - `String arguments` + + A JSON string of the arguments passed to the tool. + + - `String name` + + The name of the tool that was run. + + - `String serverLabel` + + The label of the MCP server running the tool. + + - `JsonValue; type "mcp_call"constant` + + The type of the item. Always `mcp_call`. + + - `MCP_CALL("mcp_call")` + + - `Optional approvalRequestId` + + The ID of an associated approval request, if any. + + - `Optional error` + + The error from the tool call, if any. + + - `class RealtimeMcpProtocolError:` + + - `long code` + + - `String message` + + - `JsonValue; type "protocol_error"constant` + + - `PROTOCOL_ERROR("protocol_error")` + + - `class RealtimeMcpToolExecutionError:` + + - `String message` + + - `JsonValue; type "tool_execution_error"constant` + + - `TOOL_EXECUTION_ERROR("tool_execution_error")` + + - `class RealtimeMcphttpError:` + + - `long code` + + - `String message` + + - `JsonValue; type "http_error"constant` + + - `HTTP_ERROR("http_error")` + + - `Optional output` + + The output from the tool call. + +### Realtime Mcp Tool Execution Error + +- `class RealtimeMcpToolExecutionError:` + + - `String message` + + - `JsonValue; type "tool_execution_error"constant` + + - `TOOL_EXECUTION_ERROR("tool_execution_error")` + +### Realtime Mcphttp Error + +- `class RealtimeMcphttpError:` + + - `long code` + + - `String message` + + - `JsonValue; type "http_error"constant` + + - `HTTP_ERROR("http_error")` + +### Realtime Response + +- `class RealtimeResponse:` + + The response resource. + + - `Optional id` + + The unique ID of the response, will look like `resp_1234`. + + - `Optional