Realtime server events

For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to the page URL.

These are events emitted from the OpenAI Realtime WebSocket server to the client.

error

Returned when an error occurs, which could be a client problem or a server problem. Most errors are recoverable and the session will stay open, we recommend to implementors to monitor and log error messages by default.

Schema

Schema name: RealtimeServerEventError

• error: RealtimeError

  Details of the error.

  • message: string

    A human-readable error message.

  • type: string

    The type of error (e.g., "invalid_request_error", "server_error").

  • code: optional string or null

    Error code, if any.

  • event_id: optional string or null

    The event_id of the client event that caused the error, if applicable.

  • param: optional string or null

    Parameter related to the error, if any.

• event_id: string

  The unique ID of the server event.

• type: "error"

  The event type, must be error.

  • "error"

Example

{
    "event_id": "event_890",
    "type": "error",
    "error": {
        "type": "invalid_request_error",
        "code": "invalid_event",
        "message": "The 'type' field is missing.",
        "param": null,
        "event_id": "event_567"
    }
}

session.created

Returned when a Session is created. Emitted automatically when a new connection is established as the first server event. This event will contain the default Session configuration.

Schema

Schema name: RealtimeServerEventSessionCreated

• event_id: string

  The unique ID of the server event.

• session: RealtimeSessionCreateResponse or RealtimeTranscriptionSessionCreateResponse

  The session configuration.

  • RealtimeSessionCreateResponse object { id, object, type, 13 more }

    A Realtime session configuration object.

    • id: string

      Unique identifier for the session that looks like sess_1234567890abcdef.

    • object: "realtime.session"

      The object type. Always realtime.session.

      • "realtime.session"

    • type: "realtime"

      The type of session to create. Always realtime for the Realtime API.

      • "realtime"

    • audio: optional object { input, output }

      Configuration for input and output audio.

      • input: optional object { format, noise_reduction, transcription, turn_detection }

        • format: optional RealtimeAudioFormats

          The format of the input audio.

          • PCMAudio object { rate, type }

            The PCM audio format. Only a 24kHz sample rate is supported.

            • rate: optional 24000

              The sample rate of the audio. Always 24000.

              • 24000

            • type: optional "audio/pcm"

              The audio format. Always audio/pcm.

              • "audio/pcm"

          • PCMUAudio object { type }

            The G.711 μ-law format.

            • type: optional "audio/pcmu"

              The audio format. Always audio/pcmu.

              • "audio/pcmu"

          • PCMAAudio object { type }

            The G.711 A-law format.

            • type: optional "audio/pcma"

              The audio format. Always audio/pcma.

              • "audio/pcma"

        • noise_reduction: optional object { type }

          Configuration for input audio noise reduction. This can be set to null to turn off. Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.

          • type: optional NoiseReductionType

            Type of noise reduction. near_field is for close-talking microphones such as headphones, far_field is for far-field microphones such as laptop or conference room microphones.

            • "near_field"

            • "far_field"

        • transcription: optional object { language, languages, model, prompt }

          Configuration for input audio transcription, defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through the /audio/transcriptions endpoint and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service.

          • language: optional string

            The language of the input audio.

          • languages: optional array of string

            The possible input audio languages configured for transcription, in ISO-639-1 format.

          • model: optional string or "whisper-1" or "gpt-transcribe" or "gpt-live-transcribe" or 5 more

            The model used for transcription. Current options are whisper-1, gpt-transcribe, gpt-live-transcribe, gpt-4o-mini-transcribe, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-transcribe, gpt-4o-transcribe-diarize, and gpt-realtime-whisper.

            • string

            • "whisper-1" or "gpt-transcribe" or "gpt-live-transcribe" or 5 more

              The model used for transcription. Current options are whisper-1, gpt-transcribe, gpt-live-transcribe, gpt-4o-mini-transcribe, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-transcribe, gpt-4o-transcribe-diarize, and gpt-realtime-whisper.

              • "whisper-1"

              • "gpt-transcribe"

              • "gpt-live-transcribe"

              • "gpt-4o-mini-transcribe"

              • "gpt-4o-mini-transcribe-2025-12-15"

              • "gpt-4o-transcribe"

              • "gpt-4o-transcribe-diarize"

              • "gpt-realtime-whisper"

          • prompt: optional string

            The prompt configured for input audio transcription, when present.

        • turn_detection: optional object { type, create_response, idle_timeout_ms, 4 more } or object { type, create_response, eagerness, interrupt_response } or null

          Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.

          Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.

          Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.

          For gpt-realtime-whisper transcription sessions, turn detection must be set to null; VAD is not supported.

          • ServerVad object { type, create_response, idle_timeout_ms, 4 more }

            Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence.

            • type: "server_vad"

              Type of turn detection, server_vad to turn on simple Server VAD.

              • "server_vad"

            • create_response: optional boolean

              Whether or not to automatically generate a response when a VAD stop event occurs. If interrupt_response is set to false this may fail to create a response if the model is already responding.

              If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.

            • idle_timeout_ms: optional number or null

              Optional timeout after which a model response will be triggered automatically. This is useful for situations in which a long pause from the user is unexpected, such as a phone call. The model will effectively prompt the user to continue the conversation based on the current context.

              The timeout value will be applied after the last model response's audio has finished playing, i.e. it's set to the response.done time plus audio playback duration.

              An input_audio_buffer.timeout_triggered event (plus events associated with the Response) will be emitted when the timeout is reached. Idle timeout is currently only supported for server_vad mode.

            • interrupt_response: optional boolean

              Whether or not to automatically interrupt (cancel) any ongoing response with output to the default conversation (i.e. conversation of auto) when a VAD start event occurs. If true then the response will be cancelled, otherwise it will continue until complete.

              If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.

            • prefix_padding_ms: optional number

              Used only for server_vad mode. Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.

            • silence_duration_ms: optional number

              Used only for server_vad mode. Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.

            • threshold: optional number

              Used only for server_vad mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments.

          • SemanticVad object { type, create_response, eagerness, interrupt_response }

            Server-side semantic turn detection which uses a model to determine when the user has finished speaking.

            • type: "semantic_vad"

              Type of turn detection, semantic_vad to turn on Semantic VAD.

              • "semantic_vad"

            • create_response: optional boolean

              Whether or not to automatically generate a response when a VAD stop event occurs.

            • eagerness: optional "low" or "medium" or "high" or "auto"

              Used only for semantic_vad mode. The eagerness of the model to respond. low will wait longer for the user to continue speaking, high will respond more quickly. auto is the default and is equivalent to medium. low, medium, and high have max timeouts of 8s, 4s, and 2s respectively.

              • "low"

              • "medium"

              • "high"

              • "auto"

            • interrupt_response: optional boolean

              Whether or not to automatically interrupt any ongoing response with output to the default conversation (i.e. conversation of auto) when a VAD start event occurs.

      • output: optional object { format, speed, voice }

        • format: optional RealtimeAudioFormats

          The format of the output audio.

        • speed: optional number

          The speed of the model's spoken response as a multiple of the original speed. 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress.

          This parameter is a post-processing adjustment to the audio after it is generated, it's also possible to prompt the model to speak faster or slower.

        • voice: optional string or "alloy" or "ash" or "ballad" or 7 more

          The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar. We recommend marin and cedar for best quality.

          • string

          • "alloy" or "ash" or "ballad" or 7 more

            The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar. We recommend marin and cedar for best quality.

            • "alloy"

            • "ash"

            • "ballad"

            • "coral"

            • "echo"

            • "sage"

            • "shimmer"

            • "verse"

            • "marin"

            • "cedar"

    • expires_at: optional number

      Expiration timestamp for the session, in seconds since epoch.

    • include: optional array of "item.input_audio_transcription.logprobs"

      Additional fields to include in server outputs.

      item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.

      • "item.input_audio_transcription.logprobs"

    • instructions: optional string

      The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.

      Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.

    • max_output_tokens: optional number or "inf"

      Maximum number of output tokens for a single assistant response, inclusive of tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.

      • number

      • "inf"

        • "inf"

    • model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2" or 16 more

      The Realtime model used for this session.

      • string

      • "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2" or 16 more

        The Realtime model used for this session.

        • "gpt-realtime"

        • "gpt-realtime-1.5"

        • "gpt-realtime-2"

        • "gpt-realtime-2.1"

        • "gpt-realtime-2.1-mini"

        • "gpt-realtime-2025-08-28"

        • "gpt-4o-realtime-preview"

        • "gpt-4o-realtime-preview-2024-10-01"

        • "gpt-4o-realtime-preview-2024-12-17"

        • "gpt-4o-realtime-preview-2025-06-03"

        • "gpt-4o-mini-realtime-preview"

        • "gpt-4o-mini-realtime-preview-2024-12-17"

        • "gpt-realtime-mini"

        • "gpt-realtime-mini-2025-10-06"

        • "gpt-realtime-mini-2025-12-15"

        • "gpt-audio-1.5"

        • "gpt-audio-mini"

        • "gpt-audio-mini-2025-10-06"

        • "gpt-audio-mini-2025-12-15"

    • output_modalities: optional array of "text" or "audio"

      The set of modalities the model can respond with. It defaults to ["audio"], indicating that the model will respond with audio plus a transcript. ["text"] can be used to make the model respond with text only. It is not possible to request both text and audio at the same time.

      • "text"

      • "audio"

    • prompt: optional ResponsePrompt or null

      Reference to a prompt template and its variables. Learn more.

      • id: string

        The unique identifier of the prompt template to use.

      • variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile] or null

        Optional map of values to substitute in for variables in your prompt. The substitution values can either be strings, or other Response input types like images or files.

        • string

        • ResponseInputText object { text, type, prompt_cache_breakpoint }

          A text input to the model.

          • text: string

            The text input to the model.

          • type: "input_text"

            The type of the input item. Always input_text.

            • "input_text"

          • prompt_cache_breakpoint: optional object { mode }

            Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.

            • mode: "explicit"

              The breakpoint mode. Always explicit.

              • "explicit"

        • ResponseInputImage object { detail, type, file_id, 2 more }

          An image input to the model. Learn about image inputs.

          • detail: ImageDetail

            The detail level of the image to be sent to the model. One of high, low, auto, or original. Defaults to auto.

            • "low"

            • "high"

            • "auto"

            • "original"

          • type: "input_image"

            The type of the input item. Always input_image.

            • "input_image"

          • file_id: optional string or null

            The ID of the file to be sent to the model.

          • image_url: optional string or null

            The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.

          • prompt_cache_breakpoint: optional object { mode }

            Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.

            • mode: "explicit"

              The breakpoint mode. Always explicit.

              • "explicit"

        • ResponseInputFile object { type, detail, file_data, 4 more }

          A file input to the model.

          • type: "input_file"

            The type of the input item. Always input_file.

            • "input_file"

          • detail: optional "auto" or "low" or "high"

            The detail level of the file to be sent to the model. Use auto to let the system select the detail level; for GPT-5.6 and later models, auto uses high-quality rendering, which may increase input token usage. Use low for lower-cost rendering, or high to render the file at higher quality. Defaults to auto.

            • "auto"

            • "low"

            • "high"

          • file_data: optional string

            The content of the file to be sent to the model.

          • file_id: optional string or null

            The ID of the file to be sent to the model.

          • file_url: optional string

            The URL of the file to be sent to the model.

          • filename: optional string

            The name of the file to be sent to the model.

          • prompt_cache_breakpoint: optional object { mode }

            Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.

            • mode: "explicit"

              The breakpoint mode. Always explicit.

              • "explicit"

      • version: optional string or null

        Optional version of the prompt template.

    • reasoning: optional RealtimeReasoning

      Configuration for reasoning-capable Realtime models such as gpt-realtime-2.

      • effort: optional RealtimeReasoningEffort

        Constrains effort on reasoning for reasoning-capable Realtime models such as gpt-realtime-2.

        • "minimal"

        • "low"

        • "medium"

        • "high"

        • "xhigh"

    • tool_choice: optional ToolChoiceOptions or ToolChoiceFunction or ToolChoiceMcp

      How the model chooses tools. Provide one of the string modes or force a specific function/MCP tool.

      • ToolChoiceOptions = "none" or "auto" or "required"

        Controls which (if any) tool is called by the model.

        none means the model will not call any tool and instead generates a message.

        auto means the model can pick between generating a message or calling one or more tools.

        required means the model must call one or more tools.

        • "none"

        • "auto"

        • "required"

      • ToolChoiceFunction object { name, type }

        Use this option to force the model to call a specific function.

        • name: string

          The name of the function to call.

        • type: "function"

          For function calling, the type is always function.

          • "function"

      • ToolChoiceMcp object { server_label, type, name }

        Use this option to force the model to call a specific tool on a remote MCP server.

        • server_label: string

          The label of the MCP server to use.

        • type: "mcp"

          For MCP tools, the type is always mcp.

          • "mcp"

        • name: optional string or null

          The name of the tool to call on the server.

    • tools: optional array of RealtimeFunctionTool or object { server_label, type, allowed_callers, 9 more }

      Tools available to the model.

      • RealtimeFunctionTool object { description, name, parameters, type }

        • description: optional string

          The description of the function, including guidance on when and how to call it, and guidance about what to tell the user when calling (if anything).

        • name: optional string

          The name of the function.

        • parameters: optional unknown

          Parameters of the function in JSON Schema.

        • type: optional "function"

          The type of the tool, i.e. function.

          • "function"

      • McpTool object { server_label, type, allowed_callers, 9 more }

        Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.

        • server_label: string

          A label for this MCP server, used to identify it in tool calls.

        • type: "mcp"

          The type of the MCP tool. Always mcp.

          • "mcp"

        • allowed_callers: optional array of "direct" or "programmatic" or null

          The tool invocation context(s).

          • "direct"

          • "programmatic"

        • allowed_tools: optional array of string or object { read_only, tool_names } or null

          List of allowed tool names or a filter object.

          • McpAllowedTools = array of string

            A string array of allowed tool names

          • McpToolFilter object { read_only, tool_names }

            A filter object to specify which tools are allowed.

            • read_only: optional boolean

              Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

            • tool_names: optional array of string

              List of allowed tool names.

        • authorization: optional string

          An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.

        • connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more

          Identifier for service connectors, like those available in ChatGPT. One of server_url, connector_id, or tunnel_id must be provided. Learn more about service connectors here.

          Currently supported connector_id values are:

          • Dropbox: connector_dropbox

          • Gmail: connector_gmail

          • Google Calendar: connector_googlecalendar

          • Google Drive: connector_googledrive

          • Microsoft Teams: connector_microsoftteams

          • Outlook Calendar: connector_outlookcalendar

          • Outlook Email: connector_outlookemail

          • SharePoint: connector_sharepoint

          • "connector_dropbox"

          • "connector_gmail"

          • "connector_googlecalendar"

          • "connector_googledrive"

          • "connector_microsoftteams"

          • "connector_outlookcalendar"

          • "connector_outlookemail"

          • "connector_sharepoint"

        • defer_loading: optional boolean

          Whether this MCP tool is deferred and discovered via tool search.

        • headers: optional map[string] or null

          Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.

        • require_approval: optional object { always, never } or "always" or "never" or null

          Specify which of the MCP server's tools require approval.

          • McpToolApprovalFilter object { always, never }

            Specify which of the MCP server's tools require approval. Can be always, never, or a filter object associated with tools that require approval.

            • always: optional object { read_only, tool_names }

              A filter object to specify which tools are allowed.

              • read_only: optional boolean

                Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

              • tool_names: optional array of string

                List of allowed tool names.

            • never: optional object { read_only, tool_names }

              A filter object to specify which tools are allowed.

              • read_only: optional boolean

                Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

              • tool_names: optional array of string

                List of allowed tool names.

          • McpToolApprovalSetting = "always" or "never"

            Specify a single approval policy for all tools. One of always or never. When set to always, all tools will require approval. When set to never, all tools will not require approval.

            • "always"

            • "never"

        • server_description: optional string

          Optional description of the MCP server, used to provide more context.

        • server_url: optional string

          The URL for the MCP server. One of server_url, connector_id, or tunnel_id must be provided.

        • tunnel_id: optional string

          The Secure MCP Tunnel ID to use instead of a direct server URL. One of server_url, connector_id, or tunnel_id must be provided.

    • tracing: optional "auto" or object { group_id, metadata, workflow_name } or null

      Realtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified.

      auto will create a trace for the session with default values for the workflow name, group id, and metadata.

      • Auto = "auto"

        Enables tracing and sets default values for tracing configuration options. Always auto.

        • "auto"

      • TracingConfiguration object { group_id, metadata, workflow_name }

        Granular configuration for tracing.

        • group_id: optional string

          The group id to attach to this trace to enable filtering and grouping in the Traces Dashboard.

        • metadata: optional unknown

          The arbitrary metadata to attach to this trace to enable filtering in the Traces Dashboard.

        • workflow_name: optional string

          The name of the workflow to attach to this trace. This is used to name the trace in the Traces Dashboard.

    • truncation: optional RealtimeTruncation

      When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs.

      Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost.

      Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate.

      Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit.

      • "auto" or "disabled"

        The truncation strategy to use for the session. auto is the default truncation strategy. disabled will disable truncation and emit errors when the conversation exceeds the input token limit.

        • "auto"

        • "disabled"

      • RetentionRatioTruncation object { retention_ratio, type, token_limits }

        Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage.

        • retention_ratio: number

          Fraction of post-instruction conversation tokens to retain (0.0 - 1.0) when the conversation exceeds the input token limit. Setting this to 0.8 means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates.

        • type: "retention_ratio"

          Use retention ratio truncation.

          • "retention_ratio"

        • token_limits: optional object { post_instructions }

          Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used.

          • post_instructions: optional number

            Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens.

  • RealtimeTranscriptionSessionCreateResponse object { id, object, type, 3 more }

    A Realtime transcription session configuration object.

    • id: string

      Unique identifier for the session that looks like sess_1234567890abcdef.

    • object: string

      The object type. Always realtime.transcription_session.

    • type: "transcription"

      The type of session. Always transcription for transcription sessions.

      • "transcription"

    • audio: optional object { input }

      Configuration for input audio for the session.

      • input: optional object { format, noise_reduction, transcription, turn_detection }

        • format: optional RealtimeAudioFormats

          The PCM audio format. Only a 24kHz sample rate is supported.

        • noise_reduction: optional object { type }

          Configuration for input audio noise reduction.

          • type: optional NoiseReductionType

            Type of noise reduction. near_field is for close-talking microphones such as headphones, far_field is for far-field microphones such as laptop or conference room microphones.

        • transcription: optional object { language, languages, model, prompt }

          Configuration of the transcription model.

          • language: optional string

            The language of the input audio.

          • languages: optional array of string

            The possible input audio languages configured for transcription, in ISO-639-1 format.

          • model: optional string or "whisper-1" or "gpt-transcribe" or "gpt-live-transcribe" or 5 more

            The model used for transcription. Current options are whisper-1, gpt-transcribe, gpt-live-transcribe, gpt-4o-mini-transcribe, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-transcribe, gpt-4o-transcribe-diarize, and gpt-realtime-whisper.

            • string

            • "whisper-1" or "gpt-transcribe" or "gpt-live-transcribe" or 5 more

              The model used for transcription. Current options are whisper-1, gpt-transcribe, gpt-live-transcribe, gpt-4o-mini-transcribe, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-transcribe, gpt-4o-transcribe-diarize, and gpt-realtime-whisper.

              • "whisper-1"

              • "gpt-transcribe"

              • "gpt-live-transcribe"

              • "gpt-4o-mini-transcribe"

              • "gpt-4o-mini-transcribe-2025-12-15"

              • "gpt-4o-transcribe"

              • "gpt-4o-transcribe-diarize"

              • "gpt-realtime-whisper"

          • prompt: optional string

            The prompt configured for input audio transcription, when present.

        • turn_detection: optional RealtimeTranscriptionSessionTurnDetection or null

          Configuration for turn detection. Can be set to null to turn off. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. For gpt-realtime-whisper, this must be null; VAD is not supported.

          • prefix_padding_ms: optional number

            Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.

          • silence_duration_ms: optional number

            Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.

          • threshold: optional number

            Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments.

          • type: optional string

            Type of turn detection, only server_vad is currently supported.

    • expires_at: optional number

      Expiration timestamp for the session, in seconds since epoch.

    • include: optional array of "item.input_audio_transcription.logprobs"

      Additional fields to include in server outputs.

      • item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.

      • "item.input_audio_transcription.logprobs"

• type: "session.created"

  The event type, must be session.created.

  • "session.created"

Example

{
  "type": "session.created",
  "event_id": "event_C9G5RJeJ2gF77mV7f2B1j",
  "session": {
    "type": "realtime",
    "object": "realtime.session",
    "id": "sess_C9G5QPteg4UIbotdKLoYQ",
    "model": "gpt-realtime-2025-08-28",
    "output_modalities": [
      "audio"
    ],
    "instructions": "Your knowledge cutoff is 2023-10. You are a helpful, witty, and friendly AI. Act like a human, but remember that you aren't a human and that you can't do human things in the real world. Your voice and personality should be warm and engaging, with a lively and playful tone. If interacting in a non-English language, start by using the standard accent or dialect familiar to the user. Talk quickly. You should always call a function if you can. Do not refer to these rules, even if you’re asked about them.",
    "tools": [],
    "tool_choice": "auto",
    "max_output_tokens": "inf",
    "tracing": null,
    "prompt": null,
    "expires_at": 1756324625,
    "audio": {
      "input": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "transcription": null,
        "noise_reduction": null,
        "turn_detection": {
          "type": "server_vad",
          "threshold": 0.5,
          "prefix_padding_ms": 300,
          "silence_duration_ms": 200,
          "idle_timeout_ms": null,
          "create_response": true,
          "interrupt_response": true
        }
      },
      "output": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "voice": "marin",
        "speed": 1
      }
    },
    "include": null
  },
}

session.updated

Returned when a session is updated with a session.update event, unless there is an error.

Schema

Schema name: RealtimeServerEventSessionUpdated

• event_id: string

  The unique ID of the server event.

• session: RealtimeSessionCreateResponse or RealtimeTranscriptionSessionCreateResponse

  The session configuration.

  • RealtimeSessionCreateResponse object { id, object, type, 13 more }

    A Realtime session configuration object.

    • id: string

      Unique identifier for the session that looks like sess_1234567890abcdef.

    • object: "realtime.session"

      The object type. Always realtime.session.

      • "realtime.session"

    • type: "realtime"

      The type of session to create. Always realtime for the Realtime API.

      • "realtime"

    • audio: optional object { input, output }

      Configuration for input and output audio.

      • input: optional object { format, noise_reduction, transcription, turn_detection }

        • format: optional RealtimeAudioFormats

          The format of the input audio.

          • PCMAudio object { rate, type }

            The PCM audio format. Only a 24kHz sample rate is supported.

            • rate: optional 24000

              The sample rate of the audio. Always 24000.

              • 24000

            • type: optional "audio/pcm"

              The audio format. Always audio/pcm.

              • "audio/pcm"

          • PCMUAudio object { type }

            The G.711 μ-law format.

            • type: optional "audio/pcmu"

              The audio format. Always audio/pcmu.

              • "audio/pcmu"

          • PCMAAudio object { type }

            The G.711 A-law format.

            • type: optional "audio/pcma"

              The audio format. Always audio/pcma.

              • "audio/pcma"

        • noise_reduction: optional object { type }

          Configuration for input audio noise reduction. This can be set to null to turn off. Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.

          • type: optional NoiseReductionType

            Type of noise reduction. near_field is for close-talking microphones such as headphones, far_field is for far-field microphones such as laptop or conference room microphones.

            • "near_field"

            • "far_field"

        • transcription: optional object { language, languages, model, prompt }

          Configuration for input audio transcription, defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through the /audio/transcriptions endpoint and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service.

          • language: optional string

            The language of the input audio.

          • languages: optional array of string

            The possible input audio languages configured for transcription, in ISO-639-1 format.

          • model: optional string or "whisper-1" or "gpt-transcribe" or "gpt-live-transcribe" or 5 more

            The model used for transcription. Current options are whisper-1, gpt-transcribe, gpt-live-transcribe, gpt-4o-mini-transcribe, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-transcribe, gpt-4o-transcribe-diarize, and gpt-realtime-whisper.

            • string

            • "whisper-1" or "gpt-transcribe" or "gpt-live-transcribe" or 5 more

              The model used for transcription. Current options are whisper-1, gpt-transcribe, gpt-live-transcribe, gpt-4o-mini-transcribe, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-transcribe, gpt-4o-transcribe-diarize, and gpt-realtime-whisper.

              • "whisper-1"

              • "gpt-transcribe"

              • "gpt-live-transcribe"

              • "gpt-4o-mini-transcribe"

              • "gpt-4o-mini-transcribe-2025-12-15"

              • "gpt-4o-transcribe"

              • "gpt-4o-transcribe-diarize"

              • "gpt-realtime-whisper"

          • prompt: optional string

            The prompt configured for input audio transcription, when present.

        • turn_detection: optional object { type, create_response, idle_timeout_ms, 4 more } or object { type, create_response, eagerness, interrupt_response } or null

          Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.

          Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.

          Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.

          For gpt-realtime-whisper transcription sessions, turn detection must be set to null; VAD is not supported.

          • ServerVad object { type, create_response, idle_timeout_ms, 4 more }

            Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence.

            • type: "server_vad"

              Type of turn detection, server_vad to turn on simple Server VAD.

              • "server_vad"

            • create_response: optional boolean

              Whether or not to automatically generate a response when a VAD stop event occurs. If interrupt_response is set to false this may fail to create a response if the model is already responding.

              If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.

            • idle_timeout_ms: optional number or null

              Optional timeout after which a model response will be triggered automatically. This is useful for situations in which a long pause from the user is unexpected, such as a phone call. The model will effectively prompt the user to continue the conversation based on the current context.

              The timeout value will be applied after the last model response's audio has finished playing, i.e. it's set to the response.done time plus audio playback duration.

              An input_audio_buffer.timeout_triggered event (plus events associated with the Response) will be emitted when the timeout is reached. Idle timeout is currently only supported for server_vad mode.

            • interrupt_response: optional boolean

              Whether or not to automatically interrupt (cancel) any ongoing response with output to the default conversation (i.e. conversation of auto) when a VAD start event occurs. If true then the response will be cancelled, otherwise it will continue until complete.

              If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.

            • prefix_padding_ms: optional number

              Used only for server_vad mode. Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.

            • silence_duration_ms: optional number

              Used only for server_vad mode. Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.

            • threshold: optional number

              Used only for server_vad mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments.

          • SemanticVad object { type, create_response, eagerness, interrupt_response }

            Server-side semantic turn detection which uses a model to determine when the user has finished speaking.

            • type: "semantic_vad"

              Type of turn detection, semantic_vad to turn on Semantic VAD.

              • "semantic_vad"

            • create_response: optional boolean

              Whether or not to automatically generate a response when a VAD stop event occurs.

            • eagerness: optional "low" or "medium" or "high" or "auto"

              Used only for semantic_vad mode. The eagerness of the model to respond. low will wait longer for the user to continue speaking, high will respond more quickly. auto is the default and is equivalent to medium. low, medium, and high have max timeouts of 8s, 4s, and 2s respectively.

              • "low"

              • "medium"

              • "high"

              • "auto"

            • interrupt_response: optional boolean

              Whether or not to automatically interrupt any ongoing response with output to the default conversation (i.e. conversation of auto) when a VAD start event occurs.

      • output: optional object { format, speed, voice }

        • format: optional RealtimeAudioFormats

          The format of the output audio.

        • speed: optional number

          The speed of the model's spoken response as a multiple of the original speed. 1.0 is the default speed. 0.25 is the minimum speed. 1.5 is the maximum speed. This value can only be changed in between model turns, not while a response is in progress.

          This parameter is a post-processing adjustment to the audio after it is generated, it's also possible to prompt the model to speak faster or slower.

        • voice: optional string or "alloy" or "ash" or "ballad" or 7 more

          The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar. We recommend marin and cedar for best quality.

          • string

          • "alloy" or "ash" or "ballad" or 7 more

            The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar. We recommend marin and cedar for best quality.

            • "alloy"

            • "ash"

            • "ballad"

            • "coral"

            • "echo"

            • "sage"

            • "shimmer"

            • "verse"

            • "marin"

            • "cedar"

    • expires_at: optional number

      Expiration timestamp for the session, in seconds since epoch.

    • include: optional array of "item.input_audio_transcription.logprobs"

      Additional fields to include in server outputs.

      item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.

      • "item.input_audio_transcription.logprobs"

    • instructions: optional string

      The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.

      Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.

    • max_output_tokens: optional number or "inf"

      Maximum number of output tokens for a single assistant response, inclusive of tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.

      • number

      • "inf"

        • "inf"

    • model: optional string or "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2" or 16 more

      The Realtime model used for this session.

      • string

      • "gpt-realtime" or "gpt-realtime-1.5" or "gpt-realtime-2" or 16 more

        The Realtime model used for this session.

        • "gpt-realtime"

        • "gpt-realtime-1.5"

        • "gpt-realtime-2"

        • "gpt-realtime-2.1"

        • "gpt-realtime-2.1-mini"

        • "gpt-realtime-2025-08-28"

        • "gpt-4o-realtime-preview"

        • "gpt-4o-realtime-preview-2024-10-01"

        • "gpt-4o-realtime-preview-2024-12-17"

        • "gpt-4o-realtime-preview-2025-06-03"

        • "gpt-4o-mini-realtime-preview"

        • "gpt-4o-mini-realtime-preview-2024-12-17"

        • "gpt-realtime-mini"

        • "gpt-realtime-mini-2025-10-06"

        • "gpt-realtime-mini-2025-12-15"

        • "gpt-audio-1.5"

        • "gpt-audio-mini"

        • "gpt-audio-mini-2025-10-06"

        • "gpt-audio-mini-2025-12-15"

    • output_modalities: optional array of "text" or "audio"

      The set of modalities the model can respond with. It defaults to ["audio"], indicating that the model will respond with audio plus a transcript. ["text"] can be used to make the model respond with text only. It is not possible to request both text and audio at the same time.

      • "text"

      • "audio"

    • prompt: optional ResponsePrompt or null

      Reference to a prompt template and its variables. Learn more.

      • id: string

        The unique identifier of the prompt template to use.

      • variables: optional map[string or ResponseInputText or ResponseInputImage or ResponseInputFile] or null

        Optional map of values to substitute in for variables in your prompt. The substitution values can either be strings, or other Response input types like images or files.

        • string

        • ResponseInputText object { text, type, prompt_cache_breakpoint }

          A text input to the model.

          • text: string

            The text input to the model.

          • type: "input_text"

            The type of the input item. Always input_text.

            • "input_text"

          • prompt_cache_breakpoint: optional object { mode }

            Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.

            • mode: "explicit"

              The breakpoint mode. Always explicit.

              • "explicit"

        • ResponseInputImage object { detail, type, file_id, 2 more }

          An image input to the model. Learn about image inputs.

          • detail: ImageDetail

            The detail level of the image to be sent to the model. One of high, low, auto, or original. Defaults to auto.

            • "low"

            • "high"

            • "auto"

            • "original"

          • type: "input_image"

            The type of the input item. Always input_image.

            • "input_image"

          • file_id: optional string or null

            The ID of the file to be sent to the model.

          • image_url: optional string or null

            The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.

          • prompt_cache_breakpoint: optional object { mode }

            Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.

            • mode: "explicit"

              The breakpoint mode. Always explicit.

              • "explicit"

        • ResponseInputFile object { type, detail, file_data, 4 more }

          A file input to the model.

          • type: "input_file"

            The type of the input item. Always input_file.

            • "input_file"

          • detail: optional "auto" or "low" or "high"

            The detail level of the file to be sent to the model. Use auto to let the system select the detail level; for GPT-5.6 and later models, auto uses high-quality rendering, which may increase input token usage. Use low for lower-cost rendering, or high to render the file at higher quality. Defaults to auto.

            • "auto"

            • "low"

            • "high"

          • file_data: optional string

            The content of the file to be sent to the model.

          • file_id: optional string or null

            The ID of the file to be sent to the model.

          • file_url: optional string

            The URL of the file to be sent to the model.

          • filename: optional string

            The name of the file to be sent to the model.

          • prompt_cache_breakpoint: optional object { mode }

            Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.

            • mode: "explicit"

              The breakpoint mode. Always explicit.

              • "explicit"

      • version: optional string or null

        Optional version of the prompt template.

    • reasoning: optional RealtimeReasoning

      Configuration for reasoning-capable Realtime models such as gpt-realtime-2.

      • effort: optional RealtimeReasoningEffort

        Constrains effort on reasoning for reasoning-capable Realtime models such as gpt-realtime-2.

        • "minimal"

        • "low"

        • "medium"

        • "high"

        • "xhigh"

    • tool_choice: optional ToolChoiceOptions or ToolChoiceFunction or ToolChoiceMcp

      How the model chooses tools. Provide one of the string modes or force a specific function/MCP tool.

      • ToolChoiceOptions = "none" or "auto" or "required"

        Controls which (if any) tool is called by the model.

        none means the model will not call any tool and instead generates a message.

        auto means the model can pick between generating a message or calling one or more tools.

        required means the model must call one or more tools.

        • "none"

        • "auto"

        • "required"

      • ToolChoiceFunction object { name, type }

        Use this option to force the model to call a specific function.

        • name: string

          The name of the function to call.

        • type: "function"

          For function calling, the type is always function.

          • "function"

      • ToolChoiceMcp object { server_label, type, name }

        Use this option to force the model to call a specific tool on a remote MCP server.

        • server_label: string

          The label of the MCP server to use.

        • type: "mcp"

          For MCP tools, the type is always mcp.

          • "mcp"

        • name: optional string or null

          The name of the tool to call on the server.

    • tools: optional array of RealtimeFunctionTool or object { server_label, type, allowed_callers, 9 more }

      Tools available to the model.

      • RealtimeFunctionTool object { description, name, parameters, type }

        • description: optional string

          The description of the function, including guidance on when and how to call it, and guidance about what to tell the user when calling (if anything).

        • name: optional string

          The name of the function.

        • parameters: optional unknown

          Parameters of the function in JSON Schema.

        • type: optional "function"

          The type of the tool, i.e. function.

          • "function"

      • McpTool object { server_label, type, allowed_callers, 9 more }

        Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.

        • server_label: string

          A label for this MCP server, used to identify it in tool calls.

        • type: "mcp"

          The type of the MCP tool. Always mcp.

          • "mcp"

        • allowed_callers: optional array of "direct" or "programmatic" or null

          The tool invocation context(s).

          • "direct"

          • "programmatic"

        • allowed_tools: optional array of string or object { read_only, tool_names } or null

          List of allowed tool names or a filter object.

          • McpAllowedTools = array of string

            A string array of allowed tool names

          • McpToolFilter object { read_only, tool_names }

            A filter object to specify which tools are allowed.

            • read_only: optional boolean

              Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

            • tool_names: optional array of string

              List of allowed tool names.

        • authorization: optional string

          An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.

        • connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more

          Identifier for service connectors, like those available in ChatGPT. One of server_url, connector_id, or tunnel_id must be provided. Learn more about service connectors here.

          Currently supported connector_id values are:

          • Dropbox: connector_dropbox

          • Gmail: connector_gmail

          • Google Calendar: connector_googlecalendar

          • Google Drive: connector_googledrive

          • Microsoft Teams: connector_microsoftteams

          • Outlook Calendar: connector_outlookcalendar

          • Outlook Email: connector_outlookemail

          • SharePoint: connector_sharepoint

          • "connector_dropbox"

          • "connector_gmail"

          • "connector_googlecalendar"

          • "connector_googledrive"

          • "connector_microsoftteams"

          • "connector_outlookcalendar"

          • "connector_outlookemail"

          • "connector_sharepoint"

        • defer_loading: optional boolean

          Whether this MCP tool is deferred and discovered via tool search.

        • headers: optional map[string] or null

          Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.

        • require_approval: optional object { always, never } or "always" or "never" or null

          Specify which of the MCP server's tools require approval.

          • McpToolApprovalFilter object { always, never }

            Specify which of the MCP server's tools require approval. Can be always, never, or a filter object associated with tools that require approval.

            • always: optional object { read_only, tool_names }

              A filter object to specify which tools are allowed.

              • read_only: optional boolean

                Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

              • tool_names: optional array of string

                List of allowed tool names.

            • never: optional object { read_only, tool_names }

              A filter object to specify which tools are allowed.

              • read_only: optional boolean

                Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

              • tool_names: optional array of string

                List of allowed tool names.

          • McpToolApprovalSetting = "always" or "never"

            Specify a single approval policy for all tools. One of always or never. When set to always, all tools will require approval. When set to never, all tools will not require approval.

            • "always"

            • "never"

        • server_description: optional string

          Optional description of the MCP server, used to provide more context.

        • server_url: optional string

          The URL for the MCP server. One of server_url, connector_id, or tunnel_id must be provided.

        • tunnel_id: optional string

          The Secure MCP Tunnel ID to use instead of a direct server URL. One of server_url, connector_id, or tunnel_id must be provided.

    • tracing: optional "auto" or object { group_id, metadata, workflow_name } or null

      Realtime API can write session traces to the Traces Dashboard. Set to null to disable tracing. Once tracing is enabled for a session, the configuration cannot be modified.

      auto will create a trace for the session with default values for the workflow name, group id, and metadata.

      • Auto = "auto"

        Enables tracing and sets default values for tracing configuration options. Always auto.

        • "auto"

      • TracingConfiguration object { group_id, metadata, workflow_name }

        Granular configuration for tracing.

        • group_id: optional string

          The group id to attach to this trace to enable filtering and grouping in the Traces Dashboard.

        • metadata: optional unknown

          The arbitrary metadata to attach to this trace to enable filtering in the Traces Dashboard.

        • workflow_name: optional string

          The name of the workflow to attach to this trace. This is used to name the trace in the Traces Dashboard.

    • truncation: optional RealtimeTruncation

      When the number of tokens in a conversation exceeds the model's input token limit, the conversation be truncated, meaning messages (starting from the oldest) will not be included in the model's context. A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs.

      Clients can configure truncation behavior to truncate with a lower max token limit, which is an effective way to control token usage and cost.

      Truncation will reduce the number of cached tokens on the next turn (busting the cache), since messages are dropped from the beginning of the context. However, clients can also configure truncation to retain messages up to a fraction of the maximum context size, which will reduce the need for future truncations and thus improve the cache rate.

      Truncation can be disabled entirely, which means the server will never truncate but would instead return an error if the conversation exceeds the model's input token limit.

      • "auto" or "disabled"

        The truncation strategy to use for the session. auto is the default truncation strategy. disabled will disable truncation and emit errors when the conversation exceeds the input token limit.

        • "auto"

        • "disabled"

      • RetentionRatioTruncation object { retention_ratio, type, token_limits }

        Retain a fraction of the conversation tokens when the conversation exceeds the input token limit. This allows you to amortize truncations across multiple turns, which can help improve cached token usage.

        • retention_ratio: number

          Fraction of post-instruction conversation tokens to retain (0.0 - 1.0) when the conversation exceeds the input token limit. Setting this to 0.8 means that messages will be dropped until 80% of the maximum allowed tokens are used. This helps reduce the frequency of truncations and improve cache rates.

        • type: "retention_ratio"

          Use retention ratio truncation.

          • "retention_ratio"

        • token_limits: optional object { post_instructions }

          Optional custom token limits for this truncation strategy. If not provided, the model's default token limits will be used.

          • post_instructions: optional number

            Maximum tokens allowed in the conversation after instructions (which including tool definitions). For example, setting this to 5,000 would mean that truncation would occur when the conversation exceeds 5,000 tokens after instructions. This cannot be higher than the model's context window size minus the maximum output tokens.

  • RealtimeTranscriptionSessionCreateResponse object { id, object, type, 3 more }

    A Realtime transcription session configuration object.

    • id: string

      Unique identifier for the session that looks like sess_1234567890abcdef.

    • object: string

      The object type. Always realtime.transcription_session.

    • type: "transcription"

      The type of session. Always transcription for transcription sessions.

      • "transcription"

    • audio: optional object { input }

      Configuration for input audio for the session.

      • input: optional object { format, noise_reduction, transcription, turn_detection }

        • format: optional RealtimeAudioFormats

          The PCM audio format. Only a 24kHz sample rate is supported.

        • noise_reduction: optional object { type }

          Configuration for input audio noise reduction.

          • type: optional NoiseReductionType

            Type of noise reduction. near_field is for close-talking microphones such as headphones, far_field is for far-field microphones such as laptop or conference room microphones.

        • transcription: optional object { language, languages, model, prompt }

          Configuration of the transcription model.

          • language: optional string

            The language of the input audio.

          • languages: optional array of string

            The possible input audio languages configured for transcription, in ISO-639-1 format.

          • model: optional string or "whisper-1" or "gpt-transcribe" or "gpt-live-transcribe" or 5 more

            The model used for transcription. Current options are whisper-1, gpt-transcribe, gpt-live-transcribe, gpt-4o-mini-transcribe, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-transcribe, gpt-4o-transcribe-diarize, and gpt-realtime-whisper.

            • string

            • "whisper-1" or "gpt-transcribe" or "gpt-live-transcribe" or 5 more

              The model used for transcription. Current options are whisper-1, gpt-transcribe, gpt-live-transcribe, gpt-4o-mini-transcribe, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-transcribe, gpt-4o-transcribe-diarize, and gpt-realtime-whisper.

              • "whisper-1"

              • "gpt-transcribe"

              • "gpt-live-transcribe"

              • "gpt-4o-mini-transcribe"

              • "gpt-4o-mini-transcribe-2025-12-15"

              • "gpt-4o-transcribe"

              • "gpt-4o-transcribe-diarize"

              • "gpt-realtime-whisper"

          • prompt: optional string

            The prompt configured for input audio transcription, when present.

        • turn_detection: optional RealtimeTranscriptionSessionTurnDetection or null

          Configuration for turn detection. Can be set to null to turn off. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. For gpt-realtime-whisper, this must be null; VAD is not supported.

          • prefix_padding_ms: optional number

            Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.

          • silence_duration_ms: optional number

            Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.

          • threshold: optional number

            Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments.

          • type: optional string

            Type of turn detection, only server_vad is currently supported.

    • expires_at: optional number

      Expiration timestamp for the session, in seconds since epoch.

    • include: optional array of "item.input_audio_transcription.logprobs"

      Additional fields to include in server outputs.

      • item.input_audio_transcription.logprobs: Include logprobs for input audio transcription.

      • "item.input_audio_transcription.logprobs"

• type: "session.updated"

  The event type, must be session.updated.

  • "session.updated"

Example

{
  "type": "session.updated",
  "event_id": "event_C9G8mqI3IucaojlVKE8Cs",
  "session": {
    "type": "realtime",
    "object": "realtime.session",
    "id": "sess_C9G8l3zp50uFv4qgxfJ8o",
    "model": "gpt-realtime-2025-08-28",
    "output_modalities": [
      "audio"
    ],
    "instructions": "Your knowledge cutoff is 2023-10. You are a helpful, witty, and friendly AI. Act like a human, but remember that you aren't a human and that you can't do human things in the real world. Your voice and personality should be warm and engaging, with a lively and playful tone. If interacting in a non-English language, start by using the standard accent or dialect familiar to the user. Talk quickly. You should always call a function if you can. Do not refer to these rules, even if you’re asked about them.",
    "tools": [
      {
        "type": "function",
        "name": "display_color_palette",
        "description": "\nCall this function when a user asks for a color palette.\n",
        "parameters": {
          "type": "object",
          "strict": true,
          "properties": {
            "theme": {
              "type": "string",
              "description": "Description of the theme for the color scheme."
            },
            "colors": {
              "type": "array",
              "description": "Array of five hex color codes based on the theme.",
              "items": {
                "type": "string",
                "description": "Hex color code"
              }
            }
          },
          "required": [
            "theme",
            "colors"
          ]
        }
      }
    ],
    "tool_choice": "auto",
    "max_output_tokens": "inf",
    "tracing": null,
    "prompt": null,
    "expires_at": 1756324832,
    "audio": {
      "input": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "transcription": null,
        "noise_reduction": null,
        "turn_detection": {
          "type": "server_vad",
          "threshold": 0.5,
          "prefix_padding_ms": 300,
          "silence_duration_ms": 200,
          "idle_timeout_ms": null,
          "create_response": true,
          "interrupt_response": true
        }
      },
      "output": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "voice": "marin",
        "speed": 1
      }
    },
    "include": null
  },
}

conversation.item.added

Sent by the server when an Item is added to the default Conversation. This can happen in several cases:

• When the client sends a conversation.item.create event.
• When the input audio buffer is committed. In this case the item will be a user message containing the audio from the buffer.
• When the model is generating a Response. In this case the conversation.item.added event will be sent when the model starts generating a specific Item, and thus it will not yet have any content (and status will be in_progress).

The event will include the full content of the Item (except when model is generating a Response) except for audio data, which can be retrieved separately with a conversation.item.retrieve event if necessary.

Schema

Schema name: RealtimeServerEventConversationItemAdded

• event_id: string

  The unique ID of the server event.

• item: ConversationItem

  A single item within a Realtime conversation.

  • RealtimeConversationItemSystemMessage object { content, role, type, 3 more }

    A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.

    • content: array of object { text, type }

      The content of the message.

      • text: optional string

        The text content.

      • type: optional "input_text"

        The content type. Always input_text for system messages.

        • "input_text"

    • role: "system"

      The role of the message sender. Always system.

      • "system"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemUserMessage object { content, role, type, 3 more }

    A user message item in a Realtime conversation.

    • content: array of object { audio, detail, image_url, 3 more }

      The content of the message.

      • audio: optional string

        Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

      • detail: optional "auto" or "low" or "high"

        The detail level of the image (for input_image). auto will default to high.

        • "auto"

        • "low"

        • "high"

      • image_url: optional string

        Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.

      • text: optional string

        The text content (for input_text).

      • transcript: optional string

        Transcript of the audio (for input_audio). This is not sent to the model, but will be attached to the message item for reference.

      • type: optional "input_text" or "input_audio" or "input_image"

        The content type (input_text, input_audio, or input_image).

        • "input_text"

        • "input_audio"

        • "input_image"

    • role: "user"

      The role of the message sender. Always user.

      • "user"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }

    An assistant message item in a Realtime conversation.

    • content: array of object { audio, text, transcript, type }

      The content of the message.

      • audio: optional string

        Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

      • text: optional string

        The text content.

      • transcript: optional string

        The transcript of the audio content, this will always be present if the output type is audio.

      • type: optional "output_text" or "output_audio"

        The content type, output_text or output_audio depending on the session output_modalities configuration.

        • "output_text"

        • "output_audio"

    • role: "assistant"

      The role of the message sender. Always assistant.

      • "assistant"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }

    A function call item in a Realtime conversation.

    • arguments: string

      The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.

    • name: string

      The name of the function being called.

    • type: "function_call"

      The type of the item. Always function_call.

      • "function_call"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • call_id: optional string

      The ID of the function call.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }

    A function call output item in a Realtime conversation.

    • call_id: string

      The ID of the function call this output is for.

    • output: string

      The output of the function call, this is free text and can contain any information or simply be empty.

    • type: "function_call_output"

      The type of the item. Always function_call_output.

      • "function_call_output"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }

    A Realtime item responding to an MCP approval request.

    • id: string

      The unique ID of the approval response.

    • approval_request_id: string

      The ID of the approval request being answered.

    • approve: boolean

      Whether the request was approved.

    • type: "mcp_approval_response"

      The type of the item. Always mcp_approval_response.

      • "mcp_approval_response"

    • reason: optional string or null

      Optional reason for the decision.

  • RealtimeMcpListTools object { server_label, tools, type, id }

    A Realtime item listing tools available on an MCP server.

    • server_label: string

      The label of the MCP server.

    • tools: array of object { input_schema, name, annotations, description }

      The tools available on the server.

      • input_schema: unknown

        The JSON schema describing the tool's input.

      • name: string

        The name of the tool.

      • annotations: optional unknown or null

        Additional annotations about the tool.

      • description: optional string or null

        The description of the tool.

    • type: "mcp_list_tools"

      The type of the item. Always mcp_list_tools.

      • "mcp_list_tools"

    • id: optional string

      The unique ID of the list.

  • RealtimeMcpToolCall object { id, arguments, name, 5 more }

    A Realtime item representing an invocation of a tool on an MCP server.

    • id: string

      The unique ID of the tool call.

    • arguments: string

      A JSON string of the arguments passed to the tool.

    • name: string

      The name of the tool that was run.

    • server_label: string

      The label of the MCP server running the tool.

    • type: "mcp_call"

      The type of the item. Always mcp_call.

      • "mcp_call"

    • approval_request_id: optional string or null

      The ID of an associated approval request, if any.

    • error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError or null

      The error from the tool call, if any.

      • RealtimeMcpProtocolError object { code, message, type }

        • code: number

        • message: string

        • type: "protocol_error"

          • "protocol_error"

      • RealtimeMcpToolExecutionError object { message, type }

        • message: string

        • type: "tool_execution_error"

          • "tool_execution_error"

      • RealtimeMcphttpError object { code, message, type }

        • code: number

        • message: string

        • type: "http_error"

          • "http_error"

    • output: optional string or null

      The output from the tool call.

  • RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }

    A Realtime item requesting human approval of a tool invocation.

    • id: string

      The unique ID of the approval request.

    • arguments: string

      A JSON string of arguments for the tool.

    • name: string

      The name of the tool to run.

    • server_label: string

      The label of the MCP server making the request.

    • type: "mcp_approval_request"

      The type of the item. Always mcp_approval_request.

      • "mcp_approval_request"

• type: "conversation.item.added"

  The event type, must be conversation.item.added.

  • "conversation.item.added"

• previous_item_id: optional string or null

  The ID of the item that precedes this one, if any. This is used to maintain ordering when items are inserted.

Example

{
  "type": "conversation.item.added",
  "event_id": "event_C9G8pjSJCfRNEhMEnYAVy",
  "previous_item_id": null,
  "item": {
    "id": "item_C9G8pGVKYnaZu8PH5YQ9O",
    "type": "message",
    "status": "completed",
    "role": "user",
    "content": [
      {
        "type": "input_text",
        "text": "hi"
      }
    ]
  }
}

conversation.item.done

Returned when a conversation item is finalized.

The event will include the full content of the Item except for audio data, which can be retrieved separately with a conversation.item.retrieve event if needed.

Schema

Schema name: RealtimeServerEventConversationItemDone

• event_id: string

  The unique ID of the server event.

• item: ConversationItem

  A single item within a Realtime conversation.

  • RealtimeConversationItemSystemMessage object { content, role, type, 3 more }

    A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.

    • content: array of object { text, type }

      The content of the message.

      • text: optional string

        The text content.

      • type: optional "input_text"

        The content type. Always input_text for system messages.

        • "input_text"

    • role: "system"

      The role of the message sender. Always system.

      • "system"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemUserMessage object { content, role, type, 3 more }

    A user message item in a Realtime conversation.

    • content: array of object { audio, detail, image_url, 3 more }

      The content of the message.

      • audio: optional string

        Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

      • detail: optional "auto" or "low" or "high"

        The detail level of the image (for input_image). auto will default to high.

        • "auto"

        • "low"

        • "high"

      • image_url: optional string

        Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.

      • text: optional string

        The text content (for input_text).

      • transcript: optional string

        Transcript of the audio (for input_audio). This is not sent to the model, but will be attached to the message item for reference.

      • type: optional "input_text" or "input_audio" or "input_image"

        The content type (input_text, input_audio, or input_image).

        • "input_text"

        • "input_audio"

        • "input_image"

    • role: "user"

      The role of the message sender. Always user.

      • "user"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }

    An assistant message item in a Realtime conversation.

    • content: array of object { audio, text, transcript, type }

      The content of the message.

      • audio: optional string

        Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

      • text: optional string

        The text content.

      • transcript: optional string

        The transcript of the audio content, this will always be present if the output type is audio.

      • type: optional "output_text" or "output_audio"

        The content type, output_text or output_audio depending on the session output_modalities configuration.

        • "output_text"

        • "output_audio"

    • role: "assistant"

      The role of the message sender. Always assistant.

      • "assistant"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }

    A function call item in a Realtime conversation.

    • arguments: string

      The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.

    • name: string

      The name of the function being called.

    • type: "function_call"

      The type of the item. Always function_call.

      • "function_call"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • call_id: optional string

      The ID of the function call.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }

    A function call output item in a Realtime conversation.

    • call_id: string

      The ID of the function call this output is for.

    • output: string

      The output of the function call, this is free text and can contain any information or simply be empty.

    • type: "function_call_output"

      The type of the item. Always function_call_output.

      • "function_call_output"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }

    A Realtime item responding to an MCP approval request.

    • id: string

      The unique ID of the approval response.

    • approval_request_id: string

      The ID of the approval request being answered.

    • approve: boolean

      Whether the request was approved.

    • type: "mcp_approval_response"

      The type of the item. Always mcp_approval_response.

      • "mcp_approval_response"

    • reason: optional string or null

      Optional reason for the decision.

  • RealtimeMcpListTools object { server_label, tools, type, id }

    A Realtime item listing tools available on an MCP server.

    • server_label: string

      The label of the MCP server.

    • tools: array of object { input_schema, name, annotations, description }

      The tools available on the server.

      • input_schema: unknown

        The JSON schema describing the tool's input.

      • name: string

        The name of the tool.

      • annotations: optional unknown or null

        Additional annotations about the tool.

      • description: optional string or null

        The description of the tool.

    • type: "mcp_list_tools"

      The type of the item. Always mcp_list_tools.

      • "mcp_list_tools"

    • id: optional string

      The unique ID of the list.

  • RealtimeMcpToolCall object { id, arguments, name, 5 more }

    A Realtime item representing an invocation of a tool on an MCP server.

    • id: string

      The unique ID of the tool call.

    • arguments: string

      A JSON string of the arguments passed to the tool.

    • name: string

      The name of the tool that was run.

    • server_label: string

      The label of the MCP server running the tool.

    • type: "mcp_call"

      The type of the item. Always mcp_call.

      • "mcp_call"

    • approval_request_id: optional string or null

      The ID of an associated approval request, if any.

    • error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError or null

      The error from the tool call, if any.

      • RealtimeMcpProtocolError object { code, message, type }

        • code: number

        • message: string

        • type: "protocol_error"

          • "protocol_error"

      • RealtimeMcpToolExecutionError object { message, type }

        • message: string

        • type: "tool_execution_error"

          • "tool_execution_error"

      • RealtimeMcphttpError object { code, message, type }

        • code: number

        • message: string

        • type: "http_error"

          • "http_error"

    • output: optional string or null

      The output from the tool call.

  • RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }

    A Realtime item requesting human approval of a tool invocation.

    • id: string

      The unique ID of the approval request.

    • arguments: string

      A JSON string of arguments for the tool.

    • name: string

      The name of the tool to run.

    • server_label: string

      The label of the MCP server making the request.

    • type: "mcp_approval_request"

      The type of the item. Always mcp_approval_request.

      • "mcp_approval_request"

• type: "conversation.item.done"

  The event type, must be conversation.item.done.

  • "conversation.item.done"

• previous_item_id: optional string or null

  The ID of the item that precedes this one, if any. This is used to maintain ordering when items are inserted.

Example

{
  "type": "conversation.item.done",
  "event_id": "event_CCXLgMZPo3qioWCeQa4WH",
  "previous_item_id": "item_CCXLecNJVIVR2HUy3ABLj",
  "item": {
    "id": "item_CCXLfxmM5sXVJVz4mCa2S",
    "type": "message",
    "status": "completed",
    "role": "assistant",
    "content": [
      {
        "type": "output_audio",
        "transcript": "Oh, I can hear you loud and clear! Sounds like we're connected just fine. What can I help you with today?"
      }
    ]
  }
}

conversation.item.retrieved

Returned when a conversation item is retrieved with conversation.item.retrieve. This is provided as a way to fetch the server's representation of an item, for example to get access to the post-processed audio data after noise cancellation and VAD. It includes the full content of the Item, including audio data.

Schema

Schema name: RealtimeServerEventConversationItemRetrieved

• event_id: string

  The unique ID of the server event.

• item: ConversationItem

  A single item within a Realtime conversation.

  • RealtimeConversationItemSystemMessage object { content, role, type, 3 more }

    A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.

    • content: array of object { text, type }

      The content of the message.

      • text: optional string

        The text content.

      • type: optional "input_text"

        The content type. Always input_text for system messages.

        • "input_text"

    • role: "system"

      The role of the message sender. Always system.

      • "system"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemUserMessage object { content, role, type, 3 more }

    A user message item in a Realtime conversation.

    • content: array of object { audio, detail, image_url, 3 more }

      The content of the message.

      • audio: optional string

        Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

      • detail: optional "auto" or "low" or "high"

        The detail level of the image (for input_image). auto will default to high.

        • "auto"

        • "low"

        • "high"

      • image_url: optional string

        Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.

      • text: optional string

        The text content (for input_text).

      • transcript: optional string

        Transcript of the audio (for input_audio). This is not sent to the model, but will be attached to the message item for reference.

      • type: optional "input_text" or "input_audio" or "input_image"

        The content type (input_text, input_audio, or input_image).

        • "input_text"

        • "input_audio"

        • "input_image"

    • role: "user"

      The role of the message sender. Always user.

      • "user"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }

    An assistant message item in a Realtime conversation.

    • content: array of object { audio, text, transcript, type }

      The content of the message.

      • audio: optional string

        Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

      • text: optional string

        The text content.

      • transcript: optional string

        The transcript of the audio content, this will always be present if the output type is audio.

      • type: optional "output_text" or "output_audio"

        The content type, output_text or output_audio depending on the session output_modalities configuration.

        • "output_text"

        • "output_audio"

    • role: "assistant"

      The role of the message sender. Always assistant.

      • "assistant"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }

    A function call item in a Realtime conversation.

    • arguments: string

      The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.

    • name: string

      The name of the function being called.

    • type: "function_call"

      The type of the item. Always function_call.

      • "function_call"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • call_id: optional string

      The ID of the function call.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }

    A function call output item in a Realtime conversation.

    • call_id: string

      The ID of the function call this output is for.

    • output: string

      The output of the function call, this is free text and can contain any information or simply be empty.

    • type: "function_call_output"

      The type of the item. Always function_call_output.

      • "function_call_output"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }

    A Realtime item responding to an MCP approval request.

    • id: string

      The unique ID of the approval response.

    • approval_request_id: string

      The ID of the approval request being answered.

    • approve: boolean

      Whether the request was approved.

    • type: "mcp_approval_response"

      The type of the item. Always mcp_approval_response.

      • "mcp_approval_response"

    • reason: optional string or null

      Optional reason for the decision.

  • RealtimeMcpListTools object { server_label, tools, type, id }

    A Realtime item listing tools available on an MCP server.

    • server_label: string

      The label of the MCP server.

    • tools: array of object { input_schema, name, annotations, description }

      The tools available on the server.

      • input_schema: unknown

        The JSON schema describing the tool's input.

      • name: string

        The name of the tool.

      • annotations: optional unknown or null

        Additional annotations about the tool.

      • description: optional string or null

        The description of the tool.

    • type: "mcp_list_tools"

      The type of the item. Always mcp_list_tools.

      • "mcp_list_tools"

    • id: optional string

      The unique ID of the list.

  • RealtimeMcpToolCall object { id, arguments, name, 5 more }

    A Realtime item representing an invocation of a tool on an MCP server.

    • id: string

      The unique ID of the tool call.

    • arguments: string

      A JSON string of the arguments passed to the tool.

    • name: string

      The name of the tool that was run.

    • server_label: string

      The label of the MCP server running the tool.

    • type: "mcp_call"

      The type of the item. Always mcp_call.

      • "mcp_call"

    • approval_request_id: optional string or null

      The ID of an associated approval request, if any.

    • error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError or null

      The error from the tool call, if any.

      • RealtimeMcpProtocolError object { code, message, type }

        • code: number

        • message: string

        • type: "protocol_error"

          • "protocol_error"

      • RealtimeMcpToolExecutionError object { message, type }

        • message: string

        • type: "tool_execution_error"

          • "tool_execution_error"

      • RealtimeMcphttpError object { code, message, type }

        • code: number

        • message: string

        • type: "http_error"

          • "http_error"

    • output: optional string or null

      The output from the tool call.

  • RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }

    A Realtime item requesting human approval of a tool invocation.

    • id: string

      The unique ID of the approval request.

    • arguments: string

      A JSON string of arguments for the tool.

    • name: string

      The name of the tool to run.

    • server_label: string

      The label of the MCP server making the request.

    • type: "mcp_approval_request"

      The type of the item. Always mcp_approval_request.

      • "mcp_approval_request"

• type: "conversation.item.retrieved"

  The event type, must be conversation.item.retrieved.

  • "conversation.item.retrieved"

Example

{
  "type": "conversation.item.retrieved",
  "event_id": "event_CCXGSizgEppa2d4XbKA7K",
  "item": {
    "id": "item_CCXGRxbY0n6WE4EszhF5w",
    "object": "realtime.item",
    "type": "message",
    "status": "completed",
    "role": "assistant",
    "content": [
      {
        "type": "audio",
        "transcript": "Yes, I can hear you loud and clear. How can I help you today?",
        "audio": "8//2//v/9//q/+//+P/s...",
        "format": "pcm16"
      }
    ]
  }
}

conversation.item.input_audio_transcription.completed

This event is the output of audio transcription for user audio written to the user audio buffer. Transcription begins when the input audio buffer is committed by the client or server (when VAD is enabled). Transcription runs asynchronously with Response creation, so this event may come before or after the Response events.

Realtime API models accept audio natively, and thus input transcription is a separate process run on a separate ASR (Automatic Speech Recognition) model. The transcript may diverge somewhat from the model's interpretation, and should be treated as a rough guide.

Schema

Schema name: RealtimeServerEventConversationItemInputAudioTranscriptionCompleted

• content_index: number

  The index of the content part containing the audio.

• event_id: string

  The unique ID of the server event.

• item_id: string

  The ID of the item containing the audio that is being transcribed.

• transcript: string

  The transcribed text.

• type: "conversation.item.input_audio_transcription.completed"

  The event type, must be conversation.item.input_audio_transcription.completed.

  • "conversation.item.input_audio_transcription.completed"

• usage: object { input_tokens, output_tokens, total_tokens, 2 more } or object { seconds, type }

  Usage statistics for the transcription, this is billed according to the ASR model's pricing rather than the realtime model's pricing.

  • Tokens object { input_tokens, output_tokens, total_tokens, 2 more }

    Usage statistics for models billed by token usage.

    • input_tokens: number

      Number of input tokens billed for this request.

    • output_tokens: number

      Number of output tokens generated.

    • total_tokens: number

      Total number of tokens used (input + output).

    • type: "tokens"

      The type of the usage object. Always tokens for this variant.

      • "tokens"

    • input_token_details: optional object { audio_tokens, text_tokens }

      Details about the input tokens billed for this request.

      • audio_tokens: optional number

        Number of audio tokens billed for this request.

      • text_tokens: optional number

        Number of text tokens billed for this request.

  • Duration object { seconds, type }

    Usage statistics for models billed by audio input duration.

    • seconds: number

      Duration of the input audio in seconds.

    • type: "duration"

      The type of the usage object. Always duration for this variant.

      • "duration"

• languages: optional array of TranscriptionLanguage

  The languages detected in the audio. Returned by gpt-transcribe. An empty array indicates that no language could be reliably detected.

  • code: string

    The code of a language detected in the audio.

• logprobs: optional array of LogProbProperties or null

  The log probabilities of the transcription.

  • token: string

    The token that was used to generate the log probability.

  • bytes: array of number

    The bytes that were used to generate the log probability.

  • logprob: number

    The log probability of the token.

Example

{
  "type": "conversation.item.input_audio_transcription.completed",
  "event_id": "event_CCXGRvtUVrax5SJAnNOWZ",
  "item_id": "item_CCXGQ4e1ht4cOraEYcuR2",
  "content_index": 0,
  "transcript": "Hey, can you hear me?",
  "usage": {
    "type": "tokens",
    "total_tokens": 22,
    "input_tokens": 13,
    "input_token_details": {
      "text_tokens": 0,
      "audio_tokens": 13
    },
    "output_tokens": 9
  }
}

conversation.item.input_audio_transcription.delta

Returned when the text value of an input audio transcription content part is updated with incremental transcription results.

Schema

Schema name: RealtimeServerEventConversationItemInputAudioTranscriptionDelta

• event_id: string

  The unique ID of the server event.

• item_id: string

  The ID of the item containing the audio that is being transcribed.

• type: "conversation.item.input_audio_transcription.delta"

  The event type, must be conversation.item.input_audio_transcription.delta.

  • "conversation.item.input_audio_transcription.delta"

• content_index: optional number

  The index of the content part in the item's content array.

• delta: optional string

  The text delta.

• logprobs: optional array of LogProbProperties or null

  The log probabilities of the transcription. These can be enabled by configurating the session with "include": ["item.input_audio_transcription.logprobs"]. Each entry in the array corresponds a log probability of which token would be selected for this chunk of transcription. This can help to identify if it was possible there were multiple valid options for a given chunk of transcription.

  • token: string

    The token that was used to generate the log probability.

  • bytes: array of number

    The bytes that were used to generate the log probability.

  • logprob: number

    The log probability of the token.

Example

{
  "type": "conversation.item.input_audio_transcription.delta",
  "event_id": "event_CCXGRxsAimPAs8kS2Wc7Z",
  "item_id": "item_CCXGQ4e1ht4cOraEYcuR2",
  "content_index": 0,
  "delta": "Hey",
  "obfuscation": "aLxx0jTEciOGe"
}

conversation.item.input_audio_transcription.segment

Returned when an input audio transcription segment is identified for an item.

Schema

Schema name: RealtimeServerEventConversationItemInputAudioTranscriptionSegment

• id: string

  The segment identifier.

• content_index: number

  The index of the input audio content part within the item.

• end: number

  End time of the segment in seconds.

• event_id: string

  The unique ID of the server event.

• item_id: string

  The ID of the item containing the input audio content.

• speaker: string

  The detected speaker label for this segment.

• start: number

  Start time of the segment in seconds.

• text: string

  The text for this segment.

• type: "conversation.item.input_audio_transcription.segment"

  The event type, must be conversation.item.input_audio_transcription.segment.

  • "conversation.item.input_audio_transcription.segment"

Example

{
    "event_id": "event_6501",
    "type": "conversation.item.input_audio_transcription.segment",
    "item_id": "msg_011",
    "content_index": 0,
    "text": "hello",
    "id": "seg_0001",
    "speaker": "spk_1",
    "start": 0.0,
    "end": 0.4
}

conversation.item.input_audio_transcription.failed

Returned when input audio transcription is configured, and a transcription request for a user message failed. These events are separate from other error events so that the client can identify the related Item.

Schema

Schema name: RealtimeServerEventConversationItemInputAudioTranscriptionFailed

• content_index: number

  The index of the content part containing the audio.

• error: object { code, message, param, type }

  Details of the transcription error.

  • code: optional string

    Error code, if any.

  • message: optional string

    A human-readable error message.

  • param: optional string

    Parameter related to the error, if any.

  • type: optional string

    The type of error.

• event_id: string

  The unique ID of the server event.

• item_id: string

  The ID of the user message item.

• type: "conversation.item.input_audio_transcription.failed"

  The event type, must be conversation.item.input_audio_transcription.failed.

  • "conversation.item.input_audio_transcription.failed"

Example

{
    "event_id": "event_2324",
    "type": "conversation.item.input_audio_transcription.failed",
    "item_id": "msg_003",
    "content_index": 0,
    "error": {
        "type": "transcription_error",
        "code": "audio_unintelligible",
        "message": "The audio could not be transcribed.",
        "param": null
    }
}

conversation.item.truncated

Returned when an earlier assistant audio message item is truncated by the client with a conversation.item.truncate event. This event is used to synchronize the server's understanding of the audio with the client's playback.

This action will truncate the audio and remove the server-side text transcript to ensure there is no text in the context that hasn't been heard by the user.

Schema

Schema name: RealtimeServerEventConversationItemTruncated

• audio_end_ms: number

  The duration up to which the audio was truncated, in milliseconds.

• content_index: number

  The index of the content part that was truncated.

• event_id: string

  The unique ID of the server event.

• item_id: string

  The ID of the assistant message item that was truncated.

• type: "conversation.item.truncated"

  The event type, must be conversation.item.truncated.

  • "conversation.item.truncated"

Example

{
    "event_id": "event_2526",
    "type": "conversation.item.truncated",
    "item_id": "msg_004",
    "content_index": 0,
    "audio_end_ms": 1500
}

conversation.item.deleted

Returned when an item in the conversation is deleted by the client with a conversation.item.delete event. This event is used to synchronize the server's understanding of the conversation history with the client's view.

Schema

Schema name: RealtimeServerEventConversationItemDeleted

• event_id: string

  The unique ID of the server event.

• item_id: string

  The ID of the item that was deleted.

• type: "conversation.item.deleted"

  The event type, must be conversation.item.deleted.

  • "conversation.item.deleted"

Example

{
    "event_id": "event_2728",
    "type": "conversation.item.deleted",
    "item_id": "msg_005"
}

input_audio_buffer.committed

Returned when an input audio buffer is committed, either by the client or automatically in server VAD mode. The item_id property is the ID of the user message item that will be created, thus a conversation.item.created event will also be sent to the client.

Schema

Schema name: RealtimeServerEventInputAudioBufferCommitted

• event_id: string

  The unique ID of the server event.

• item_id: string

  The ID of the user message item that will be created.

• type: "input_audio_buffer.committed"

  The event type, must be input_audio_buffer.committed.

  • "input_audio_buffer.committed"

• previous_item_id: optional string or null

  The ID of the preceding item after which the new item will be inserted. Can be null if the item has no predecessor.

Example

{
    "event_id": "event_1121",
    "type": "input_audio_buffer.committed",
    "previous_item_id": "msg_001",
    "item_id": "msg_002"
}

input_audio_buffer.dtmf_event_received

SIP Only: Returned when an DTMF event is received. A DTMF event is a message that represents a telephone keypad press (0–9, *, #, A–D). The event property is the keypad that the user press. The received_at is the UTC Unix Timestamp that the server received the event.

Schema

Schema name: RealtimeServerEventInputAudioBufferDtmfEventReceived

• event: string

  The telephone keypad that was pressed by the user.

• received_at: number

  UTC Unix Timestamp when DTMF Event was received by server.

• type: "input_audio_buffer.dtmf_event_received"

  The event type, must be input_audio_buffer.dtmf_event_received.

  • "input_audio_buffer.dtmf_event_received"

Example

{
    "type":" input_audio_buffer.dtmf_event_received",
    "event": "9",
    "received_at": 1763605109,
}

input_audio_buffer.cleared

Returned when the input audio buffer is cleared by the client with a input_audio_buffer.clear event.

Schema

Schema name: RealtimeServerEventInputAudioBufferCleared

• event_id: string

  The unique ID of the server event.

• type: "input_audio_buffer.cleared"

  The event type, must be input_audio_buffer.cleared.

  • "input_audio_buffer.cleared"

Example

{
    "event_id": "event_1314",
    "type": "input_audio_buffer.cleared"
}

input_audio_buffer.speech_started

Sent by the server when in server_vad mode to indicate that speech has been detected in the audio buffer. This can happen any time audio is added to the buffer (unless speech is already detected). The client may want to use this event to interrupt audio playback or provide visual feedback to the user.

The client should expect to receive a input_audio_buffer.speech_stopped event when speech stops. The item_id property is the ID of the user message item that will be created when speech stops and will also be included in the input_audio_buffer.speech_stopped event (unless the client manually commits the audio buffer during VAD activation).

Schema

Schema name: RealtimeServerEventInputAudioBufferSpeechStarted

• audio_start_ms: number

  Milliseconds from the start of all audio written to the buffer during the session when speech was first detected. This will correspond to the beginning of audio sent to the model, and thus includes the prefix_padding_ms configured in the Session.

• event_id: string

  The unique ID of the server event.

• item_id: string

  The ID of the user message item that will be created when speech stops.

• type: "input_audio_buffer.speech_started"

  The event type, must be input_audio_buffer.speech_started.

  • "input_audio_buffer.speech_started"

Example

{
    "event_id": "event_1516",
    "type": "input_audio_buffer.speech_started",
    "audio_start_ms": 1000,
    "item_id": "msg_003"
}

input_audio_buffer.speech_stopped

Returned in server_vad mode when the server detects the end of speech in the audio buffer. The server will also send an conversation.item.created event with the user message item that is created from the audio buffer.

Schema

Schema name: RealtimeServerEventInputAudioBufferSpeechStopped

• audio_end_ms: number

  Milliseconds since the session started when speech stopped. This will correspond to the end of audio sent to the model, and thus includes the min_silence_duration_ms configured in the Session.

• event_id: string

  The unique ID of the server event.

• item_id: string

  The ID of the user message item that will be created.

• type: "input_audio_buffer.speech_stopped"

  The event type, must be input_audio_buffer.speech_stopped.

  • "input_audio_buffer.speech_stopped"

Example

{
    "event_id": "event_1718",
    "type": "input_audio_buffer.speech_stopped",
    "audio_end_ms": 2000,
    "item_id": "msg_003"
}

input_audio_buffer.timeout_triggered

Returned when the Server VAD timeout is triggered for the input audio buffer. This is configured with idle_timeout_ms in the turn_detection settings of the session, and it indicates that there hasn't been any speech detected for the configured duration.

The audio_start_ms and audio_end_ms fields indicate the segment of audio after the last model response up to the triggering time, as an offset from the beginning of audio written to the input audio buffer. This means it demarcates the segment of audio that was silent and the difference between the start and end values will roughly match the configured timeout.

The empty audio will be committed to the conversation as an input_audio item (there will be a input_audio_buffer.committed event) and a model response will be generated. There may be speech that didn't trigger VAD but is still detected by the model, so the model may respond with something relevant to the conversation or a prompt to continue speaking.

Schema

Schema name: RealtimeServerEventInputAudioBufferTimeoutTriggered

• audio_end_ms: number

  Millisecond offset of audio written to the input audio buffer at the time the timeout was triggered.

• audio_start_ms: number

  Millisecond offset of audio written to the input audio buffer that was after the playback time of the last model response.

• event_id: string

  The unique ID of the server event.

• item_id: string

  The ID of the item associated with this segment.

• type: "input_audio_buffer.timeout_triggered"

  The event type, must be input_audio_buffer.timeout_triggered.

  • "input_audio_buffer.timeout_triggered"

Example

{
    "type":"input_audio_buffer.timeout_triggered",
    "event_id":"event_CEKKrf1KTGvemCPyiJTJ2",
    "audio_start_ms":13216,
    "audio_end_ms":19232,
    "item_id":"item_CEKKrWH0GiwN0ET97NUZc"
}

output_audio_buffer.started

WebRTC/SIP Only: Emitted when the server begins streaming audio to the client. This event is emitted after an audio content part has been added (response.content_part.added) to the response. Learn more.

Schema

Schema name: RealtimeServerEventOutputAudioBufferStarted

• event_id: string

  The unique ID of the server event.

• response_id: string

  The unique ID of the response that produced the audio.

• type: "output_audio_buffer.started"

  The event type, must be output_audio_buffer.started.

  • "output_audio_buffer.started"

Example

{
    "event_id": "event_abc123",
    "type": "output_audio_buffer.started",
    "response_id": "resp_abc123"
}

output_audio_buffer.stopped

WebRTC/SIP Only: Emitted when the output audio buffer has been completely drained on the server, and no more audio is forthcoming. This event is emitted after the full response data has been sent to the client (response.done). Learn more.

Schema

Schema name: RealtimeServerEventOutputAudioBufferStopped

• event_id: string

  The unique ID of the server event.

• response_id: string

  The unique ID of the response that produced the audio.

• type: "output_audio_buffer.stopped"

  The event type, must be output_audio_buffer.stopped.

  • "output_audio_buffer.stopped"

Example

{
    "event_id": "event_abc123",
    "type": "output_audio_buffer.stopped",
    "response_id": "resp_abc123"
}

output_audio_buffer.cleared

WebRTC/SIP Only: Emitted when the output audio buffer is cleared. This happens either in VAD mode when the user has interrupted (input_audio_buffer.speech_started), or when the client has emitted the output_audio_buffer.clear event to manually cut off the current audio response. Learn more.

Schema

Schema name: RealtimeServerEventOutputAudioBufferCleared

• event_id: string

  The unique ID of the server event.

• response_id: string

  The unique ID of the response that produced the audio.

• type: "output_audio_buffer.cleared"

  The event type, must be output_audio_buffer.cleared.

  • "output_audio_buffer.cleared"

Example

{
    "event_id": "event_abc123",
    "type": "output_audio_buffer.cleared",
    "response_id": "resp_abc123"
}

response.created

Returned when a new Response is created. The first event of response creation, where the response is in an initial state of in_progress.

Schema

Schema name: RealtimeServerEventResponseCreated

• event_id: string

  The unique ID of the server event.

• response: RealtimeResponse

  The response resource.

  • id: optional string

    The unique ID of the response, will look like resp_1234.

  • audio: optional object { output }

    Configuration for audio output.

    • output: optional object { format, voice }

      • format: optional RealtimeAudioFormats

        The format of the output audio.

        • PCMAudio object { rate, type }

          The PCM audio format. Only a 24kHz sample rate is supported.

          • rate: optional 24000

            The sample rate of the audio. Always 24000.

            • 24000

          • type: optional "audio/pcm"

            The audio format. Always audio/pcm.

            • "audio/pcm"

        • PCMUAudio object { type }

          The G.711 μ-law format.

          • type: optional "audio/pcmu"

            The audio format. Always audio/pcmu.

            • "audio/pcmu"

        • PCMAAudio object { type }

          The G.711 A-law format.

          • type: optional "audio/pcma"

            The audio format. Always audio/pcma.

            • "audio/pcma"

      • voice: optional string or "alloy" or "ash" or "ballad" or 7 more

        The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar. We recommend marin and cedar for best quality.

        • string

        • "alloy" or "ash" or "ballad" or 7 more

          The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar. We recommend marin and cedar for best quality.

          • "alloy"

          • "ash"

          • "ballad"

          • "coral"

          • "echo"

          • "sage"

          • "shimmer"

          • "verse"

          • "marin"

          • "cedar"

  • conversation_id: optional string

    Which conversation the response is added to, determined by the conversation field in the response.create event. If auto, the response will be added to the default conversation and the value of conversation_id will be an id like conv_1234. If none, the response will not be added to any conversation and the value of conversation_id will be null. If responses are being triggered automatically by VAD the response will be added to the default conversation

  • max_output_tokens: optional number or "inf"

    Maximum number of output tokens for a single assistant response, inclusive of tool calls, that was used in this response.

    • number

    • "inf"

      • "inf"

  • metadata: optional Metadata or null

    Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

    Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • object: optional "realtime.response"

    The object type, must be realtime.response.

    • "realtime.response"

  • output: optional array of ConversationItem

    The list of output items generated by the response.

    • RealtimeConversationItemSystemMessage object { content, role, type, 3 more }

      A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.

      • content: array of object { text, type }

        The content of the message.

        • text: optional string

          The text content.

        • type: optional "input_text"

          The content type. Always input_text for system messages.

          • "input_text"

      • role: "system"

        The role of the message sender. Always system.

        • "system"

      • type: "message"

        The type of the item. Always message.

        • "message"

      • id: optional string

        The unique ID of the item. This may be provided by the client or generated by the server.

      • object: optional "realtime.item"

        Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

        • "realtime.item"

      • status: optional "completed" or "incomplete" or "in_progress"

        The status of the item. Has no effect on the conversation.

        • "completed"

        • "incomplete"

        • "in_progress"

    • RealtimeConversationItemUserMessage object { content, role, type, 3 more }

      A user message item in a Realtime conversation.

      • content: array of object { audio, detail, image_url, 3 more }

        The content of the message.

        • audio: optional string

          Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

        • detail: optional "auto" or "low" or "high"

          The detail level of the image (for input_image). auto will default to high.

          • "auto"

          • "low"

          • "high"

        • image_url: optional string

          Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.

        • text: optional string

          The text content (for input_text).

        • transcript: optional string

          Transcript of the audio (for input_audio). This is not sent to the model, but will be attached to the message item for reference.

        • type: optional "input_text" or "input_audio" or "input_image"

          The content type (input_text, input_audio, or input_image).

          • "input_text"

          • "input_audio"

          • "input_image"

      • role: "user"

        The role of the message sender. Always user.

        • "user"

      • type: "message"

        The type of the item. Always message.

        • "message"

      • id: optional string

        The unique ID of the item. This may be provided by the client or generated by the server.

      • object: optional "realtime.item"

        Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

        • "realtime.item"

      • status: optional "completed" or "incomplete" or "in_progress"

        The status of the item. Has no effect on the conversation.

        • "completed"

        • "incomplete"

        • "in_progress"

    • RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }

      An assistant message item in a Realtime conversation.

      • content: array of object { audio, text, transcript, type }

        The content of the message.

        • audio: optional string

          Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

        • text: optional string

          The text content.

        • transcript: optional string

          The transcript of the audio content, this will always be present if the output type is audio.

        • type: optional "output_text" or "output_audio"

          The content type, output_text or output_audio depending on the session output_modalities configuration.

          • "output_text"

          • "output_audio"

      • role: "assistant"

        The role of the message sender. Always assistant.

        • "assistant"

      • type: "message"

        The type of the item. Always message.

        • "message"

      • id: optional string

        The unique ID of the item. This may be provided by the client or generated by the server.

      • object: optional "realtime.item"

        Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

        • "realtime.item"

      • status: optional "completed" or "incomplete" or "in_progress"

        The status of the item. Has no effect on the conversation.

        • "completed"

        • "incomplete"

        • "in_progress"

    • RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }

      A function call item in a Realtime conversation.

      • arguments: string

        The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.

      • name: string

        The name of the function being called.

      • type: "function_call"

        The type of the item. Always function_call.

        • "function_call"

      • id: optional string

        The unique ID of the item. This may be provided by the client or generated by the server.

      • call_id: optional string

        The ID of the function call.

      • object: optional "realtime.item"

        Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

        • "realtime.item"

      • status: optional "completed" or "incomplete" or "in_progress"

        The status of the item. Has no effect on the conversation.

        • "completed"

        • "incomplete"

        • "in_progress"

    • RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }

      A function call output item in a Realtime conversation.

      • call_id: string

        The ID of the function call this output is for.

      • output: string

        The output of the function call, this is free text and can contain any information or simply be empty.

      • type: "function_call_output"

        The type of the item. Always function_call_output.

        • "function_call_output"

      • id: optional string

        The unique ID of the item. This may be provided by the client or generated by the server.

      • object: optional "realtime.item"

        Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

        • "realtime.item"

      • status: optional "completed" or "incomplete" or "in_progress"

        The status of the item. Has no effect on the conversation.

        • "completed"

        • "incomplete"

        • "in_progress"

    • RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }

      A Realtime item responding to an MCP approval request.

      • id: string

        The unique ID of the approval response.

      • approval_request_id: string

        The ID of the approval request being answered.

      • approve: boolean

        Whether the request was approved.

      • type: "mcp_approval_response"

        The type of the item. Always mcp_approval_response.

        • "mcp_approval_response"

      • reason: optional string or null

        Optional reason for the decision.

    • RealtimeMcpListTools object { server_label, tools, type, id }

      A Realtime item listing tools available on an MCP server.

      • server_label: string

        The label of the MCP server.

      • tools: array of object { input_schema, name, annotations, description }

        The tools available on the server.

        • input_schema: unknown

          The JSON schema describing the tool's input.

        • name: string

          The name of the tool.

        • annotations: optional unknown or null

          Additional annotations about the tool.

        • description: optional string or null

          The description of the tool.

      • type: "mcp_list_tools"

        The type of the item. Always mcp_list_tools.

        • "mcp_list_tools"

      • id: optional string

        The unique ID of the list.

    • RealtimeMcpToolCall object { id, arguments, name, 5 more }

      A Realtime item representing an invocation of a tool on an MCP server.

      • id: string

        The unique ID of the tool call.

      • arguments: string

        A JSON string of the arguments passed to the tool.

      • name: string

        The name of the tool that was run.

      • server_label: string

        The label of the MCP server running the tool.

      • type: "mcp_call"

        The type of the item. Always mcp_call.

        • "mcp_call"

      • approval_request_id: optional string or null

        The ID of an associated approval request, if any.

      • error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError or null

        The error from the tool call, if any.

        • RealtimeMcpProtocolError object { code, message, type }

          • code: number

          • message: string

          • type: "protocol_error"

            • "protocol_error"

        • RealtimeMcpToolExecutionError object { message, type }

          • message: string

          • type: "tool_execution_error"

            • "tool_execution_error"

        • RealtimeMcphttpError object { code, message, type }

          • code: number

          • message: string

          • type: "http_error"

            • "http_error"

      • output: optional string or null

        The output from the tool call.

    • RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }

      A Realtime item requesting human approval of a tool invocation.

      • id: string

        The unique ID of the approval request.

      • arguments: string

        A JSON string of arguments for the tool.

      • name: string

        The name of the tool to run.

      • server_label: string

        The label of the MCP server making the request.

      • type: "mcp_approval_request"

        The type of the item. Always mcp_approval_request.

        • "mcp_approval_request"

  • output_modalities: optional array of "text" or "audio"

    The set of modalities the model used to respond, currently the only possible values are [\"audio\"], [\"text\"]. Audio output always include a text transcript. Setting the output to mode text will disable audio output from the model.

    • "text"

    • "audio"

  • status: optional "completed" or "cancelled" or "failed" or 2 more

    The final status of the response (completed, cancelled, failed, or incomplete, in_progress).

    • "completed"

    • "cancelled"

    • "failed"

    • "incomplete"

    • "in_progress"

  • status_details: optional RealtimeResponseStatus

    Additional details about the status.

    • error: optional object { code, type }

      A description of the error that caused the response to fail, populated when the status is failed.

      • code: optional string

        Error code, if any.

      • type: optional string

        The type of error.

    • reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"

      The reason the Response did not complete. For a cancelled Response, one of turn_detected (the server VAD detected a new start of speech) or client_cancelled (the client sent a cancel event). For an incomplete Response, one of max_output_tokens or content_filter (the server-side safety filter activated and cut off the response).

      • "turn_detected"

      • "client_cancelled"

      • "max_output_tokens"

      • "content_filter"

    • type: optional "completed" or "cancelled" or "failed" or "incomplete"

      The type of error that caused the response to fail, corresponding with the status field (completed, cancelled, incomplete, failed).

      • "completed"

      • "cancelled"

      • "failed"

      • "incomplete"

  • usage: optional RealtimeResponseUsage

    Usage statistics for the Response, this will correspond to billing. A Realtime API session will maintain a conversation context and append new Items to the Conversation, thus output from previous turns (text and audio tokens) will become the input for later turns.

    • input_token_details: optional RealtimeResponseUsageInputTokenDetails

      Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens.

      • audio_tokens: optional number

        The number of audio tokens used as input for the Response.

      • cached_tokens: optional number

        The number of cached tokens used as input for the Response.

      • cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }

        Details about the cached tokens used as input for the Response.

        • audio_tokens: optional number

          The number of cached audio tokens used as input for the Response.

        • image_tokens: optional number

          The number of cached image tokens used as input for the Response.

        • text_tokens: optional number

          The number of cached text tokens used as input for the Response.

      • image_tokens: optional number

        The number of image tokens used as input for the Response.

      • text_tokens: optional number

        The number of text tokens used as input for the Response.

    • input_tokens: optional number

      The number of input tokens used in the Response, including text and audio tokens.

    • output_token_details: optional RealtimeResponseUsageOutputTokenDetails

      Details about the output tokens used in the Response.

      • audio_tokens: optional number

        The number of audio tokens used in the Response.

      • text_tokens: optional number

        The number of text tokens used in the Response.

    • output_tokens: optional number

      The number of output tokens sent in the Response, including text and audio tokens.

    • total_tokens: optional number

      The total number of tokens in the Response including input and output text and audio tokens.

• type: "response.created"

  The event type, must be response.created.

  • "response.created"

Example

{
  "type": "response.created",
  "event_id": "event_C9G8pqbTEddBSIxbBN6Os",
  "response": {
    "object": "realtime.response",
    "id": "resp_C9G8p7IH2WxLbkgPNouYL",
    "status": "in_progress",
    "status_details": null,
    "output": [],
    "conversation_id": "conv_C9G8mmBkLhQJwCon3hoJN",
    "output_modalities": [
      "audio"
    ],
    "max_output_tokens": "inf",
    "audio": {
      "output": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "voice": "marin"
      }
    },
    "usage": null,
    "metadata": null
  },
}

response.done

Returned when a Response is done streaming. Always emitted, no matter the final state. The Response object included in the response.done event will include all output Items in the Response but will omit the raw audio data.

Clients should check the status field of the Response to determine if it was successful (completed) or if there was another outcome: cancelled, failed, or incomplete.

A response will contain all output items that were generated during the response, excluding any audio content.

Schema

Schema name: RealtimeServerEventResponseDone

• event_id: string

  The unique ID of the server event.

• response: RealtimeResponse

  The response resource.

  • id: optional string

    The unique ID of the response, will look like resp_1234.

  • audio: optional object { output }

    Configuration for audio output.

    • output: optional object { format, voice }

      • format: optional RealtimeAudioFormats

        The format of the output audio.

        • PCMAudio object { rate, type }

          The PCM audio format. Only a 24kHz sample rate is supported.

          • rate: optional 24000

            The sample rate of the audio. Always 24000.

            • 24000

          • type: optional "audio/pcm"

            The audio format. Always audio/pcm.

            • "audio/pcm"

        • PCMUAudio object { type }

          The G.711 μ-law format.

          • type: optional "audio/pcmu"

            The audio format. Always audio/pcmu.

            • "audio/pcmu"

        • PCMAAudio object { type }

          The G.711 A-law format.

          • type: optional "audio/pcma"

            The audio format. Always audio/pcma.

            • "audio/pcma"

      • voice: optional string or "alloy" or "ash" or "ballad" or 7 more

        The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar. We recommend marin and cedar for best quality.

        • string

        • "alloy" or "ash" or "ballad" or 7 more

          The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar. We recommend marin and cedar for best quality.

          • "alloy"

          • "ash"

          • "ballad"

          • "coral"

          • "echo"

          • "sage"

          • "shimmer"

          • "verse"

          • "marin"

          • "cedar"

  • conversation_id: optional string

    Which conversation the response is added to, determined by the conversation field in the response.create event. If auto, the response will be added to the default conversation and the value of conversation_id will be an id like conv_1234. If none, the response will not be added to any conversation and the value of conversation_id will be null. If responses are being triggered automatically by VAD the response will be added to the default conversation

  • max_output_tokens: optional number or "inf"

    Maximum number of output tokens for a single assistant response, inclusive of tool calls, that was used in this response.

    • number

    • "inf"

      • "inf"

  • metadata: optional Metadata or null

    Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

    Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • object: optional "realtime.response"

    The object type, must be realtime.response.

    • "realtime.response"

  • output: optional array of ConversationItem

    The list of output items generated by the response.

    • RealtimeConversationItemSystemMessage object { content, role, type, 3 more }

      A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.

      • content: array of object { text, type }

        The content of the message.

        • text: optional string

          The text content.

        • type: optional "input_text"

          The content type. Always input_text for system messages.

          • "input_text"

      • role: "system"

        The role of the message sender. Always system.

        • "system"

      • type: "message"

        The type of the item. Always message.

        • "message"

      • id: optional string

        The unique ID of the item. This may be provided by the client or generated by the server.

      • object: optional "realtime.item"

        Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

        • "realtime.item"

      • status: optional "completed" or "incomplete" or "in_progress"

        The status of the item. Has no effect on the conversation.

        • "completed"

        • "incomplete"

        • "in_progress"

    • RealtimeConversationItemUserMessage object { content, role, type, 3 more }

      A user message item in a Realtime conversation.

      • content: array of object { audio, detail, image_url, 3 more }

        The content of the message.

        • audio: optional string

          Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

        • detail: optional "auto" or "low" or "high"

          The detail level of the image (for input_image). auto will default to high.

          • "auto"

          • "low"

          • "high"

        • image_url: optional string

          Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.

        • text: optional string

          The text content (for input_text).

        • transcript: optional string

          Transcript of the audio (for input_audio). This is not sent to the model, but will be attached to the message item for reference.

        • type: optional "input_text" or "input_audio" or "input_image"

          The content type (input_text, input_audio, or input_image).

          • "input_text"

          • "input_audio"

          • "input_image"

      • role: "user"

        The role of the message sender. Always user.

        • "user"

      • type: "message"

        The type of the item. Always message.

        • "message"

      • id: optional string

        The unique ID of the item. This may be provided by the client or generated by the server.

      • object: optional "realtime.item"

        Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

        • "realtime.item"

      • status: optional "completed" or "incomplete" or "in_progress"

        The status of the item. Has no effect on the conversation.

        • "completed"

        • "incomplete"

        • "in_progress"

    • RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }

      An assistant message item in a Realtime conversation.

      • content: array of object { audio, text, transcript, type }

        The content of the message.

        • audio: optional string

          Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

        • text: optional string

          The text content.

        • transcript: optional string

          The transcript of the audio content, this will always be present if the output type is audio.

        • type: optional "output_text" or "output_audio"

          The content type, output_text or output_audio depending on the session output_modalities configuration.

          • "output_text"

          • "output_audio"

      • role: "assistant"

        The role of the message sender. Always assistant.

        • "assistant"

      • type: "message"

        The type of the item. Always message.

        • "message"

      • id: optional string

        The unique ID of the item. This may be provided by the client or generated by the server.

      • object: optional "realtime.item"

        Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

        • "realtime.item"

      • status: optional "completed" or "incomplete" or "in_progress"

        The status of the item. Has no effect on the conversation.

        • "completed"

        • "incomplete"

        • "in_progress"

    • RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }

      A function call item in a Realtime conversation.

      • arguments: string

        The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.

      • name: string

        The name of the function being called.

      • type: "function_call"

        The type of the item. Always function_call.

        • "function_call"

      • id: optional string

        The unique ID of the item. This may be provided by the client or generated by the server.

      • call_id: optional string

        The ID of the function call.

      • object: optional "realtime.item"

        Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

        • "realtime.item"

      • status: optional "completed" or "incomplete" or "in_progress"

        The status of the item. Has no effect on the conversation.

        • "completed"

        • "incomplete"

        • "in_progress"

    • RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }

      A function call output item in a Realtime conversation.

      • call_id: string

        The ID of the function call this output is for.

      • output: string

        The output of the function call, this is free text and can contain any information or simply be empty.

      • type: "function_call_output"

        The type of the item. Always function_call_output.

        • "function_call_output"

      • id: optional string

        The unique ID of the item. This may be provided by the client or generated by the server.

      • object: optional "realtime.item"

        Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

        • "realtime.item"

      • status: optional "completed" or "incomplete" or "in_progress"

        The status of the item. Has no effect on the conversation.

        • "completed"

        • "incomplete"

        • "in_progress"

    • RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }

      A Realtime item responding to an MCP approval request.

      • id: string

        The unique ID of the approval response.

      • approval_request_id: string

        The ID of the approval request being answered.

      • approve: boolean

        Whether the request was approved.

      • type: "mcp_approval_response"

        The type of the item. Always mcp_approval_response.

        • "mcp_approval_response"

      • reason: optional string or null

        Optional reason for the decision.

    • RealtimeMcpListTools object { server_label, tools, type, id }

      A Realtime item listing tools available on an MCP server.

      • server_label: string

        The label of the MCP server.

      • tools: array of object { input_schema, name, annotations, description }

        The tools available on the server.

        • input_schema: unknown

          The JSON schema describing the tool's input.

        • name: string

          The name of the tool.

        • annotations: optional unknown or null

          Additional annotations about the tool.

        • description: optional string or null

          The description of the tool.

      • type: "mcp_list_tools"

        The type of the item. Always mcp_list_tools.

        • "mcp_list_tools"

      • id: optional string

        The unique ID of the list.

    • RealtimeMcpToolCall object { id, arguments, name, 5 more }

      A Realtime item representing an invocation of a tool on an MCP server.

      • id: string

        The unique ID of the tool call.

      • arguments: string

        A JSON string of the arguments passed to the tool.

      • name: string

        The name of the tool that was run.

      • server_label: string

        The label of the MCP server running the tool.

      • type: "mcp_call"

        The type of the item. Always mcp_call.

        • "mcp_call"

      • approval_request_id: optional string or null

        The ID of an associated approval request, if any.

      • error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError or null

        The error from the tool call, if any.

        • RealtimeMcpProtocolError object { code, message, type }

          • code: number

          • message: string

          • type: "protocol_error"

            • "protocol_error"

        • RealtimeMcpToolExecutionError object { message, type }

          • message: string

          • type: "tool_execution_error"

            • "tool_execution_error"

        • RealtimeMcphttpError object { code, message, type }

          • code: number

          • message: string

          • type: "http_error"

            • "http_error"

      • output: optional string or null

        The output from the tool call.

    • RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }

      A Realtime item requesting human approval of a tool invocation.

      • id: string

        The unique ID of the approval request.

      • arguments: string

        A JSON string of arguments for the tool.

      • name: string

        The name of the tool to run.

      • server_label: string

        The label of the MCP server making the request.

      • type: "mcp_approval_request"

        The type of the item. Always mcp_approval_request.

        • "mcp_approval_request"

  • output_modalities: optional array of "text" or "audio"

    The set of modalities the model used to respond, currently the only possible values are [\"audio\"], [\"text\"]. Audio output always include a text transcript. Setting the output to mode text will disable audio output from the model.

    • "text"

    • "audio"

  • status: optional "completed" or "cancelled" or "failed" or 2 more

    The final status of the response (completed, cancelled, failed, or incomplete, in_progress).

    • "completed"

    • "cancelled"

    • "failed"

    • "incomplete"

    • "in_progress"

  • status_details: optional RealtimeResponseStatus

    Additional details about the status.

    • error: optional object { code, type }

      A description of the error that caused the response to fail, populated when the status is failed.

      • code: optional string

        Error code, if any.

      • type: optional string

        The type of error.

    • reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"

      The reason the Response did not complete. For a cancelled Response, one of turn_detected (the server VAD detected a new start of speech) or client_cancelled (the client sent a cancel event). For an incomplete Response, one of max_output_tokens or content_filter (the server-side safety filter activated and cut off the response).

      • "turn_detected"

      • "client_cancelled"

      • "max_output_tokens"

      • "content_filter"

    • type: optional "completed" or "cancelled" or "failed" or "incomplete"

      The type of error that caused the response to fail, corresponding with the status field (completed, cancelled, incomplete, failed).

      • "completed"

      • "cancelled"

      • "failed"

      • "incomplete"

  • usage: optional RealtimeResponseUsage

    Usage statistics for the Response, this will correspond to billing. A Realtime API session will maintain a conversation context and append new Items to the Conversation, thus output from previous turns (text and audio tokens) will become the input for later turns.

    • input_token_details: optional RealtimeResponseUsageInputTokenDetails

      Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens.

      • audio_tokens: optional number

        The number of audio tokens used as input for the Response.

      • cached_tokens: optional number

        The number of cached tokens used as input for the Response.

      • cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }

        Details about the cached tokens used as input for the Response.

        • audio_tokens: optional number

          The number of cached audio tokens used as input for the Response.

        • image_tokens: optional number

          The number of cached image tokens used as input for the Response.

        • text_tokens: optional number

          The number of cached text tokens used as input for the Response.

      • image_tokens: optional number

        The number of image tokens used as input for the Response.

      • text_tokens: optional number

        The number of text tokens used as input for the Response.

    • input_tokens: optional number

      The number of input tokens used in the Response, including text and audio tokens.

    • output_token_details: optional RealtimeResponseUsageOutputTokenDetails

      Details about the output tokens used in the Response.

      • audio_tokens: optional number

        The number of audio tokens used in the Response.

      • text_tokens: optional number

        The number of text tokens used in the Response.

    • output_tokens: optional number

      The number of output tokens sent in the Response, including text and audio tokens.

    • total_tokens: optional number

      The total number of tokens in the Response including input and output text and audio tokens.

• type: "response.done"

  The event type, must be response.done.

  • "response.done"

Example

{
  "type": "response.done",
  "event_id": "event_CCXHxcMy86rrKhBLDdqCh",
  "response": {
    "object": "realtime.response",
    "id": "resp_CCXHw0UJld10EzIUXQCNh",
    "status": "completed",
    "status_details": null,
    "output": [
      {
        "id": "item_CCXHwGjjDUfOXbiySlK7i",
        "type": "message",
        "status": "completed",
        "role": "assistant",
        "content": [
          {
            "type": "output_audio",
            "transcript": "Loud and clear! I can hear you perfectly. How can I help you today?"
          }
        ]
      }
    ],
    "conversation_id": "conv_CCXHsurMKcaVxIZvaCI5m",
    "output_modalities": [
      "audio"
    ],
    "max_output_tokens": "inf",
    "audio": {
      "output": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "voice": "alloy"
      }
    },
    "usage": {
      "total_tokens": 253,
      "input_tokens": 132,
      "output_tokens": 121,
      "input_token_details": {
        "text_tokens": 119,
        "audio_tokens": 13,
        "image_tokens": 0,
        "cached_tokens": 64,
        "cached_tokens_details": {
          "text_tokens": 64,
          "audio_tokens": 0,
          "image_tokens": 0
        }
      },
      "output_token_details": {
        "text_tokens": 30,
        "audio_tokens": 91
      }
    },
    "metadata": null
  }
}

response.output_item.added

Returned when a new Item is created during Response generation.

Schema

Schema name: RealtimeServerEventResponseOutputItemAdded

• event_id: string

  The unique ID of the server event.

• item: ConversationItem

  A single item within a Realtime conversation.

  • RealtimeConversationItemSystemMessage object { content, role, type, 3 more }

    A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.

    • content: array of object { text, type }

      The content of the message.

      • text: optional string

        The text content.

      • type: optional "input_text"

        The content type. Always input_text for system messages.

        • "input_text"

    • role: "system"

      The role of the message sender. Always system.

      • "system"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemUserMessage object { content, role, type, 3 more }

    A user message item in a Realtime conversation.

    • content: array of object { audio, detail, image_url, 3 more }

      The content of the message.

      • audio: optional string

        Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

      • detail: optional "auto" or "low" or "high"

        The detail level of the image (for input_image). auto will default to high.

        • "auto"

        • "low"

        • "high"

      • image_url: optional string

        Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.

      • text: optional string

        The text content (for input_text).

      • transcript: optional string

        Transcript of the audio (for input_audio). This is not sent to the model, but will be attached to the message item for reference.

      • type: optional "input_text" or "input_audio" or "input_image"

        The content type (input_text, input_audio, or input_image).

        • "input_text"

        • "input_audio"

        • "input_image"

    • role: "user"

      The role of the message sender. Always user.

      • "user"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }

    An assistant message item in a Realtime conversation.

    • content: array of object { audio, text, transcript, type }

      The content of the message.

      • audio: optional string

        Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

      • text: optional string

        The text content.

      • transcript: optional string

        The transcript of the audio content, this will always be present if the output type is audio.

      • type: optional "output_text" or "output_audio"

        The content type, output_text or output_audio depending on the session output_modalities configuration.

        • "output_text"

        • "output_audio"

    • role: "assistant"

      The role of the message sender. Always assistant.

      • "assistant"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }

    A function call item in a Realtime conversation.

    • arguments: string

      The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.

    • name: string

      The name of the function being called.

    • type: "function_call"

      The type of the item. Always function_call.

      • "function_call"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • call_id: optional string

      The ID of the function call.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }

    A function call output item in a Realtime conversation.

    • call_id: string

      The ID of the function call this output is for.

    • output: string

      The output of the function call, this is free text and can contain any information or simply be empty.

    • type: "function_call_output"

      The type of the item. Always function_call_output.

      • "function_call_output"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }

    A Realtime item responding to an MCP approval request.

    • id: string

      The unique ID of the approval response.

    • approval_request_id: string

      The ID of the approval request being answered.

    • approve: boolean

      Whether the request was approved.

    • type: "mcp_approval_response"

      The type of the item. Always mcp_approval_response.

      • "mcp_approval_response"

    • reason: optional string or null

      Optional reason for the decision.

  • RealtimeMcpListTools object { server_label, tools, type, id }

    A Realtime item listing tools available on an MCP server.

    • server_label: string

      The label of the MCP server.

    • tools: array of object { input_schema, name, annotations, description }

      The tools available on the server.

      • input_schema: unknown

        The JSON schema describing the tool's input.

      • name: string

        The name of the tool.

      • annotations: optional unknown or null

        Additional annotations about the tool.

      • description: optional string or null

        The description of the tool.

    • type: "mcp_list_tools"

      The type of the item. Always mcp_list_tools.

      • "mcp_list_tools"

    • id: optional string

      The unique ID of the list.

  • RealtimeMcpToolCall object { id, arguments, name, 5 more }

    A Realtime item representing an invocation of a tool on an MCP server.

    • id: string

      The unique ID of the tool call.

    • arguments: string

      A JSON string of the arguments passed to the tool.

    • name: string

      The name of the tool that was run.

    • server_label: string

      The label of the MCP server running the tool.

    • type: "mcp_call"

      The type of the item. Always mcp_call.

      • "mcp_call"

    • approval_request_id: optional string or null

      The ID of an associated approval request, if any.

    • error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError or null

      The error from the tool call, if any.

      • RealtimeMcpProtocolError object { code, message, type }

        • code: number

        • message: string

        • type: "protocol_error"

          • "protocol_error"

      • RealtimeMcpToolExecutionError object { message, type }

        • message: string

        • type: "tool_execution_error"

          • "tool_execution_error"

      • RealtimeMcphttpError object { code, message, type }

        • code: number

        • message: string

        • type: "http_error"

          • "http_error"

    • output: optional string or null

      The output from the tool call.

  • RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }

    A Realtime item requesting human approval of a tool invocation.

    • id: string

      The unique ID of the approval request.

    • arguments: string

      A JSON string of arguments for the tool.

    • name: string

      The name of the tool to run.

    • server_label: string

      The label of the MCP server making the request.

    • type: "mcp_approval_request"

      The type of the item. Always mcp_approval_request.

      • "mcp_approval_request"

• output_index: number

  The index of the output item in the Response.

• response_id: string

  The ID of the Response to which the item belongs.

• type: "response.output_item.added"

  The event type, must be response.output_item.added.

  • "response.output_item.added"

Example

{
    "event_id": "event_3334",
    "type": "response.output_item.added",
    "response_id": "resp_001",
    "output_index": 0,
    "item": {
        "id": "msg_007",
        "object": "realtime.item",
        "type": "message",
        "status": "in_progress",
        "role": "assistant",
        "content": []
    }
}

response.output_item.done

Returned when an Item is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Schema

Schema name: RealtimeServerEventResponseOutputItemDone

• event_id: string

  The unique ID of the server event.

• item: ConversationItem

  A single item within a Realtime conversation.

  • RealtimeConversationItemSystemMessage object { content, role, type, 3 more }

    A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.

    • content: array of object { text, type }

      The content of the message.

      • text: optional string

        The text content.

      • type: optional "input_text"

        The content type. Always input_text for system messages.

        • "input_text"

    • role: "system"

      The role of the message sender. Always system.

      • "system"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemUserMessage object { content, role, type, 3 more }

    A user message item in a Realtime conversation.

    • content: array of object { audio, detail, image_url, 3 more }

      The content of the message.

      • audio: optional string

        Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

      • detail: optional "auto" or "low" or "high"

        The detail level of the image (for input_image). auto will default to high.

        • "auto"

        • "low"

        • "high"

      • image_url: optional string

        Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.

      • text: optional string

        The text content (for input_text).

      • transcript: optional string

        Transcript of the audio (for input_audio). This is not sent to the model, but will be attached to the message item for reference.

      • type: optional "input_text" or "input_audio" or "input_image"

        The content type (input_text, input_audio, or input_image).

        • "input_text"

        • "input_audio"

        • "input_image"

    • role: "user"

      The role of the message sender. Always user.

      • "user"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }

    An assistant message item in a Realtime conversation.

    • content: array of object { audio, text, transcript, type }

      The content of the message.

      • audio: optional string

        Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

      • text: optional string

        The text content.

      • transcript: optional string

        The transcript of the audio content, this will always be present if the output type is audio.

      • type: optional "output_text" or "output_audio"

        The content type, output_text or output_audio depending on the session output_modalities configuration.

        • "output_text"

        • "output_audio"

    • role: "assistant"

      The role of the message sender. Always assistant.

      • "assistant"

    • type: "message"

      The type of the item. Always message.

      • "message"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }

    A function call item in a Realtime conversation.

    • arguments: string

      The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.

    • name: string

      The name of the function being called.

    • type: "function_call"

      The type of the item. Always function_call.

      • "function_call"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • call_id: optional string

      The ID of the function call.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }

    A function call output item in a Realtime conversation.

    • call_id: string

      The ID of the function call this output is for.

    • output: string

      The output of the function call, this is free text and can contain any information or simply be empty.

    • type: "function_call_output"

      The type of the item. Always function_call_output.

      • "function_call_output"

    • id: optional string

      The unique ID of the item. This may be provided by the client or generated by the server.

    • object: optional "realtime.item"

      Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

      • "realtime.item"

    • status: optional "completed" or "incomplete" or "in_progress"

      The status of the item. Has no effect on the conversation.

      • "completed"

      • "incomplete"

      • "in_progress"

  • RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }

    A Realtime item responding to an MCP approval request.

    • id: string

      The unique ID of the approval response.

    • approval_request_id: string

      The ID of the approval request being answered.

    • approve: boolean

      Whether the request was approved.

    • type: "mcp_approval_response"

      The type of the item. Always mcp_approval_response.

      • "mcp_approval_response"

    • reason: optional string or null

      Optional reason for the decision.

  • RealtimeMcpListTools object { server_label, tools, type, id }

    A Realtime item listing tools available on an MCP server.

    • server_label: string

      The label of the MCP server.

    • tools: array of object { input_schema, name, annotations, description }

      The tools available on the server.

      • input_schema: unknown

        The JSON schema describing the tool's input.

      • name: string

        The name of the tool.

      • annotations: optional unknown or null

        Additional annotations about the tool.

      • description: optional string or null

        The description of the tool.

    • type: "mcp_list_tools"

      The type of the item. Always mcp_list_tools.

      • "mcp_list_tools"

    • id: optional string

      The unique ID of the list.

  • RealtimeMcpToolCall object { id, arguments, name, 5 more }

    A Realtime item representing an invocation of a tool on an MCP server.

    • id: string

      The unique ID of the tool call.

    • arguments: string

      A JSON string of the arguments passed to the tool.

    • name: string

      The name of the tool that was run.

    • server_label: string

      The label of the MCP server running the tool.

    • type: "mcp_call"

      The type of the item. Always mcp_call.

      • "mcp_call"

    • approval_request_id: optional string or null

      The ID of an associated approval request, if any.

    • error: optional RealtimeMcpProtocolError or RealtimeMcpToolExecutionError or RealtimeMcphttpError or null

      The error from the tool call, if any.

      • RealtimeMcpProtocolError object { code, message, type }

        • code: number

        • message: string

        • type: "protocol_error"

          • "protocol_error"

      • RealtimeMcpToolExecutionError object { message, type }

        • message: string

        • type: "tool_execution_error"

          • "tool_execution_error"

      • RealtimeMcphttpError object { code, message, type }

        • code: number

        • message: string

        • type: "http_error"

          • "http_error"

    • output: optional string or null

      The output from the tool call.

  • RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }

    A Realtime item requesting human approval of a tool invocation.

    • id: string

      The unique ID of the approval request.

    • arguments: string

      A JSON string of arguments for the tool.

    • name: string

      The name of the tool to run.

    • server_label: string

      The label of the MCP server making the request.

    • type: "mcp_approval_request"

      The type of the item. Always mcp_approval_request.

      • "mcp_approval_request"

• output_index: number

  The index of the output item in the Response.

• response_id: string

  The ID of the Response to which the item belongs.

• type: "response.output_item.done"

  The event type, must be response.output_item.done.

  • "response.output_item.done"