diff --git a/en/ruby/resources/evals/subresources/runs/index.md b/en/ruby/resources/evals/subresources/runs/index.md deleted file mode 100644 index f593732..0000000 --- a/en/ruby/resources/evals/subresources/runs/index.md +++ /dev/null @@ -1,19766 +0,0 @@ -# Runs - -## Get eval runs - -`evals.runs.list(eval_id, **kwargs) -> CursorPage` - -**get** `/evals/{eval_id}/runs` - -Get a list of runs for an evaluation. - -### Parameters - -- `eval_id: String` - -- `after: String` - - Identifier for the last run from the previous pagination request. - -- `limit: Integer` - - Number of runs to retrieve. - -- `order: :asc | :desc` - - Sort order for runs by timestamp. Use `asc` for ascending order or `desc` for descending order. Defaults to `asc`. - - - `:asc` - - - `:desc` - -- `status: :queued | :in_progress | :completed | 2 more` - - Filter runs by status. One of `queued` | `in_progress` | `failed` | `completed` | `canceled`. - - - `:queued` - - - `:in_progress` - - - `:completed` - - - `:canceled` - - - `:failed` - -### Returns - -- `class RunListResponse` - - A schema representing an evaluation run. - - - `id: String` - - Unique identifier for the evaluation run. - - - `created_at: Integer` - - Unix timestamp (in seconds) when the evaluation run was created. - - - `data_source: CreateEvalJSONLRunDataSource | CreateEvalCompletionsRunDataSource | Responses{ source, type, input_messages, 2 more}` - - Information about the run's data source. - - - `class CreateEvalJSONLRunDataSource` - - A JsonlRunDataSource object with that specifies a JSONL file that matches the eval - - - `source: FileContent{ content, type} | FileID{ id, type}` - - Determines what populates the `item` namespace in the data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `type: :jsonl` - - The type of data source. Always `jsonl`. - - - `:jsonl` - - - `class CreateEvalCompletionsRunDataSource` - - A CompletionsRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | StoredCompletions{ type, created_after, created_before, 3 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class StoredCompletions` - - A StoredCompletionsRunDataSource configuration describing a set of filters - - - `type: :stored_completions` - - The type of source. Always `stored_completions`. - - - `:stored_completions` - - - `created_after: Integer` - - An optional Unix timestamp to filter items created after this time. - - - `created_before: Integer` - - An optional Unix timestamp to filter items created before this time. - - - `limit: Integer` - - An optional maximum number of items to return. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - An optional model to filter by (e.g., 'gpt-4o'). - - - `type: :completions` - - The type of run data source. Always `completions`. - - - `:completions` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[EasyInputMessage | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class EasyInputMessage` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputMessageContentList` - - Text, image, or audio input to the model, used to generate a response. - Can also contain previous assistant responses. - - - `String = String` - - A text input to the model. - - - `ResponseInputMessageContentList = Array[ResponseInputContent]` - - A list of one or many input items to the model, containing different content - types. - - - `class ResponseInputText` - - A text input to the model. - - - `text: String` - - The text input to the model. - - - `type: :input_text` - - The type of the input item. Always `input_text`. - - - `:input_text` - - - `class ResponseInputImage` - - An image input to the model. Learn about [image inputs](https://platform.openai.com/docs/guides/vision). - - - `detail: :low | :high | :auto | :original` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `:low` - - - `:high` - - - `:auto` - - - `:original` - - - `type: :input_image` - - The type of the input item. Always `input_image`. - - - `:input_image` - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `image_url: String` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `class ResponseInputFile` - - A file input to the model. - - - `type: :input_file` - - The type of the input item. Always `input_file`. - - - `:input_file` - - - `detail: :low | :high` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `:low` - - - `:high` - - - `file_data: String` - - The content of the file to be sent to the model. - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `file_url: String` - - The URL of the file to be sent to the model. - - - `filename: String` - - The name of the file to be sent to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `phase: :commentary | :final_answer` - - Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`). - For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend - phase on all assistant messages — dropping it can degrade performance. Not used for user messages. - - - `:commentary` - - - `:final_answer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `input_audio: InputAudio{ data, format_}` - - - `data: String` - - Base64-encoded audio data. - - - `format_: :mp3 | :wav` - - The format of the audio data. Currently supported formats are `mp3` and - `wav`. - - - `:mp3` - - - `:wav` - - - `type: :input_audio` - - The type of the input item. Always `input_audio`. - - - `:input_audio` - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.input_trajectory" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, response_format, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `:none` - - - `:minimal` - - - `:low` - - - `:medium` - - - `:high` - - - `:xhigh` - - - `response_format: ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject` - - An object specifying the format that the model must output. - - Setting to `{ "type": "json_schema", "json_schema": {...} }` enables - Structured Outputs which ensures the model will match your supplied JSON - schema. Learn more in the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `type: :text` - - The type of response format being defined. Always `text`. - - - `:text` - - - `class ResponseFormatJSONSchema` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `json_schema: JSONSchema{ name, description, schema, strict}` - - Structured Outputs configuration options, including a JSON Schema. - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `type: :json_object` - - The type of response format being defined. Always `json_object`. - - - `:json_object` - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `tools: Array[ChatCompletionFunctionTool]` - - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported. - - - `function: FunctionDefinition` - - - `name: String` - - The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the function does, used by the model to choose when and how to call the function. - - - `parameters: FunctionParameters` - - The parameters the functions accepts, described as a JSON Schema object. See the [guide](https://platform.openai.com/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format. - - Omitting `parameters` defines a function with an empty parameter list. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](https://platform.openai.com/docs/guides/function-calling). - - - `type: :function` - - The type of the tool. Currently, only `function` is supported. - - - `:function` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `class Responses` - - A ResponsesRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | Responses{ type, created_after, created_before, 8 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class Responses` - - A EvalResponsesSource object describing a run data source configuration. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `created_after: Integer` - - Only include items created after this timestamp (inclusive). This is a query parameter used to select responses. - - - `created_before: Integer` - - Only include items created before this timestamp (inclusive). This is a query parameter used to select responses. - - - `instructions_search: String` - - Optional string to search the 'instructions' field. This is a query parameter used to select responses. - - - `metadata: untyped` - - Metadata filter for the responses. This is a query parameter used to select responses. - - - `model: String` - - The name of the model to find responses for. This is a query parameter used to select responses. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `temperature: Float` - - Sampling temperature. This is a query parameter used to select responses. - - - `tools: Array[String]` - - List of tool names. This is a query parameter used to select responses. - - - `top_p: Float` - - Nucleus sampling parameter. This is a query parameter used to select responses. - - - `users: Array[String]` - - List of user identifiers. This is a query parameter used to select responses. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[ChatMessage{ content, role} | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class ChatMessage` - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.name" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, seed, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `text: Text{ format_}` - - Configuration options for a text response from the model. Can be plain - text or structured JSON data. Learn more: - - - [Text inputs and outputs](https://platform.openai.com/docs/guides/text) - - [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) - - - `format_: ResponseFormatTextConfig` - - An object specifying the format that the model must output. - - Configuring `{ "type": "json_schema" }` enables Structured Outputs, - which ensures the model will match your supplied JSON schema. Learn more in the - [Structured Outputs guide](https://platform.openai.com/docs/guides/structured-outputs). - - The default format is `{ "type": "text" }` with no additional options. - - **Not recommended for gpt-4o and newer models:** - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `class ResponseFormatTextJSONSchemaConfig` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `tools: Array[Tool]` - - An array of tools the model may call while generating a response. You - can specify which tool to use by setting the `tool_choice` parameter. - - The two categories of tools you can provide the model are: - - - **Built-in tools**: Tools that are provided by OpenAI that extend the - model's capabilities, like [web search](https://platform.openai.com/docs/guides/tools-web-search) - or [file search](https://platform.openai.com/docs/guides/tools-file-search). Learn more about - [built-in tools](https://platform.openai.com/docs/guides/tools). - - **Function calls (custom tools)**: Functions that are defined by you, - enabling the model to call your own code. Learn more about - [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `class FunctionTool` - - Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `name: String` - - The name of the function to call. - - - `parameters: Hash[Symbol, untyped]` - - A JSON schema object describing the parameters of the function. - - - `strict: bool` - - Whether to enforce strict parameter validation. Default `true`. - - - `type: :function` - - The type of the function tool. Always `function`. - - - `:function` - - - `defer_loading: bool` - - Whether this function is deferred and loaded via tool search. - - - `description: String` - - A description of the function. Used by the model to determine whether or not to call the function. - - - `class FileSearchTool` - - A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search). - - - `type: :file_search` - - The type of the file search tool. Always `file_search`. - - - `:file_search` - - - `vector_store_ids: Array[String]` - - The IDs of the vector stores to search. - - - `filters: ComparisonFilter | CompoundFilter` - - A filter to apply. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `key: String` - - The key to compare against the value. - - - `type: :eq | :ne | :gt | 5 more` - - Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`. - - - `eq`: equals - - `ne`: not equal - - `gt`: greater than - - `gte`: greater than or equal - - `lt`: less than - - `lte`: less than or equal - - `in`: in - - `nin`: not in - - - `:eq` - - - `:ne` - - - `:gt` - - - `:gte` - - - `:lt` - - - `:lte` - - - `:in` - - - `:nin` - - - `value: String | Float | bool | Array[String | Float]` - - The value to compare against the attribute key; supports string, number, or boolean types. - - - `String = String` - - - `Float = Float` - - - `UnionMember2 = bool` - - - `UnionMember3 = Array[String | Float]` - - - `String = String` - - - `Float = Float` - - - `class CompoundFilter` - - Combine multiple filters using `and` or `or`. - - - `filters: Array[ComparisonFilter | untyped]` - - Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `UnionMember1 = untyped` - - - `type: :and | :or` - - Type of operation: `and` or `or`. - - - `:and` - - - `:or` - - - `max_num_results: Integer` - - The maximum number of results to return. This number should be between 1 and 50 inclusive. - - - `ranking_options: RankingOptions{ hybrid_search, ranker, score_threshold}` - - Ranking options for search. - - - `hybrid_search: HybridSearch{ embedding_weight, text_weight}` - - Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled. - - - `embedding_weight: Float` - - The weight of the embedding in the reciprocal ranking fusion. - - - `text_weight: Float` - - The weight of the text in the reciprocal ranking fusion. - - - `ranker: :auto | :"default-2024-11-15"` - - The ranker to use for the file search. - - - `:auto` - - - `:"default-2024-11-15"` - - - `score_threshold: Float` - - The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results. - - - `class ComputerTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `type: :computer` - - The type of the computer tool. Always `computer`. - - - `:computer` - - - `class ComputerUsePreviewTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `display_height: Integer` - - The height of the computer display. - - - `display_width: Integer` - - The width of the computer display. - - - `environment: :windows | :mac | :linux | 2 more` - - The type of computer environment to control. - - - `:windows` - - - `:mac` - - - `:linux` - - - `:ubuntu` - - - `:browser` - - - `type: :computer_use_preview` - - The type of the computer use tool. Always `computer_use_preview`. - - - `:computer_use_preview` - - - `class WebSearchTool` - - Search the Internet for sources related to the prompt. Learn more about the - [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search | :web_search_2025_08_26` - - The type of the web search tool. One of `web_search` or `web_search_2025_08_26`. - - - `:web_search` - - - `:web_search_2025_08_26` - - - `filters: Filters{ allowed_domains}` - - Filters for the search. - - - `allowed_domains: Array[String]` - - Allowed domains for the search. If not provided, all domains are allowed. - Subdomains of the provided domains are allowed as well. - - Example: `["pubmed.ncbi.nlm.nih.gov"]` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ city, country, region, 2 more}` - - The approximate location of the user. - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `class Mcp` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](https://platform.openai.com/docs/guides/tools-remote-mcp). - - - `server_label: String` - - A label for this MCP server, used to identify it in tool calls. - - - `type: :mcp` - - The type of the MCP tool. Always `mcp`. - - - `:mcp` - - - `allowed_tools: Array[String] | McpToolFilter{ read_only, tool_names}` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = Array[String]` - - A string array of allowed tool names - - - `class McpToolFilter` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `authorization: String` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: :connector_dropbox | :connector_gmail | :connector_googlecalendar | 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](https://platform.openai.com/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `:connector_dropbox` - - - `:connector_gmail` - - - `:connector_googlecalendar` - - - `:connector_googledrive` - - - `:connector_microsoftteams` - - - `:connector_outlookcalendar` - - - `:connector_outlookemail` - - - `:connector_sharepoint` - - - `defer_loading: bool` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: Hash[Symbol, String]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: McpToolApprovalFilter{ always, never} | :always | :never` - - Specify which of the MCP server's tools require approval. - - - `class McpToolApprovalFilter` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: Always{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `never: Never{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `McpToolApprovalSetting = :always | :never` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `:always` - - - `:never` - - - `server_description: String` - - Optional description of the MCP server, used to provide more context. - - - `server_url: String` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `class CodeInterpreter` - - A tool that runs Python code to help generate a response to a prompt. - - - `container: String | CodeInterpreterToolAuto{ type, file_ids, memory_limit, network_policy}` - - The code interpreter container. Can be a container ID or an object that - specifies uploaded file IDs to make available to your code, along with an - optional `memory_limit` setting. - - - `String = String` - - The container ID. - - - `class CodeInterpreterToolAuto` - - Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on. - - - `type: :auto` - - Always `auto`. - - - `:auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the code interpreter container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `type: :disabled` - - Disable outbound network access. Always `disabled`. - - - `:disabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `allowed_domains: Array[String]` - - A list of allowed domains when type is `allowlist`. - - - `type: :allowlist` - - Allow outbound network access only to specified domains. Always `allowlist`. - - - `:allowlist` - - - `domain_secrets: Array[ContainerNetworkPolicyDomainSecret]` - - Optional domain-scoped secrets for allowlisted domains. - - - `domain: String` - - The domain associated with the secret. - - - `name: String` - - The name of the secret to inject for the domain. - - - `value: String` - - The secret value to inject for the domain. - - - `type: :code_interpreter` - - The type of the code interpreter tool. Always `code_interpreter`. - - - `:code_interpreter` - - - `class ImageGeneration` - - A tool that generates images using the GPT image models. - - - `type: :image_generation` - - The type of the image generation tool. Always `image_generation`. - - - `:image_generation` - - - `action: :generate | :edit | :auto` - - Whether to generate a new image or edit an existing image. Default: `auto`. - - - `:generate` - - - `:edit` - - - `:auto` - - - `background: :transparent | :opaque | :auto` - - Allows to set transparency for the background of the generated image(s). - This parameter is only supported for GPT image models that support - transparent backgrounds. Must be one of `transparent`, `opaque`, or - `auto` (default value). When `auto` is used, the model will - automatically determine the best background for the image. - - `gpt-image-2` and `gpt-image-2-2026-04-21` do not support - transparent backgrounds. Requests with `background` set to - `transparent` will return an error for these models; use `opaque` or - `auto` instead. - - If `transparent`, the output format needs to support transparency, - so it should be set to either `png` (default value) or `webp`. - - - `:transparent` - - - `:opaque` - - - `:auto` - - - `input_fidelity: :high | :low` - - Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`. - - - `:high` - - - `:low` - - - `input_image_mask: InputImageMask{ file_id, image_url}` - - Optional mask for inpainting. Contains `image_url` - (string, optional) and `file_id` (string, optional). - - - `file_id: String` - - File ID for the mask image. - - - `image_url: String` - - Base64-encoded mask image. - - - `model: String | :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `String = String` - - - `Model = :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `:"gpt-image-1"` - - - `:"gpt-image-1-mini"` - - - `:"gpt-image-2"` - - - `:"gpt-image-2-2026-04-21"` - - - `:"gpt-image-1.5"` - - - `:"chatgpt-image-latest"` - - - `moderation: :auto | :low` - - Moderation level for the generated image. Default: `auto`. - - - `:auto` - - - `:low` - - - `output_compression: Integer` - - Compression level for the output image. Default: 100. - - - `output_format: :png | :webp | :jpeg` - - The output format of the generated image. One of `png`, `webp`, or - `jpeg`. Default: `png`. - - - `:png` - - - `:webp` - - - `:jpeg` - - - `partial_images: Integer` - - Number of partial images to generate in streaming mode, from 0 (default value) to 3. - - - `quality: :low | :medium | :high | :auto` - - The quality of the generated image. One of `low`, `medium`, `high`, - or `auto`. Default: `auto`. - - - `:low` - - - `:medium` - - - `:high` - - - `:auto` - - - `size: String | :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `String = String` - - - `Size = :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `:"1024x1024"` - - - `:"1024x1536"` - - - `:"1536x1024"` - - - `:auto` - - - `class LocalShell` - - A tool that allows the model to execute shell commands in a local environment. - - - `type: :local_shell` - - The type of the local shell tool. Always `local_shell`. - - - `:local_shell` - - - `class FunctionShellTool` - - A tool that allows the model to execute shell commands. - - - `type: :shell` - - The type of the shell tool. Always `shell`. - - - `:shell` - - - `environment: ContainerAuto | LocalEnvironment | ContainerReference` - - - `class ContainerAuto` - - - `type: :container_auto` - - Automatically creates a container for this request - - - `:container_auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `skills: Array[SkillReference | InlineSkill]` - - An optional list of skills referenced by id or inline data. - - - `class SkillReference` - - - `skill_id: String` - - The ID of the referenced skill. - - - `type: :skill_reference` - - References a skill created with the /v1/skills endpoint. - - - `:skill_reference` - - - `version: String` - - Optional skill version. Use a positive integer or 'latest'. Omit for default. - - - `class InlineSkill` - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `source: InlineSkillSource` - - Inline skill payload - - - `data: String` - - Base64-encoded skill zip bundle. - - - `media_type: :"application/zip"` - - The media type of the inline skill payload. Must be `application/zip`. - - - `:"application/zip"` - - - `type: :base64` - - The type of the inline skill source. Must be `base64`. - - - `:base64` - - - `type: :inline` - - Defines an inline skill for this request. - - - `:inline` - - - `class LocalEnvironment` - - - `type: :local` - - Use a local computer environment. - - - `:local` - - - `skills: Array[LocalSkill]` - - An optional list of skills. - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `path: String` - - The path to the directory containing the skill. - - - `class ContainerReference` - - - `container_id: String` - - The ID of the referenced container. - - - `type: :container_reference` - - References a container created with the /v1/containers endpoint - - - `:container_reference` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `name: String` - - The name of the custom tool, used to identify it in tool calls. - - - `type: :custom` - - The type of the custom tool. Always `custom`. - - - `:custom` - - - `defer_loading: bool` - - Whether this tool should be deferred and discovered via tool search. - - - `description: String` - - Optional description of the custom tool, used to provide more context. - - - `format_: CustomToolInputFormat` - - The input format for the custom tool. Default is unconstrained text. - - - `class Text` - - Unconstrained free-form text. - - - `type: :text` - - Unconstrained text format. Always `text`. - - - `:text` - - - `class Grammar` - - A grammar defined by the user. - - - `definition: String` - - The grammar definition. - - - `syntax: :lark | :regex` - - The syntax of the grammar definition. One of `lark` or `regex`. - - - `:lark` - - - `:regex` - - - `type: :grammar` - - Grammar format. Always `grammar`. - - - `:grammar` - - - `class NamespaceTool` - - Groups function/custom tools under a shared namespace. - - - `description: String` - - A description of the namespace shown to the model. - - - `name: String` - - The namespace name used in tool calls (for example, `crm`). - - - `tools: Array[Function{ name, type, defer_loading, 3 more} | CustomTool]` - - The function/custom tools available inside this namespace. - - - `class Function` - - - `name: String` - - - `type: :function` - - - `:function` - - - `defer_loading: bool` - - Whether this function should be deferred and discovered via tool search. - - - `description: String` - - - `parameters: untyped` - - - `strict: bool` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `type: :namespace` - - The type of the tool. Always `namespace`. - - - `:namespace` - - - `class ToolSearchTool` - - Hosted or BYOT tool search configuration for deferred tools. - - - `type: :tool_search` - - The type of the tool. Always `tool_search`. - - - `:tool_search` - - - `description: String` - - Description shown to the model for a client-executed tool search tool. - - - `execution: :server | :client` - - Whether tool search is executed by the server or by the client. - - - `:server` - - - `:client` - - - `parameters: untyped` - - Parameter schema for a client-executed tool search tool. - - - `class WebSearchPreviewTool` - - This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search_preview | :web_search_preview_2025_03_11` - - The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`. - - - `:web_search_preview` - - - `:web_search_preview_2025_03_11` - - - `search_content_types: Array[:text | :image]` - - - `:text` - - - `:image` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ type, city, country, 2 more}` - - The user's location. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `class ApplyPatchTool` - - Allows the assistant to create, delete, or update files using unified diffs. - - - `type: :apply_patch` - - The type of the tool. Always `apply_patch`. - - - `:apply_patch` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `error: EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - - - `eval_id: String` - - The identifier of the associated evaluation. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - The model that is evaluated, if applicable. - - - `name: String` - - The name of the evaluation run. - - - `object: :"eval.run"` - - The type of the object. Always "eval.run". - - - `:"eval.run"` - - - `per_model_usage: Array[PerModelUsage{ cached_tokens, completion_tokens, invocation_count, 3 more}]` - - Usage statistics for each model during the evaluation run. - - - `cached_tokens: Integer` - - The number of tokens retrieved from cache. - - - `completion_tokens: Integer` - - The number of completion tokens generated. - - - `invocation_count: Integer` - - The number of invocations. - - - `model_name: String` - - The name of the model. - - - `prompt_tokens: Integer` - - The number of prompt tokens used. - - - `total_tokens: Integer` - - The total number of tokens used. - - - `per_testing_criteria_results: Array[PerTestingCriteriaResult{ failed, passed, testing_criteria}]` - - Results per testing criteria applied during the evaluation run. - - - `failed: Integer` - - Number of tests failed for this criteria. - - - `passed: Integer` - - Number of tests passed for this criteria. - - - `testing_criteria: String` - - A description of the testing criteria. - - - `report_url: String` - - The URL to the rendered evaluation run report on the UI dashboard. - - - `result_counts: ResultCounts{ errored, failed, passed, total}` - - Counters summarizing the outcomes of the evaluation run. - - - `errored: Integer` - - Number of output items that resulted in an error. - - - `failed: Integer` - - Number of output items that failed to pass the evaluation. - - - `passed: Integer` - - Number of output items that passed the evaluation. - - - `total: Integer` - - Total number of executed output items. - - - `status: String` - - The status of the evaluation run. - -### Example - -```ruby -require "openai" - -openai = OpenAI::Client.new(api_key: "My API Key") - -page = openai.evals.runs.list("eval_id") - -puts(page) -``` - -#### Response - -```json -{ - "data": [ - { - "id": "id", - "created_at": 0, - "data_source": { - "source": { - "content": [ - { - "item": { - "foo": "bar" - }, - "sample": { - "foo": "bar" - } - } - ], - "type": "file_content" - }, - "type": "jsonl" - }, - "error": { - "code": "code", - "message": "message" - }, - "eval_id": "eval_id", - "metadata": { - "foo": "string" - }, - "model": "model", - "name": "name", - "object": "eval.run", - "per_model_usage": [ - { - "cached_tokens": 0, - "completion_tokens": 0, - "invocation_count": 0, - "model_name": "model_name", - "prompt_tokens": 0, - "total_tokens": 0 - } - ], - "per_testing_criteria_results": [ - { - "failed": 0, - "passed": 0, - "testing_criteria": "testing_criteria" - } - ], - "report_url": "https://example.com", - "result_counts": { - "errored": 0, - "failed": 0, - "passed": 0, - "total": 0 - }, - "status": "status" - } - ], - "first_id": "first_id", - "has_more": true, - "last_id": "last_id", - "object": "list" -} -``` - -## Create eval run - -`evals.runs.create(eval_id, **kwargs) -> RunCreateResponse` - -**post** `/evals/{eval_id}/runs` - -Kicks off a new run for a given evaluation, specifying the data source, and what model configuration to use to test. The datasource will be validated against the schema specified in the config of the evaluation. - -### Parameters - -- `eval_id: String` - -- `data_source: CreateEvalJSONLRunDataSource | CreateEvalCompletionsRunDataSource | CreateEvalResponsesRunDataSource{ source, type, input_messages, 2 more}` - - Details about the run's data source. - - - `class CreateEvalJSONLRunDataSource` - - A JsonlRunDataSource object with that specifies a JSONL file that matches the eval - - - `source: FileContent{ content, type} | FileID{ id, type}` - - Determines what populates the `item` namespace in the data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `type: :jsonl` - - The type of data source. Always `jsonl`. - - - `:jsonl` - - - `class CreateEvalCompletionsRunDataSource` - - A CompletionsRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | StoredCompletions{ type, created_after, created_before, 3 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class StoredCompletions` - - A StoredCompletionsRunDataSource configuration describing a set of filters - - - `type: :stored_completions` - - The type of source. Always `stored_completions`. - - - `:stored_completions` - - - `created_after: Integer` - - An optional Unix timestamp to filter items created after this time. - - - `created_before: Integer` - - An optional Unix timestamp to filter items created before this time. - - - `limit: Integer` - - An optional maximum number of items to return. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - An optional model to filter by (e.g., 'gpt-4o'). - - - `type: :completions` - - The type of run data source. Always `completions`. - - - `:completions` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[EasyInputMessage | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class EasyInputMessage` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputMessageContentList` - - Text, image, or audio input to the model, used to generate a response. - Can also contain previous assistant responses. - - - `String = String` - - A text input to the model. - - - `ResponseInputMessageContentList = Array[ResponseInputContent]` - - A list of one or many input items to the model, containing different content - types. - - - `class ResponseInputText` - - A text input to the model. - - - `text: String` - - The text input to the model. - - - `type: :input_text` - - The type of the input item. Always `input_text`. - - - `:input_text` - - - `class ResponseInputImage` - - An image input to the model. Learn about [image inputs](https://platform.openai.com/docs/guides/vision). - - - `detail: :low | :high | :auto | :original` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `:low` - - - `:high` - - - `:auto` - - - `:original` - - - `type: :input_image` - - The type of the input item. Always `input_image`. - - - `:input_image` - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `image_url: String` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `class ResponseInputFile` - - A file input to the model. - - - `type: :input_file` - - The type of the input item. Always `input_file`. - - - `:input_file` - - - `detail: :low | :high` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `:low` - - - `:high` - - - `file_data: String` - - The content of the file to be sent to the model. - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `file_url: String` - - The URL of the file to be sent to the model. - - - `filename: String` - - The name of the file to be sent to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `phase: :commentary | :final_answer` - - Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`). - For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend - phase on all assistant messages — dropping it can degrade performance. Not used for user messages. - - - `:commentary` - - - `:final_answer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `input_audio: InputAudio{ data, format_}` - - - `data: String` - - Base64-encoded audio data. - - - `format_: :mp3 | :wav` - - The format of the audio data. Currently supported formats are `mp3` and - `wav`. - - - `:mp3` - - - `:wav` - - - `type: :input_audio` - - The type of the input item. Always `input_audio`. - - - `:input_audio` - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.input_trajectory" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, response_format, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `:none` - - - `:minimal` - - - `:low` - - - `:medium` - - - `:high` - - - `:xhigh` - - - `response_format: ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject` - - An object specifying the format that the model must output. - - Setting to `{ "type": "json_schema", "json_schema": {...} }` enables - Structured Outputs which ensures the model will match your supplied JSON - schema. Learn more in the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `type: :text` - - The type of response format being defined. Always `text`. - - - `:text` - - - `class ResponseFormatJSONSchema` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `json_schema: JSONSchema{ name, description, schema, strict}` - - Structured Outputs configuration options, including a JSON Schema. - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `type: :json_object` - - The type of response format being defined. Always `json_object`. - - - `:json_object` - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `tools: Array[ChatCompletionFunctionTool]` - - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported. - - - `function: FunctionDefinition` - - - `name: String` - - The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the function does, used by the model to choose when and how to call the function. - - - `parameters: FunctionParameters` - - The parameters the functions accepts, described as a JSON Schema object. See the [guide](https://platform.openai.com/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format. - - Omitting `parameters` defines a function with an empty parameter list. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](https://platform.openai.com/docs/guides/function-calling). - - - `type: :function` - - The type of the tool. Currently, only `function` is supported. - - - `:function` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `class CreateEvalResponsesRunDataSource` - - A ResponsesRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | Responses{ type, created_after, created_before, 8 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class Responses` - - A EvalResponsesSource object describing a run data source configuration. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `created_after: Integer` - - Only include items created after this timestamp (inclusive). This is a query parameter used to select responses. - - - `created_before: Integer` - - Only include items created before this timestamp (inclusive). This is a query parameter used to select responses. - - - `instructions_search: String` - - Optional string to search the 'instructions' field. This is a query parameter used to select responses. - - - `metadata: untyped` - - Metadata filter for the responses. This is a query parameter used to select responses. - - - `model: String` - - The name of the model to find responses for. This is a query parameter used to select responses. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `temperature: Float` - - Sampling temperature. This is a query parameter used to select responses. - - - `tools: Array[String]` - - List of tool names. This is a query parameter used to select responses. - - - `top_p: Float` - - Nucleus sampling parameter. This is a query parameter used to select responses. - - - `users: Array[String]` - - List of user identifiers. This is a query parameter used to select responses. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[ChatMessage{ content, role} | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class ChatMessage` - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.name" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, seed, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `text: Text{ format_}` - - Configuration options for a text response from the model. Can be plain - text or structured JSON data. Learn more: - - - [Text inputs and outputs](https://platform.openai.com/docs/guides/text) - - [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) - - - `format_: ResponseFormatTextConfig` - - An object specifying the format that the model must output. - - Configuring `{ "type": "json_schema" }` enables Structured Outputs, - which ensures the model will match your supplied JSON schema. Learn more in the - [Structured Outputs guide](https://platform.openai.com/docs/guides/structured-outputs). - - The default format is `{ "type": "text" }` with no additional options. - - **Not recommended for gpt-4o and newer models:** - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `class ResponseFormatTextJSONSchemaConfig` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `tools: Array[Tool]` - - An array of tools the model may call while generating a response. You - can specify which tool to use by setting the `tool_choice` parameter. - - The two categories of tools you can provide the model are: - - - **Built-in tools**: Tools that are provided by OpenAI that extend the - model's capabilities, like [web search](https://platform.openai.com/docs/guides/tools-web-search) - or [file search](https://platform.openai.com/docs/guides/tools-file-search). Learn more about - [built-in tools](https://platform.openai.com/docs/guides/tools). - - **Function calls (custom tools)**: Functions that are defined by you, - enabling the model to call your own code. Learn more about - [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `class FunctionTool` - - Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `name: String` - - The name of the function to call. - - - `parameters: Hash[Symbol, untyped]` - - A JSON schema object describing the parameters of the function. - - - `strict: bool` - - Whether to enforce strict parameter validation. Default `true`. - - - `type: :function` - - The type of the function tool. Always `function`. - - - `:function` - - - `defer_loading: bool` - - Whether this function is deferred and loaded via tool search. - - - `description: String` - - A description of the function. Used by the model to determine whether or not to call the function. - - - `class FileSearchTool` - - A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search). - - - `type: :file_search` - - The type of the file search tool. Always `file_search`. - - - `:file_search` - - - `vector_store_ids: Array[String]` - - The IDs of the vector stores to search. - - - `filters: ComparisonFilter | CompoundFilter` - - A filter to apply. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `key: String` - - The key to compare against the value. - - - `type: :eq | :ne | :gt | 5 more` - - Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`. - - - `eq`: equals - - `ne`: not equal - - `gt`: greater than - - `gte`: greater than or equal - - `lt`: less than - - `lte`: less than or equal - - `in`: in - - `nin`: not in - - - `:eq` - - - `:ne` - - - `:gt` - - - `:gte` - - - `:lt` - - - `:lte` - - - `:in` - - - `:nin` - - - `value: String | Float | bool | Array[String | Float]` - - The value to compare against the attribute key; supports string, number, or boolean types. - - - `String = String` - - - `Float = Float` - - - `UnionMember2 = bool` - - - `UnionMember3 = Array[String | Float]` - - - `String = String` - - - `Float = Float` - - - `class CompoundFilter` - - Combine multiple filters using `and` or `or`. - - - `filters: Array[ComparisonFilter | untyped]` - - Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `UnionMember1 = untyped` - - - `type: :and | :or` - - Type of operation: `and` or `or`. - - - `:and` - - - `:or` - - - `max_num_results: Integer` - - The maximum number of results to return. This number should be between 1 and 50 inclusive. - - - `ranking_options: RankingOptions{ hybrid_search, ranker, score_threshold}` - - Ranking options for search. - - - `hybrid_search: HybridSearch{ embedding_weight, text_weight}` - - Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled. - - - `embedding_weight: Float` - - The weight of the embedding in the reciprocal ranking fusion. - - - `text_weight: Float` - - The weight of the text in the reciprocal ranking fusion. - - - `ranker: :auto | :"default-2024-11-15"` - - The ranker to use for the file search. - - - `:auto` - - - `:"default-2024-11-15"` - - - `score_threshold: Float` - - The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results. - - - `class ComputerTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `type: :computer` - - The type of the computer tool. Always `computer`. - - - `:computer` - - - `class ComputerUsePreviewTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `display_height: Integer` - - The height of the computer display. - - - `display_width: Integer` - - The width of the computer display. - - - `environment: :windows | :mac | :linux | 2 more` - - The type of computer environment to control. - - - `:windows` - - - `:mac` - - - `:linux` - - - `:ubuntu` - - - `:browser` - - - `type: :computer_use_preview` - - The type of the computer use tool. Always `computer_use_preview`. - - - `:computer_use_preview` - - - `class WebSearchTool` - - Search the Internet for sources related to the prompt. Learn more about the - [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search | :web_search_2025_08_26` - - The type of the web search tool. One of `web_search` or `web_search_2025_08_26`. - - - `:web_search` - - - `:web_search_2025_08_26` - - - `filters: Filters{ allowed_domains}` - - Filters for the search. - - - `allowed_domains: Array[String]` - - Allowed domains for the search. If not provided, all domains are allowed. - Subdomains of the provided domains are allowed as well. - - Example: `["pubmed.ncbi.nlm.nih.gov"]` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ city, country, region, 2 more}` - - The approximate location of the user. - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `class Mcp` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](https://platform.openai.com/docs/guides/tools-remote-mcp). - - - `server_label: String` - - A label for this MCP server, used to identify it in tool calls. - - - `type: :mcp` - - The type of the MCP tool. Always `mcp`. - - - `:mcp` - - - `allowed_tools: Array[String] | McpToolFilter{ read_only, tool_names}` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = Array[String]` - - A string array of allowed tool names - - - `class McpToolFilter` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `authorization: String` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: :connector_dropbox | :connector_gmail | :connector_googlecalendar | 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](https://platform.openai.com/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `:connector_dropbox` - - - `:connector_gmail` - - - `:connector_googlecalendar` - - - `:connector_googledrive` - - - `:connector_microsoftteams` - - - `:connector_outlookcalendar` - - - `:connector_outlookemail` - - - `:connector_sharepoint` - - - `defer_loading: bool` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: Hash[Symbol, String]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: McpToolApprovalFilter{ always, never} | :always | :never` - - Specify which of the MCP server's tools require approval. - - - `class McpToolApprovalFilter` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: Always{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `never: Never{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `McpToolApprovalSetting = :always | :never` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `:always` - - - `:never` - - - `server_description: String` - - Optional description of the MCP server, used to provide more context. - - - `server_url: String` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `class CodeInterpreter` - - A tool that runs Python code to help generate a response to a prompt. - - - `container: String | CodeInterpreterToolAuto{ type, file_ids, memory_limit, network_policy}` - - The code interpreter container. Can be a container ID or an object that - specifies uploaded file IDs to make available to your code, along with an - optional `memory_limit` setting. - - - `String = String` - - The container ID. - - - `class CodeInterpreterToolAuto` - - Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on. - - - `type: :auto` - - Always `auto`. - - - `:auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the code interpreter container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `type: :disabled` - - Disable outbound network access. Always `disabled`. - - - `:disabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `allowed_domains: Array[String]` - - A list of allowed domains when type is `allowlist`. - - - `type: :allowlist` - - Allow outbound network access only to specified domains. Always `allowlist`. - - - `:allowlist` - - - `domain_secrets: Array[ContainerNetworkPolicyDomainSecret]` - - Optional domain-scoped secrets for allowlisted domains. - - - `domain: String` - - The domain associated with the secret. - - - `name: String` - - The name of the secret to inject for the domain. - - - `value: String` - - The secret value to inject for the domain. - - - `type: :code_interpreter` - - The type of the code interpreter tool. Always `code_interpreter`. - - - `:code_interpreter` - - - `class ImageGeneration` - - A tool that generates images using the GPT image models. - - - `type: :image_generation` - - The type of the image generation tool. Always `image_generation`. - - - `:image_generation` - - - `action: :generate | :edit | :auto` - - Whether to generate a new image or edit an existing image. Default: `auto`. - - - `:generate` - - - `:edit` - - - `:auto` - - - `background: :transparent | :opaque | :auto` - - Allows to set transparency for the background of the generated image(s). - This parameter is only supported for GPT image models that support - transparent backgrounds. Must be one of `transparent`, `opaque`, or - `auto` (default value). When `auto` is used, the model will - automatically determine the best background for the image. - - `gpt-image-2` and `gpt-image-2-2026-04-21` do not support - transparent backgrounds. Requests with `background` set to - `transparent` will return an error for these models; use `opaque` or - `auto` instead. - - If `transparent`, the output format needs to support transparency, - so it should be set to either `png` (default value) or `webp`. - - - `:transparent` - - - `:opaque` - - - `:auto` - - - `input_fidelity: :high | :low` - - Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`. - - - `:high` - - - `:low` - - - `input_image_mask: InputImageMask{ file_id, image_url}` - - Optional mask for inpainting. Contains `image_url` - (string, optional) and `file_id` (string, optional). - - - `file_id: String` - - File ID for the mask image. - - - `image_url: String` - - Base64-encoded mask image. - - - `model: String | :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `String = String` - - - `Model = :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `:"gpt-image-1"` - - - `:"gpt-image-1-mini"` - - - `:"gpt-image-2"` - - - `:"gpt-image-2-2026-04-21"` - - - `:"gpt-image-1.5"` - - - `:"chatgpt-image-latest"` - - - `moderation: :auto | :low` - - Moderation level for the generated image. Default: `auto`. - - - `:auto` - - - `:low` - - - `output_compression: Integer` - - Compression level for the output image. Default: 100. - - - `output_format: :png | :webp | :jpeg` - - The output format of the generated image. One of `png`, `webp`, or - `jpeg`. Default: `png`. - - - `:png` - - - `:webp` - - - `:jpeg` - - - `partial_images: Integer` - - Number of partial images to generate in streaming mode, from 0 (default value) to 3. - - - `quality: :low | :medium | :high | :auto` - - The quality of the generated image. One of `low`, `medium`, `high`, - or `auto`. Default: `auto`. - - - `:low` - - - `:medium` - - - `:high` - - - `:auto` - - - `size: String | :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `String = String` - - - `Size = :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `:"1024x1024"` - - - `:"1024x1536"` - - - `:"1536x1024"` - - - `:auto` - - - `class LocalShell` - - A tool that allows the model to execute shell commands in a local environment. - - - `type: :local_shell` - - The type of the local shell tool. Always `local_shell`. - - - `:local_shell` - - - `class FunctionShellTool` - - A tool that allows the model to execute shell commands. - - - `type: :shell` - - The type of the shell tool. Always `shell`. - - - `:shell` - - - `environment: ContainerAuto | LocalEnvironment | ContainerReference` - - - `class ContainerAuto` - - - `type: :container_auto` - - Automatically creates a container for this request - - - `:container_auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `skills: Array[SkillReference | InlineSkill]` - - An optional list of skills referenced by id or inline data. - - - `class SkillReference` - - - `skill_id: String` - - The ID of the referenced skill. - - - `type: :skill_reference` - - References a skill created with the /v1/skills endpoint. - - - `:skill_reference` - - - `version: String` - - Optional skill version. Use a positive integer or 'latest'. Omit for default. - - - `class InlineSkill` - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `source: InlineSkillSource` - - Inline skill payload - - - `data: String` - - Base64-encoded skill zip bundle. - - - `media_type: :"application/zip"` - - The media type of the inline skill payload. Must be `application/zip`. - - - `:"application/zip"` - - - `type: :base64` - - The type of the inline skill source. Must be `base64`. - - - `:base64` - - - `type: :inline` - - Defines an inline skill for this request. - - - `:inline` - - - `class LocalEnvironment` - - - `type: :local` - - Use a local computer environment. - - - `:local` - - - `skills: Array[LocalSkill]` - - An optional list of skills. - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `path: String` - - The path to the directory containing the skill. - - - `class ContainerReference` - - - `container_id: String` - - The ID of the referenced container. - - - `type: :container_reference` - - References a container created with the /v1/containers endpoint - - - `:container_reference` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `name: String` - - The name of the custom tool, used to identify it in tool calls. - - - `type: :custom` - - The type of the custom tool. Always `custom`. - - - `:custom` - - - `defer_loading: bool` - - Whether this tool should be deferred and discovered via tool search. - - - `description: String` - - Optional description of the custom tool, used to provide more context. - - - `format_: CustomToolInputFormat` - - The input format for the custom tool. Default is unconstrained text. - - - `class Text` - - Unconstrained free-form text. - - - `type: :text` - - Unconstrained text format. Always `text`. - - - `:text` - - - `class Grammar` - - A grammar defined by the user. - - - `definition: String` - - The grammar definition. - - - `syntax: :lark | :regex` - - The syntax of the grammar definition. One of `lark` or `regex`. - - - `:lark` - - - `:regex` - - - `type: :grammar` - - Grammar format. Always `grammar`. - - - `:grammar` - - - `class NamespaceTool` - - Groups function/custom tools under a shared namespace. - - - `description: String` - - A description of the namespace shown to the model. - - - `name: String` - - The namespace name used in tool calls (for example, `crm`). - - - `tools: Array[Function{ name, type, defer_loading, 3 more} | CustomTool]` - - The function/custom tools available inside this namespace. - - - `class Function` - - - `name: String` - - - `type: :function` - - - `:function` - - - `defer_loading: bool` - - Whether this function should be deferred and discovered via tool search. - - - `description: String` - - - `parameters: untyped` - - - `strict: bool` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `type: :namespace` - - The type of the tool. Always `namespace`. - - - `:namespace` - - - `class ToolSearchTool` - - Hosted or BYOT tool search configuration for deferred tools. - - - `type: :tool_search` - - The type of the tool. Always `tool_search`. - - - `:tool_search` - - - `description: String` - - Description shown to the model for a client-executed tool search tool. - - - `execution: :server | :client` - - Whether tool search is executed by the server or by the client. - - - `:server` - - - `:client` - - - `parameters: untyped` - - Parameter schema for a client-executed tool search tool. - - - `class WebSearchPreviewTool` - - This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search_preview | :web_search_preview_2025_03_11` - - The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`. - - - `:web_search_preview` - - - `:web_search_preview_2025_03_11` - - - `search_content_types: Array[:text | :image]` - - - `:text` - - - `:image` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ type, city, country, 2 more}` - - The user's location. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `class ApplyPatchTool` - - Allows the assistant to create, delete, or update files using unified diffs. - - - `type: :apply_patch` - - The type of the tool. Always `apply_patch`. - - - `:apply_patch` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - -- `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - -- `name: String` - - The name of the run. - -### Returns - -- `class RunCreateResponse` - - A schema representing an evaluation run. - - - `id: String` - - Unique identifier for the evaluation run. - - - `created_at: Integer` - - Unix timestamp (in seconds) when the evaluation run was created. - - - `data_source: CreateEvalJSONLRunDataSource | CreateEvalCompletionsRunDataSource | Responses{ source, type, input_messages, 2 more}` - - Information about the run's data source. - - - `class CreateEvalJSONLRunDataSource` - - A JsonlRunDataSource object with that specifies a JSONL file that matches the eval - - - `source: FileContent{ content, type} | FileID{ id, type}` - - Determines what populates the `item` namespace in the data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `type: :jsonl` - - The type of data source. Always `jsonl`. - - - `:jsonl` - - - `class CreateEvalCompletionsRunDataSource` - - A CompletionsRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | StoredCompletions{ type, created_after, created_before, 3 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class StoredCompletions` - - A StoredCompletionsRunDataSource configuration describing a set of filters - - - `type: :stored_completions` - - The type of source. Always `stored_completions`. - - - `:stored_completions` - - - `created_after: Integer` - - An optional Unix timestamp to filter items created after this time. - - - `created_before: Integer` - - An optional Unix timestamp to filter items created before this time. - - - `limit: Integer` - - An optional maximum number of items to return. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - An optional model to filter by (e.g., 'gpt-4o'). - - - `type: :completions` - - The type of run data source. Always `completions`. - - - `:completions` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[EasyInputMessage | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class EasyInputMessage` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputMessageContentList` - - Text, image, or audio input to the model, used to generate a response. - Can also contain previous assistant responses. - - - `String = String` - - A text input to the model. - - - `ResponseInputMessageContentList = Array[ResponseInputContent]` - - A list of one or many input items to the model, containing different content - types. - - - `class ResponseInputText` - - A text input to the model. - - - `text: String` - - The text input to the model. - - - `type: :input_text` - - The type of the input item. Always `input_text`. - - - `:input_text` - - - `class ResponseInputImage` - - An image input to the model. Learn about [image inputs](https://platform.openai.com/docs/guides/vision). - - - `detail: :low | :high | :auto | :original` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `:low` - - - `:high` - - - `:auto` - - - `:original` - - - `type: :input_image` - - The type of the input item. Always `input_image`. - - - `:input_image` - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `image_url: String` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `class ResponseInputFile` - - A file input to the model. - - - `type: :input_file` - - The type of the input item. Always `input_file`. - - - `:input_file` - - - `detail: :low | :high` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `:low` - - - `:high` - - - `file_data: String` - - The content of the file to be sent to the model. - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `file_url: String` - - The URL of the file to be sent to the model. - - - `filename: String` - - The name of the file to be sent to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `phase: :commentary | :final_answer` - - Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`). - For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend - phase on all assistant messages — dropping it can degrade performance. Not used for user messages. - - - `:commentary` - - - `:final_answer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `input_audio: InputAudio{ data, format_}` - - - `data: String` - - Base64-encoded audio data. - - - `format_: :mp3 | :wav` - - The format of the audio data. Currently supported formats are `mp3` and - `wav`. - - - `:mp3` - - - `:wav` - - - `type: :input_audio` - - The type of the input item. Always `input_audio`. - - - `:input_audio` - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.input_trajectory" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, response_format, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `:none` - - - `:minimal` - - - `:low` - - - `:medium` - - - `:high` - - - `:xhigh` - - - `response_format: ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject` - - An object specifying the format that the model must output. - - Setting to `{ "type": "json_schema", "json_schema": {...} }` enables - Structured Outputs which ensures the model will match your supplied JSON - schema. Learn more in the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `type: :text` - - The type of response format being defined. Always `text`. - - - `:text` - - - `class ResponseFormatJSONSchema` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `json_schema: JSONSchema{ name, description, schema, strict}` - - Structured Outputs configuration options, including a JSON Schema. - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `type: :json_object` - - The type of response format being defined. Always `json_object`. - - - `:json_object` - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `tools: Array[ChatCompletionFunctionTool]` - - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported. - - - `function: FunctionDefinition` - - - `name: String` - - The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the function does, used by the model to choose when and how to call the function. - - - `parameters: FunctionParameters` - - The parameters the functions accepts, described as a JSON Schema object. See the [guide](https://platform.openai.com/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format. - - Omitting `parameters` defines a function with an empty parameter list. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](https://platform.openai.com/docs/guides/function-calling). - - - `type: :function` - - The type of the tool. Currently, only `function` is supported. - - - `:function` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `class Responses` - - A ResponsesRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | Responses{ type, created_after, created_before, 8 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class Responses` - - A EvalResponsesSource object describing a run data source configuration. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `created_after: Integer` - - Only include items created after this timestamp (inclusive). This is a query parameter used to select responses. - - - `created_before: Integer` - - Only include items created before this timestamp (inclusive). This is a query parameter used to select responses. - - - `instructions_search: String` - - Optional string to search the 'instructions' field. This is a query parameter used to select responses. - - - `metadata: untyped` - - Metadata filter for the responses. This is a query parameter used to select responses. - - - `model: String` - - The name of the model to find responses for. This is a query parameter used to select responses. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `temperature: Float` - - Sampling temperature. This is a query parameter used to select responses. - - - `tools: Array[String]` - - List of tool names. This is a query parameter used to select responses. - - - `top_p: Float` - - Nucleus sampling parameter. This is a query parameter used to select responses. - - - `users: Array[String]` - - List of user identifiers. This is a query parameter used to select responses. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[ChatMessage{ content, role} | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class ChatMessage` - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.name" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, seed, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `text: Text{ format_}` - - Configuration options for a text response from the model. Can be plain - text or structured JSON data. Learn more: - - - [Text inputs and outputs](https://platform.openai.com/docs/guides/text) - - [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) - - - `format_: ResponseFormatTextConfig` - - An object specifying the format that the model must output. - - Configuring `{ "type": "json_schema" }` enables Structured Outputs, - which ensures the model will match your supplied JSON schema. Learn more in the - [Structured Outputs guide](https://platform.openai.com/docs/guides/structured-outputs). - - The default format is `{ "type": "text" }` with no additional options. - - **Not recommended for gpt-4o and newer models:** - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `class ResponseFormatTextJSONSchemaConfig` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `tools: Array[Tool]` - - An array of tools the model may call while generating a response. You - can specify which tool to use by setting the `tool_choice` parameter. - - The two categories of tools you can provide the model are: - - - **Built-in tools**: Tools that are provided by OpenAI that extend the - model's capabilities, like [web search](https://platform.openai.com/docs/guides/tools-web-search) - or [file search](https://platform.openai.com/docs/guides/tools-file-search). Learn more about - [built-in tools](https://platform.openai.com/docs/guides/tools). - - **Function calls (custom tools)**: Functions that are defined by you, - enabling the model to call your own code. Learn more about - [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `class FunctionTool` - - Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `name: String` - - The name of the function to call. - - - `parameters: Hash[Symbol, untyped]` - - A JSON schema object describing the parameters of the function. - - - `strict: bool` - - Whether to enforce strict parameter validation. Default `true`. - - - `type: :function` - - The type of the function tool. Always `function`. - - - `:function` - - - `defer_loading: bool` - - Whether this function is deferred and loaded via tool search. - - - `description: String` - - A description of the function. Used by the model to determine whether or not to call the function. - - - `class FileSearchTool` - - A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search). - - - `type: :file_search` - - The type of the file search tool. Always `file_search`. - - - `:file_search` - - - `vector_store_ids: Array[String]` - - The IDs of the vector stores to search. - - - `filters: ComparisonFilter | CompoundFilter` - - A filter to apply. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `key: String` - - The key to compare against the value. - - - `type: :eq | :ne | :gt | 5 more` - - Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`. - - - `eq`: equals - - `ne`: not equal - - `gt`: greater than - - `gte`: greater than or equal - - `lt`: less than - - `lte`: less than or equal - - `in`: in - - `nin`: not in - - - `:eq` - - - `:ne` - - - `:gt` - - - `:gte` - - - `:lt` - - - `:lte` - - - `:in` - - - `:nin` - - - `value: String | Float | bool | Array[String | Float]` - - The value to compare against the attribute key; supports string, number, or boolean types. - - - `String = String` - - - `Float = Float` - - - `UnionMember2 = bool` - - - `UnionMember3 = Array[String | Float]` - - - `String = String` - - - `Float = Float` - - - `class CompoundFilter` - - Combine multiple filters using `and` or `or`. - - - `filters: Array[ComparisonFilter | untyped]` - - Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `UnionMember1 = untyped` - - - `type: :and | :or` - - Type of operation: `and` or `or`. - - - `:and` - - - `:or` - - - `max_num_results: Integer` - - The maximum number of results to return. This number should be between 1 and 50 inclusive. - - - `ranking_options: RankingOptions{ hybrid_search, ranker, score_threshold}` - - Ranking options for search. - - - `hybrid_search: HybridSearch{ embedding_weight, text_weight}` - - Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled. - - - `embedding_weight: Float` - - The weight of the embedding in the reciprocal ranking fusion. - - - `text_weight: Float` - - The weight of the text in the reciprocal ranking fusion. - - - `ranker: :auto | :"default-2024-11-15"` - - The ranker to use for the file search. - - - `:auto` - - - `:"default-2024-11-15"` - - - `score_threshold: Float` - - The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results. - - - `class ComputerTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `type: :computer` - - The type of the computer tool. Always `computer`. - - - `:computer` - - - `class ComputerUsePreviewTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `display_height: Integer` - - The height of the computer display. - - - `display_width: Integer` - - The width of the computer display. - - - `environment: :windows | :mac | :linux | 2 more` - - The type of computer environment to control. - - - `:windows` - - - `:mac` - - - `:linux` - - - `:ubuntu` - - - `:browser` - - - `type: :computer_use_preview` - - The type of the computer use tool. Always `computer_use_preview`. - - - `:computer_use_preview` - - - `class WebSearchTool` - - Search the Internet for sources related to the prompt. Learn more about the - [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search | :web_search_2025_08_26` - - The type of the web search tool. One of `web_search` or `web_search_2025_08_26`. - - - `:web_search` - - - `:web_search_2025_08_26` - - - `filters: Filters{ allowed_domains}` - - Filters for the search. - - - `allowed_domains: Array[String]` - - Allowed domains for the search. If not provided, all domains are allowed. - Subdomains of the provided domains are allowed as well. - - Example: `["pubmed.ncbi.nlm.nih.gov"]` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ city, country, region, 2 more}` - - The approximate location of the user. - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `class Mcp` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](https://platform.openai.com/docs/guides/tools-remote-mcp). - - - `server_label: String` - - A label for this MCP server, used to identify it in tool calls. - - - `type: :mcp` - - The type of the MCP tool. Always `mcp`. - - - `:mcp` - - - `allowed_tools: Array[String] | McpToolFilter{ read_only, tool_names}` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = Array[String]` - - A string array of allowed tool names - - - `class McpToolFilter` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `authorization: String` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: :connector_dropbox | :connector_gmail | :connector_googlecalendar | 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](https://platform.openai.com/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `:connector_dropbox` - - - `:connector_gmail` - - - `:connector_googlecalendar` - - - `:connector_googledrive` - - - `:connector_microsoftteams` - - - `:connector_outlookcalendar` - - - `:connector_outlookemail` - - - `:connector_sharepoint` - - - `defer_loading: bool` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: Hash[Symbol, String]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: McpToolApprovalFilter{ always, never} | :always | :never` - - Specify which of the MCP server's tools require approval. - - - `class McpToolApprovalFilter` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: Always{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `never: Never{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `McpToolApprovalSetting = :always | :never` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `:always` - - - `:never` - - - `server_description: String` - - Optional description of the MCP server, used to provide more context. - - - `server_url: String` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `class CodeInterpreter` - - A tool that runs Python code to help generate a response to a prompt. - - - `container: String | CodeInterpreterToolAuto{ type, file_ids, memory_limit, network_policy}` - - The code interpreter container. Can be a container ID or an object that - specifies uploaded file IDs to make available to your code, along with an - optional `memory_limit` setting. - - - `String = String` - - The container ID. - - - `class CodeInterpreterToolAuto` - - Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on. - - - `type: :auto` - - Always `auto`. - - - `:auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the code interpreter container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `type: :disabled` - - Disable outbound network access. Always `disabled`. - - - `:disabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `allowed_domains: Array[String]` - - A list of allowed domains when type is `allowlist`. - - - `type: :allowlist` - - Allow outbound network access only to specified domains. Always `allowlist`. - - - `:allowlist` - - - `domain_secrets: Array[ContainerNetworkPolicyDomainSecret]` - - Optional domain-scoped secrets for allowlisted domains. - - - `domain: String` - - The domain associated with the secret. - - - `name: String` - - The name of the secret to inject for the domain. - - - `value: String` - - The secret value to inject for the domain. - - - `type: :code_interpreter` - - The type of the code interpreter tool. Always `code_interpreter`. - - - `:code_interpreter` - - - `class ImageGeneration` - - A tool that generates images using the GPT image models. - - - `type: :image_generation` - - The type of the image generation tool. Always `image_generation`. - - - `:image_generation` - - - `action: :generate | :edit | :auto` - - Whether to generate a new image or edit an existing image. Default: `auto`. - - - `:generate` - - - `:edit` - - - `:auto` - - - `background: :transparent | :opaque | :auto` - - Allows to set transparency for the background of the generated image(s). - This parameter is only supported for GPT image models that support - transparent backgrounds. Must be one of `transparent`, `opaque`, or - `auto` (default value). When `auto` is used, the model will - automatically determine the best background for the image. - - `gpt-image-2` and `gpt-image-2-2026-04-21` do not support - transparent backgrounds. Requests with `background` set to - `transparent` will return an error for these models; use `opaque` or - `auto` instead. - - If `transparent`, the output format needs to support transparency, - so it should be set to either `png` (default value) or `webp`. - - - `:transparent` - - - `:opaque` - - - `:auto` - - - `input_fidelity: :high | :low` - - Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`. - - - `:high` - - - `:low` - - - `input_image_mask: InputImageMask{ file_id, image_url}` - - Optional mask for inpainting. Contains `image_url` - (string, optional) and `file_id` (string, optional). - - - `file_id: String` - - File ID for the mask image. - - - `image_url: String` - - Base64-encoded mask image. - - - `model: String | :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `String = String` - - - `Model = :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `:"gpt-image-1"` - - - `:"gpt-image-1-mini"` - - - `:"gpt-image-2"` - - - `:"gpt-image-2-2026-04-21"` - - - `:"gpt-image-1.5"` - - - `:"chatgpt-image-latest"` - - - `moderation: :auto | :low` - - Moderation level for the generated image. Default: `auto`. - - - `:auto` - - - `:low` - - - `output_compression: Integer` - - Compression level for the output image. Default: 100. - - - `output_format: :png | :webp | :jpeg` - - The output format of the generated image. One of `png`, `webp`, or - `jpeg`. Default: `png`. - - - `:png` - - - `:webp` - - - `:jpeg` - - - `partial_images: Integer` - - Number of partial images to generate in streaming mode, from 0 (default value) to 3. - - - `quality: :low | :medium | :high | :auto` - - The quality of the generated image. One of `low`, `medium`, `high`, - or `auto`. Default: `auto`. - - - `:low` - - - `:medium` - - - `:high` - - - `:auto` - - - `size: String | :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `String = String` - - - `Size = :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `:"1024x1024"` - - - `:"1024x1536"` - - - `:"1536x1024"` - - - `:auto` - - - `class LocalShell` - - A tool that allows the model to execute shell commands in a local environment. - - - `type: :local_shell` - - The type of the local shell tool. Always `local_shell`. - - - `:local_shell` - - - `class FunctionShellTool` - - A tool that allows the model to execute shell commands. - - - `type: :shell` - - The type of the shell tool. Always `shell`. - - - `:shell` - - - `environment: ContainerAuto | LocalEnvironment | ContainerReference` - - - `class ContainerAuto` - - - `type: :container_auto` - - Automatically creates a container for this request - - - `:container_auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `skills: Array[SkillReference | InlineSkill]` - - An optional list of skills referenced by id or inline data. - - - `class SkillReference` - - - `skill_id: String` - - The ID of the referenced skill. - - - `type: :skill_reference` - - References a skill created with the /v1/skills endpoint. - - - `:skill_reference` - - - `version: String` - - Optional skill version. Use a positive integer or 'latest'. Omit for default. - - - `class InlineSkill` - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `source: InlineSkillSource` - - Inline skill payload - - - `data: String` - - Base64-encoded skill zip bundle. - - - `media_type: :"application/zip"` - - The media type of the inline skill payload. Must be `application/zip`. - - - `:"application/zip"` - - - `type: :base64` - - The type of the inline skill source. Must be `base64`. - - - `:base64` - - - `type: :inline` - - Defines an inline skill for this request. - - - `:inline` - - - `class LocalEnvironment` - - - `type: :local` - - Use a local computer environment. - - - `:local` - - - `skills: Array[LocalSkill]` - - An optional list of skills. - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `path: String` - - The path to the directory containing the skill. - - - `class ContainerReference` - - - `container_id: String` - - The ID of the referenced container. - - - `type: :container_reference` - - References a container created with the /v1/containers endpoint - - - `:container_reference` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `name: String` - - The name of the custom tool, used to identify it in tool calls. - - - `type: :custom` - - The type of the custom tool. Always `custom`. - - - `:custom` - - - `defer_loading: bool` - - Whether this tool should be deferred and discovered via tool search. - - - `description: String` - - Optional description of the custom tool, used to provide more context. - - - `format_: CustomToolInputFormat` - - The input format for the custom tool. Default is unconstrained text. - - - `class Text` - - Unconstrained free-form text. - - - `type: :text` - - Unconstrained text format. Always `text`. - - - `:text` - - - `class Grammar` - - A grammar defined by the user. - - - `definition: String` - - The grammar definition. - - - `syntax: :lark | :regex` - - The syntax of the grammar definition. One of `lark` or `regex`. - - - `:lark` - - - `:regex` - - - `type: :grammar` - - Grammar format. Always `grammar`. - - - `:grammar` - - - `class NamespaceTool` - - Groups function/custom tools under a shared namespace. - - - `description: String` - - A description of the namespace shown to the model. - - - `name: String` - - The namespace name used in tool calls (for example, `crm`). - - - `tools: Array[Function{ name, type, defer_loading, 3 more} | CustomTool]` - - The function/custom tools available inside this namespace. - - - `class Function` - - - `name: String` - - - `type: :function` - - - `:function` - - - `defer_loading: bool` - - Whether this function should be deferred and discovered via tool search. - - - `description: String` - - - `parameters: untyped` - - - `strict: bool` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `type: :namespace` - - The type of the tool. Always `namespace`. - - - `:namespace` - - - `class ToolSearchTool` - - Hosted or BYOT tool search configuration for deferred tools. - - - `type: :tool_search` - - The type of the tool. Always `tool_search`. - - - `:tool_search` - - - `description: String` - - Description shown to the model for a client-executed tool search tool. - - - `execution: :server | :client` - - Whether tool search is executed by the server or by the client. - - - `:server` - - - `:client` - - - `parameters: untyped` - - Parameter schema for a client-executed tool search tool. - - - `class WebSearchPreviewTool` - - This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search_preview | :web_search_preview_2025_03_11` - - The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`. - - - `:web_search_preview` - - - `:web_search_preview_2025_03_11` - - - `search_content_types: Array[:text | :image]` - - - `:text` - - - `:image` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ type, city, country, 2 more}` - - The user's location. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `class ApplyPatchTool` - - Allows the assistant to create, delete, or update files using unified diffs. - - - `type: :apply_patch` - - The type of the tool. Always `apply_patch`. - - - `:apply_patch` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `error: EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - - - `eval_id: String` - - The identifier of the associated evaluation. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - The model that is evaluated, if applicable. - - - `name: String` - - The name of the evaluation run. - - - `object: :"eval.run"` - - The type of the object. Always "eval.run". - - - `:"eval.run"` - - - `per_model_usage: Array[PerModelUsage{ cached_tokens, completion_tokens, invocation_count, 3 more}]` - - Usage statistics for each model during the evaluation run. - - - `cached_tokens: Integer` - - The number of tokens retrieved from cache. - - - `completion_tokens: Integer` - - The number of completion tokens generated. - - - `invocation_count: Integer` - - The number of invocations. - - - `model_name: String` - - The name of the model. - - - `prompt_tokens: Integer` - - The number of prompt tokens used. - - - `total_tokens: Integer` - - The total number of tokens used. - - - `per_testing_criteria_results: Array[PerTestingCriteriaResult{ failed, passed, testing_criteria}]` - - Results per testing criteria applied during the evaluation run. - - - `failed: Integer` - - Number of tests failed for this criteria. - - - `passed: Integer` - - Number of tests passed for this criteria. - - - `testing_criteria: String` - - A description of the testing criteria. - - - `report_url: String` - - The URL to the rendered evaluation run report on the UI dashboard. - - - `result_counts: ResultCounts{ errored, failed, passed, total}` - - Counters summarizing the outcomes of the evaluation run. - - - `errored: Integer` - - Number of output items that resulted in an error. - - - `failed: Integer` - - Number of output items that failed to pass the evaluation. - - - `passed: Integer` - - Number of output items that passed the evaluation. - - - `total: Integer` - - Total number of executed output items. - - - `status: String` - - The status of the evaluation run. - -### Example - -```ruby -require "openai" - -openai = OpenAI::Client.new(api_key: "My API Key") - -run = openai.evals.runs.create( - "eval_id", - data_source: {source: {content: [{item: {foo: "bar"}}], type: :file_content}, type: :jsonl} -) - -puts(run) -``` - -#### Response - -```json -{ - "id": "id", - "created_at": 0, - "data_source": { - "source": { - "content": [ - { - "item": { - "foo": "bar" - }, - "sample": { - "foo": "bar" - } - } - ], - "type": "file_content" - }, - "type": "jsonl" - }, - "error": { - "code": "code", - "message": "message" - }, - "eval_id": "eval_id", - "metadata": { - "foo": "string" - }, - "model": "model", - "name": "name", - "object": "eval.run", - "per_model_usage": [ - { - "cached_tokens": 0, - "completion_tokens": 0, - "invocation_count": 0, - "model_name": "model_name", - "prompt_tokens": 0, - "total_tokens": 0 - } - ], - "per_testing_criteria_results": [ - { - "failed": 0, - "passed": 0, - "testing_criteria": "testing_criteria" - } - ], - "report_url": "https://example.com", - "result_counts": { - "errored": 0, - "failed": 0, - "passed": 0, - "total": 0 - }, - "status": "status" -} -``` - -## Get an eval run - -`evals.runs.retrieve(run_id, **kwargs) -> RunRetrieveResponse` - -**get** `/evals/{eval_id}/runs/{run_id}` - -Get an evaluation run by ID. - -### Parameters - -- `eval_id: String` - -- `run_id: String` - -### Returns - -- `class RunRetrieveResponse` - - A schema representing an evaluation run. - - - `id: String` - - Unique identifier for the evaluation run. - - - `created_at: Integer` - - Unix timestamp (in seconds) when the evaluation run was created. - - - `data_source: CreateEvalJSONLRunDataSource | CreateEvalCompletionsRunDataSource | Responses{ source, type, input_messages, 2 more}` - - Information about the run's data source. - - - `class CreateEvalJSONLRunDataSource` - - A JsonlRunDataSource object with that specifies a JSONL file that matches the eval - - - `source: FileContent{ content, type} | FileID{ id, type}` - - Determines what populates the `item` namespace in the data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `type: :jsonl` - - The type of data source. Always `jsonl`. - - - `:jsonl` - - - `class CreateEvalCompletionsRunDataSource` - - A CompletionsRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | StoredCompletions{ type, created_after, created_before, 3 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class StoredCompletions` - - A StoredCompletionsRunDataSource configuration describing a set of filters - - - `type: :stored_completions` - - The type of source. Always `stored_completions`. - - - `:stored_completions` - - - `created_after: Integer` - - An optional Unix timestamp to filter items created after this time. - - - `created_before: Integer` - - An optional Unix timestamp to filter items created before this time. - - - `limit: Integer` - - An optional maximum number of items to return. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - An optional model to filter by (e.g., 'gpt-4o'). - - - `type: :completions` - - The type of run data source. Always `completions`. - - - `:completions` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[EasyInputMessage | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class EasyInputMessage` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputMessageContentList` - - Text, image, or audio input to the model, used to generate a response. - Can also contain previous assistant responses. - - - `String = String` - - A text input to the model. - - - `ResponseInputMessageContentList = Array[ResponseInputContent]` - - A list of one or many input items to the model, containing different content - types. - - - `class ResponseInputText` - - A text input to the model. - - - `text: String` - - The text input to the model. - - - `type: :input_text` - - The type of the input item. Always `input_text`. - - - `:input_text` - - - `class ResponseInputImage` - - An image input to the model. Learn about [image inputs](https://platform.openai.com/docs/guides/vision). - - - `detail: :low | :high | :auto | :original` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `:low` - - - `:high` - - - `:auto` - - - `:original` - - - `type: :input_image` - - The type of the input item. Always `input_image`. - - - `:input_image` - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `image_url: String` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `class ResponseInputFile` - - A file input to the model. - - - `type: :input_file` - - The type of the input item. Always `input_file`. - - - `:input_file` - - - `detail: :low | :high` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `:low` - - - `:high` - - - `file_data: String` - - The content of the file to be sent to the model. - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `file_url: String` - - The URL of the file to be sent to the model. - - - `filename: String` - - The name of the file to be sent to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `phase: :commentary | :final_answer` - - Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`). - For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend - phase on all assistant messages — dropping it can degrade performance. Not used for user messages. - - - `:commentary` - - - `:final_answer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `input_audio: InputAudio{ data, format_}` - - - `data: String` - - Base64-encoded audio data. - - - `format_: :mp3 | :wav` - - The format of the audio data. Currently supported formats are `mp3` and - `wav`. - - - `:mp3` - - - `:wav` - - - `type: :input_audio` - - The type of the input item. Always `input_audio`. - - - `:input_audio` - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.input_trajectory" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, response_format, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `:none` - - - `:minimal` - - - `:low` - - - `:medium` - - - `:high` - - - `:xhigh` - - - `response_format: ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject` - - An object specifying the format that the model must output. - - Setting to `{ "type": "json_schema", "json_schema": {...} }` enables - Structured Outputs which ensures the model will match your supplied JSON - schema. Learn more in the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `type: :text` - - The type of response format being defined. Always `text`. - - - `:text` - - - `class ResponseFormatJSONSchema` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `json_schema: JSONSchema{ name, description, schema, strict}` - - Structured Outputs configuration options, including a JSON Schema. - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `type: :json_object` - - The type of response format being defined. Always `json_object`. - - - `:json_object` - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `tools: Array[ChatCompletionFunctionTool]` - - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported. - - - `function: FunctionDefinition` - - - `name: String` - - The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the function does, used by the model to choose when and how to call the function. - - - `parameters: FunctionParameters` - - The parameters the functions accepts, described as a JSON Schema object. See the [guide](https://platform.openai.com/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format. - - Omitting `parameters` defines a function with an empty parameter list. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](https://platform.openai.com/docs/guides/function-calling). - - - `type: :function` - - The type of the tool. Currently, only `function` is supported. - - - `:function` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `class Responses` - - A ResponsesRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | Responses{ type, created_after, created_before, 8 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class Responses` - - A EvalResponsesSource object describing a run data source configuration. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `created_after: Integer` - - Only include items created after this timestamp (inclusive). This is a query parameter used to select responses. - - - `created_before: Integer` - - Only include items created before this timestamp (inclusive). This is a query parameter used to select responses. - - - `instructions_search: String` - - Optional string to search the 'instructions' field. This is a query parameter used to select responses. - - - `metadata: untyped` - - Metadata filter for the responses. This is a query parameter used to select responses. - - - `model: String` - - The name of the model to find responses for. This is a query parameter used to select responses. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `temperature: Float` - - Sampling temperature. This is a query parameter used to select responses. - - - `tools: Array[String]` - - List of tool names. This is a query parameter used to select responses. - - - `top_p: Float` - - Nucleus sampling parameter. This is a query parameter used to select responses. - - - `users: Array[String]` - - List of user identifiers. This is a query parameter used to select responses. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[ChatMessage{ content, role} | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class ChatMessage` - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.name" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, seed, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `text: Text{ format_}` - - Configuration options for a text response from the model. Can be plain - text or structured JSON data. Learn more: - - - [Text inputs and outputs](https://platform.openai.com/docs/guides/text) - - [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) - - - `format_: ResponseFormatTextConfig` - - An object specifying the format that the model must output. - - Configuring `{ "type": "json_schema" }` enables Structured Outputs, - which ensures the model will match your supplied JSON schema. Learn more in the - [Structured Outputs guide](https://platform.openai.com/docs/guides/structured-outputs). - - The default format is `{ "type": "text" }` with no additional options. - - **Not recommended for gpt-4o and newer models:** - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `class ResponseFormatTextJSONSchemaConfig` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `tools: Array[Tool]` - - An array of tools the model may call while generating a response. You - can specify which tool to use by setting the `tool_choice` parameter. - - The two categories of tools you can provide the model are: - - - **Built-in tools**: Tools that are provided by OpenAI that extend the - model's capabilities, like [web search](https://platform.openai.com/docs/guides/tools-web-search) - or [file search](https://platform.openai.com/docs/guides/tools-file-search). Learn more about - [built-in tools](https://platform.openai.com/docs/guides/tools). - - **Function calls (custom tools)**: Functions that are defined by you, - enabling the model to call your own code. Learn more about - [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `class FunctionTool` - - Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `name: String` - - The name of the function to call. - - - `parameters: Hash[Symbol, untyped]` - - A JSON schema object describing the parameters of the function. - - - `strict: bool` - - Whether to enforce strict parameter validation. Default `true`. - - - `type: :function` - - The type of the function tool. Always `function`. - - - `:function` - - - `defer_loading: bool` - - Whether this function is deferred and loaded via tool search. - - - `description: String` - - A description of the function. Used by the model to determine whether or not to call the function. - - - `class FileSearchTool` - - A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search). - - - `type: :file_search` - - The type of the file search tool. Always `file_search`. - - - `:file_search` - - - `vector_store_ids: Array[String]` - - The IDs of the vector stores to search. - - - `filters: ComparisonFilter | CompoundFilter` - - A filter to apply. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `key: String` - - The key to compare against the value. - - - `type: :eq | :ne | :gt | 5 more` - - Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`. - - - `eq`: equals - - `ne`: not equal - - `gt`: greater than - - `gte`: greater than or equal - - `lt`: less than - - `lte`: less than or equal - - `in`: in - - `nin`: not in - - - `:eq` - - - `:ne` - - - `:gt` - - - `:gte` - - - `:lt` - - - `:lte` - - - `:in` - - - `:nin` - - - `value: String | Float | bool | Array[String | Float]` - - The value to compare against the attribute key; supports string, number, or boolean types. - - - `String = String` - - - `Float = Float` - - - `UnionMember2 = bool` - - - `UnionMember3 = Array[String | Float]` - - - `String = String` - - - `Float = Float` - - - `class CompoundFilter` - - Combine multiple filters using `and` or `or`. - - - `filters: Array[ComparisonFilter | untyped]` - - Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `UnionMember1 = untyped` - - - `type: :and | :or` - - Type of operation: `and` or `or`. - - - `:and` - - - `:or` - - - `max_num_results: Integer` - - The maximum number of results to return. This number should be between 1 and 50 inclusive. - - - `ranking_options: RankingOptions{ hybrid_search, ranker, score_threshold}` - - Ranking options for search. - - - `hybrid_search: HybridSearch{ embedding_weight, text_weight}` - - Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled. - - - `embedding_weight: Float` - - The weight of the embedding in the reciprocal ranking fusion. - - - `text_weight: Float` - - The weight of the text in the reciprocal ranking fusion. - - - `ranker: :auto | :"default-2024-11-15"` - - The ranker to use for the file search. - - - `:auto` - - - `:"default-2024-11-15"` - - - `score_threshold: Float` - - The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results. - - - `class ComputerTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `type: :computer` - - The type of the computer tool. Always `computer`. - - - `:computer` - - - `class ComputerUsePreviewTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `display_height: Integer` - - The height of the computer display. - - - `display_width: Integer` - - The width of the computer display. - - - `environment: :windows | :mac | :linux | 2 more` - - The type of computer environment to control. - - - `:windows` - - - `:mac` - - - `:linux` - - - `:ubuntu` - - - `:browser` - - - `type: :computer_use_preview` - - The type of the computer use tool. Always `computer_use_preview`. - - - `:computer_use_preview` - - - `class WebSearchTool` - - Search the Internet for sources related to the prompt. Learn more about the - [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search | :web_search_2025_08_26` - - The type of the web search tool. One of `web_search` or `web_search_2025_08_26`. - - - `:web_search` - - - `:web_search_2025_08_26` - - - `filters: Filters{ allowed_domains}` - - Filters for the search. - - - `allowed_domains: Array[String]` - - Allowed domains for the search. If not provided, all domains are allowed. - Subdomains of the provided domains are allowed as well. - - Example: `["pubmed.ncbi.nlm.nih.gov"]` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ city, country, region, 2 more}` - - The approximate location of the user. - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `class Mcp` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](https://platform.openai.com/docs/guides/tools-remote-mcp). - - - `server_label: String` - - A label for this MCP server, used to identify it in tool calls. - - - `type: :mcp` - - The type of the MCP tool. Always `mcp`. - - - `:mcp` - - - `allowed_tools: Array[String] | McpToolFilter{ read_only, tool_names}` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = Array[String]` - - A string array of allowed tool names - - - `class McpToolFilter` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `authorization: String` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: :connector_dropbox | :connector_gmail | :connector_googlecalendar | 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](https://platform.openai.com/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `:connector_dropbox` - - - `:connector_gmail` - - - `:connector_googlecalendar` - - - `:connector_googledrive` - - - `:connector_microsoftteams` - - - `:connector_outlookcalendar` - - - `:connector_outlookemail` - - - `:connector_sharepoint` - - - `defer_loading: bool` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: Hash[Symbol, String]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: McpToolApprovalFilter{ always, never} | :always | :never` - - Specify which of the MCP server's tools require approval. - - - `class McpToolApprovalFilter` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: Always{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `never: Never{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `McpToolApprovalSetting = :always | :never` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `:always` - - - `:never` - - - `server_description: String` - - Optional description of the MCP server, used to provide more context. - - - `server_url: String` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `class CodeInterpreter` - - A tool that runs Python code to help generate a response to a prompt. - - - `container: String | CodeInterpreterToolAuto{ type, file_ids, memory_limit, network_policy}` - - The code interpreter container. Can be a container ID or an object that - specifies uploaded file IDs to make available to your code, along with an - optional `memory_limit` setting. - - - `String = String` - - The container ID. - - - `class CodeInterpreterToolAuto` - - Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on. - - - `type: :auto` - - Always `auto`. - - - `:auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the code interpreter container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `type: :disabled` - - Disable outbound network access. Always `disabled`. - - - `:disabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `allowed_domains: Array[String]` - - A list of allowed domains when type is `allowlist`. - - - `type: :allowlist` - - Allow outbound network access only to specified domains. Always `allowlist`. - - - `:allowlist` - - - `domain_secrets: Array[ContainerNetworkPolicyDomainSecret]` - - Optional domain-scoped secrets for allowlisted domains. - - - `domain: String` - - The domain associated with the secret. - - - `name: String` - - The name of the secret to inject for the domain. - - - `value: String` - - The secret value to inject for the domain. - - - `type: :code_interpreter` - - The type of the code interpreter tool. Always `code_interpreter`. - - - `:code_interpreter` - - - `class ImageGeneration` - - A tool that generates images using the GPT image models. - - - `type: :image_generation` - - The type of the image generation tool. Always `image_generation`. - - - `:image_generation` - - - `action: :generate | :edit | :auto` - - Whether to generate a new image or edit an existing image. Default: `auto`. - - - `:generate` - - - `:edit` - - - `:auto` - - - `background: :transparent | :opaque | :auto` - - Allows to set transparency for the background of the generated image(s). - This parameter is only supported for GPT image models that support - transparent backgrounds. Must be one of `transparent`, `opaque`, or - `auto` (default value). When `auto` is used, the model will - automatically determine the best background for the image. - - `gpt-image-2` and `gpt-image-2-2026-04-21` do not support - transparent backgrounds. Requests with `background` set to - `transparent` will return an error for these models; use `opaque` or - `auto` instead. - - If `transparent`, the output format needs to support transparency, - so it should be set to either `png` (default value) or `webp`. - - - `:transparent` - - - `:opaque` - - - `:auto` - - - `input_fidelity: :high | :low` - - Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`. - - - `:high` - - - `:low` - - - `input_image_mask: InputImageMask{ file_id, image_url}` - - Optional mask for inpainting. Contains `image_url` - (string, optional) and `file_id` (string, optional). - - - `file_id: String` - - File ID for the mask image. - - - `image_url: String` - - Base64-encoded mask image. - - - `model: String | :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `String = String` - - - `Model = :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `:"gpt-image-1"` - - - `:"gpt-image-1-mini"` - - - `:"gpt-image-2"` - - - `:"gpt-image-2-2026-04-21"` - - - `:"gpt-image-1.5"` - - - `:"chatgpt-image-latest"` - - - `moderation: :auto | :low` - - Moderation level for the generated image. Default: `auto`. - - - `:auto` - - - `:low` - - - `output_compression: Integer` - - Compression level for the output image. Default: 100. - - - `output_format: :png | :webp | :jpeg` - - The output format of the generated image. One of `png`, `webp`, or - `jpeg`. Default: `png`. - - - `:png` - - - `:webp` - - - `:jpeg` - - - `partial_images: Integer` - - Number of partial images to generate in streaming mode, from 0 (default value) to 3. - - - `quality: :low | :medium | :high | :auto` - - The quality of the generated image. One of `low`, `medium`, `high`, - or `auto`. Default: `auto`. - - - `:low` - - - `:medium` - - - `:high` - - - `:auto` - - - `size: String | :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `String = String` - - - `Size = :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `:"1024x1024"` - - - `:"1024x1536"` - - - `:"1536x1024"` - - - `:auto` - - - `class LocalShell` - - A tool that allows the model to execute shell commands in a local environment. - - - `type: :local_shell` - - The type of the local shell tool. Always `local_shell`. - - - `:local_shell` - - - `class FunctionShellTool` - - A tool that allows the model to execute shell commands. - - - `type: :shell` - - The type of the shell tool. Always `shell`. - - - `:shell` - - - `environment: ContainerAuto | LocalEnvironment | ContainerReference` - - - `class ContainerAuto` - - - `type: :container_auto` - - Automatically creates a container for this request - - - `:container_auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `skills: Array[SkillReference | InlineSkill]` - - An optional list of skills referenced by id or inline data. - - - `class SkillReference` - - - `skill_id: String` - - The ID of the referenced skill. - - - `type: :skill_reference` - - References a skill created with the /v1/skills endpoint. - - - `:skill_reference` - - - `version: String` - - Optional skill version. Use a positive integer or 'latest'. Omit for default. - - - `class InlineSkill` - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `source: InlineSkillSource` - - Inline skill payload - - - `data: String` - - Base64-encoded skill zip bundle. - - - `media_type: :"application/zip"` - - The media type of the inline skill payload. Must be `application/zip`. - - - `:"application/zip"` - - - `type: :base64` - - The type of the inline skill source. Must be `base64`. - - - `:base64` - - - `type: :inline` - - Defines an inline skill for this request. - - - `:inline` - - - `class LocalEnvironment` - - - `type: :local` - - Use a local computer environment. - - - `:local` - - - `skills: Array[LocalSkill]` - - An optional list of skills. - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `path: String` - - The path to the directory containing the skill. - - - `class ContainerReference` - - - `container_id: String` - - The ID of the referenced container. - - - `type: :container_reference` - - References a container created with the /v1/containers endpoint - - - `:container_reference` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `name: String` - - The name of the custom tool, used to identify it in tool calls. - - - `type: :custom` - - The type of the custom tool. Always `custom`. - - - `:custom` - - - `defer_loading: bool` - - Whether this tool should be deferred and discovered via tool search. - - - `description: String` - - Optional description of the custom tool, used to provide more context. - - - `format_: CustomToolInputFormat` - - The input format for the custom tool. Default is unconstrained text. - - - `class Text` - - Unconstrained free-form text. - - - `type: :text` - - Unconstrained text format. Always `text`. - - - `:text` - - - `class Grammar` - - A grammar defined by the user. - - - `definition: String` - - The grammar definition. - - - `syntax: :lark | :regex` - - The syntax of the grammar definition. One of `lark` or `regex`. - - - `:lark` - - - `:regex` - - - `type: :grammar` - - Grammar format. Always `grammar`. - - - `:grammar` - - - `class NamespaceTool` - - Groups function/custom tools under a shared namespace. - - - `description: String` - - A description of the namespace shown to the model. - - - `name: String` - - The namespace name used in tool calls (for example, `crm`). - - - `tools: Array[Function{ name, type, defer_loading, 3 more} | CustomTool]` - - The function/custom tools available inside this namespace. - - - `class Function` - - - `name: String` - - - `type: :function` - - - `:function` - - - `defer_loading: bool` - - Whether this function should be deferred and discovered via tool search. - - - `description: String` - - - `parameters: untyped` - - - `strict: bool` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `type: :namespace` - - The type of the tool. Always `namespace`. - - - `:namespace` - - - `class ToolSearchTool` - - Hosted or BYOT tool search configuration for deferred tools. - - - `type: :tool_search` - - The type of the tool. Always `tool_search`. - - - `:tool_search` - - - `description: String` - - Description shown to the model for a client-executed tool search tool. - - - `execution: :server | :client` - - Whether tool search is executed by the server or by the client. - - - `:server` - - - `:client` - - - `parameters: untyped` - - Parameter schema for a client-executed tool search tool. - - - `class WebSearchPreviewTool` - - This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search_preview | :web_search_preview_2025_03_11` - - The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`. - - - `:web_search_preview` - - - `:web_search_preview_2025_03_11` - - - `search_content_types: Array[:text | :image]` - - - `:text` - - - `:image` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ type, city, country, 2 more}` - - The user's location. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `class ApplyPatchTool` - - Allows the assistant to create, delete, or update files using unified diffs. - - - `type: :apply_patch` - - The type of the tool. Always `apply_patch`. - - - `:apply_patch` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `error: EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - - - `eval_id: String` - - The identifier of the associated evaluation. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - The model that is evaluated, if applicable. - - - `name: String` - - The name of the evaluation run. - - - `object: :"eval.run"` - - The type of the object. Always "eval.run". - - - `:"eval.run"` - - - `per_model_usage: Array[PerModelUsage{ cached_tokens, completion_tokens, invocation_count, 3 more}]` - - Usage statistics for each model during the evaluation run. - - - `cached_tokens: Integer` - - The number of tokens retrieved from cache. - - - `completion_tokens: Integer` - - The number of completion tokens generated. - - - `invocation_count: Integer` - - The number of invocations. - - - `model_name: String` - - The name of the model. - - - `prompt_tokens: Integer` - - The number of prompt tokens used. - - - `total_tokens: Integer` - - The total number of tokens used. - - - `per_testing_criteria_results: Array[PerTestingCriteriaResult{ failed, passed, testing_criteria}]` - - Results per testing criteria applied during the evaluation run. - - - `failed: Integer` - - Number of tests failed for this criteria. - - - `passed: Integer` - - Number of tests passed for this criteria. - - - `testing_criteria: String` - - A description of the testing criteria. - - - `report_url: String` - - The URL to the rendered evaluation run report on the UI dashboard. - - - `result_counts: ResultCounts{ errored, failed, passed, total}` - - Counters summarizing the outcomes of the evaluation run. - - - `errored: Integer` - - Number of output items that resulted in an error. - - - `failed: Integer` - - Number of output items that failed to pass the evaluation. - - - `passed: Integer` - - Number of output items that passed the evaluation. - - - `total: Integer` - - Total number of executed output items. - - - `status: String` - - The status of the evaluation run. - -### Example - -```ruby -require "openai" - -openai = OpenAI::Client.new(api_key: "My API Key") - -run = openai.evals.runs.retrieve("run_id", eval_id: "eval_id") - -puts(run) -``` - -#### Response - -```json -{ - "id": "id", - "created_at": 0, - "data_source": { - "source": { - "content": [ - { - "item": { - "foo": "bar" - }, - "sample": { - "foo": "bar" - } - } - ], - "type": "file_content" - }, - "type": "jsonl" - }, - "error": { - "code": "code", - "message": "message" - }, - "eval_id": "eval_id", - "metadata": { - "foo": "string" - }, - "model": "model", - "name": "name", - "object": "eval.run", - "per_model_usage": [ - { - "cached_tokens": 0, - "completion_tokens": 0, - "invocation_count": 0, - "model_name": "model_name", - "prompt_tokens": 0, - "total_tokens": 0 - } - ], - "per_testing_criteria_results": [ - { - "failed": 0, - "passed": 0, - "testing_criteria": "testing_criteria" - } - ], - "report_url": "https://example.com", - "result_counts": { - "errored": 0, - "failed": 0, - "passed": 0, - "total": 0 - }, - "status": "status" -} -``` - -## Cancel eval run - -`evals.runs.cancel(run_id, **kwargs) -> RunCancelResponse` - -**post** `/evals/{eval_id}/runs/{run_id}` - -Cancel an ongoing evaluation run. - -### Parameters - -- `eval_id: String` - -- `run_id: String` - -### Returns - -- `class RunCancelResponse` - - A schema representing an evaluation run. - - - `id: String` - - Unique identifier for the evaluation run. - - - `created_at: Integer` - - Unix timestamp (in seconds) when the evaluation run was created. - - - `data_source: CreateEvalJSONLRunDataSource | CreateEvalCompletionsRunDataSource | Responses{ source, type, input_messages, 2 more}` - - Information about the run's data source. - - - `class CreateEvalJSONLRunDataSource` - - A JsonlRunDataSource object with that specifies a JSONL file that matches the eval - - - `source: FileContent{ content, type} | FileID{ id, type}` - - Determines what populates the `item` namespace in the data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `type: :jsonl` - - The type of data source. Always `jsonl`. - - - `:jsonl` - - - `class CreateEvalCompletionsRunDataSource` - - A CompletionsRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | StoredCompletions{ type, created_after, created_before, 3 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class StoredCompletions` - - A StoredCompletionsRunDataSource configuration describing a set of filters - - - `type: :stored_completions` - - The type of source. Always `stored_completions`. - - - `:stored_completions` - - - `created_after: Integer` - - An optional Unix timestamp to filter items created after this time. - - - `created_before: Integer` - - An optional Unix timestamp to filter items created before this time. - - - `limit: Integer` - - An optional maximum number of items to return. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - An optional model to filter by (e.g., 'gpt-4o'). - - - `type: :completions` - - The type of run data source. Always `completions`. - - - `:completions` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[EasyInputMessage | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class EasyInputMessage` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputMessageContentList` - - Text, image, or audio input to the model, used to generate a response. - Can also contain previous assistant responses. - - - `String = String` - - A text input to the model. - - - `ResponseInputMessageContentList = Array[ResponseInputContent]` - - A list of one or many input items to the model, containing different content - types. - - - `class ResponseInputText` - - A text input to the model. - - - `text: String` - - The text input to the model. - - - `type: :input_text` - - The type of the input item. Always `input_text`. - - - `:input_text` - - - `class ResponseInputImage` - - An image input to the model. Learn about [image inputs](https://platform.openai.com/docs/guides/vision). - - - `detail: :low | :high | :auto | :original` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `:low` - - - `:high` - - - `:auto` - - - `:original` - - - `type: :input_image` - - The type of the input item. Always `input_image`. - - - `:input_image` - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `image_url: String` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `class ResponseInputFile` - - A file input to the model. - - - `type: :input_file` - - The type of the input item. Always `input_file`. - - - `:input_file` - - - `detail: :low | :high` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `:low` - - - `:high` - - - `file_data: String` - - The content of the file to be sent to the model. - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `file_url: String` - - The URL of the file to be sent to the model. - - - `filename: String` - - The name of the file to be sent to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `phase: :commentary | :final_answer` - - Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`). - For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend - phase on all assistant messages — dropping it can degrade performance. Not used for user messages. - - - `:commentary` - - - `:final_answer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `input_audio: InputAudio{ data, format_}` - - - `data: String` - - Base64-encoded audio data. - - - `format_: :mp3 | :wav` - - The format of the audio data. Currently supported formats are `mp3` and - `wav`. - - - `:mp3` - - - `:wav` - - - `type: :input_audio` - - The type of the input item. Always `input_audio`. - - - `:input_audio` - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.input_trajectory" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, response_format, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `:none` - - - `:minimal` - - - `:low` - - - `:medium` - - - `:high` - - - `:xhigh` - - - `response_format: ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject` - - An object specifying the format that the model must output. - - Setting to `{ "type": "json_schema", "json_schema": {...} }` enables - Structured Outputs which ensures the model will match your supplied JSON - schema. Learn more in the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `type: :text` - - The type of response format being defined. Always `text`. - - - `:text` - - - `class ResponseFormatJSONSchema` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `json_schema: JSONSchema{ name, description, schema, strict}` - - Structured Outputs configuration options, including a JSON Schema. - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `type: :json_object` - - The type of response format being defined. Always `json_object`. - - - `:json_object` - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `tools: Array[ChatCompletionFunctionTool]` - - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported. - - - `function: FunctionDefinition` - - - `name: String` - - The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the function does, used by the model to choose when and how to call the function. - - - `parameters: FunctionParameters` - - The parameters the functions accepts, described as a JSON Schema object. See the [guide](https://platform.openai.com/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format. - - Omitting `parameters` defines a function with an empty parameter list. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](https://platform.openai.com/docs/guides/function-calling). - - - `type: :function` - - The type of the tool. Currently, only `function` is supported. - - - `:function` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `class Responses` - - A ResponsesRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | Responses{ type, created_after, created_before, 8 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class Responses` - - A EvalResponsesSource object describing a run data source configuration. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `created_after: Integer` - - Only include items created after this timestamp (inclusive). This is a query parameter used to select responses. - - - `created_before: Integer` - - Only include items created before this timestamp (inclusive). This is a query parameter used to select responses. - - - `instructions_search: String` - - Optional string to search the 'instructions' field. This is a query parameter used to select responses. - - - `metadata: untyped` - - Metadata filter for the responses. This is a query parameter used to select responses. - - - `model: String` - - The name of the model to find responses for. This is a query parameter used to select responses. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `temperature: Float` - - Sampling temperature. This is a query parameter used to select responses. - - - `tools: Array[String]` - - List of tool names. This is a query parameter used to select responses. - - - `top_p: Float` - - Nucleus sampling parameter. This is a query parameter used to select responses. - - - `users: Array[String]` - - List of user identifiers. This is a query parameter used to select responses. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[ChatMessage{ content, role} | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class ChatMessage` - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.name" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, seed, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `text: Text{ format_}` - - Configuration options for a text response from the model. Can be plain - text or structured JSON data. Learn more: - - - [Text inputs and outputs](https://platform.openai.com/docs/guides/text) - - [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) - - - `format_: ResponseFormatTextConfig` - - An object specifying the format that the model must output. - - Configuring `{ "type": "json_schema" }` enables Structured Outputs, - which ensures the model will match your supplied JSON schema. Learn more in the - [Structured Outputs guide](https://platform.openai.com/docs/guides/structured-outputs). - - The default format is `{ "type": "text" }` with no additional options. - - **Not recommended for gpt-4o and newer models:** - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `class ResponseFormatTextJSONSchemaConfig` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `tools: Array[Tool]` - - An array of tools the model may call while generating a response. You - can specify which tool to use by setting the `tool_choice` parameter. - - The two categories of tools you can provide the model are: - - - **Built-in tools**: Tools that are provided by OpenAI that extend the - model's capabilities, like [web search](https://platform.openai.com/docs/guides/tools-web-search) - or [file search](https://platform.openai.com/docs/guides/tools-file-search). Learn more about - [built-in tools](https://platform.openai.com/docs/guides/tools). - - **Function calls (custom tools)**: Functions that are defined by you, - enabling the model to call your own code. Learn more about - [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `class FunctionTool` - - Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `name: String` - - The name of the function to call. - - - `parameters: Hash[Symbol, untyped]` - - A JSON schema object describing the parameters of the function. - - - `strict: bool` - - Whether to enforce strict parameter validation. Default `true`. - - - `type: :function` - - The type of the function tool. Always `function`. - - - `:function` - - - `defer_loading: bool` - - Whether this function is deferred and loaded via tool search. - - - `description: String` - - A description of the function. Used by the model to determine whether or not to call the function. - - - `class FileSearchTool` - - A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search). - - - `type: :file_search` - - The type of the file search tool. Always `file_search`. - - - `:file_search` - - - `vector_store_ids: Array[String]` - - The IDs of the vector stores to search. - - - `filters: ComparisonFilter | CompoundFilter` - - A filter to apply. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `key: String` - - The key to compare against the value. - - - `type: :eq | :ne | :gt | 5 more` - - Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`. - - - `eq`: equals - - `ne`: not equal - - `gt`: greater than - - `gte`: greater than or equal - - `lt`: less than - - `lte`: less than or equal - - `in`: in - - `nin`: not in - - - `:eq` - - - `:ne` - - - `:gt` - - - `:gte` - - - `:lt` - - - `:lte` - - - `:in` - - - `:nin` - - - `value: String | Float | bool | Array[String | Float]` - - The value to compare against the attribute key; supports string, number, or boolean types. - - - `String = String` - - - `Float = Float` - - - `UnionMember2 = bool` - - - `UnionMember3 = Array[String | Float]` - - - `String = String` - - - `Float = Float` - - - `class CompoundFilter` - - Combine multiple filters using `and` or `or`. - - - `filters: Array[ComparisonFilter | untyped]` - - Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `UnionMember1 = untyped` - - - `type: :and | :or` - - Type of operation: `and` or `or`. - - - `:and` - - - `:or` - - - `max_num_results: Integer` - - The maximum number of results to return. This number should be between 1 and 50 inclusive. - - - `ranking_options: RankingOptions{ hybrid_search, ranker, score_threshold}` - - Ranking options for search. - - - `hybrid_search: HybridSearch{ embedding_weight, text_weight}` - - Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled. - - - `embedding_weight: Float` - - The weight of the embedding in the reciprocal ranking fusion. - - - `text_weight: Float` - - The weight of the text in the reciprocal ranking fusion. - - - `ranker: :auto | :"default-2024-11-15"` - - The ranker to use for the file search. - - - `:auto` - - - `:"default-2024-11-15"` - - - `score_threshold: Float` - - The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results. - - - `class ComputerTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `type: :computer` - - The type of the computer tool. Always `computer`. - - - `:computer` - - - `class ComputerUsePreviewTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `display_height: Integer` - - The height of the computer display. - - - `display_width: Integer` - - The width of the computer display. - - - `environment: :windows | :mac | :linux | 2 more` - - The type of computer environment to control. - - - `:windows` - - - `:mac` - - - `:linux` - - - `:ubuntu` - - - `:browser` - - - `type: :computer_use_preview` - - The type of the computer use tool. Always `computer_use_preview`. - - - `:computer_use_preview` - - - `class WebSearchTool` - - Search the Internet for sources related to the prompt. Learn more about the - [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search | :web_search_2025_08_26` - - The type of the web search tool. One of `web_search` or `web_search_2025_08_26`. - - - `:web_search` - - - `:web_search_2025_08_26` - - - `filters: Filters{ allowed_domains}` - - Filters for the search. - - - `allowed_domains: Array[String]` - - Allowed domains for the search. If not provided, all domains are allowed. - Subdomains of the provided domains are allowed as well. - - Example: `["pubmed.ncbi.nlm.nih.gov"]` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ city, country, region, 2 more}` - - The approximate location of the user. - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `class Mcp` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](https://platform.openai.com/docs/guides/tools-remote-mcp). - - - `server_label: String` - - A label for this MCP server, used to identify it in tool calls. - - - `type: :mcp` - - The type of the MCP tool. Always `mcp`. - - - `:mcp` - - - `allowed_tools: Array[String] | McpToolFilter{ read_only, tool_names}` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = Array[String]` - - A string array of allowed tool names - - - `class McpToolFilter` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `authorization: String` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: :connector_dropbox | :connector_gmail | :connector_googlecalendar | 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](https://platform.openai.com/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `:connector_dropbox` - - - `:connector_gmail` - - - `:connector_googlecalendar` - - - `:connector_googledrive` - - - `:connector_microsoftteams` - - - `:connector_outlookcalendar` - - - `:connector_outlookemail` - - - `:connector_sharepoint` - - - `defer_loading: bool` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: Hash[Symbol, String]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: McpToolApprovalFilter{ always, never} | :always | :never` - - Specify which of the MCP server's tools require approval. - - - `class McpToolApprovalFilter` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: Always{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `never: Never{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `McpToolApprovalSetting = :always | :never` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `:always` - - - `:never` - - - `server_description: String` - - Optional description of the MCP server, used to provide more context. - - - `server_url: String` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `class CodeInterpreter` - - A tool that runs Python code to help generate a response to a prompt. - - - `container: String | CodeInterpreterToolAuto{ type, file_ids, memory_limit, network_policy}` - - The code interpreter container. Can be a container ID or an object that - specifies uploaded file IDs to make available to your code, along with an - optional `memory_limit` setting. - - - `String = String` - - The container ID. - - - `class CodeInterpreterToolAuto` - - Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on. - - - `type: :auto` - - Always `auto`. - - - `:auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the code interpreter container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `type: :disabled` - - Disable outbound network access. Always `disabled`. - - - `:disabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `allowed_domains: Array[String]` - - A list of allowed domains when type is `allowlist`. - - - `type: :allowlist` - - Allow outbound network access only to specified domains. Always `allowlist`. - - - `:allowlist` - - - `domain_secrets: Array[ContainerNetworkPolicyDomainSecret]` - - Optional domain-scoped secrets for allowlisted domains. - - - `domain: String` - - The domain associated with the secret. - - - `name: String` - - The name of the secret to inject for the domain. - - - `value: String` - - The secret value to inject for the domain. - - - `type: :code_interpreter` - - The type of the code interpreter tool. Always `code_interpreter`. - - - `:code_interpreter` - - - `class ImageGeneration` - - A tool that generates images using the GPT image models. - - - `type: :image_generation` - - The type of the image generation tool. Always `image_generation`. - - - `:image_generation` - - - `action: :generate | :edit | :auto` - - Whether to generate a new image or edit an existing image. Default: `auto`. - - - `:generate` - - - `:edit` - - - `:auto` - - - `background: :transparent | :opaque | :auto` - - Allows to set transparency for the background of the generated image(s). - This parameter is only supported for GPT image models that support - transparent backgrounds. Must be one of `transparent`, `opaque`, or - `auto` (default value). When `auto` is used, the model will - automatically determine the best background for the image. - - `gpt-image-2` and `gpt-image-2-2026-04-21` do not support - transparent backgrounds. Requests with `background` set to - `transparent` will return an error for these models; use `opaque` or - `auto` instead. - - If `transparent`, the output format needs to support transparency, - so it should be set to either `png` (default value) or `webp`. - - - `:transparent` - - - `:opaque` - - - `:auto` - - - `input_fidelity: :high | :low` - - Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`. - - - `:high` - - - `:low` - - - `input_image_mask: InputImageMask{ file_id, image_url}` - - Optional mask for inpainting. Contains `image_url` - (string, optional) and `file_id` (string, optional). - - - `file_id: String` - - File ID for the mask image. - - - `image_url: String` - - Base64-encoded mask image. - - - `model: String | :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `String = String` - - - `Model = :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `:"gpt-image-1"` - - - `:"gpt-image-1-mini"` - - - `:"gpt-image-2"` - - - `:"gpt-image-2-2026-04-21"` - - - `:"gpt-image-1.5"` - - - `:"chatgpt-image-latest"` - - - `moderation: :auto | :low` - - Moderation level for the generated image. Default: `auto`. - - - `:auto` - - - `:low` - - - `output_compression: Integer` - - Compression level for the output image. Default: 100. - - - `output_format: :png | :webp | :jpeg` - - The output format of the generated image. One of `png`, `webp`, or - `jpeg`. Default: `png`. - - - `:png` - - - `:webp` - - - `:jpeg` - - - `partial_images: Integer` - - Number of partial images to generate in streaming mode, from 0 (default value) to 3. - - - `quality: :low | :medium | :high | :auto` - - The quality of the generated image. One of `low`, `medium`, `high`, - or `auto`. Default: `auto`. - - - `:low` - - - `:medium` - - - `:high` - - - `:auto` - - - `size: String | :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `String = String` - - - `Size = :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `:"1024x1024"` - - - `:"1024x1536"` - - - `:"1536x1024"` - - - `:auto` - - - `class LocalShell` - - A tool that allows the model to execute shell commands in a local environment. - - - `type: :local_shell` - - The type of the local shell tool. Always `local_shell`. - - - `:local_shell` - - - `class FunctionShellTool` - - A tool that allows the model to execute shell commands. - - - `type: :shell` - - The type of the shell tool. Always `shell`. - - - `:shell` - - - `environment: ContainerAuto | LocalEnvironment | ContainerReference` - - - `class ContainerAuto` - - - `type: :container_auto` - - Automatically creates a container for this request - - - `:container_auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `skills: Array[SkillReference | InlineSkill]` - - An optional list of skills referenced by id or inline data. - - - `class SkillReference` - - - `skill_id: String` - - The ID of the referenced skill. - - - `type: :skill_reference` - - References a skill created with the /v1/skills endpoint. - - - `:skill_reference` - - - `version: String` - - Optional skill version. Use a positive integer or 'latest'. Omit for default. - - - `class InlineSkill` - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `source: InlineSkillSource` - - Inline skill payload - - - `data: String` - - Base64-encoded skill zip bundle. - - - `media_type: :"application/zip"` - - The media type of the inline skill payload. Must be `application/zip`. - - - `:"application/zip"` - - - `type: :base64` - - The type of the inline skill source. Must be `base64`. - - - `:base64` - - - `type: :inline` - - Defines an inline skill for this request. - - - `:inline` - - - `class LocalEnvironment` - - - `type: :local` - - Use a local computer environment. - - - `:local` - - - `skills: Array[LocalSkill]` - - An optional list of skills. - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `path: String` - - The path to the directory containing the skill. - - - `class ContainerReference` - - - `container_id: String` - - The ID of the referenced container. - - - `type: :container_reference` - - References a container created with the /v1/containers endpoint - - - `:container_reference` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `name: String` - - The name of the custom tool, used to identify it in tool calls. - - - `type: :custom` - - The type of the custom tool. Always `custom`. - - - `:custom` - - - `defer_loading: bool` - - Whether this tool should be deferred and discovered via tool search. - - - `description: String` - - Optional description of the custom tool, used to provide more context. - - - `format_: CustomToolInputFormat` - - The input format for the custom tool. Default is unconstrained text. - - - `class Text` - - Unconstrained free-form text. - - - `type: :text` - - Unconstrained text format. Always `text`. - - - `:text` - - - `class Grammar` - - A grammar defined by the user. - - - `definition: String` - - The grammar definition. - - - `syntax: :lark | :regex` - - The syntax of the grammar definition. One of `lark` or `regex`. - - - `:lark` - - - `:regex` - - - `type: :grammar` - - Grammar format. Always `grammar`. - - - `:grammar` - - - `class NamespaceTool` - - Groups function/custom tools under a shared namespace. - - - `description: String` - - A description of the namespace shown to the model. - - - `name: String` - - The namespace name used in tool calls (for example, `crm`). - - - `tools: Array[Function{ name, type, defer_loading, 3 more} | CustomTool]` - - The function/custom tools available inside this namespace. - - - `class Function` - - - `name: String` - - - `type: :function` - - - `:function` - - - `defer_loading: bool` - - Whether this function should be deferred and discovered via tool search. - - - `description: String` - - - `parameters: untyped` - - - `strict: bool` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `type: :namespace` - - The type of the tool. Always `namespace`. - - - `:namespace` - - - `class ToolSearchTool` - - Hosted or BYOT tool search configuration for deferred tools. - - - `type: :tool_search` - - The type of the tool. Always `tool_search`. - - - `:tool_search` - - - `description: String` - - Description shown to the model for a client-executed tool search tool. - - - `execution: :server | :client` - - Whether tool search is executed by the server or by the client. - - - `:server` - - - `:client` - - - `parameters: untyped` - - Parameter schema for a client-executed tool search tool. - - - `class WebSearchPreviewTool` - - This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search_preview | :web_search_preview_2025_03_11` - - The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`. - - - `:web_search_preview` - - - `:web_search_preview_2025_03_11` - - - `search_content_types: Array[:text | :image]` - - - `:text` - - - `:image` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ type, city, country, 2 more}` - - The user's location. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `class ApplyPatchTool` - - Allows the assistant to create, delete, or update files using unified diffs. - - - `type: :apply_patch` - - The type of the tool. Always `apply_patch`. - - - `:apply_patch` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `error: EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - - - `eval_id: String` - - The identifier of the associated evaluation. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - The model that is evaluated, if applicable. - - - `name: String` - - The name of the evaluation run. - - - `object: :"eval.run"` - - The type of the object. Always "eval.run". - - - `:"eval.run"` - - - `per_model_usage: Array[PerModelUsage{ cached_tokens, completion_tokens, invocation_count, 3 more}]` - - Usage statistics for each model during the evaluation run. - - - `cached_tokens: Integer` - - The number of tokens retrieved from cache. - - - `completion_tokens: Integer` - - The number of completion tokens generated. - - - `invocation_count: Integer` - - The number of invocations. - - - `model_name: String` - - The name of the model. - - - `prompt_tokens: Integer` - - The number of prompt tokens used. - - - `total_tokens: Integer` - - The total number of tokens used. - - - `per_testing_criteria_results: Array[PerTestingCriteriaResult{ failed, passed, testing_criteria}]` - - Results per testing criteria applied during the evaluation run. - - - `failed: Integer` - - Number of tests failed for this criteria. - - - `passed: Integer` - - Number of tests passed for this criteria. - - - `testing_criteria: String` - - A description of the testing criteria. - - - `report_url: String` - - The URL to the rendered evaluation run report on the UI dashboard. - - - `result_counts: ResultCounts{ errored, failed, passed, total}` - - Counters summarizing the outcomes of the evaluation run. - - - `errored: Integer` - - Number of output items that resulted in an error. - - - `failed: Integer` - - Number of output items that failed to pass the evaluation. - - - `passed: Integer` - - Number of output items that passed the evaluation. - - - `total: Integer` - - Total number of executed output items. - - - `status: String` - - The status of the evaluation run. - -### Example - -```ruby -require "openai" - -openai = OpenAI::Client.new(api_key: "My API Key") - -response = openai.evals.runs.cancel("run_id", eval_id: "eval_id") - -puts(response) -``` - -#### Response - -```json -{ - "id": "id", - "created_at": 0, - "data_source": { - "source": { - "content": [ - { - "item": { - "foo": "bar" - }, - "sample": { - "foo": "bar" - } - } - ], - "type": "file_content" - }, - "type": "jsonl" - }, - "error": { - "code": "code", - "message": "message" - }, - "eval_id": "eval_id", - "metadata": { - "foo": "string" - }, - "model": "model", - "name": "name", - "object": "eval.run", - "per_model_usage": [ - { - "cached_tokens": 0, - "completion_tokens": 0, - "invocation_count": 0, - "model_name": "model_name", - "prompt_tokens": 0, - "total_tokens": 0 - } - ], - "per_testing_criteria_results": [ - { - "failed": 0, - "passed": 0, - "testing_criteria": "testing_criteria" - } - ], - "report_url": "https://example.com", - "result_counts": { - "errored": 0, - "failed": 0, - "passed": 0, - "total": 0 - }, - "status": "status" -} -``` - -## Delete eval run - -`evals.runs.delete(run_id, **kwargs) -> RunDeleteResponse` - -**delete** `/evals/{eval_id}/runs/{run_id}` - -Delete an eval run. - -### Parameters - -- `eval_id: String` - -- `run_id: String` - -### Returns - -- `class RunDeleteResponse` - - - `deleted: bool` - - - `object: String` - - - `run_id: String` - -### Example - -```ruby -require "openai" - -openai = OpenAI::Client.new(api_key: "My API Key") - -run = openai.evals.runs.delete("run_id", eval_id: "eval_id") - -puts(run) -``` - -#### Response - -```json -{ - "deleted": true, - "object": "eval.run.deleted", - "run_id": "evalrun_677469f564d48190807532a852da3afb" -} -``` - -## Domain Types - -### Create Eval Completions Run Data Source - -- `class CreateEvalCompletionsRunDataSource` - - A CompletionsRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | StoredCompletions{ type, created_after, created_before, 3 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class StoredCompletions` - - A StoredCompletionsRunDataSource configuration describing a set of filters - - - `type: :stored_completions` - - The type of source. Always `stored_completions`. - - - `:stored_completions` - - - `created_after: Integer` - - An optional Unix timestamp to filter items created after this time. - - - `created_before: Integer` - - An optional Unix timestamp to filter items created before this time. - - - `limit: Integer` - - An optional maximum number of items to return. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - An optional model to filter by (e.g., 'gpt-4o'). - - - `type: :completions` - - The type of run data source. Always `completions`. - - - `:completions` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[EasyInputMessage | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class EasyInputMessage` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputMessageContentList` - - Text, image, or audio input to the model, used to generate a response. - Can also contain previous assistant responses. - - - `String = String` - - A text input to the model. - - - `ResponseInputMessageContentList = Array[ResponseInputContent]` - - A list of one or many input items to the model, containing different content - types. - - - `class ResponseInputText` - - A text input to the model. - - - `text: String` - - The text input to the model. - - - `type: :input_text` - - The type of the input item. Always `input_text`. - - - `:input_text` - - - `class ResponseInputImage` - - An image input to the model. Learn about [image inputs](https://platform.openai.com/docs/guides/vision). - - - `detail: :low | :high | :auto | :original` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `:low` - - - `:high` - - - `:auto` - - - `:original` - - - `type: :input_image` - - The type of the input item. Always `input_image`. - - - `:input_image` - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `image_url: String` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `class ResponseInputFile` - - A file input to the model. - - - `type: :input_file` - - The type of the input item. Always `input_file`. - - - `:input_file` - - - `detail: :low | :high` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `:low` - - - `:high` - - - `file_data: String` - - The content of the file to be sent to the model. - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `file_url: String` - - The URL of the file to be sent to the model. - - - `filename: String` - - The name of the file to be sent to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `phase: :commentary | :final_answer` - - Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`). - For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend - phase on all assistant messages — dropping it can degrade performance. Not used for user messages. - - - `:commentary` - - - `:final_answer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `input_audio: InputAudio{ data, format_}` - - - `data: String` - - Base64-encoded audio data. - - - `format_: :mp3 | :wav` - - The format of the audio data. Currently supported formats are `mp3` and - `wav`. - - - `:mp3` - - - `:wav` - - - `type: :input_audio` - - The type of the input item. Always `input_audio`. - - - `:input_audio` - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.input_trajectory" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, response_format, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `:none` - - - `:minimal` - - - `:low` - - - `:medium` - - - `:high` - - - `:xhigh` - - - `response_format: ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject` - - An object specifying the format that the model must output. - - Setting to `{ "type": "json_schema", "json_schema": {...} }` enables - Structured Outputs which ensures the model will match your supplied JSON - schema. Learn more in the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `type: :text` - - The type of response format being defined. Always `text`. - - - `:text` - - - `class ResponseFormatJSONSchema` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `json_schema: JSONSchema{ name, description, schema, strict}` - - Structured Outputs configuration options, including a JSON Schema. - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `type: :json_object` - - The type of response format being defined. Always `json_object`. - - - `:json_object` - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `tools: Array[ChatCompletionFunctionTool]` - - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported. - - - `function: FunctionDefinition` - - - `name: String` - - The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the function does, used by the model to choose when and how to call the function. - - - `parameters: FunctionParameters` - - The parameters the functions accepts, described as a JSON Schema object. See the [guide](https://platform.openai.com/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format. - - Omitting `parameters` defines a function with an empty parameter list. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](https://platform.openai.com/docs/guides/function-calling). - - - `type: :function` - - The type of the tool. Currently, only `function` is supported. - - - `:function` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - -### Create Eval JSONL Run Data Source - -- `class CreateEvalJSONLRunDataSource` - - A JsonlRunDataSource object with that specifies a JSONL file that matches the eval - - - `source: FileContent{ content, type} | FileID{ id, type}` - - Determines what populates the `item` namespace in the data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `type: :jsonl` - - The type of data source. Always `jsonl`. - - - `:jsonl` - -### Eval API Error - -- `class EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - -### Run List Response - -- `class RunListResponse` - - A schema representing an evaluation run. - - - `id: String` - - Unique identifier for the evaluation run. - - - `created_at: Integer` - - Unix timestamp (in seconds) when the evaluation run was created. - - - `data_source: CreateEvalJSONLRunDataSource | CreateEvalCompletionsRunDataSource | Responses{ source, type, input_messages, 2 more}` - - Information about the run's data source. - - - `class CreateEvalJSONLRunDataSource` - - A JsonlRunDataSource object with that specifies a JSONL file that matches the eval - - - `source: FileContent{ content, type} | FileID{ id, type}` - - Determines what populates the `item` namespace in the data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `type: :jsonl` - - The type of data source. Always `jsonl`. - - - `:jsonl` - - - `class CreateEvalCompletionsRunDataSource` - - A CompletionsRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | StoredCompletions{ type, created_after, created_before, 3 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class StoredCompletions` - - A StoredCompletionsRunDataSource configuration describing a set of filters - - - `type: :stored_completions` - - The type of source. Always `stored_completions`. - - - `:stored_completions` - - - `created_after: Integer` - - An optional Unix timestamp to filter items created after this time. - - - `created_before: Integer` - - An optional Unix timestamp to filter items created before this time. - - - `limit: Integer` - - An optional maximum number of items to return. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - An optional model to filter by (e.g., 'gpt-4o'). - - - `type: :completions` - - The type of run data source. Always `completions`. - - - `:completions` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[EasyInputMessage | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class EasyInputMessage` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputMessageContentList` - - Text, image, or audio input to the model, used to generate a response. - Can also contain previous assistant responses. - - - `String = String` - - A text input to the model. - - - `ResponseInputMessageContentList = Array[ResponseInputContent]` - - A list of one or many input items to the model, containing different content - types. - - - `class ResponseInputText` - - A text input to the model. - - - `text: String` - - The text input to the model. - - - `type: :input_text` - - The type of the input item. Always `input_text`. - - - `:input_text` - - - `class ResponseInputImage` - - An image input to the model. Learn about [image inputs](https://platform.openai.com/docs/guides/vision). - - - `detail: :low | :high | :auto | :original` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `:low` - - - `:high` - - - `:auto` - - - `:original` - - - `type: :input_image` - - The type of the input item. Always `input_image`. - - - `:input_image` - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `image_url: String` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `class ResponseInputFile` - - A file input to the model. - - - `type: :input_file` - - The type of the input item. Always `input_file`. - - - `:input_file` - - - `detail: :low | :high` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `:low` - - - `:high` - - - `file_data: String` - - The content of the file to be sent to the model. - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `file_url: String` - - The URL of the file to be sent to the model. - - - `filename: String` - - The name of the file to be sent to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `phase: :commentary | :final_answer` - - Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`). - For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend - phase on all assistant messages — dropping it can degrade performance. Not used for user messages. - - - `:commentary` - - - `:final_answer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `input_audio: InputAudio{ data, format_}` - - - `data: String` - - Base64-encoded audio data. - - - `format_: :mp3 | :wav` - - The format of the audio data. Currently supported formats are `mp3` and - `wav`. - - - `:mp3` - - - `:wav` - - - `type: :input_audio` - - The type of the input item. Always `input_audio`. - - - `:input_audio` - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.input_trajectory" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, response_format, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `:none` - - - `:minimal` - - - `:low` - - - `:medium` - - - `:high` - - - `:xhigh` - - - `response_format: ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject` - - An object specifying the format that the model must output. - - Setting to `{ "type": "json_schema", "json_schema": {...} }` enables - Structured Outputs which ensures the model will match your supplied JSON - schema. Learn more in the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `type: :text` - - The type of response format being defined. Always `text`. - - - `:text` - - - `class ResponseFormatJSONSchema` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `json_schema: JSONSchema{ name, description, schema, strict}` - - Structured Outputs configuration options, including a JSON Schema. - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `type: :json_object` - - The type of response format being defined. Always `json_object`. - - - `:json_object` - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `tools: Array[ChatCompletionFunctionTool]` - - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported. - - - `function: FunctionDefinition` - - - `name: String` - - The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the function does, used by the model to choose when and how to call the function. - - - `parameters: FunctionParameters` - - The parameters the functions accepts, described as a JSON Schema object. See the [guide](https://platform.openai.com/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format. - - Omitting `parameters` defines a function with an empty parameter list. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](https://platform.openai.com/docs/guides/function-calling). - - - `type: :function` - - The type of the tool. Currently, only `function` is supported. - - - `:function` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `class Responses` - - A ResponsesRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | Responses{ type, created_after, created_before, 8 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class Responses` - - A EvalResponsesSource object describing a run data source configuration. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `created_after: Integer` - - Only include items created after this timestamp (inclusive). This is a query parameter used to select responses. - - - `created_before: Integer` - - Only include items created before this timestamp (inclusive). This is a query parameter used to select responses. - - - `instructions_search: String` - - Optional string to search the 'instructions' field. This is a query parameter used to select responses. - - - `metadata: untyped` - - Metadata filter for the responses. This is a query parameter used to select responses. - - - `model: String` - - The name of the model to find responses for. This is a query parameter used to select responses. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `temperature: Float` - - Sampling temperature. This is a query parameter used to select responses. - - - `tools: Array[String]` - - List of tool names. This is a query parameter used to select responses. - - - `top_p: Float` - - Nucleus sampling parameter. This is a query parameter used to select responses. - - - `users: Array[String]` - - List of user identifiers. This is a query parameter used to select responses. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[ChatMessage{ content, role} | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class ChatMessage` - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.name" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, seed, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `text: Text{ format_}` - - Configuration options for a text response from the model. Can be plain - text or structured JSON data. Learn more: - - - [Text inputs and outputs](https://platform.openai.com/docs/guides/text) - - [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) - - - `format_: ResponseFormatTextConfig` - - An object specifying the format that the model must output. - - Configuring `{ "type": "json_schema" }` enables Structured Outputs, - which ensures the model will match your supplied JSON schema. Learn more in the - [Structured Outputs guide](https://platform.openai.com/docs/guides/structured-outputs). - - The default format is `{ "type": "text" }` with no additional options. - - **Not recommended for gpt-4o and newer models:** - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `class ResponseFormatTextJSONSchemaConfig` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `tools: Array[Tool]` - - An array of tools the model may call while generating a response. You - can specify which tool to use by setting the `tool_choice` parameter. - - The two categories of tools you can provide the model are: - - - **Built-in tools**: Tools that are provided by OpenAI that extend the - model's capabilities, like [web search](https://platform.openai.com/docs/guides/tools-web-search) - or [file search](https://platform.openai.com/docs/guides/tools-file-search). Learn more about - [built-in tools](https://platform.openai.com/docs/guides/tools). - - **Function calls (custom tools)**: Functions that are defined by you, - enabling the model to call your own code. Learn more about - [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `class FunctionTool` - - Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `name: String` - - The name of the function to call. - - - `parameters: Hash[Symbol, untyped]` - - A JSON schema object describing the parameters of the function. - - - `strict: bool` - - Whether to enforce strict parameter validation. Default `true`. - - - `type: :function` - - The type of the function tool. Always `function`. - - - `:function` - - - `defer_loading: bool` - - Whether this function is deferred and loaded via tool search. - - - `description: String` - - A description of the function. Used by the model to determine whether or not to call the function. - - - `class FileSearchTool` - - A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search). - - - `type: :file_search` - - The type of the file search tool. Always `file_search`. - - - `:file_search` - - - `vector_store_ids: Array[String]` - - The IDs of the vector stores to search. - - - `filters: ComparisonFilter | CompoundFilter` - - A filter to apply. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `key: String` - - The key to compare against the value. - - - `type: :eq | :ne | :gt | 5 more` - - Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`. - - - `eq`: equals - - `ne`: not equal - - `gt`: greater than - - `gte`: greater than or equal - - `lt`: less than - - `lte`: less than or equal - - `in`: in - - `nin`: not in - - - `:eq` - - - `:ne` - - - `:gt` - - - `:gte` - - - `:lt` - - - `:lte` - - - `:in` - - - `:nin` - - - `value: String | Float | bool | Array[String | Float]` - - The value to compare against the attribute key; supports string, number, or boolean types. - - - `String = String` - - - `Float = Float` - - - `UnionMember2 = bool` - - - `UnionMember3 = Array[String | Float]` - - - `String = String` - - - `Float = Float` - - - `class CompoundFilter` - - Combine multiple filters using `and` or `or`. - - - `filters: Array[ComparisonFilter | untyped]` - - Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `UnionMember1 = untyped` - - - `type: :and | :or` - - Type of operation: `and` or `or`. - - - `:and` - - - `:or` - - - `max_num_results: Integer` - - The maximum number of results to return. This number should be between 1 and 50 inclusive. - - - `ranking_options: RankingOptions{ hybrid_search, ranker, score_threshold}` - - Ranking options for search. - - - `hybrid_search: HybridSearch{ embedding_weight, text_weight}` - - Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled. - - - `embedding_weight: Float` - - The weight of the embedding in the reciprocal ranking fusion. - - - `text_weight: Float` - - The weight of the text in the reciprocal ranking fusion. - - - `ranker: :auto | :"default-2024-11-15"` - - The ranker to use for the file search. - - - `:auto` - - - `:"default-2024-11-15"` - - - `score_threshold: Float` - - The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results. - - - `class ComputerTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `type: :computer` - - The type of the computer tool. Always `computer`. - - - `:computer` - - - `class ComputerUsePreviewTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `display_height: Integer` - - The height of the computer display. - - - `display_width: Integer` - - The width of the computer display. - - - `environment: :windows | :mac | :linux | 2 more` - - The type of computer environment to control. - - - `:windows` - - - `:mac` - - - `:linux` - - - `:ubuntu` - - - `:browser` - - - `type: :computer_use_preview` - - The type of the computer use tool. Always `computer_use_preview`. - - - `:computer_use_preview` - - - `class WebSearchTool` - - Search the Internet for sources related to the prompt. Learn more about the - [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search | :web_search_2025_08_26` - - The type of the web search tool. One of `web_search` or `web_search_2025_08_26`. - - - `:web_search` - - - `:web_search_2025_08_26` - - - `filters: Filters{ allowed_domains}` - - Filters for the search. - - - `allowed_domains: Array[String]` - - Allowed domains for the search. If not provided, all domains are allowed. - Subdomains of the provided domains are allowed as well. - - Example: `["pubmed.ncbi.nlm.nih.gov"]` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ city, country, region, 2 more}` - - The approximate location of the user. - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `class Mcp` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](https://platform.openai.com/docs/guides/tools-remote-mcp). - - - `server_label: String` - - A label for this MCP server, used to identify it in tool calls. - - - `type: :mcp` - - The type of the MCP tool. Always `mcp`. - - - `:mcp` - - - `allowed_tools: Array[String] | McpToolFilter{ read_only, tool_names}` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = Array[String]` - - A string array of allowed tool names - - - `class McpToolFilter` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `authorization: String` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: :connector_dropbox | :connector_gmail | :connector_googlecalendar | 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](https://platform.openai.com/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `:connector_dropbox` - - - `:connector_gmail` - - - `:connector_googlecalendar` - - - `:connector_googledrive` - - - `:connector_microsoftteams` - - - `:connector_outlookcalendar` - - - `:connector_outlookemail` - - - `:connector_sharepoint` - - - `defer_loading: bool` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: Hash[Symbol, String]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: McpToolApprovalFilter{ always, never} | :always | :never` - - Specify which of the MCP server's tools require approval. - - - `class McpToolApprovalFilter` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: Always{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `never: Never{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `McpToolApprovalSetting = :always | :never` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `:always` - - - `:never` - - - `server_description: String` - - Optional description of the MCP server, used to provide more context. - - - `server_url: String` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `class CodeInterpreter` - - A tool that runs Python code to help generate a response to a prompt. - - - `container: String | CodeInterpreterToolAuto{ type, file_ids, memory_limit, network_policy}` - - The code interpreter container. Can be a container ID or an object that - specifies uploaded file IDs to make available to your code, along with an - optional `memory_limit` setting. - - - `String = String` - - The container ID. - - - `class CodeInterpreterToolAuto` - - Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on. - - - `type: :auto` - - Always `auto`. - - - `:auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the code interpreter container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `type: :disabled` - - Disable outbound network access. Always `disabled`. - - - `:disabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `allowed_domains: Array[String]` - - A list of allowed domains when type is `allowlist`. - - - `type: :allowlist` - - Allow outbound network access only to specified domains. Always `allowlist`. - - - `:allowlist` - - - `domain_secrets: Array[ContainerNetworkPolicyDomainSecret]` - - Optional domain-scoped secrets for allowlisted domains. - - - `domain: String` - - The domain associated with the secret. - - - `name: String` - - The name of the secret to inject for the domain. - - - `value: String` - - The secret value to inject for the domain. - - - `type: :code_interpreter` - - The type of the code interpreter tool. Always `code_interpreter`. - - - `:code_interpreter` - - - `class ImageGeneration` - - A tool that generates images using the GPT image models. - - - `type: :image_generation` - - The type of the image generation tool. Always `image_generation`. - - - `:image_generation` - - - `action: :generate | :edit | :auto` - - Whether to generate a new image or edit an existing image. Default: `auto`. - - - `:generate` - - - `:edit` - - - `:auto` - - - `background: :transparent | :opaque | :auto` - - Allows to set transparency for the background of the generated image(s). - This parameter is only supported for GPT image models that support - transparent backgrounds. Must be one of `transparent`, `opaque`, or - `auto` (default value). When `auto` is used, the model will - automatically determine the best background for the image. - - `gpt-image-2` and `gpt-image-2-2026-04-21` do not support - transparent backgrounds. Requests with `background` set to - `transparent` will return an error for these models; use `opaque` or - `auto` instead. - - If `transparent`, the output format needs to support transparency, - so it should be set to either `png` (default value) or `webp`. - - - `:transparent` - - - `:opaque` - - - `:auto` - - - `input_fidelity: :high | :low` - - Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`. - - - `:high` - - - `:low` - - - `input_image_mask: InputImageMask{ file_id, image_url}` - - Optional mask for inpainting. Contains `image_url` - (string, optional) and `file_id` (string, optional). - - - `file_id: String` - - File ID for the mask image. - - - `image_url: String` - - Base64-encoded mask image. - - - `model: String | :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `String = String` - - - `Model = :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `:"gpt-image-1"` - - - `:"gpt-image-1-mini"` - - - `:"gpt-image-2"` - - - `:"gpt-image-2-2026-04-21"` - - - `:"gpt-image-1.5"` - - - `:"chatgpt-image-latest"` - - - `moderation: :auto | :low` - - Moderation level for the generated image. Default: `auto`. - - - `:auto` - - - `:low` - - - `output_compression: Integer` - - Compression level for the output image. Default: 100. - - - `output_format: :png | :webp | :jpeg` - - The output format of the generated image. One of `png`, `webp`, or - `jpeg`. Default: `png`. - - - `:png` - - - `:webp` - - - `:jpeg` - - - `partial_images: Integer` - - Number of partial images to generate in streaming mode, from 0 (default value) to 3. - - - `quality: :low | :medium | :high | :auto` - - The quality of the generated image. One of `low`, `medium`, `high`, - or `auto`. Default: `auto`. - - - `:low` - - - `:medium` - - - `:high` - - - `:auto` - - - `size: String | :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `String = String` - - - `Size = :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `:"1024x1024"` - - - `:"1024x1536"` - - - `:"1536x1024"` - - - `:auto` - - - `class LocalShell` - - A tool that allows the model to execute shell commands in a local environment. - - - `type: :local_shell` - - The type of the local shell tool. Always `local_shell`. - - - `:local_shell` - - - `class FunctionShellTool` - - A tool that allows the model to execute shell commands. - - - `type: :shell` - - The type of the shell tool. Always `shell`. - - - `:shell` - - - `environment: ContainerAuto | LocalEnvironment | ContainerReference` - - - `class ContainerAuto` - - - `type: :container_auto` - - Automatically creates a container for this request - - - `:container_auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `skills: Array[SkillReference | InlineSkill]` - - An optional list of skills referenced by id or inline data. - - - `class SkillReference` - - - `skill_id: String` - - The ID of the referenced skill. - - - `type: :skill_reference` - - References a skill created with the /v1/skills endpoint. - - - `:skill_reference` - - - `version: String` - - Optional skill version. Use a positive integer or 'latest'. Omit for default. - - - `class InlineSkill` - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `source: InlineSkillSource` - - Inline skill payload - - - `data: String` - - Base64-encoded skill zip bundle. - - - `media_type: :"application/zip"` - - The media type of the inline skill payload. Must be `application/zip`. - - - `:"application/zip"` - - - `type: :base64` - - The type of the inline skill source. Must be `base64`. - - - `:base64` - - - `type: :inline` - - Defines an inline skill for this request. - - - `:inline` - - - `class LocalEnvironment` - - - `type: :local` - - Use a local computer environment. - - - `:local` - - - `skills: Array[LocalSkill]` - - An optional list of skills. - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `path: String` - - The path to the directory containing the skill. - - - `class ContainerReference` - - - `container_id: String` - - The ID of the referenced container. - - - `type: :container_reference` - - References a container created with the /v1/containers endpoint - - - `:container_reference` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `name: String` - - The name of the custom tool, used to identify it in tool calls. - - - `type: :custom` - - The type of the custom tool. Always `custom`. - - - `:custom` - - - `defer_loading: bool` - - Whether this tool should be deferred and discovered via tool search. - - - `description: String` - - Optional description of the custom tool, used to provide more context. - - - `format_: CustomToolInputFormat` - - The input format for the custom tool. Default is unconstrained text. - - - `class Text` - - Unconstrained free-form text. - - - `type: :text` - - Unconstrained text format. Always `text`. - - - `:text` - - - `class Grammar` - - A grammar defined by the user. - - - `definition: String` - - The grammar definition. - - - `syntax: :lark | :regex` - - The syntax of the grammar definition. One of `lark` or `regex`. - - - `:lark` - - - `:regex` - - - `type: :grammar` - - Grammar format. Always `grammar`. - - - `:grammar` - - - `class NamespaceTool` - - Groups function/custom tools under a shared namespace. - - - `description: String` - - A description of the namespace shown to the model. - - - `name: String` - - The namespace name used in tool calls (for example, `crm`). - - - `tools: Array[Function{ name, type, defer_loading, 3 more} | CustomTool]` - - The function/custom tools available inside this namespace. - - - `class Function` - - - `name: String` - - - `type: :function` - - - `:function` - - - `defer_loading: bool` - - Whether this function should be deferred and discovered via tool search. - - - `description: String` - - - `parameters: untyped` - - - `strict: bool` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `type: :namespace` - - The type of the tool. Always `namespace`. - - - `:namespace` - - - `class ToolSearchTool` - - Hosted or BYOT tool search configuration for deferred tools. - - - `type: :tool_search` - - The type of the tool. Always `tool_search`. - - - `:tool_search` - - - `description: String` - - Description shown to the model for a client-executed tool search tool. - - - `execution: :server | :client` - - Whether tool search is executed by the server or by the client. - - - `:server` - - - `:client` - - - `parameters: untyped` - - Parameter schema for a client-executed tool search tool. - - - `class WebSearchPreviewTool` - - This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search_preview | :web_search_preview_2025_03_11` - - The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`. - - - `:web_search_preview` - - - `:web_search_preview_2025_03_11` - - - `search_content_types: Array[:text | :image]` - - - `:text` - - - `:image` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ type, city, country, 2 more}` - - The user's location. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `class ApplyPatchTool` - - Allows the assistant to create, delete, or update files using unified diffs. - - - `type: :apply_patch` - - The type of the tool. Always `apply_patch`. - - - `:apply_patch` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `error: EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - - - `eval_id: String` - - The identifier of the associated evaluation. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - The model that is evaluated, if applicable. - - - `name: String` - - The name of the evaluation run. - - - `object: :"eval.run"` - - The type of the object. Always "eval.run". - - - `:"eval.run"` - - - `per_model_usage: Array[PerModelUsage{ cached_tokens, completion_tokens, invocation_count, 3 more}]` - - Usage statistics for each model during the evaluation run. - - - `cached_tokens: Integer` - - The number of tokens retrieved from cache. - - - `completion_tokens: Integer` - - The number of completion tokens generated. - - - `invocation_count: Integer` - - The number of invocations. - - - `model_name: String` - - The name of the model. - - - `prompt_tokens: Integer` - - The number of prompt tokens used. - - - `total_tokens: Integer` - - The total number of tokens used. - - - `per_testing_criteria_results: Array[PerTestingCriteriaResult{ failed, passed, testing_criteria}]` - - Results per testing criteria applied during the evaluation run. - - - `failed: Integer` - - Number of tests failed for this criteria. - - - `passed: Integer` - - Number of tests passed for this criteria. - - - `testing_criteria: String` - - A description of the testing criteria. - - - `report_url: String` - - The URL to the rendered evaluation run report on the UI dashboard. - - - `result_counts: ResultCounts{ errored, failed, passed, total}` - - Counters summarizing the outcomes of the evaluation run. - - - `errored: Integer` - - Number of output items that resulted in an error. - - - `failed: Integer` - - Number of output items that failed to pass the evaluation. - - - `passed: Integer` - - Number of output items that passed the evaluation. - - - `total: Integer` - - Total number of executed output items. - - - `status: String` - - The status of the evaluation run. - -### Run Create Response - -- `class RunCreateResponse` - - A schema representing an evaluation run. - - - `id: String` - - Unique identifier for the evaluation run. - - - `created_at: Integer` - - Unix timestamp (in seconds) when the evaluation run was created. - - - `data_source: CreateEvalJSONLRunDataSource | CreateEvalCompletionsRunDataSource | Responses{ source, type, input_messages, 2 more}` - - Information about the run's data source. - - - `class CreateEvalJSONLRunDataSource` - - A JsonlRunDataSource object with that specifies a JSONL file that matches the eval - - - `source: FileContent{ content, type} | FileID{ id, type}` - - Determines what populates the `item` namespace in the data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `type: :jsonl` - - The type of data source. Always `jsonl`. - - - `:jsonl` - - - `class CreateEvalCompletionsRunDataSource` - - A CompletionsRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | StoredCompletions{ type, created_after, created_before, 3 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class StoredCompletions` - - A StoredCompletionsRunDataSource configuration describing a set of filters - - - `type: :stored_completions` - - The type of source. Always `stored_completions`. - - - `:stored_completions` - - - `created_after: Integer` - - An optional Unix timestamp to filter items created after this time. - - - `created_before: Integer` - - An optional Unix timestamp to filter items created before this time. - - - `limit: Integer` - - An optional maximum number of items to return. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - An optional model to filter by (e.g., 'gpt-4o'). - - - `type: :completions` - - The type of run data source. Always `completions`. - - - `:completions` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[EasyInputMessage | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class EasyInputMessage` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputMessageContentList` - - Text, image, or audio input to the model, used to generate a response. - Can also contain previous assistant responses. - - - `String = String` - - A text input to the model. - - - `ResponseInputMessageContentList = Array[ResponseInputContent]` - - A list of one or many input items to the model, containing different content - types. - - - `class ResponseInputText` - - A text input to the model. - - - `text: String` - - The text input to the model. - - - `type: :input_text` - - The type of the input item. Always `input_text`. - - - `:input_text` - - - `class ResponseInputImage` - - An image input to the model. Learn about [image inputs](https://platform.openai.com/docs/guides/vision). - - - `detail: :low | :high | :auto | :original` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `:low` - - - `:high` - - - `:auto` - - - `:original` - - - `type: :input_image` - - The type of the input item. Always `input_image`. - - - `:input_image` - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `image_url: String` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `class ResponseInputFile` - - A file input to the model. - - - `type: :input_file` - - The type of the input item. Always `input_file`. - - - `:input_file` - - - `detail: :low | :high` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `:low` - - - `:high` - - - `file_data: String` - - The content of the file to be sent to the model. - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `file_url: String` - - The URL of the file to be sent to the model. - - - `filename: String` - - The name of the file to be sent to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `phase: :commentary | :final_answer` - - Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`). - For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend - phase on all assistant messages — dropping it can degrade performance. Not used for user messages. - - - `:commentary` - - - `:final_answer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `input_audio: InputAudio{ data, format_}` - - - `data: String` - - Base64-encoded audio data. - - - `format_: :mp3 | :wav` - - The format of the audio data. Currently supported formats are `mp3` and - `wav`. - - - `:mp3` - - - `:wav` - - - `type: :input_audio` - - The type of the input item. Always `input_audio`. - - - `:input_audio` - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.input_trajectory" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, response_format, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `:none` - - - `:minimal` - - - `:low` - - - `:medium` - - - `:high` - - - `:xhigh` - - - `response_format: ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject` - - An object specifying the format that the model must output. - - Setting to `{ "type": "json_schema", "json_schema": {...} }` enables - Structured Outputs which ensures the model will match your supplied JSON - schema. Learn more in the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `type: :text` - - The type of response format being defined. Always `text`. - - - `:text` - - - `class ResponseFormatJSONSchema` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `json_schema: JSONSchema{ name, description, schema, strict}` - - Structured Outputs configuration options, including a JSON Schema. - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `type: :json_object` - - The type of response format being defined. Always `json_object`. - - - `:json_object` - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `tools: Array[ChatCompletionFunctionTool]` - - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported. - - - `function: FunctionDefinition` - - - `name: String` - - The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the function does, used by the model to choose when and how to call the function. - - - `parameters: FunctionParameters` - - The parameters the functions accepts, described as a JSON Schema object. See the [guide](https://platform.openai.com/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format. - - Omitting `parameters` defines a function with an empty parameter list. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](https://platform.openai.com/docs/guides/function-calling). - - - `type: :function` - - The type of the tool. Currently, only `function` is supported. - - - `:function` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `class Responses` - - A ResponsesRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | Responses{ type, created_after, created_before, 8 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class Responses` - - A EvalResponsesSource object describing a run data source configuration. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `created_after: Integer` - - Only include items created after this timestamp (inclusive). This is a query parameter used to select responses. - - - `created_before: Integer` - - Only include items created before this timestamp (inclusive). This is a query parameter used to select responses. - - - `instructions_search: String` - - Optional string to search the 'instructions' field. This is a query parameter used to select responses. - - - `metadata: untyped` - - Metadata filter for the responses. This is a query parameter used to select responses. - - - `model: String` - - The name of the model to find responses for. This is a query parameter used to select responses. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `temperature: Float` - - Sampling temperature. This is a query parameter used to select responses. - - - `tools: Array[String]` - - List of tool names. This is a query parameter used to select responses. - - - `top_p: Float` - - Nucleus sampling parameter. This is a query parameter used to select responses. - - - `users: Array[String]` - - List of user identifiers. This is a query parameter used to select responses. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[ChatMessage{ content, role} | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class ChatMessage` - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.name" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, seed, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `text: Text{ format_}` - - Configuration options for a text response from the model. Can be plain - text or structured JSON data. Learn more: - - - [Text inputs and outputs](https://platform.openai.com/docs/guides/text) - - [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) - - - `format_: ResponseFormatTextConfig` - - An object specifying the format that the model must output. - - Configuring `{ "type": "json_schema" }` enables Structured Outputs, - which ensures the model will match your supplied JSON schema. Learn more in the - [Structured Outputs guide](https://platform.openai.com/docs/guides/structured-outputs). - - The default format is `{ "type": "text" }` with no additional options. - - **Not recommended for gpt-4o and newer models:** - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `class ResponseFormatTextJSONSchemaConfig` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `tools: Array[Tool]` - - An array of tools the model may call while generating a response. You - can specify which tool to use by setting the `tool_choice` parameter. - - The two categories of tools you can provide the model are: - - - **Built-in tools**: Tools that are provided by OpenAI that extend the - model's capabilities, like [web search](https://platform.openai.com/docs/guides/tools-web-search) - or [file search](https://platform.openai.com/docs/guides/tools-file-search). Learn more about - [built-in tools](https://platform.openai.com/docs/guides/tools). - - **Function calls (custom tools)**: Functions that are defined by you, - enabling the model to call your own code. Learn more about - [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `class FunctionTool` - - Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `name: String` - - The name of the function to call. - - - `parameters: Hash[Symbol, untyped]` - - A JSON schema object describing the parameters of the function. - - - `strict: bool` - - Whether to enforce strict parameter validation. Default `true`. - - - `type: :function` - - The type of the function tool. Always `function`. - - - `:function` - - - `defer_loading: bool` - - Whether this function is deferred and loaded via tool search. - - - `description: String` - - A description of the function. Used by the model to determine whether or not to call the function. - - - `class FileSearchTool` - - A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search). - - - `type: :file_search` - - The type of the file search tool. Always `file_search`. - - - `:file_search` - - - `vector_store_ids: Array[String]` - - The IDs of the vector stores to search. - - - `filters: ComparisonFilter | CompoundFilter` - - A filter to apply. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `key: String` - - The key to compare against the value. - - - `type: :eq | :ne | :gt | 5 more` - - Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`. - - - `eq`: equals - - `ne`: not equal - - `gt`: greater than - - `gte`: greater than or equal - - `lt`: less than - - `lte`: less than or equal - - `in`: in - - `nin`: not in - - - `:eq` - - - `:ne` - - - `:gt` - - - `:gte` - - - `:lt` - - - `:lte` - - - `:in` - - - `:nin` - - - `value: String | Float | bool | Array[String | Float]` - - The value to compare against the attribute key; supports string, number, or boolean types. - - - `String = String` - - - `Float = Float` - - - `UnionMember2 = bool` - - - `UnionMember3 = Array[String | Float]` - - - `String = String` - - - `Float = Float` - - - `class CompoundFilter` - - Combine multiple filters using `and` or `or`. - - - `filters: Array[ComparisonFilter | untyped]` - - Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `UnionMember1 = untyped` - - - `type: :and | :or` - - Type of operation: `and` or `or`. - - - `:and` - - - `:or` - - - `max_num_results: Integer` - - The maximum number of results to return. This number should be between 1 and 50 inclusive. - - - `ranking_options: RankingOptions{ hybrid_search, ranker, score_threshold}` - - Ranking options for search. - - - `hybrid_search: HybridSearch{ embedding_weight, text_weight}` - - Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled. - - - `embedding_weight: Float` - - The weight of the embedding in the reciprocal ranking fusion. - - - `text_weight: Float` - - The weight of the text in the reciprocal ranking fusion. - - - `ranker: :auto | :"default-2024-11-15"` - - The ranker to use for the file search. - - - `:auto` - - - `:"default-2024-11-15"` - - - `score_threshold: Float` - - The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results. - - - `class ComputerTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `type: :computer` - - The type of the computer tool. Always `computer`. - - - `:computer` - - - `class ComputerUsePreviewTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `display_height: Integer` - - The height of the computer display. - - - `display_width: Integer` - - The width of the computer display. - - - `environment: :windows | :mac | :linux | 2 more` - - The type of computer environment to control. - - - `:windows` - - - `:mac` - - - `:linux` - - - `:ubuntu` - - - `:browser` - - - `type: :computer_use_preview` - - The type of the computer use tool. Always `computer_use_preview`. - - - `:computer_use_preview` - - - `class WebSearchTool` - - Search the Internet for sources related to the prompt. Learn more about the - [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search | :web_search_2025_08_26` - - The type of the web search tool. One of `web_search` or `web_search_2025_08_26`. - - - `:web_search` - - - `:web_search_2025_08_26` - - - `filters: Filters{ allowed_domains}` - - Filters for the search. - - - `allowed_domains: Array[String]` - - Allowed domains for the search. If not provided, all domains are allowed. - Subdomains of the provided domains are allowed as well. - - Example: `["pubmed.ncbi.nlm.nih.gov"]` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ city, country, region, 2 more}` - - The approximate location of the user. - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `class Mcp` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](https://platform.openai.com/docs/guides/tools-remote-mcp). - - - `server_label: String` - - A label for this MCP server, used to identify it in tool calls. - - - `type: :mcp` - - The type of the MCP tool. Always `mcp`. - - - `:mcp` - - - `allowed_tools: Array[String] | McpToolFilter{ read_only, tool_names}` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = Array[String]` - - A string array of allowed tool names - - - `class McpToolFilter` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `authorization: String` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: :connector_dropbox | :connector_gmail | :connector_googlecalendar | 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](https://platform.openai.com/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `:connector_dropbox` - - - `:connector_gmail` - - - `:connector_googlecalendar` - - - `:connector_googledrive` - - - `:connector_microsoftteams` - - - `:connector_outlookcalendar` - - - `:connector_outlookemail` - - - `:connector_sharepoint` - - - `defer_loading: bool` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: Hash[Symbol, String]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: McpToolApprovalFilter{ always, never} | :always | :never` - - Specify which of the MCP server's tools require approval. - - - `class McpToolApprovalFilter` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: Always{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `never: Never{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `McpToolApprovalSetting = :always | :never` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `:always` - - - `:never` - - - `server_description: String` - - Optional description of the MCP server, used to provide more context. - - - `server_url: String` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `class CodeInterpreter` - - A tool that runs Python code to help generate a response to a prompt. - - - `container: String | CodeInterpreterToolAuto{ type, file_ids, memory_limit, network_policy}` - - The code interpreter container. Can be a container ID or an object that - specifies uploaded file IDs to make available to your code, along with an - optional `memory_limit` setting. - - - `String = String` - - The container ID. - - - `class CodeInterpreterToolAuto` - - Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on. - - - `type: :auto` - - Always `auto`. - - - `:auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the code interpreter container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `type: :disabled` - - Disable outbound network access. Always `disabled`. - - - `:disabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `allowed_domains: Array[String]` - - A list of allowed domains when type is `allowlist`. - - - `type: :allowlist` - - Allow outbound network access only to specified domains. Always `allowlist`. - - - `:allowlist` - - - `domain_secrets: Array[ContainerNetworkPolicyDomainSecret]` - - Optional domain-scoped secrets for allowlisted domains. - - - `domain: String` - - The domain associated with the secret. - - - `name: String` - - The name of the secret to inject for the domain. - - - `value: String` - - The secret value to inject for the domain. - - - `type: :code_interpreter` - - The type of the code interpreter tool. Always `code_interpreter`. - - - `:code_interpreter` - - - `class ImageGeneration` - - A tool that generates images using the GPT image models. - - - `type: :image_generation` - - The type of the image generation tool. Always `image_generation`. - - - `:image_generation` - - - `action: :generate | :edit | :auto` - - Whether to generate a new image or edit an existing image. Default: `auto`. - - - `:generate` - - - `:edit` - - - `:auto` - - - `background: :transparent | :opaque | :auto` - - Allows to set transparency for the background of the generated image(s). - This parameter is only supported for GPT image models that support - transparent backgrounds. Must be one of `transparent`, `opaque`, or - `auto` (default value). When `auto` is used, the model will - automatically determine the best background for the image. - - `gpt-image-2` and `gpt-image-2-2026-04-21` do not support - transparent backgrounds. Requests with `background` set to - `transparent` will return an error for these models; use `opaque` or - `auto` instead. - - If `transparent`, the output format needs to support transparency, - so it should be set to either `png` (default value) or `webp`. - - - `:transparent` - - - `:opaque` - - - `:auto` - - - `input_fidelity: :high | :low` - - Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`. - - - `:high` - - - `:low` - - - `input_image_mask: InputImageMask{ file_id, image_url}` - - Optional mask for inpainting. Contains `image_url` - (string, optional) and `file_id` (string, optional). - - - `file_id: String` - - File ID for the mask image. - - - `image_url: String` - - Base64-encoded mask image. - - - `model: String | :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `String = String` - - - `Model = :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `:"gpt-image-1"` - - - `:"gpt-image-1-mini"` - - - `:"gpt-image-2"` - - - `:"gpt-image-2-2026-04-21"` - - - `:"gpt-image-1.5"` - - - `:"chatgpt-image-latest"` - - - `moderation: :auto | :low` - - Moderation level for the generated image. Default: `auto`. - - - `:auto` - - - `:low` - - - `output_compression: Integer` - - Compression level for the output image. Default: 100. - - - `output_format: :png | :webp | :jpeg` - - The output format of the generated image. One of `png`, `webp`, or - `jpeg`. Default: `png`. - - - `:png` - - - `:webp` - - - `:jpeg` - - - `partial_images: Integer` - - Number of partial images to generate in streaming mode, from 0 (default value) to 3. - - - `quality: :low | :medium | :high | :auto` - - The quality of the generated image. One of `low`, `medium`, `high`, - or `auto`. Default: `auto`. - - - `:low` - - - `:medium` - - - `:high` - - - `:auto` - - - `size: String | :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `String = String` - - - `Size = :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `:"1024x1024"` - - - `:"1024x1536"` - - - `:"1536x1024"` - - - `:auto` - - - `class LocalShell` - - A tool that allows the model to execute shell commands in a local environment. - - - `type: :local_shell` - - The type of the local shell tool. Always `local_shell`. - - - `:local_shell` - - - `class FunctionShellTool` - - A tool that allows the model to execute shell commands. - - - `type: :shell` - - The type of the shell tool. Always `shell`. - - - `:shell` - - - `environment: ContainerAuto | LocalEnvironment | ContainerReference` - - - `class ContainerAuto` - - - `type: :container_auto` - - Automatically creates a container for this request - - - `:container_auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `skills: Array[SkillReference | InlineSkill]` - - An optional list of skills referenced by id or inline data. - - - `class SkillReference` - - - `skill_id: String` - - The ID of the referenced skill. - - - `type: :skill_reference` - - References a skill created with the /v1/skills endpoint. - - - `:skill_reference` - - - `version: String` - - Optional skill version. Use a positive integer or 'latest'. Omit for default. - - - `class InlineSkill` - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `source: InlineSkillSource` - - Inline skill payload - - - `data: String` - - Base64-encoded skill zip bundle. - - - `media_type: :"application/zip"` - - The media type of the inline skill payload. Must be `application/zip`. - - - `:"application/zip"` - - - `type: :base64` - - The type of the inline skill source. Must be `base64`. - - - `:base64` - - - `type: :inline` - - Defines an inline skill for this request. - - - `:inline` - - - `class LocalEnvironment` - - - `type: :local` - - Use a local computer environment. - - - `:local` - - - `skills: Array[LocalSkill]` - - An optional list of skills. - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `path: String` - - The path to the directory containing the skill. - - - `class ContainerReference` - - - `container_id: String` - - The ID of the referenced container. - - - `type: :container_reference` - - References a container created with the /v1/containers endpoint - - - `:container_reference` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `name: String` - - The name of the custom tool, used to identify it in tool calls. - - - `type: :custom` - - The type of the custom tool. Always `custom`. - - - `:custom` - - - `defer_loading: bool` - - Whether this tool should be deferred and discovered via tool search. - - - `description: String` - - Optional description of the custom tool, used to provide more context. - - - `format_: CustomToolInputFormat` - - The input format for the custom tool. Default is unconstrained text. - - - `class Text` - - Unconstrained free-form text. - - - `type: :text` - - Unconstrained text format. Always `text`. - - - `:text` - - - `class Grammar` - - A grammar defined by the user. - - - `definition: String` - - The grammar definition. - - - `syntax: :lark | :regex` - - The syntax of the grammar definition. One of `lark` or `regex`. - - - `:lark` - - - `:regex` - - - `type: :grammar` - - Grammar format. Always `grammar`. - - - `:grammar` - - - `class NamespaceTool` - - Groups function/custom tools under a shared namespace. - - - `description: String` - - A description of the namespace shown to the model. - - - `name: String` - - The namespace name used in tool calls (for example, `crm`). - - - `tools: Array[Function{ name, type, defer_loading, 3 more} | CustomTool]` - - The function/custom tools available inside this namespace. - - - `class Function` - - - `name: String` - - - `type: :function` - - - `:function` - - - `defer_loading: bool` - - Whether this function should be deferred and discovered via tool search. - - - `description: String` - - - `parameters: untyped` - - - `strict: bool` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `type: :namespace` - - The type of the tool. Always `namespace`. - - - `:namespace` - - - `class ToolSearchTool` - - Hosted or BYOT tool search configuration for deferred tools. - - - `type: :tool_search` - - The type of the tool. Always `tool_search`. - - - `:tool_search` - - - `description: String` - - Description shown to the model for a client-executed tool search tool. - - - `execution: :server | :client` - - Whether tool search is executed by the server or by the client. - - - `:server` - - - `:client` - - - `parameters: untyped` - - Parameter schema for a client-executed tool search tool. - - - `class WebSearchPreviewTool` - - This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search_preview | :web_search_preview_2025_03_11` - - The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`. - - - `:web_search_preview` - - - `:web_search_preview_2025_03_11` - - - `search_content_types: Array[:text | :image]` - - - `:text` - - - `:image` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ type, city, country, 2 more}` - - The user's location. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `class ApplyPatchTool` - - Allows the assistant to create, delete, or update files using unified diffs. - - - `type: :apply_patch` - - The type of the tool. Always `apply_patch`. - - - `:apply_patch` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `error: EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - - - `eval_id: String` - - The identifier of the associated evaluation. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - The model that is evaluated, if applicable. - - - `name: String` - - The name of the evaluation run. - - - `object: :"eval.run"` - - The type of the object. Always "eval.run". - - - `:"eval.run"` - - - `per_model_usage: Array[PerModelUsage{ cached_tokens, completion_tokens, invocation_count, 3 more}]` - - Usage statistics for each model during the evaluation run. - - - `cached_tokens: Integer` - - The number of tokens retrieved from cache. - - - `completion_tokens: Integer` - - The number of completion tokens generated. - - - `invocation_count: Integer` - - The number of invocations. - - - `model_name: String` - - The name of the model. - - - `prompt_tokens: Integer` - - The number of prompt tokens used. - - - `total_tokens: Integer` - - The total number of tokens used. - - - `per_testing_criteria_results: Array[PerTestingCriteriaResult{ failed, passed, testing_criteria}]` - - Results per testing criteria applied during the evaluation run. - - - `failed: Integer` - - Number of tests failed for this criteria. - - - `passed: Integer` - - Number of tests passed for this criteria. - - - `testing_criteria: String` - - A description of the testing criteria. - - - `report_url: String` - - The URL to the rendered evaluation run report on the UI dashboard. - - - `result_counts: ResultCounts{ errored, failed, passed, total}` - - Counters summarizing the outcomes of the evaluation run. - - - `errored: Integer` - - Number of output items that resulted in an error. - - - `failed: Integer` - - Number of output items that failed to pass the evaluation. - - - `passed: Integer` - - Number of output items that passed the evaluation. - - - `total: Integer` - - Total number of executed output items. - - - `status: String` - - The status of the evaluation run. - -### Run Retrieve Response - -- `class RunRetrieveResponse` - - A schema representing an evaluation run. - - - `id: String` - - Unique identifier for the evaluation run. - - - `created_at: Integer` - - Unix timestamp (in seconds) when the evaluation run was created. - - - `data_source: CreateEvalJSONLRunDataSource | CreateEvalCompletionsRunDataSource | Responses{ source, type, input_messages, 2 more}` - - Information about the run's data source. - - - `class CreateEvalJSONLRunDataSource` - - A JsonlRunDataSource object with that specifies a JSONL file that matches the eval - - - `source: FileContent{ content, type} | FileID{ id, type}` - - Determines what populates the `item` namespace in the data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `type: :jsonl` - - The type of data source. Always `jsonl`. - - - `:jsonl` - - - `class CreateEvalCompletionsRunDataSource` - - A CompletionsRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | StoredCompletions{ type, created_after, created_before, 3 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class StoredCompletions` - - A StoredCompletionsRunDataSource configuration describing a set of filters - - - `type: :stored_completions` - - The type of source. Always `stored_completions`. - - - `:stored_completions` - - - `created_after: Integer` - - An optional Unix timestamp to filter items created after this time. - - - `created_before: Integer` - - An optional Unix timestamp to filter items created before this time. - - - `limit: Integer` - - An optional maximum number of items to return. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - An optional model to filter by (e.g., 'gpt-4o'). - - - `type: :completions` - - The type of run data source. Always `completions`. - - - `:completions` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[EasyInputMessage | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class EasyInputMessage` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputMessageContentList` - - Text, image, or audio input to the model, used to generate a response. - Can also contain previous assistant responses. - - - `String = String` - - A text input to the model. - - - `ResponseInputMessageContentList = Array[ResponseInputContent]` - - A list of one or many input items to the model, containing different content - types. - - - `class ResponseInputText` - - A text input to the model. - - - `text: String` - - The text input to the model. - - - `type: :input_text` - - The type of the input item. Always `input_text`. - - - `:input_text` - - - `class ResponseInputImage` - - An image input to the model. Learn about [image inputs](https://platform.openai.com/docs/guides/vision). - - - `detail: :low | :high | :auto | :original` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `:low` - - - `:high` - - - `:auto` - - - `:original` - - - `type: :input_image` - - The type of the input item. Always `input_image`. - - - `:input_image` - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `image_url: String` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `class ResponseInputFile` - - A file input to the model. - - - `type: :input_file` - - The type of the input item. Always `input_file`. - - - `:input_file` - - - `detail: :low | :high` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `:low` - - - `:high` - - - `file_data: String` - - The content of the file to be sent to the model. - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `file_url: String` - - The URL of the file to be sent to the model. - - - `filename: String` - - The name of the file to be sent to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `phase: :commentary | :final_answer` - - Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`). - For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend - phase on all assistant messages — dropping it can degrade performance. Not used for user messages. - - - `:commentary` - - - `:final_answer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `input_audio: InputAudio{ data, format_}` - - - `data: String` - - Base64-encoded audio data. - - - `format_: :mp3 | :wav` - - The format of the audio data. Currently supported formats are `mp3` and - `wav`. - - - `:mp3` - - - `:wav` - - - `type: :input_audio` - - The type of the input item. Always `input_audio`. - - - `:input_audio` - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.input_trajectory" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, response_format, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `:none` - - - `:minimal` - - - `:low` - - - `:medium` - - - `:high` - - - `:xhigh` - - - `response_format: ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject` - - An object specifying the format that the model must output. - - Setting to `{ "type": "json_schema", "json_schema": {...} }` enables - Structured Outputs which ensures the model will match your supplied JSON - schema. Learn more in the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `type: :text` - - The type of response format being defined. Always `text`. - - - `:text` - - - `class ResponseFormatJSONSchema` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `json_schema: JSONSchema{ name, description, schema, strict}` - - Structured Outputs configuration options, including a JSON Schema. - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `type: :json_object` - - The type of response format being defined. Always `json_object`. - - - `:json_object` - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `tools: Array[ChatCompletionFunctionTool]` - - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported. - - - `function: FunctionDefinition` - - - `name: String` - - The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the function does, used by the model to choose when and how to call the function. - - - `parameters: FunctionParameters` - - The parameters the functions accepts, described as a JSON Schema object. See the [guide](https://platform.openai.com/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format. - - Omitting `parameters` defines a function with an empty parameter list. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](https://platform.openai.com/docs/guides/function-calling). - - - `type: :function` - - The type of the tool. Currently, only `function` is supported. - - - `:function` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `class Responses` - - A ResponsesRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | Responses{ type, created_after, created_before, 8 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class Responses` - - A EvalResponsesSource object describing a run data source configuration. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `created_after: Integer` - - Only include items created after this timestamp (inclusive). This is a query parameter used to select responses. - - - `created_before: Integer` - - Only include items created before this timestamp (inclusive). This is a query parameter used to select responses. - - - `instructions_search: String` - - Optional string to search the 'instructions' field. This is a query parameter used to select responses. - - - `metadata: untyped` - - Metadata filter for the responses. This is a query parameter used to select responses. - - - `model: String` - - The name of the model to find responses for. This is a query parameter used to select responses. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `temperature: Float` - - Sampling temperature. This is a query parameter used to select responses. - - - `tools: Array[String]` - - List of tool names. This is a query parameter used to select responses. - - - `top_p: Float` - - Nucleus sampling parameter. This is a query parameter used to select responses. - - - `users: Array[String]` - - List of user identifiers. This is a query parameter used to select responses. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[ChatMessage{ content, role} | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class ChatMessage` - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.name" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, seed, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `text: Text{ format_}` - - Configuration options for a text response from the model. Can be plain - text or structured JSON data. Learn more: - - - [Text inputs and outputs](https://platform.openai.com/docs/guides/text) - - [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) - - - `format_: ResponseFormatTextConfig` - - An object specifying the format that the model must output. - - Configuring `{ "type": "json_schema" }` enables Structured Outputs, - which ensures the model will match your supplied JSON schema. Learn more in the - [Structured Outputs guide](https://platform.openai.com/docs/guides/structured-outputs). - - The default format is `{ "type": "text" }` with no additional options. - - **Not recommended for gpt-4o and newer models:** - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `class ResponseFormatTextJSONSchemaConfig` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `tools: Array[Tool]` - - An array of tools the model may call while generating a response. You - can specify which tool to use by setting the `tool_choice` parameter. - - The two categories of tools you can provide the model are: - - - **Built-in tools**: Tools that are provided by OpenAI that extend the - model's capabilities, like [web search](https://platform.openai.com/docs/guides/tools-web-search) - or [file search](https://platform.openai.com/docs/guides/tools-file-search). Learn more about - [built-in tools](https://platform.openai.com/docs/guides/tools). - - **Function calls (custom tools)**: Functions that are defined by you, - enabling the model to call your own code. Learn more about - [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `class FunctionTool` - - Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `name: String` - - The name of the function to call. - - - `parameters: Hash[Symbol, untyped]` - - A JSON schema object describing the parameters of the function. - - - `strict: bool` - - Whether to enforce strict parameter validation. Default `true`. - - - `type: :function` - - The type of the function tool. Always `function`. - - - `:function` - - - `defer_loading: bool` - - Whether this function is deferred and loaded via tool search. - - - `description: String` - - A description of the function. Used by the model to determine whether or not to call the function. - - - `class FileSearchTool` - - A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search). - - - `type: :file_search` - - The type of the file search tool. Always `file_search`. - - - `:file_search` - - - `vector_store_ids: Array[String]` - - The IDs of the vector stores to search. - - - `filters: ComparisonFilter | CompoundFilter` - - A filter to apply. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `key: String` - - The key to compare against the value. - - - `type: :eq | :ne | :gt | 5 more` - - Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`. - - - `eq`: equals - - `ne`: not equal - - `gt`: greater than - - `gte`: greater than or equal - - `lt`: less than - - `lte`: less than or equal - - `in`: in - - `nin`: not in - - - `:eq` - - - `:ne` - - - `:gt` - - - `:gte` - - - `:lt` - - - `:lte` - - - `:in` - - - `:nin` - - - `value: String | Float | bool | Array[String | Float]` - - The value to compare against the attribute key; supports string, number, or boolean types. - - - `String = String` - - - `Float = Float` - - - `UnionMember2 = bool` - - - `UnionMember3 = Array[String | Float]` - - - `String = String` - - - `Float = Float` - - - `class CompoundFilter` - - Combine multiple filters using `and` or `or`. - - - `filters: Array[ComparisonFilter | untyped]` - - Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `UnionMember1 = untyped` - - - `type: :and | :or` - - Type of operation: `and` or `or`. - - - `:and` - - - `:or` - - - `max_num_results: Integer` - - The maximum number of results to return. This number should be between 1 and 50 inclusive. - - - `ranking_options: RankingOptions{ hybrid_search, ranker, score_threshold}` - - Ranking options for search. - - - `hybrid_search: HybridSearch{ embedding_weight, text_weight}` - - Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled. - - - `embedding_weight: Float` - - The weight of the embedding in the reciprocal ranking fusion. - - - `text_weight: Float` - - The weight of the text in the reciprocal ranking fusion. - - - `ranker: :auto | :"default-2024-11-15"` - - The ranker to use for the file search. - - - `:auto` - - - `:"default-2024-11-15"` - - - `score_threshold: Float` - - The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results. - - - `class ComputerTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `type: :computer` - - The type of the computer tool. Always `computer`. - - - `:computer` - - - `class ComputerUsePreviewTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `display_height: Integer` - - The height of the computer display. - - - `display_width: Integer` - - The width of the computer display. - - - `environment: :windows | :mac | :linux | 2 more` - - The type of computer environment to control. - - - `:windows` - - - `:mac` - - - `:linux` - - - `:ubuntu` - - - `:browser` - - - `type: :computer_use_preview` - - The type of the computer use tool. Always `computer_use_preview`. - - - `:computer_use_preview` - - - `class WebSearchTool` - - Search the Internet for sources related to the prompt. Learn more about the - [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search | :web_search_2025_08_26` - - The type of the web search tool. One of `web_search` or `web_search_2025_08_26`. - - - `:web_search` - - - `:web_search_2025_08_26` - - - `filters: Filters{ allowed_domains}` - - Filters for the search. - - - `allowed_domains: Array[String]` - - Allowed domains for the search. If not provided, all domains are allowed. - Subdomains of the provided domains are allowed as well. - - Example: `["pubmed.ncbi.nlm.nih.gov"]` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ city, country, region, 2 more}` - - The approximate location of the user. - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `class Mcp` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](https://platform.openai.com/docs/guides/tools-remote-mcp). - - - `server_label: String` - - A label for this MCP server, used to identify it in tool calls. - - - `type: :mcp` - - The type of the MCP tool. Always `mcp`. - - - `:mcp` - - - `allowed_tools: Array[String] | McpToolFilter{ read_only, tool_names}` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = Array[String]` - - A string array of allowed tool names - - - `class McpToolFilter` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `authorization: String` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: :connector_dropbox | :connector_gmail | :connector_googlecalendar | 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](https://platform.openai.com/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `:connector_dropbox` - - - `:connector_gmail` - - - `:connector_googlecalendar` - - - `:connector_googledrive` - - - `:connector_microsoftteams` - - - `:connector_outlookcalendar` - - - `:connector_outlookemail` - - - `:connector_sharepoint` - - - `defer_loading: bool` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: Hash[Symbol, String]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: McpToolApprovalFilter{ always, never} | :always | :never` - - Specify which of the MCP server's tools require approval. - - - `class McpToolApprovalFilter` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: Always{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `never: Never{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `McpToolApprovalSetting = :always | :never` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `:always` - - - `:never` - - - `server_description: String` - - Optional description of the MCP server, used to provide more context. - - - `server_url: String` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `class CodeInterpreter` - - A tool that runs Python code to help generate a response to a prompt. - - - `container: String | CodeInterpreterToolAuto{ type, file_ids, memory_limit, network_policy}` - - The code interpreter container. Can be a container ID or an object that - specifies uploaded file IDs to make available to your code, along with an - optional `memory_limit` setting. - - - `String = String` - - The container ID. - - - `class CodeInterpreterToolAuto` - - Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on. - - - `type: :auto` - - Always `auto`. - - - `:auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the code interpreter container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `type: :disabled` - - Disable outbound network access. Always `disabled`. - - - `:disabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `allowed_domains: Array[String]` - - A list of allowed domains when type is `allowlist`. - - - `type: :allowlist` - - Allow outbound network access only to specified domains. Always `allowlist`. - - - `:allowlist` - - - `domain_secrets: Array[ContainerNetworkPolicyDomainSecret]` - - Optional domain-scoped secrets for allowlisted domains. - - - `domain: String` - - The domain associated with the secret. - - - `name: String` - - The name of the secret to inject for the domain. - - - `value: String` - - The secret value to inject for the domain. - - - `type: :code_interpreter` - - The type of the code interpreter tool. Always `code_interpreter`. - - - `:code_interpreter` - - - `class ImageGeneration` - - A tool that generates images using the GPT image models. - - - `type: :image_generation` - - The type of the image generation tool. Always `image_generation`. - - - `:image_generation` - - - `action: :generate | :edit | :auto` - - Whether to generate a new image or edit an existing image. Default: `auto`. - - - `:generate` - - - `:edit` - - - `:auto` - - - `background: :transparent | :opaque | :auto` - - Allows to set transparency for the background of the generated image(s). - This parameter is only supported for GPT image models that support - transparent backgrounds. Must be one of `transparent`, `opaque`, or - `auto` (default value). When `auto` is used, the model will - automatically determine the best background for the image. - - `gpt-image-2` and `gpt-image-2-2026-04-21` do not support - transparent backgrounds. Requests with `background` set to - `transparent` will return an error for these models; use `opaque` or - `auto` instead. - - If `transparent`, the output format needs to support transparency, - so it should be set to either `png` (default value) or `webp`. - - - `:transparent` - - - `:opaque` - - - `:auto` - - - `input_fidelity: :high | :low` - - Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`. - - - `:high` - - - `:low` - - - `input_image_mask: InputImageMask{ file_id, image_url}` - - Optional mask for inpainting. Contains `image_url` - (string, optional) and `file_id` (string, optional). - - - `file_id: String` - - File ID for the mask image. - - - `image_url: String` - - Base64-encoded mask image. - - - `model: String | :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `String = String` - - - `Model = :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `:"gpt-image-1"` - - - `:"gpt-image-1-mini"` - - - `:"gpt-image-2"` - - - `:"gpt-image-2-2026-04-21"` - - - `:"gpt-image-1.5"` - - - `:"chatgpt-image-latest"` - - - `moderation: :auto | :low` - - Moderation level for the generated image. Default: `auto`. - - - `:auto` - - - `:low` - - - `output_compression: Integer` - - Compression level for the output image. Default: 100. - - - `output_format: :png | :webp | :jpeg` - - The output format of the generated image. One of `png`, `webp`, or - `jpeg`. Default: `png`. - - - `:png` - - - `:webp` - - - `:jpeg` - - - `partial_images: Integer` - - Number of partial images to generate in streaming mode, from 0 (default value) to 3. - - - `quality: :low | :medium | :high | :auto` - - The quality of the generated image. One of `low`, `medium`, `high`, - or `auto`. Default: `auto`. - - - `:low` - - - `:medium` - - - `:high` - - - `:auto` - - - `size: String | :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `String = String` - - - `Size = :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `:"1024x1024"` - - - `:"1024x1536"` - - - `:"1536x1024"` - - - `:auto` - - - `class LocalShell` - - A tool that allows the model to execute shell commands in a local environment. - - - `type: :local_shell` - - The type of the local shell tool. Always `local_shell`. - - - `:local_shell` - - - `class FunctionShellTool` - - A tool that allows the model to execute shell commands. - - - `type: :shell` - - The type of the shell tool. Always `shell`. - - - `:shell` - - - `environment: ContainerAuto | LocalEnvironment | ContainerReference` - - - `class ContainerAuto` - - - `type: :container_auto` - - Automatically creates a container for this request - - - `:container_auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `skills: Array[SkillReference | InlineSkill]` - - An optional list of skills referenced by id or inline data. - - - `class SkillReference` - - - `skill_id: String` - - The ID of the referenced skill. - - - `type: :skill_reference` - - References a skill created with the /v1/skills endpoint. - - - `:skill_reference` - - - `version: String` - - Optional skill version. Use a positive integer or 'latest'. Omit for default. - - - `class InlineSkill` - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `source: InlineSkillSource` - - Inline skill payload - - - `data: String` - - Base64-encoded skill zip bundle. - - - `media_type: :"application/zip"` - - The media type of the inline skill payload. Must be `application/zip`. - - - `:"application/zip"` - - - `type: :base64` - - The type of the inline skill source. Must be `base64`. - - - `:base64` - - - `type: :inline` - - Defines an inline skill for this request. - - - `:inline` - - - `class LocalEnvironment` - - - `type: :local` - - Use a local computer environment. - - - `:local` - - - `skills: Array[LocalSkill]` - - An optional list of skills. - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `path: String` - - The path to the directory containing the skill. - - - `class ContainerReference` - - - `container_id: String` - - The ID of the referenced container. - - - `type: :container_reference` - - References a container created with the /v1/containers endpoint - - - `:container_reference` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `name: String` - - The name of the custom tool, used to identify it in tool calls. - - - `type: :custom` - - The type of the custom tool. Always `custom`. - - - `:custom` - - - `defer_loading: bool` - - Whether this tool should be deferred and discovered via tool search. - - - `description: String` - - Optional description of the custom tool, used to provide more context. - - - `format_: CustomToolInputFormat` - - The input format for the custom tool. Default is unconstrained text. - - - `class Text` - - Unconstrained free-form text. - - - `type: :text` - - Unconstrained text format. Always `text`. - - - `:text` - - - `class Grammar` - - A grammar defined by the user. - - - `definition: String` - - The grammar definition. - - - `syntax: :lark | :regex` - - The syntax of the grammar definition. One of `lark` or `regex`. - - - `:lark` - - - `:regex` - - - `type: :grammar` - - Grammar format. Always `grammar`. - - - `:grammar` - - - `class NamespaceTool` - - Groups function/custom tools under a shared namespace. - - - `description: String` - - A description of the namespace shown to the model. - - - `name: String` - - The namespace name used in tool calls (for example, `crm`). - - - `tools: Array[Function{ name, type, defer_loading, 3 more} | CustomTool]` - - The function/custom tools available inside this namespace. - - - `class Function` - - - `name: String` - - - `type: :function` - - - `:function` - - - `defer_loading: bool` - - Whether this function should be deferred and discovered via tool search. - - - `description: String` - - - `parameters: untyped` - - - `strict: bool` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `type: :namespace` - - The type of the tool. Always `namespace`. - - - `:namespace` - - - `class ToolSearchTool` - - Hosted or BYOT tool search configuration for deferred tools. - - - `type: :tool_search` - - The type of the tool. Always `tool_search`. - - - `:tool_search` - - - `description: String` - - Description shown to the model for a client-executed tool search tool. - - - `execution: :server | :client` - - Whether tool search is executed by the server or by the client. - - - `:server` - - - `:client` - - - `parameters: untyped` - - Parameter schema for a client-executed tool search tool. - - - `class WebSearchPreviewTool` - - This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search_preview | :web_search_preview_2025_03_11` - - The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`. - - - `:web_search_preview` - - - `:web_search_preview_2025_03_11` - - - `search_content_types: Array[:text | :image]` - - - `:text` - - - `:image` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ type, city, country, 2 more}` - - The user's location. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `class ApplyPatchTool` - - Allows the assistant to create, delete, or update files using unified diffs. - - - `type: :apply_patch` - - The type of the tool. Always `apply_patch`. - - - `:apply_patch` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `error: EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - - - `eval_id: String` - - The identifier of the associated evaluation. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - The model that is evaluated, if applicable. - - - `name: String` - - The name of the evaluation run. - - - `object: :"eval.run"` - - The type of the object. Always "eval.run". - - - `:"eval.run"` - - - `per_model_usage: Array[PerModelUsage{ cached_tokens, completion_tokens, invocation_count, 3 more}]` - - Usage statistics for each model during the evaluation run. - - - `cached_tokens: Integer` - - The number of tokens retrieved from cache. - - - `completion_tokens: Integer` - - The number of completion tokens generated. - - - `invocation_count: Integer` - - The number of invocations. - - - `model_name: String` - - The name of the model. - - - `prompt_tokens: Integer` - - The number of prompt tokens used. - - - `total_tokens: Integer` - - The total number of tokens used. - - - `per_testing_criteria_results: Array[PerTestingCriteriaResult{ failed, passed, testing_criteria}]` - - Results per testing criteria applied during the evaluation run. - - - `failed: Integer` - - Number of tests failed for this criteria. - - - `passed: Integer` - - Number of tests passed for this criteria. - - - `testing_criteria: String` - - A description of the testing criteria. - - - `report_url: String` - - The URL to the rendered evaluation run report on the UI dashboard. - - - `result_counts: ResultCounts{ errored, failed, passed, total}` - - Counters summarizing the outcomes of the evaluation run. - - - `errored: Integer` - - Number of output items that resulted in an error. - - - `failed: Integer` - - Number of output items that failed to pass the evaluation. - - - `passed: Integer` - - Number of output items that passed the evaluation. - - - `total: Integer` - - Total number of executed output items. - - - `status: String` - - The status of the evaluation run. - -### Run Cancel Response - -- `class RunCancelResponse` - - A schema representing an evaluation run. - - - `id: String` - - Unique identifier for the evaluation run. - - - `created_at: Integer` - - Unix timestamp (in seconds) when the evaluation run was created. - - - `data_source: CreateEvalJSONLRunDataSource | CreateEvalCompletionsRunDataSource | Responses{ source, type, input_messages, 2 more}` - - Information about the run's data source. - - - `class CreateEvalJSONLRunDataSource` - - A JsonlRunDataSource object with that specifies a JSONL file that matches the eval - - - `source: FileContent{ content, type} | FileID{ id, type}` - - Determines what populates the `item` namespace in the data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `type: :jsonl` - - The type of data source. Always `jsonl`. - - - `:jsonl` - - - `class CreateEvalCompletionsRunDataSource` - - A CompletionsRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | StoredCompletions{ type, created_after, created_before, 3 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class StoredCompletions` - - A StoredCompletionsRunDataSource configuration describing a set of filters - - - `type: :stored_completions` - - The type of source. Always `stored_completions`. - - - `:stored_completions` - - - `created_after: Integer` - - An optional Unix timestamp to filter items created after this time. - - - `created_before: Integer` - - An optional Unix timestamp to filter items created before this time. - - - `limit: Integer` - - An optional maximum number of items to return. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - An optional model to filter by (e.g., 'gpt-4o'). - - - `type: :completions` - - The type of run data source. Always `completions`. - - - `:completions` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[EasyInputMessage | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class EasyInputMessage` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputMessageContentList` - - Text, image, or audio input to the model, used to generate a response. - Can also contain previous assistant responses. - - - `String = String` - - A text input to the model. - - - `ResponseInputMessageContentList = Array[ResponseInputContent]` - - A list of one or many input items to the model, containing different content - types. - - - `class ResponseInputText` - - A text input to the model. - - - `text: String` - - The text input to the model. - - - `type: :input_text` - - The type of the input item. Always `input_text`. - - - `:input_text` - - - `class ResponseInputImage` - - An image input to the model. Learn about [image inputs](https://platform.openai.com/docs/guides/vision). - - - `detail: :low | :high | :auto | :original` - - The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`. - - - `:low` - - - `:high` - - - `:auto` - - - `:original` - - - `type: :input_image` - - The type of the input item. Always `input_image`. - - - `:input_image` - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `image_url: String` - - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL. - - - `class ResponseInputFile` - - A file input to the model. - - - `type: :input_file` - - The type of the input item. Always `input_file`. - - - `:input_file` - - - `detail: :low | :high` - - The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`. - - - `:low` - - - `:high` - - - `file_data: String` - - The content of the file to be sent to the model. - - - `file_id: String` - - The ID of the file to be sent to the model. - - - `file_url: String` - - The URL of the file to be sent to the model. - - - `filename: String` - - The name of the file to be sent to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `phase: :commentary | :final_answer` - - Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`). - For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend - phase on all assistant messages — dropping it can degrade performance. Not used for user messages. - - - `:commentary` - - - `:final_answer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `input_audio: InputAudio{ data, format_}` - - - `data: String` - - Base64-encoded audio data. - - - `format_: :mp3 | :wav` - - The format of the audio data. Currently supported formats are `mp3` and - `wav`. - - - `:mp3` - - - `:wav` - - - `type: :input_audio` - - The type of the input item. Always `input_audio`. - - - `:input_audio` - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.input_trajectory" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, response_format, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `:none` - - - `:minimal` - - - `:low` - - - `:medium` - - - `:high` - - - `:xhigh` - - - `response_format: ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject` - - An object specifying the format that the model must output. - - Setting to `{ "type": "json_schema", "json_schema": {...} }` enables - Structured Outputs which ensures the model will match your supplied JSON - schema. Learn more in the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `type: :text` - - The type of response format being defined. Always `text`. - - - `:text` - - - `class ResponseFormatJSONSchema` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `json_schema: JSONSchema{ name, description, schema, strict}` - - Structured Outputs configuration options, including a JSON Schema. - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `type: :json_object` - - The type of response format being defined. Always `json_object`. - - - `:json_object` - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `tools: Array[ChatCompletionFunctionTool]` - - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported. - - - `function: FunctionDefinition` - - - `name: String` - - The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. - - - `description: String` - - A description of what the function does, used by the model to choose when and how to call the function. - - - `parameters: FunctionParameters` - - The parameters the functions accepts, described as a JSON Schema object. See the [guide](https://platform.openai.com/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format. - - Omitting `parameters` defines a function with an empty parameter list. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](https://platform.openai.com/docs/guides/function-calling). - - - `type: :function` - - The type of the tool. Currently, only `function` is supported. - - - `:function` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `class Responses` - - A ResponsesRunDataSource object describing a model sampling configuration. - - - `source: FileContent{ content, type} | FileID{ id, type} | Responses{ type, created_after, created_before, 8 more}` - - Determines what populates the `item` namespace in this run's data source. - - - `class FileContent` - - - `content: Array[Content{ item, sample}]` - - The content of the jsonl file. - - - `item: Hash[Symbol, untyped]` - - - `sample: Hash[Symbol, untyped]` - - - `type: :file_content` - - The type of jsonl source. Always `file_content`. - - - `:file_content` - - - `class FileID` - - - `id: String` - - The identifier of the file. - - - `type: :file_id` - - The type of jsonl source. Always `file_id`. - - - `:file_id` - - - `class Responses` - - A EvalResponsesSource object describing a run data source configuration. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `created_after: Integer` - - Only include items created after this timestamp (inclusive). This is a query parameter used to select responses. - - - `created_before: Integer` - - Only include items created before this timestamp (inclusive). This is a query parameter used to select responses. - - - `instructions_search: String` - - Optional string to search the 'instructions' field. This is a query parameter used to select responses. - - - `metadata: untyped` - - Metadata filter for the responses. This is a query parameter used to select responses. - - - `model: String` - - The name of the model to find responses for. This is a query parameter used to select responses. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `temperature: Float` - - Sampling temperature. This is a query parameter used to select responses. - - - `tools: Array[String]` - - List of tool names. This is a query parameter used to select responses. - - - `top_p: Float` - - Nucleus sampling parameter. This is a query parameter used to select responses. - - - `users: Array[String]` - - List of user identifiers. This is a query parameter used to select responses. - - - `type: :responses` - - The type of run data source. Always `responses`. - - - `:responses` - - - `input_messages: Template{ template, type} | ItemReference{ item_reference, type}` - - Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace. - - - `class Template` - - - `template: Array[ChatMessage{ content, role} | EvalItem{ content, role, type}]` - - A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}. - - - `class ChatMessage` - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `class EvalItem` - - A message input to the model with a role indicating instruction following - hierarchy. Instructions given with the `developer` or `system` role take - precedence over instructions given with the `user` role. Messages with the - `assistant` role are presumed to have been generated by the model in previous - interactions. - - - `content: String | ResponseInputText | OutputText{ text, type} | 3 more` - - Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. - - - `String = String` - - A text input to the model. - - - `class ResponseInputText` - - A text input to the model. - - - `class OutputText` - - A text output from the model. - - - `text: String` - - The text output from the model. - - - `type: :output_text` - - The type of the output text. Always `output_text`. - - - `:output_text` - - - `class InputImage` - - An image input block used within EvalItem content arrays. - - - `image_url: String` - - The URL of the image input. - - - `type: :input_image` - - The type of the image input. Always `input_image`. - - - `:input_image` - - - `detail: String` - - The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`. - - - `class ResponseInputAudio` - - An audio input to the model. - - - `GraderInputs = Array[GraderInputItem]` - - A list of inputs, each of which may be either an input text, output text, input - image, or input audio object. - - - `role: :user | :assistant | :system | :developer` - - The role of the message input. One of `user`, `assistant`, `system`, or - `developer`. - - - `:user` - - - `:assistant` - - - `:system` - - - `:developer` - - - `type: :message` - - The type of the message input. Always `message`. - - - `:message` - - - `type: :template` - - The type of input messages. Always `template`. - - - `:template` - - - `class ItemReference` - - - `item_reference: String` - - A reference to a variable in the `item` namespace. Ie, "item.name" - - - `type: :item_reference` - - The type of input messages. Always `item_reference`. - - - `:item_reference` - - - `model: String` - - The name of the model to use for generating completions (e.g. "o3-mini"). - - - `sampling_params: SamplingParams{ max_completion_tokens, reasoning_effort, seed, 4 more}` - - - `max_completion_tokens: Integer` - - The maximum number of tokens in the generated output. - - - `reasoning_effort: ReasoningEffort` - - Constrains effort on reasoning for - [reasoning models](https://platform.openai.com/docs/guides/reasoning). - Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing - reasoning effort can result in faster responses and fewer tokens used - on reasoning in a response. - - - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - - `xhigh` is supported for all models after `gpt-5.1-codex-max`. - - - `seed: Integer` - - A seed value to initialize the randomness, during sampling. - - - `temperature: Float` - - A higher temperature increases randomness in the outputs. - - - `text: Text{ format_}` - - Configuration options for a text response from the model. Can be plain - text or structured JSON data. Learn more: - - - [Text inputs and outputs](https://platform.openai.com/docs/guides/text) - - [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) - - - `format_: ResponseFormatTextConfig` - - An object specifying the format that the model must output. - - Configuring `{ "type": "json_schema" }` enables Structured Outputs, - which ensures the model will match your supplied JSON schema. Learn more in the - [Structured Outputs guide](https://platform.openai.com/docs/guides/structured-outputs). - - The default format is `{ "type": "text" }` with no additional options. - - **Not recommended for gpt-4o and newer models:** - - Setting to `{ "type": "json_object" }` enables the older JSON mode, which - ensures the message the model generates is valid JSON. Using `json_schema` - is preferred for models that support it. - - - `class ResponseFormatText` - - Default response format. Used to generate text responses. - - - `class ResponseFormatTextJSONSchemaConfig` - - JSON Schema response format. Used to generate structured JSON responses. - Learn more about [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs). - - - `name: String` - - The name of the response format. Must be a-z, A-Z, 0-9, or contain - underscores and dashes, with a maximum length of 64. - - - `schema: Hash[Symbol, untyped]` - - The schema for the response format, described as a JSON Schema object. - Learn how to build JSON schemas [here](https://json-schema.org/). - - - `type: :json_schema` - - The type of response format being defined. Always `json_schema`. - - - `:json_schema` - - - `description: String` - - A description of what the response format is for, used by the model to - determine how to respond in the format. - - - `strict: bool` - - Whether to enable strict schema adherence when generating the output. - If set to true, the model will always follow the exact schema defined - in the `schema` field. Only a subset of JSON Schema is supported when - `strict` is `true`. To learn more, read the [Structured Outputs - guide](https://platform.openai.com/docs/guides/structured-outputs). - - - `class ResponseFormatJSONObject` - - JSON object response format. An older method of generating JSON responses. - Using `json_schema` is recommended for models that support it. Note that the - model will not generate JSON without a system or user message instructing it - to do so. - - - `tools: Array[Tool]` - - An array of tools the model may call while generating a response. You - can specify which tool to use by setting the `tool_choice` parameter. - - The two categories of tools you can provide the model are: - - - **Built-in tools**: Tools that are provided by OpenAI that extend the - model's capabilities, like [web search](https://platform.openai.com/docs/guides/tools-web-search) - or [file search](https://platform.openai.com/docs/guides/tools-file-search). Learn more about - [built-in tools](https://platform.openai.com/docs/guides/tools). - - **Function calls (custom tools)**: Functions that are defined by you, - enabling the model to call your own code. Learn more about - [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `class FunctionTool` - - Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling). - - - `name: String` - - The name of the function to call. - - - `parameters: Hash[Symbol, untyped]` - - A JSON schema object describing the parameters of the function. - - - `strict: bool` - - Whether to enforce strict parameter validation. Default `true`. - - - `type: :function` - - The type of the function tool. Always `function`. - - - `:function` - - - `defer_loading: bool` - - Whether this function is deferred and loaded via tool search. - - - `description: String` - - A description of the function. Used by the model to determine whether or not to call the function. - - - `class FileSearchTool` - - A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search). - - - `type: :file_search` - - The type of the file search tool. Always `file_search`. - - - `:file_search` - - - `vector_store_ids: Array[String]` - - The IDs of the vector stores to search. - - - `filters: ComparisonFilter | CompoundFilter` - - A filter to apply. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `key: String` - - The key to compare against the value. - - - `type: :eq | :ne | :gt | 5 more` - - Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`. - - - `eq`: equals - - `ne`: not equal - - `gt`: greater than - - `gte`: greater than or equal - - `lt`: less than - - `lte`: less than or equal - - `in`: in - - `nin`: not in - - - `:eq` - - - `:ne` - - - `:gt` - - - `:gte` - - - `:lt` - - - `:lte` - - - `:in` - - - `:nin` - - - `value: String | Float | bool | Array[String | Float]` - - The value to compare against the attribute key; supports string, number, or boolean types. - - - `String = String` - - - `Float = Float` - - - `UnionMember2 = bool` - - - `UnionMember3 = Array[String | Float]` - - - `String = String` - - - `Float = Float` - - - `class CompoundFilter` - - Combine multiple filters using `and` or `or`. - - - `filters: Array[ComparisonFilter | untyped]` - - Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`. - - - `class ComparisonFilter` - - A filter used to compare a specified attribute key to a given value using a defined comparison operation. - - - `UnionMember1 = untyped` - - - `type: :and | :or` - - Type of operation: `and` or `or`. - - - `:and` - - - `:or` - - - `max_num_results: Integer` - - The maximum number of results to return. This number should be between 1 and 50 inclusive. - - - `ranking_options: RankingOptions{ hybrid_search, ranker, score_threshold}` - - Ranking options for search. - - - `hybrid_search: HybridSearch{ embedding_weight, text_weight}` - - Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled. - - - `embedding_weight: Float` - - The weight of the embedding in the reciprocal ranking fusion. - - - `text_weight: Float` - - The weight of the text in the reciprocal ranking fusion. - - - `ranker: :auto | :"default-2024-11-15"` - - The ranker to use for the file search. - - - `:auto` - - - `:"default-2024-11-15"` - - - `score_threshold: Float` - - The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results. - - - `class ComputerTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `type: :computer` - - The type of the computer tool. Always `computer`. - - - `:computer` - - - `class ComputerUsePreviewTool` - - A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use). - - - `display_height: Integer` - - The height of the computer display. - - - `display_width: Integer` - - The width of the computer display. - - - `environment: :windows | :mac | :linux | 2 more` - - The type of computer environment to control. - - - `:windows` - - - `:mac` - - - `:linux` - - - `:ubuntu` - - - `:browser` - - - `type: :computer_use_preview` - - The type of the computer use tool. Always `computer_use_preview`. - - - `:computer_use_preview` - - - `class WebSearchTool` - - Search the Internet for sources related to the prompt. Learn more about the - [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search | :web_search_2025_08_26` - - The type of the web search tool. One of `web_search` or `web_search_2025_08_26`. - - - `:web_search` - - - `:web_search_2025_08_26` - - - `filters: Filters{ allowed_domains}` - - Filters for the search. - - - `allowed_domains: Array[String]` - - Allowed domains for the search. If not provided, all domains are allowed. - Subdomains of the provided domains are allowed as well. - - Example: `["pubmed.ncbi.nlm.nih.gov"]` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ city, country, region, 2 more}` - - The approximate location of the user. - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `class Mcp` - - Give the model access to additional tools via remote Model Context Protocol - (MCP) servers. [Learn more about MCP](https://platform.openai.com/docs/guides/tools-remote-mcp). - - - `server_label: String` - - A label for this MCP server, used to identify it in tool calls. - - - `type: :mcp` - - The type of the MCP tool. Always `mcp`. - - - `:mcp` - - - `allowed_tools: Array[String] | McpToolFilter{ read_only, tool_names}` - - List of allowed tool names or a filter object. - - - `McpAllowedTools = Array[String]` - - A string array of allowed tool names - - - `class McpToolFilter` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `authorization: String` - - An OAuth access token that can be used with a remote MCP server, either - with a custom MCP server URL or a service connector. Your application - must handle the OAuth authorization flow and provide the token here. - - - `connector_id: :connector_dropbox | :connector_gmail | :connector_googlecalendar | 5 more` - - Identifier for service connectors, like those available in ChatGPT. One of - `server_url` or `connector_id` must be provided. Learn more about service - connectors [here](https://platform.openai.com/docs/guides/tools-remote-mcp#connectors). - - Currently supported `connector_id` values are: - - - Dropbox: `connector_dropbox` - - Gmail: `connector_gmail` - - Google Calendar: `connector_googlecalendar` - - Google Drive: `connector_googledrive` - - Microsoft Teams: `connector_microsoftteams` - - Outlook Calendar: `connector_outlookcalendar` - - Outlook Email: `connector_outlookemail` - - SharePoint: `connector_sharepoint` - - - `:connector_dropbox` - - - `:connector_gmail` - - - `:connector_googlecalendar` - - - `:connector_googledrive` - - - `:connector_microsoftteams` - - - `:connector_outlookcalendar` - - - `:connector_outlookemail` - - - `:connector_sharepoint` - - - `defer_loading: bool` - - Whether this MCP tool is deferred and discovered via tool search. - - - `headers: Hash[Symbol, String]` - - Optional HTTP headers to send to the MCP server. Use for authentication - or other purposes. - - - `require_approval: McpToolApprovalFilter{ always, never} | :always | :never` - - Specify which of the MCP server's tools require approval. - - - `class McpToolApprovalFilter` - - Specify which of the MCP server's tools require approval. Can be - `always`, `never`, or a filter object associated with tools - that require approval. - - - `always: Always{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `never: Never{ read_only, tool_names}` - - A filter object to specify which tools are allowed. - - - `read_only: bool` - - Indicates whether or not a tool modifies data or is read-only. If an - MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint), - it will match this filter. - - - `tool_names: Array[String]` - - List of allowed tool names. - - - `McpToolApprovalSetting = :always | :never` - - Specify a single approval policy for all tools. One of `always` or - `never`. When set to `always`, all tools will require approval. When - set to `never`, all tools will not require approval. - - - `:always` - - - `:never` - - - `server_description: String` - - Optional description of the MCP server, used to provide more context. - - - `server_url: String` - - The URL for the MCP server. One of `server_url` or `connector_id` must be - provided. - - - `class CodeInterpreter` - - A tool that runs Python code to help generate a response to a prompt. - - - `container: String | CodeInterpreterToolAuto{ type, file_ids, memory_limit, network_policy}` - - The code interpreter container. Can be a container ID or an object that - specifies uploaded file IDs to make available to your code, along with an - optional `memory_limit` setting. - - - `String = String` - - The container ID. - - - `class CodeInterpreterToolAuto` - - Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on. - - - `type: :auto` - - Always `auto`. - - - `:auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the code interpreter container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `type: :disabled` - - Disable outbound network access. Always `disabled`. - - - `:disabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `allowed_domains: Array[String]` - - A list of allowed domains when type is `allowlist`. - - - `type: :allowlist` - - Allow outbound network access only to specified domains. Always `allowlist`. - - - `:allowlist` - - - `domain_secrets: Array[ContainerNetworkPolicyDomainSecret]` - - Optional domain-scoped secrets for allowlisted domains. - - - `domain: String` - - The domain associated with the secret. - - - `name: String` - - The name of the secret to inject for the domain. - - - `value: String` - - The secret value to inject for the domain. - - - `type: :code_interpreter` - - The type of the code interpreter tool. Always `code_interpreter`. - - - `:code_interpreter` - - - `class ImageGeneration` - - A tool that generates images using the GPT image models. - - - `type: :image_generation` - - The type of the image generation tool. Always `image_generation`. - - - `:image_generation` - - - `action: :generate | :edit | :auto` - - Whether to generate a new image or edit an existing image. Default: `auto`. - - - `:generate` - - - `:edit` - - - `:auto` - - - `background: :transparent | :opaque | :auto` - - Allows to set transparency for the background of the generated image(s). - This parameter is only supported for GPT image models that support - transparent backgrounds. Must be one of `transparent`, `opaque`, or - `auto` (default value). When `auto` is used, the model will - automatically determine the best background for the image. - - `gpt-image-2` and `gpt-image-2-2026-04-21` do not support - transparent backgrounds. Requests with `background` set to - `transparent` will return an error for these models; use `opaque` or - `auto` instead. - - If `transparent`, the output format needs to support transparency, - so it should be set to either `png` (default value) or `webp`. - - - `:transparent` - - - `:opaque` - - - `:auto` - - - `input_fidelity: :high | :low` - - Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`. - - - `:high` - - - `:low` - - - `input_image_mask: InputImageMask{ file_id, image_url}` - - Optional mask for inpainting. Contains `image_url` - (string, optional) and `file_id` (string, optional). - - - `file_id: String` - - File ID for the mask image. - - - `image_url: String` - - Base64-encoded mask image. - - - `model: String | :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `String = String` - - - `Model = :"gpt-image-1" | :"gpt-image-1-mini" | :"gpt-image-2" | 3 more` - - The image generation model to use. Default: `gpt-image-1`. - - - `:"gpt-image-1"` - - - `:"gpt-image-1-mini"` - - - `:"gpt-image-2"` - - - `:"gpt-image-2-2026-04-21"` - - - `:"gpt-image-1.5"` - - - `:"chatgpt-image-latest"` - - - `moderation: :auto | :low` - - Moderation level for the generated image. Default: `auto`. - - - `:auto` - - - `:low` - - - `output_compression: Integer` - - Compression level for the output image. Default: 100. - - - `output_format: :png | :webp | :jpeg` - - The output format of the generated image. One of `png`, `webp`, or - `jpeg`. Default: `png`. - - - `:png` - - - `:webp` - - - `:jpeg` - - - `partial_images: Integer` - - Number of partial images to generate in streaming mode, from 0 (default value) to 3. - - - `quality: :low | :medium | :high | :auto` - - The quality of the generated image. One of `low`, `medium`, `high`, - or `auto`. Default: `auto`. - - - `:low` - - - `:medium` - - - `:high` - - - `:auto` - - - `size: String | :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `String = String` - - - `Size = :"1024x1024" | :"1024x1536" | :"1536x1024" | :auto` - - The size of the generated images. For `gpt-image-2` and `gpt-image-2-2026-04-21`, arbitrary resolutions are supported as `WIDTHxHEIGHT` strings, for example `1536x864`. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above `2560x1440` are experimental, and the maximum supported resolution is `3840x2160`. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes `1024x1024`, `1536x1024`, and `1024x1536` are supported by the GPT image models; `auto` is supported for models that allow automatic sizing. For `dall-e-2`, use one of `256x256`, `512x512`, or `1024x1024`. For `dall-e-3`, use one of `1024x1024`, `1792x1024`, or `1024x1792`. - - - `:"1024x1024"` - - - `:"1024x1536"` - - - `:"1536x1024"` - - - `:auto` - - - `class LocalShell` - - A tool that allows the model to execute shell commands in a local environment. - - - `type: :local_shell` - - The type of the local shell tool. Always `local_shell`. - - - `:local_shell` - - - `class FunctionShellTool` - - A tool that allows the model to execute shell commands. - - - `type: :shell` - - The type of the shell tool. Always `shell`. - - - `:shell` - - - `environment: ContainerAuto | LocalEnvironment | ContainerReference` - - - `class ContainerAuto` - - - `type: :container_auto` - - Automatically creates a container for this request - - - `:container_auto` - - - `file_ids: Array[String]` - - An optional list of uploaded files to make available to your code. - - - `memory_limit: :"1g" | :"4g" | :"16g" | :"64g"` - - The memory limit for the container. - - - `:"1g"` - - - `:"4g"` - - - `:"16g"` - - - `:"64g"` - - - `network_policy: ContainerNetworkPolicyDisabled | ContainerNetworkPolicyAllowlist` - - Network access policy for the container. - - - `class ContainerNetworkPolicyDisabled` - - - `class ContainerNetworkPolicyAllowlist` - - - `skills: Array[SkillReference | InlineSkill]` - - An optional list of skills referenced by id or inline data. - - - `class SkillReference` - - - `skill_id: String` - - The ID of the referenced skill. - - - `type: :skill_reference` - - References a skill created with the /v1/skills endpoint. - - - `:skill_reference` - - - `version: String` - - Optional skill version. Use a positive integer or 'latest'. Omit for default. - - - `class InlineSkill` - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `source: InlineSkillSource` - - Inline skill payload - - - `data: String` - - Base64-encoded skill zip bundle. - - - `media_type: :"application/zip"` - - The media type of the inline skill payload. Must be `application/zip`. - - - `:"application/zip"` - - - `type: :base64` - - The type of the inline skill source. Must be `base64`. - - - `:base64` - - - `type: :inline` - - Defines an inline skill for this request. - - - `:inline` - - - `class LocalEnvironment` - - - `type: :local` - - Use a local computer environment. - - - `:local` - - - `skills: Array[LocalSkill]` - - An optional list of skills. - - - `description: String` - - The description of the skill. - - - `name: String` - - The name of the skill. - - - `path: String` - - The path to the directory containing the skill. - - - `class ContainerReference` - - - `container_id: String` - - The ID of the referenced container. - - - `type: :container_reference` - - References a container created with the /v1/containers endpoint - - - `:container_reference` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `name: String` - - The name of the custom tool, used to identify it in tool calls. - - - `type: :custom` - - The type of the custom tool. Always `custom`. - - - `:custom` - - - `defer_loading: bool` - - Whether this tool should be deferred and discovered via tool search. - - - `description: String` - - Optional description of the custom tool, used to provide more context. - - - `format_: CustomToolInputFormat` - - The input format for the custom tool. Default is unconstrained text. - - - `class Text` - - Unconstrained free-form text. - - - `type: :text` - - Unconstrained text format. Always `text`. - - - `:text` - - - `class Grammar` - - A grammar defined by the user. - - - `definition: String` - - The grammar definition. - - - `syntax: :lark | :regex` - - The syntax of the grammar definition. One of `lark` or `regex`. - - - `:lark` - - - `:regex` - - - `type: :grammar` - - Grammar format. Always `grammar`. - - - `:grammar` - - - `class NamespaceTool` - - Groups function/custom tools under a shared namespace. - - - `description: String` - - A description of the namespace shown to the model. - - - `name: String` - - The namespace name used in tool calls (for example, `crm`). - - - `tools: Array[Function{ name, type, defer_loading, 3 more} | CustomTool]` - - The function/custom tools available inside this namespace. - - - `class Function` - - - `name: String` - - - `type: :function` - - - `:function` - - - `defer_loading: bool` - - Whether this function should be deferred and discovered via tool search. - - - `description: String` - - - `parameters: untyped` - - - `strict: bool` - - - `class CustomTool` - - A custom tool that processes input using a specified format. Learn more about [custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) - - - `type: :namespace` - - The type of the tool. Always `namespace`. - - - `:namespace` - - - `class ToolSearchTool` - - Hosted or BYOT tool search configuration for deferred tools. - - - `type: :tool_search` - - The type of the tool. Always `tool_search`. - - - `:tool_search` - - - `description: String` - - Description shown to the model for a client-executed tool search tool. - - - `execution: :server | :client` - - Whether tool search is executed by the server or by the client. - - - `:server` - - - `:client` - - - `parameters: untyped` - - Parameter schema for a client-executed tool search tool. - - - `class WebSearchPreviewTool` - - This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search). - - - `type: :web_search_preview | :web_search_preview_2025_03_11` - - The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`. - - - `:web_search_preview` - - - `:web_search_preview_2025_03_11` - - - `search_content_types: Array[:text | :image]` - - - `:text` - - - `:image` - - - `search_context_size: :low | :medium | :high` - - High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default. - - - `:low` - - - `:medium` - - - `:high` - - - `user_location: UserLocation{ type, city, country, 2 more}` - - The user's location. - - - `type: :approximate` - - The type of location approximation. Always `approximate`. - - - `:approximate` - - - `city: String` - - Free text input for the city of the user, e.g. `San Francisco`. - - - `country: String` - - The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`. - - - `region: String` - - Free text input for the region of the user, e.g. `California`. - - - `timezone: String` - - The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`. - - - `class ApplyPatchTool` - - Allows the assistant to create, delete, or update files using unified diffs. - - - `type: :apply_patch` - - The type of the tool. Always `apply_patch`. - - - `:apply_patch` - - - `top_p: Float` - - An alternative to temperature for nucleus sampling; 1.0 includes all tokens. - - - `error: EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - - - `eval_id: String` - - The identifier of the associated evaluation. - - - `metadata: Metadata` - - Set of 16 key-value pairs that can be attached to an object. This can be - useful for storing additional information about the object in a structured - format, and querying for objects via API or the dashboard. - - Keys are strings with a maximum length of 64 characters. Values are strings - with a maximum length of 512 characters. - - - `model: String` - - The model that is evaluated, if applicable. - - - `name: String` - - The name of the evaluation run. - - - `object: :"eval.run"` - - The type of the object. Always "eval.run". - - - `:"eval.run"` - - - `per_model_usage: Array[PerModelUsage{ cached_tokens, completion_tokens, invocation_count, 3 more}]` - - Usage statistics for each model during the evaluation run. - - - `cached_tokens: Integer` - - The number of tokens retrieved from cache. - - - `completion_tokens: Integer` - - The number of completion tokens generated. - - - `invocation_count: Integer` - - The number of invocations. - - - `model_name: String` - - The name of the model. - - - `prompt_tokens: Integer` - - The number of prompt tokens used. - - - `total_tokens: Integer` - - The total number of tokens used. - - - `per_testing_criteria_results: Array[PerTestingCriteriaResult{ failed, passed, testing_criteria}]` - - Results per testing criteria applied during the evaluation run. - - - `failed: Integer` - - Number of tests failed for this criteria. - - - `passed: Integer` - - Number of tests passed for this criteria. - - - `testing_criteria: String` - - A description of the testing criteria. - - - `report_url: String` - - The URL to the rendered evaluation run report on the UI dashboard. - - - `result_counts: ResultCounts{ errored, failed, passed, total}` - - Counters summarizing the outcomes of the evaluation run. - - - `errored: Integer` - - Number of output items that resulted in an error. - - - `failed: Integer` - - Number of output items that failed to pass the evaluation. - - - `passed: Integer` - - Number of output items that passed the evaluation. - - - `total: Integer` - - Total number of executed output items. - - - `status: String` - - The status of the evaluation run. - -### Run Delete Response - -- `class RunDeleteResponse` - - - `deleted: bool` - - - `object: String` - - - `run_id: String` - -# Output Items - -## Get eval run output items - -`evals.runs.output_items.list(run_id, **kwargs) -> CursorPage` - -**get** `/evals/{eval_id}/runs/{run_id}/output_items` - -Get a list of output items for an evaluation run. - -### Parameters - -- `eval_id: String` - -- `run_id: String` - -- `after: String` - - Identifier for the last output item from the previous pagination request. - -- `limit: Integer` - - Number of output items to retrieve. - -- `order: :asc | :desc` - - Sort order for output items by timestamp. Use `asc` for ascending order or `desc` for descending order. Defaults to `asc`. - - - `:asc` - - - `:desc` - -- `status: :fail | :pass` - - Filter output items by status. Use `failed` to filter by failed output - items or `pass` to filter by passed output items. - - - `:fail` - - - `:pass` - -### Returns - -- `class OutputItemListResponse` - - A schema representing an evaluation run output item. - - - `id: String` - - Unique identifier for the evaluation run output item. - - - `created_at: Integer` - - Unix timestamp (in seconds) when the evaluation run was created. - - - `datasource_item: Hash[Symbol, untyped]` - - Details of the input data source item. - - - `datasource_item_id: Integer` - - The identifier for the data source item. - - - `eval_id: String` - - The identifier of the evaluation group. - - - `object: :"eval.run.output_item"` - - The type of the object. Always "eval.run.output_item". - - - `:"eval.run.output_item"` - - - `results: Array[Result{ name, passed, score, 2 more}]` - - A list of grader results for this output item. - - - `name: String` - - The name of the grader. - - - `passed: bool` - - Whether the grader considered the output a pass. - - - `score: Float` - - The numeric score produced by the grader. - - - `sample: Hash[Symbol, untyped]` - - Optional sample or intermediate data produced by the grader. - - - `type: String` - - The grader type (for example, "string-check-grader"). - - - `run_id: String` - - The identifier of the evaluation run associated with this output item. - - - `sample: Sample{ error, finish_reason, input, 7 more}` - - A sample containing the input and output of the evaluation run. - - - `error: EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - - - `finish_reason: String` - - The reason why the sample generation was finished. - - - `input: Array[Input{ content, role}]` - - An array of input messages. - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message sender (e.g., system, user, developer). - - - `max_completion_tokens: Integer` - - The maximum number of tokens allowed for completion. - - - `model: String` - - The model used for generating the sample. - - - `output: Array[Output{ content, role}]` - - An array of output messages. - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `seed: Integer` - - The seed used for generating the sample. - - - `temperature: Float` - - The sampling temperature used. - - - `top_p: Float` - - The top_p value used for sampling. - - - `usage: Usage{ cached_tokens, completion_tokens, prompt_tokens, total_tokens}` - - Token usage details for the sample. - - - `cached_tokens: Integer` - - The number of tokens retrieved from cache. - - - `completion_tokens: Integer` - - The number of completion tokens generated. - - - `prompt_tokens: Integer` - - The number of prompt tokens used. - - - `total_tokens: Integer` - - The total number of tokens used. - - - `status: String` - - The status of the evaluation run. - -### Example - -```ruby -require "openai" - -openai = OpenAI::Client.new(api_key: "My API Key") - -page = openai.evals.runs.output_items.list("run_id", eval_id: "eval_id") - -puts(page) -``` - -#### Response - -```json -{ - "data": [ - { - "id": "id", - "created_at": 0, - "datasource_item": { - "foo": "bar" - }, - "datasource_item_id": 0, - "eval_id": "eval_id", - "object": "eval.run.output_item", - "results": [ - { - "name": "name", - "passed": true, - "score": 0, - "sample": { - "foo": "bar" - }, - "type": "type" - } - ], - "run_id": "run_id", - "sample": { - "error": { - "code": "code", - "message": "message" - }, - "finish_reason": "finish_reason", - "input": [ - { - "content": "content", - "role": "role" - } - ], - "max_completion_tokens": 0, - "model": "model", - "output": [ - { - "content": "content", - "role": "role" - } - ], - "seed": 0, - "temperature": 0, - "top_p": 0, - "usage": { - "cached_tokens": 0, - "completion_tokens": 0, - "prompt_tokens": 0, - "total_tokens": 0 - } - }, - "status": "status" - } - ], - "first_id": "first_id", - "has_more": true, - "last_id": "last_id", - "object": "list" -} -``` - -## Get an output item of an eval run - -`evals.runs.output_items.retrieve(output_item_id, **kwargs) -> OutputItemRetrieveResponse` - -**get** `/evals/{eval_id}/runs/{run_id}/output_items/{output_item_id}` - -Get an evaluation run output item by ID. - -### Parameters - -- `eval_id: String` - -- `run_id: String` - -- `output_item_id: String` - -### Returns - -- `class OutputItemRetrieveResponse` - - A schema representing an evaluation run output item. - - - `id: String` - - Unique identifier for the evaluation run output item. - - - `created_at: Integer` - - Unix timestamp (in seconds) when the evaluation run was created. - - - `datasource_item: Hash[Symbol, untyped]` - - Details of the input data source item. - - - `datasource_item_id: Integer` - - The identifier for the data source item. - - - `eval_id: String` - - The identifier of the evaluation group. - - - `object: :"eval.run.output_item"` - - The type of the object. Always "eval.run.output_item". - - - `:"eval.run.output_item"` - - - `results: Array[Result{ name, passed, score, 2 more}]` - - A list of grader results for this output item. - - - `name: String` - - The name of the grader. - - - `passed: bool` - - Whether the grader considered the output a pass. - - - `score: Float` - - The numeric score produced by the grader. - - - `sample: Hash[Symbol, untyped]` - - Optional sample or intermediate data produced by the grader. - - - `type: String` - - The grader type (for example, "string-check-grader"). - - - `run_id: String` - - The identifier of the evaluation run associated with this output item. - - - `sample: Sample{ error, finish_reason, input, 7 more}` - - A sample containing the input and output of the evaluation run. - - - `error: EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - - - `finish_reason: String` - - The reason why the sample generation was finished. - - - `input: Array[Input{ content, role}]` - - An array of input messages. - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message sender (e.g., system, user, developer). - - - `max_completion_tokens: Integer` - - The maximum number of tokens allowed for completion. - - - `model: String` - - The model used for generating the sample. - - - `output: Array[Output{ content, role}]` - - An array of output messages. - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `seed: Integer` - - The seed used for generating the sample. - - - `temperature: Float` - - The sampling temperature used. - - - `top_p: Float` - - The top_p value used for sampling. - - - `usage: Usage{ cached_tokens, completion_tokens, prompt_tokens, total_tokens}` - - Token usage details for the sample. - - - `cached_tokens: Integer` - - The number of tokens retrieved from cache. - - - `completion_tokens: Integer` - - The number of completion tokens generated. - - - `prompt_tokens: Integer` - - The number of prompt tokens used. - - - `total_tokens: Integer` - - The total number of tokens used. - - - `status: String` - - The status of the evaluation run. - -### Example - -```ruby -require "openai" - -openai = OpenAI::Client.new(api_key: "My API Key") - -output_item = openai.evals.runs.output_items.retrieve("output_item_id", eval_id: "eval_id", run_id: "run_id") - -puts(output_item) -``` - -#### Response - -```json -{ - "id": "id", - "created_at": 0, - "datasource_item": { - "foo": "bar" - }, - "datasource_item_id": 0, - "eval_id": "eval_id", - "object": "eval.run.output_item", - "results": [ - { - "name": "name", - "passed": true, - "score": 0, - "sample": { - "foo": "bar" - }, - "type": "type" - } - ], - "run_id": "run_id", - "sample": { - "error": { - "code": "code", - "message": "message" - }, - "finish_reason": "finish_reason", - "input": [ - { - "content": "content", - "role": "role" - } - ], - "max_completion_tokens": 0, - "model": "model", - "output": [ - { - "content": "content", - "role": "role" - } - ], - "seed": 0, - "temperature": 0, - "top_p": 0, - "usage": { - "cached_tokens": 0, - "completion_tokens": 0, - "prompt_tokens": 0, - "total_tokens": 0 - } - }, - "status": "status" -} -``` - -## Domain Types - -### Output Item List Response - -- `class OutputItemListResponse` - - A schema representing an evaluation run output item. - - - `id: String` - - Unique identifier for the evaluation run output item. - - - `created_at: Integer` - - Unix timestamp (in seconds) when the evaluation run was created. - - - `datasource_item: Hash[Symbol, untyped]` - - Details of the input data source item. - - - `datasource_item_id: Integer` - - The identifier for the data source item. - - - `eval_id: String` - - The identifier of the evaluation group. - - - `object: :"eval.run.output_item"` - - The type of the object. Always "eval.run.output_item". - - - `:"eval.run.output_item"` - - - `results: Array[Result{ name, passed, score, 2 more}]` - - A list of grader results for this output item. - - - `name: String` - - The name of the grader. - - - `passed: bool` - - Whether the grader considered the output a pass. - - - `score: Float` - - The numeric score produced by the grader. - - - `sample: Hash[Symbol, untyped]` - - Optional sample or intermediate data produced by the grader. - - - `type: String` - - The grader type (for example, "string-check-grader"). - - - `run_id: String` - - The identifier of the evaluation run associated with this output item. - - - `sample: Sample{ error, finish_reason, input, 7 more}` - - A sample containing the input and output of the evaluation run. - - - `error: EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - - - `finish_reason: String` - - The reason why the sample generation was finished. - - - `input: Array[Input{ content, role}]` - - An array of input messages. - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message sender (e.g., system, user, developer). - - - `max_completion_tokens: Integer` - - The maximum number of tokens allowed for completion. - - - `model: String` - - The model used for generating the sample. - - - `output: Array[Output{ content, role}]` - - An array of output messages. - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `seed: Integer` - - The seed used for generating the sample. - - - `temperature: Float` - - The sampling temperature used. - - - `top_p: Float` - - The top_p value used for sampling. - - - `usage: Usage{ cached_tokens, completion_tokens, prompt_tokens, total_tokens}` - - Token usage details for the sample. - - - `cached_tokens: Integer` - - The number of tokens retrieved from cache. - - - `completion_tokens: Integer` - - The number of completion tokens generated. - - - `prompt_tokens: Integer` - - The number of prompt tokens used. - - - `total_tokens: Integer` - - The total number of tokens used. - - - `status: String` - - The status of the evaluation run. - -### Output Item Retrieve Response - -- `class OutputItemRetrieveResponse` - - A schema representing an evaluation run output item. - - - `id: String` - - Unique identifier for the evaluation run output item. - - - `created_at: Integer` - - Unix timestamp (in seconds) when the evaluation run was created. - - - `datasource_item: Hash[Symbol, untyped]` - - Details of the input data source item. - - - `datasource_item_id: Integer` - - The identifier for the data source item. - - - `eval_id: String` - - The identifier of the evaluation group. - - - `object: :"eval.run.output_item"` - - The type of the object. Always "eval.run.output_item". - - - `:"eval.run.output_item"` - - - `results: Array[Result{ name, passed, score, 2 more}]` - - A list of grader results for this output item. - - - `name: String` - - The name of the grader. - - - `passed: bool` - - Whether the grader considered the output a pass. - - - `score: Float` - - The numeric score produced by the grader. - - - `sample: Hash[Symbol, untyped]` - - Optional sample or intermediate data produced by the grader. - - - `type: String` - - The grader type (for example, "string-check-grader"). - - - `run_id: String` - - The identifier of the evaluation run associated with this output item. - - - `sample: Sample{ error, finish_reason, input, 7 more}` - - A sample containing the input and output of the evaluation run. - - - `error: EvalAPIError` - - An object representing an error response from the Eval API. - - - `code: String` - - The error code. - - - `message: String` - - The error message. - - - `finish_reason: String` - - The reason why the sample generation was finished. - - - `input: Array[Input{ content, role}]` - - An array of input messages. - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message sender (e.g., system, user, developer). - - - `max_completion_tokens: Integer` - - The maximum number of tokens allowed for completion. - - - `model: String` - - The model used for generating the sample. - - - `output: Array[Output{ content, role}]` - - An array of output messages. - - - `content: String` - - The content of the message. - - - `role: String` - - The role of the message (e.g. "system", "assistant", "user"). - - - `seed: Integer` - - The seed used for generating the sample. - - - `temperature: Float` - - The sampling temperature used. - - - `top_p: Float` - - The top_p value used for sampling. - - - `usage: Usage{ cached_tokens, completion_tokens, prompt_tokens, total_tokens}` - - Token usage details for the sample. - - - `cached_tokens: Integer` - - The number of tokens retrieved from cache. - - - `completion_tokens: Integer` - - The number of completion tokens generated. - - - `prompt_tokens: Integer` - - The number of prompt tokens used. - - - `total_tokens: Integer` - - The total number of tokens used. - - - `status: String` - - The status of the evaluation run.