diff --git a/en/resources/evals/index.md b/en/resources/evals/index.md
index d41596b..9da53de 100644
--- a/en/resources/evals/index.md
+++ b/en/resources/evals/index.md
@@ -51,7 +51,7 @@ List evaluations for a project.
Configuration of data sources used in runs of the evaluation.
- - `EvalCustomDataSourceConfig = object { schema, type }`
+ - `EvalCustomDataSourceConfig object { schema, type }`
A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
The response schema defines the shape of the data that will be:
@@ -70,7 +70,7 @@ List evaluations for a project.
- `"custom"`
- - `LogsDataSourceConfig = object { schema, type, metadata }`
+ - `LogsDataSourceConfig object { schema, type, metadata }`
A LogsDataSourceConfig which specifies the metadata property of your logs query.
This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
@@ -97,7 +97,7 @@ List evaluations for a project.
Keys are strings with a maximum length of 64 characters. Values are strings
with a maximum length of 512 characters.
- - `EvalStoredCompletionsDataSourceConfig = object { schema, type, metadata }`
+ - `EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }`
Deprecated in favor of LogsDataSourceConfig.
@@ -144,7 +144,7 @@ List evaluations for a project.
A list of testing criteria.
- - `LabelModelGrader = object { input, labels, model, 3 more }`
+ - `LabelModelGrader object { input, labels, model, 3 more }`
A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
@@ -159,7 +159,7 @@ List evaluations for a project.
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
@@ -173,7 +173,7 @@ List evaluations for a project.
- `"input_text"`
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -187,7 +187,7 @@ List evaluations for a project.
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -205,7 +205,7 @@ List evaluations for a project.
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
@@ -239,21 +239,11 @@ List evaluations for a project.
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
- - `text: string`
-
- The text input to the model.
-
- - `type: "input_text"`
-
- The type of the input item. Always `input_text`.
-
- - `"input_text"`
-
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -267,7 +257,7 @@ List evaluations for a project.
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -285,31 +275,10 @@ List evaluations for a project.
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
- - `input_audio: object { data, format }`
-
- - `data: string`
-
- Base64-encoded audio data.
-
- - `format: "mp3" or "wav"`
-
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
-
- - `"mp3"`
-
- - `"wav"`
-
- - `type: "input_audio"`
-
- The type of the input item. Always `input_audio`.
-
- - `"input_audio"`
-
- `role: "user" or "assistant" or "system" or "developer"`
The role of the message input. One of `user`, `assistant`, `system`, or
@@ -351,7 +320,7 @@ List evaluations for a project.
- `"label_model"`
- - `StringCheckGrader = object { input, name, operation, 2 more }`
+ - `StringCheckGrader object { input, name, operation, 2 more }`
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
@@ -581,7 +550,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
- - `CustomDataSourceConfig = object { item_schema, type, include_sample_schema }`
+ - `CustomDataSourceConfig object { item_schema, type, include_sample_schema }`
A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs.
This schema is used to define the shape of the data that will be:
@@ -603,7 +572,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)
- - `LogsDataSourceConfig = object { type, metadata }`
+ - `LogsDataSourceConfig object { type, metadata }`
A data source config which specifies the metadata property of your logs query.
This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
@@ -618,7 +587,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
Metadata filters for the logs data source.
- - `StoredCompletionsDataSourceConfig = object { type, metadata }`
+ - `StoredCompletionsDataSourceConfig object { type, metadata }`
Deprecated in favor of LogsDataSourceConfig.
@@ -636,7 +605,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).
- - `LabelModelGrader = object { input, labels, model, 3 more }`
+ - `LabelModelGrader object { input, labels, model, 3 more }`
A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
@@ -645,7 +614,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
- - `SimpleInputMessage = object { content, role }`
+ - `SimpleInputMessage object { content, role }`
- `content: string`
@@ -655,7 +624,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
The role of the message (e.g. "system", "assistant", "user").
- - `EvalMessageObject = object { content, role, type }`
+ - `EvalMessageObject object { content, role, type }`
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the `developer` or `system` role take
@@ -671,7 +640,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
@@ -685,7 +654,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
- `"input_text"`
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -699,7 +668,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -717,7 +686,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
@@ -751,21 +720,11 @@ For more information, see the [Evals guide](/docs/guides/evals).
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
- - `text: string`
-
- The text input to the model.
-
- - `type: "input_text"`
-
- The type of the input item. Always `input_text`.
-
- - `"input_text"`
-
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -779,7 +738,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -797,31 +756,10 @@ For more information, see the [Evals guide](/docs/guides/evals).
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
- - `input_audio: object { data, format }`
-
- - `data: string`
-
- Base64-encoded audio data.
-
- - `format: "mp3" or "wav"`
-
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
-
- - `"mp3"`
-
- - `"wav"`
-
- - `type: "input_audio"`
-
- The type of the input item. Always `input_audio`.
-
- - `"input_audio"`
-
- `role: "user" or "assistant" or "system" or "developer"`
The role of the message input. One of `user`, `assistant`, `system`, or
@@ -863,7 +801,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
- `"label_model"`
- - `StringCheckGrader = object { input, name, operation, 2 more }`
+ - `StringCheckGrader object { input, name, operation, 2 more }`
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
@@ -948,7 +886,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
Configuration of data sources used in runs of the evaluation.
- - `EvalCustomDataSourceConfig = object { schema, type }`
+ - `EvalCustomDataSourceConfig object { schema, type }`
A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
The response schema defines the shape of the data that will be:
@@ -967,7 +905,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
- `"custom"`
- - `LogsDataSourceConfig = object { schema, type, metadata }`
+ - `LogsDataSourceConfig object { schema, type, metadata }`
A LogsDataSourceConfig which specifies the metadata property of your logs query.
This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
@@ -994,7 +932,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
Keys are strings with a maximum length of 64 characters. Values are strings
with a maximum length of 512 characters.
- - `EvalStoredCompletionsDataSourceConfig = object { schema, type, metadata }`
+ - `EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }`
Deprecated in favor of LogsDataSourceConfig.
@@ -1041,7 +979,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
A list of testing criteria.
- - `LabelModelGrader = object { input, labels, model, 3 more }`
+ - `LabelModelGrader object { input, labels, model, 3 more }`
A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
@@ -1056,7 +994,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
@@ -1070,7 +1008,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
- `"input_text"`
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -1084,7 +1022,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -1102,7 +1040,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
@@ -1136,21 +1074,11 @@ For more information, see the [Evals guide](/docs/guides/evals).
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
- - `text: string`
-
- The text input to the model.
-
- - `type: "input_text"`
-
- The type of the input item. Always `input_text`.
-
- - `"input_text"`
-
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -1164,7 +1092,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -1182,31 +1110,10 @@ For more information, see the [Evals guide](/docs/guides/evals).
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
- - `input_audio: object { data, format }`
-
- - `data: string`
-
- Base64-encoded audio data.
-
- - `format: "mp3" or "wav"`
-
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
-
- - `"mp3"`
-
- - `"wav"`
-
- - `type: "input_audio"`
-
- The type of the input item. Always `input_audio`.
-
- - `"input_audio"`
-
- `role: "user" or "assistant" or "system" or "developer"`
The role of the message input. One of `user`, `assistant`, `system`, or
@@ -1248,7 +1155,7 @@ For more information, see the [Evals guide](/docs/guides/evals).
- `"label_model"`
- - `StringCheckGrader = object { input, name, operation, 2 more }`
+ - `StringCheckGrader object { input, name, operation, 2 more }`
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
@@ -1514,7 +1421,7 @@ Get an evaluation by ID.
Configuration of data sources used in runs of the evaluation.
- - `EvalCustomDataSourceConfig = object { schema, type }`
+ - `EvalCustomDataSourceConfig object { schema, type }`
A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
The response schema defines the shape of the data that will be:
@@ -1533,7 +1440,7 @@ Get an evaluation by ID.
- `"custom"`
- - `LogsDataSourceConfig = object { schema, type, metadata }`
+ - `LogsDataSourceConfig object { schema, type, metadata }`
A LogsDataSourceConfig which specifies the metadata property of your logs query.
This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
@@ -1560,7 +1467,7 @@ Get an evaluation by ID.
Keys are strings with a maximum length of 64 characters. Values are strings
with a maximum length of 512 characters.
- - `EvalStoredCompletionsDataSourceConfig = object { schema, type, metadata }`
+ - `EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }`
Deprecated in favor of LogsDataSourceConfig.
@@ -1607,7 +1514,7 @@ Get an evaluation by ID.
A list of testing criteria.
- - `LabelModelGrader = object { input, labels, model, 3 more }`
+ - `LabelModelGrader object { input, labels, model, 3 more }`
A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
@@ -1622,7 +1529,7 @@ Get an evaluation by ID.
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
@@ -1636,7 +1543,7 @@ Get an evaluation by ID.
- `"input_text"`
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -1650,7 +1557,7 @@ Get an evaluation by ID.
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -1668,7 +1575,7 @@ Get an evaluation by ID.
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
@@ -1702,21 +1609,11 @@ Get an evaluation by ID.
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
- - `text: string`
-
- The text input to the model.
-
- - `type: "input_text"`
-
- The type of the input item. Always `input_text`.
-
- - `"input_text"`
-
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -1730,7 +1627,7 @@ Get an evaluation by ID.
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -1748,31 +1645,10 @@ Get an evaluation by ID.
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
- - `input_audio: object { data, format }`
-
- - `data: string`
-
- Base64-encoded audio data.
-
- - `format: "mp3" or "wav"`
-
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
-
- - `"mp3"`
-
- - `"wav"`
-
- - `type: "input_audio"`
-
- The type of the input item. Always `input_audio`.
-
- - `"input_audio"`
-
- `role: "user" or "assistant" or "system" or "developer"`
The role of the message input. One of `user`, `assistant`, `system`, or
@@ -1814,7 +1690,7 @@ Get an evaluation by ID.
- `"label_model"`
- - `StringCheckGrader = object { input, name, operation, 2 more }`
+ - `StringCheckGrader object { input, name, operation, 2 more }`
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
@@ -2014,7 +1890,7 @@ Update certain properties of an evaluation.
Configuration of data sources used in runs of the evaluation.
- - `EvalCustomDataSourceConfig = object { schema, type }`
+ - `EvalCustomDataSourceConfig object { schema, type }`
A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
The response schema defines the shape of the data that will be:
@@ -2033,7 +1909,7 @@ Update certain properties of an evaluation.
- `"custom"`
- - `LogsDataSourceConfig = object { schema, type, metadata }`
+ - `LogsDataSourceConfig object { schema, type, metadata }`
A LogsDataSourceConfig which specifies the metadata property of your logs query.
This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
@@ -2060,7 +1936,7 @@ Update certain properties of an evaluation.
Keys are strings with a maximum length of 64 characters. Values are strings
with a maximum length of 512 characters.
- - `EvalStoredCompletionsDataSourceConfig = object { schema, type, metadata }`
+ - `EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }`
Deprecated in favor of LogsDataSourceConfig.
@@ -2107,7 +1983,7 @@ Update certain properties of an evaluation.
A list of testing criteria.
- - `LabelModelGrader = object { input, labels, model, 3 more }`
+ - `LabelModelGrader object { input, labels, model, 3 more }`
A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
@@ -2122,7 +1998,7 @@ Update certain properties of an evaluation.
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
@@ -2136,7 +2012,7 @@ Update certain properties of an evaluation.
- `"input_text"`
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -2150,7 +2026,7 @@ Update certain properties of an evaluation.
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -2168,7 +2044,7 @@ Update certain properties of an evaluation.
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
@@ -2202,21 +2078,11 @@ Update certain properties of an evaluation.
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
- - `text: string`
-
- The text input to the model.
-
- - `type: "input_text"`
-
- The type of the input item. Always `input_text`.
-
- - `"input_text"`
-
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -2230,7 +2096,7 @@ Update certain properties of an evaluation.
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -2248,31 +2114,10 @@ Update certain properties of an evaluation.
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
- - `input_audio: object { data, format }`
-
- - `data: string`
-
- Base64-encoded audio data.
-
- - `format: "mp3" or "wav"`
-
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
-
- - `"mp3"`
-
- - `"wav"`
-
- - `type: "input_audio"`
-
- The type of the input item. Always `input_audio`.
-
- - `"input_audio"`
-
- `role: "user" or "assistant" or "system" or "developer"`
The role of the message input. One of `user`, `assistant`, `system`, or
@@ -2314,7 +2159,7 @@ Update certain properties of an evaluation.
- `"label_model"`
- - `StringCheckGrader = object { input, name, operation, 2 more }`
+ - `StringCheckGrader object { input, name, operation, 2 more }`
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
@@ -2536,7 +2381,7 @@ curl https://api.openai.com/v1/evals/eval_abc123 \
### Eval Custom Data Source Config
-- `EvalCustomDataSourceConfig = object { schema, type }`
+- `EvalCustomDataSourceConfig object { schema, type }`
A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
The response schema defines the shape of the data that will be:
@@ -2557,7 +2402,7 @@ curl https://api.openai.com/v1/evals/eval_abc123 \
### Eval Stored Completions Data Source Config
-- `EvalStoredCompletionsDataSourceConfig = object { schema, type, metadata }`
+- `EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }`
Deprecated in favor of LogsDataSourceConfig.
@@ -2581,333 +2426,129 @@ curl https://api.openai.com/v1/evals/eval_abc123 \
Keys are strings with a maximum length of 64 characters. Values are strings
with a maximum length of 512 characters.
-# Runs
+### Eval List Response
-## Get eval runs
+- `EvalListResponse object { id, created_at, data_source_config, 4 more }`
-**get** `/evals/{eval_id}/runs`
+ An Eval object with a data source config and testing criteria.
+ An Eval represents a task to be done for your LLM integration.
+ Like:
-Get a list of runs for an evaluation.
+ - Improve the quality of my chatbot
+ - See how well my chatbot handles customer support
+ - Check if o4-mini is better at my usecase than gpt-4o
-### Path Parameters
+ - `id: string`
-- `eval_id: string`
+ Unique identifier for the evaluation.
-### Query Parameters
+ - `created_at: number`
-- `after: optional string`
+ The Unix timestamp (in seconds) for when the eval was created.
- Identifier for the last run from the previous pagination request.
+ - `data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig`
-- `limit: optional number`
+ Configuration of data sources used in runs of the evaluation.
- Number of runs to retrieve.
+ - `EvalCustomDataSourceConfig object { schema, type }`
-- `order: optional "asc" or "desc"`
+ A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
+ The response schema defines the shape of the data that will be:
- Sort order for runs by timestamp. Use `asc` for ascending order or `desc` for descending order. Defaults to `asc`.
+ - Used to define your testing criteria and
+ - What data is required when creating a run
- - `"asc"`
+ - `schema: map[unknown]`
- - `"desc"`
+ The json schema for the run data source items.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
-- `status: optional "queued" or "in_progress" or "completed" or 2 more`
+ - `type: "custom"`
- Filter runs by status. One of `queued` | `in_progress` | `failed` | `completed` | `canceled`.
+ The type of data source. Always `custom`.
- - `"queued"`
+ - `"custom"`
- - `"in_progress"`
+ - `LogsDataSourceConfig object { schema, type, metadata }`
- - `"completed"`
+ A LogsDataSourceConfig which specifies the metadata property of your logs query.
+ This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
+ The schema returned by this data source config is used to defined what variables are available in your evals.
+ `item` and `sample` are both defined when using this data source config.
- - `"canceled"`
+ - `schema: map[unknown]`
- - `"failed"`
+ The json schema for the run data source items.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
-### Returns
+ - `type: "logs"`
-- `data: array of object { id, created_at, data_source, 11 more }`
+ The type of data source. Always `logs`.
- An array of eval run objects.
+ - `"logs"`
- - `id: string`
+ - `metadata: optional Metadata`
- Unique identifier for the evaluation run.
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - `created_at: number`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- Unix timestamp (in seconds) when the evaluation run was created.
+ - `EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }`
- - `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
+ Deprecated in favor of LogsDataSourceConfig.
- Information about the run's data source.
+ - `schema: map[unknown]`
- - `CreateEvalJSONLRunDataSource = object { source, type }`
+ The json schema for the run data source items.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
+ - `type: "stored_completions"`
- - `source: object { content, type } or object { id, type }`
+ The type of data source. Always `stored_completions`.
- Determines what populates the `item` namespace in the data source.
+ - `"stored_completions"`
- - `EvalJSONLFileContentSource = object { content, type }`
+ - `metadata: optional Metadata`
- - `content: array of object { item, sample }`
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- The content of the jsonl file.
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `item: map[unknown]`
+ - `metadata: Metadata`
- - `sample: optional map[unknown]`
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - `type: "file_content"`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- The type of jsonl source. Always `file_content`.
+ - `name: string`
- - `"file_content"`
+ The name of the evaluation.
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `object: "eval"`
- - `id: string`
+ The object type.
- The identifier of the file.
+ - `"eval"`
- - `type: "file_id"`
+ - `testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more`
- The type of jsonl source. Always `file_id`.
-
- - `"file_id"`
-
- - `type: "jsonl"`
-
- The type of data source. Always `jsonl`.
-
- - `"jsonl"`
-
- - `CreateEvalCompletionsRunDataSource = object { source, type, input_messages, 2 more }`
-
- A CompletionsRunDataSource object describing a model sampling configuration.
-
- - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
-
- Determines what populates the `item` namespace in this run's data source.
-
- - `EvalJSONLFileContentSource = object { content, type }`
-
- - `content: array of object { item, sample }`
-
- The content of the jsonl file.
-
- - `item: map[unknown]`
-
- - `sample: optional map[unknown]`
-
- - `type: "file_content"`
-
- The type of jsonl source. Always `file_content`.
-
- - `"file_content"`
-
- - `EvalJSONLFileIDSource = object { id, type }`
-
- - `id: string`
-
- The identifier of the file.
-
- - `type: "file_id"`
-
- The type of jsonl source. Always `file_id`.
-
- - `"file_id"`
-
- - `StoredCompletionsRunDataSource = object { type, created_after, created_before, 3 more }`
-
- A StoredCompletionsRunDataSource configuration describing a set of filters
-
- - `type: "stored_completions"`
-
- The type of source. Always `stored_completions`.
-
- - `"stored_completions"`
-
- - `created_after: optional number`
-
- An optional Unix timestamp to filter items created after this time.
-
- - `created_before: optional number`
-
- An optional Unix timestamp to filter items created before this time.
-
- - `limit: optional number`
-
- An optional maximum number of items to return.
-
- - `metadata: optional Metadata`
-
- Set of 16 key-value pairs that can be attached to an object. This can be
- useful for storing additional information about the object in a structured
- format, and querying for objects via API or the dashboard.
-
- Keys are strings with a maximum length of 64 characters. Values are strings
- with a maximum length of 512 characters.
-
- - `model: optional string`
-
- An optional model to filter by (e.g., 'gpt-4o').
-
- - `type: "completions"`
-
- The type of run data source. Always `completions`.
-
- - `"completions"`
-
- - `input_messages: optional object { template, type } or object { item_reference, type }`
-
- Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
-
- - `TemplateInputMessages = object { template, type }`
-
- - `template: array of EasyInputMessage or object { content, role, type }`
-
- A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
-
- - `EasyInputMessage = object { content, role, phase, type }`
-
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
-
- - `content: string or ResponseInputMessageContentList`
-
- Text, image, or audio input to the model, used to generate a response.
- Can also contain previous assistant responses.
-
- - `TextInput = string`
-
- A text input to the model.
-
- - `ResponseInputMessageContentList = array of ResponseInputContent`
-
- A list of one or many input items to the model, containing different content
- types.
-
- - `ResponseInputText = object { text, type }`
-
- A text input to the model.
-
- - `text: string`
-
- The text input to the model.
-
- - `type: "input_text"`
-
- The type of the input item. Always `input_text`.
-
- - `"input_text"`
-
- - `ResponseInputImage = object { detail, type, file_id, image_url }`
-
- An image input to the model. Learn about [image inputs](/docs/guides/vision).
-
- - `detail: "low" or "high" or "auto" or "original"`
-
- The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
-
- - `"low"`
-
- - `"high"`
-
- - `"auto"`
-
- - `"original"`
-
- - `type: "input_image"`
-
- The type of the input item. Always `input_image`.
-
- - `"input_image"`
-
- - `file_id: optional string`
-
- The ID of the file to be sent to the model.
-
- - `image_url: optional string`
-
- The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
-
- - `ResponseInputFile = object { type, detail, file_data, 3 more }`
-
- A file input to the model.
-
- - `type: "input_file"`
-
- The type of the input item. Always `input_file`.
-
- - `"input_file"`
-
- - `detail: optional "low" or "high"`
-
- The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
-
- - `"low"`
-
- - `"high"`
-
- - `file_data: optional string`
-
- The content of the file to be sent to the model.
-
- - `file_id: optional string`
-
- The ID of the file to be sent to the model.
-
- - `file_url: optional string`
-
- The URL of the file to be sent to the model.
-
- - `filename: optional string`
-
- The name of the file to be sent to the model.
-
- - `role: "user" or "assistant" or "system" or "developer"`
-
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
-
- - `"user"`
-
- - `"assistant"`
-
- - `"system"`
-
- - `"developer"`
-
- - `phase: optional "commentary" or "final_answer"`
-
- Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
- For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
- phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
-
- - `"commentary"`
-
- - `"final_answer"`
-
- - `type: optional "message"`
-
- The type of the message input. Always `message`.
+ A list of testing criteria.
- - `"message"`
+ - `LabelModelGrader object { input, labels, model, 3 more }`
- - `EvalMessageObject = object { content, role, type }`
+ A LabelModelGrader object which uses a model to assign labels to each item
+ in the evaluation.
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ - `input: array of object { content, role, type }`
- `content: string or ResponseInputText or object { text, type } or 3 more`
@@ -2917,7 +2558,7 @@ Get a list of runs for an evaluation.
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
@@ -2931,7 +2572,7 @@ Get a list of runs for an evaluation.
- `"input_text"`
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -2945,7 +2586,7 @@ Get a list of runs for an evaluation.
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -2963,7 +2604,7 @@ Get a list of runs for an evaluation.
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
@@ -2997,21 +2638,11 @@ Get a list of runs for an evaluation.
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
- - `text: string`
-
- The text input to the model.
-
- - `type: "input_text"`
-
- The type of the input item. Always `input_text`.
-
- - `"input_text"`
-
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -3025,7 +2656,7 @@ Get a list of runs for an evaluation.
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -3043,31 +2674,10 @@ Get a list of runs for an evaluation.
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
- - `input_audio: object { data, format }`
-
- - `data: string`
-
- Base64-encoded audio data.
-
- - `format: "mp3" or "wav"`
-
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
-
- - `"mp3"`
-
- - `"wav"`
-
- - `type: "input_audio"`
-
- The type of the input item. Always `input_audio`.
-
- - `"input_audio"`
-
- `role: "user" or "assistant" or "system" or "developer"`
The role of the message input. One of `user`, `assistant`, `system`, or
@@ -3087,361 +2697,255 @@ Get a list of runs for an evaluation.
- `"message"`
- - `type: "template"`
+ - `labels: array of string`
- The type of input messages. Always `template`.
+ The labels to assign to each item in the evaluation.
- - `"template"`
+ - `model: string`
- - `ItemReferenceInputMessages = object { item_reference, type }`
+ The model to use for the evaluation. Must support structured outputs.
- - `item_reference: string`
+ - `name: string`
- A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
+ The name of the grader.
- - `type: "item_reference"`
+ - `passing_labels: array of string`
- The type of input messages. Always `item_reference`.
+ The labels that indicate a passing result. Must be a subset of labels.
- - `"item_reference"`
+ - `type: "label_model"`
- - `model: optional string`
+ The object type, which is always `label_model`.
- The name of the model to use for generating completions (e.g. "o3-mini").
+ - `"label_model"`
- - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
+ - `StringCheckGrader object { input, name, operation, 2 more }`
- - `max_completion_tokens: optional number`
+ A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
- The maximum number of tokens in the generated output.
+ - `input: string`
- - `reasoning_effort: optional ReasoningEffort`
+ The input text. This may include template strings.
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ - `name: string`
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ The name of the grader.
- - `"none"`
+ - `operation: "eq" or "ne" or "like" or "ilike"`
- - `"minimal"`
+ The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
- - `"low"`
+ - `"eq"`
- - `"medium"`
+ - `"ne"`
- - `"high"`
+ - `"like"`
- - `"xhigh"`
+ - `"ilike"`
- - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
+ - `reference: string`
- An object specifying the format that the model must output.
+ The reference text. This may include template strings.
- Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
- Structured Outputs which ensures the model will match your supplied JSON
- schema. Learn more in the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ - `type: "string_check"`
- Setting to `{ "type": "json_object" }` enables the older JSON mode, which
- ensures the message the model generates is valid JSON. Using `json_schema`
- is preferred for models that support it.
+ The object type, which is always `string_check`.
- - `ResponseFormatText = object { type }`
+ - `"string_check"`
- Default response format. Used to generate text responses.
+ - `TextSimilarityGrader = TextSimilarityGrader`
- - `type: "text"`
+ A TextSimilarityGrader object which grades text based on similarity metrics.
- The type of response format being defined. Always `text`.
+ - `pass_threshold: number`
- - `"text"`
+ The threshold for the score.
- - `ResponseFormatJSONSchema = object { json_schema, type }`
+ - `PythonGrader = PythonGrader`
- JSON Schema response format. Used to generate structured JSON responses.
- Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+ A PythonGrader object that runs a python script on the input.
- - `json_schema: object { name, description, schema, strict }`
+ - `pass_threshold: optional number`
- Structured Outputs configuration options, including a JSON Schema.
+ The threshold for the score.
- - `name: string`
+ - `ScoreModelGrader = ScoreModelGrader`
- The name of the response format. Must be a-z, A-Z, 0-9, or contain
- underscores and dashes, with a maximum length of 64.
+ A ScoreModelGrader object that uses a model to assign a score to the input.
- - `description: optional string`
+ - `pass_threshold: optional number`
- A description of what the response format is for, used by the model to
- determine how to respond in the format.
+ The threshold for the score.
- - `schema: optional map[unknown]`
+### Eval Create Response
- The schema for the response format, described as a JSON Schema object.
- Learn how to build JSON schemas [here](https://json-schema.org/).
+- `EvalCreateResponse object { id, created_at, data_source_config, 4 more }`
- - `strict: optional boolean`
+ An Eval object with a data source config and testing criteria.
+ An Eval represents a task to be done for your LLM integration.
+ Like:
- Whether to enable strict schema adherence when generating the output.
- If set to true, the model will always follow the exact schema defined
- in the `schema` field. Only a subset of JSON Schema is supported when
- `strict` is `true`. To learn more, read the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ - Improve the quality of my chatbot
+ - See how well my chatbot handles customer support
+ - Check if o4-mini is better at my usecase than gpt-4o
- - `type: "json_schema"`
+ - `id: string`
- The type of response format being defined. Always `json_schema`.
+ Unique identifier for the evaluation.
- - `"json_schema"`
+ - `created_at: number`
- - `ResponseFormatJSONObject = object { type }`
+ The Unix timestamp (in seconds) for when the eval was created.
- JSON object response format. An older method of generating JSON responses.
- Using `json_schema` is recommended for models that support it. Note that the
- model will not generate JSON without a system or user message instructing it
- to do so.
+ - `data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig`
- - `type: "json_object"`
+ Configuration of data sources used in runs of the evaluation.
- The type of response format being defined. Always `json_object`.
+ - `EvalCustomDataSourceConfig object { schema, type }`
- - `"json_object"`
+ A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
+ The response schema defines the shape of the data that will be:
- - `seed: optional number`
+ - Used to define your testing criteria and
+ - What data is required when creating a run
- A seed value to initialize the randomness, during sampling.
+ - `schema: map[unknown]`
- - `temperature: optional number`
+ The json schema for the run data source items.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- A higher temperature increases randomness in the outputs.
+ - `type: "custom"`
- - `tools: optional array of ChatCompletionFunctionTool`
+ The type of data source. Always `custom`.
- A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
+ - `"custom"`
- - `function: FunctionDefinition`
+ - `LogsDataSourceConfig object { schema, type, metadata }`
- - `name: string`
+ A LogsDataSourceConfig which specifies the metadata property of your logs query.
+ This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
+ The schema returned by this data source config is used to defined what variables are available in your evals.
+ `item` and `sample` are both defined when using this data source config.
- The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
+ - `schema: map[unknown]`
- - `description: optional string`
+ The json schema for the run data source items.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- A description of what the function does, used by the model to choose when and how to call the function.
+ - `type: "logs"`
- - `parameters: optional FunctionParameters`
+ The type of data source. Always `logs`.
- The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
+ - `"logs"`
- Omitting `parameters` defines a function with an empty parameter list.
+ - `metadata: optional Metadata`
- - `strict: optional boolean`
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `type: "function"`
+ - `EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }`
- The type of the tool. Currently, only `function` is supported.
+ Deprecated in favor of LogsDataSourceConfig.
- - `"function"`
+ - `schema: map[unknown]`
- - `top_p: optional number`
+ The json schema for the run data source items.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+ - `type: "stored_completions"`
- - `ResponsesRunDataSource = object { source, type, input_messages, 2 more }`
+ The type of data source. Always `stored_completions`.
- A ResponsesRunDataSource object describing a model sampling configuration.
+ - `"stored_completions"`
- - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
+ - `metadata: optional Metadata`
- Determines what populates the `item` namespace in this run's data source.
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - `EvalJSONLFileContentSource = object { content, type }`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `content: array of object { item, sample }`
+ - `metadata: Metadata`
- The content of the jsonl file.
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - `item: map[unknown]`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `sample: optional map[unknown]`
+ - `name: string`
- - `type: "file_content"`
+ The name of the evaluation.
- The type of jsonl source. Always `file_content`.
+ - `object: "eval"`
- - `"file_content"`
+ The object type.
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `"eval"`
- - `id: string`
+ - `testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more`
- The identifier of the file.
+ A list of testing criteria.
- - `type: "file_id"`
+ - `LabelModelGrader object { input, labels, model, 3 more }`
- The type of jsonl source. Always `file_id`.
+ A LabelModelGrader object which uses a model to assign labels to each item
+ in the evaluation.
- - `"file_id"`
+ - `input: array of object { content, role, type }`
- - `EvalResponsesSource = object { type, created_after, created_before, 8 more }`
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- A EvalResponsesSource object describing a run data source configuration.
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- - `type: "responses"`
+ - `TextInput = string`
- The type of run data source. Always `responses`.
+ A text input to the model.
- - `"responses"`
+ - `ResponseInputText object { text, type }`
- - `created_after: optional number`
+ A text input to the model.
- Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
+ - `text: string`
- - `created_before: optional number`
+ The text input to the model.
- Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
+ - `type: "input_text"`
- - `instructions_search: optional string`
+ The type of the input item. Always `input_text`.
- Optional string to search the 'instructions' field. This is a query parameter used to select responses.
+ - `"input_text"`
- - `metadata: optional unknown`
+ - `OutputText object { text, type }`
- Metadata filter for the responses. This is a query parameter used to select responses.
+ A text output from the model.
- - `model: optional string`
+ - `text: string`
- The name of the model to find responses for. This is a query parameter used to select responses.
+ The text output from the model.
- - `reasoning_effort: optional ReasoningEffort`
+ - `type: "output_text"`
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ The type of the output text. Always `output_text`.
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ - `"output_text"`
- - `"none"`
+ - `InputImage object { image_url, type, detail }`
- - `"minimal"`
+ An image input block used within EvalItem content arrays.
- - `"low"`
+ - `image_url: string`
- - `"medium"`
+ The URL of the image input.
- - `"high"`
-
- - `"xhigh"`
-
- - `temperature: optional number`
-
- Sampling temperature. This is a query parameter used to select responses.
-
- - `tools: optional array of string`
-
- List of tool names. This is a query parameter used to select responses.
-
- - `top_p: optional number`
-
- Nucleus sampling parameter. This is a query parameter used to select responses.
-
- - `users: optional array of string`
-
- List of user identifiers. This is a query parameter used to select responses.
-
- - `type: "responses"`
-
- The type of run data source. Always `responses`.
-
- - `"responses"`
-
- - `input_messages: optional object { template, type } or object { item_reference, type }`
-
- Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
-
- - `InputMessagesTemplate = object { template, type }`
-
- - `template: array of object { content, role } or object { content, role, type }`
-
- A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
-
- - `ChatMessage = object { content, role }`
-
- - `content: string`
-
- The content of the message.
-
- - `role: string`
-
- The role of the message (e.g. "system", "assistant", "user").
-
- - `EvalMessageObject = object { content, role, type }`
-
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
-
- - `content: string or ResponseInputText or object { text, type } or 3 more`
-
- Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
-
- - `TextInput = string`
-
- A text input to the model.
-
- - `ResponseInputText = object { text, type }`
-
- A text input to the model.
-
- - `text: string`
-
- The text input to the model.
-
- - `type: "input_text"`
-
- The type of the input item. Always `input_text`.
-
- - `"input_text"`
-
- - `OutputText = object { text, type }`
-
- A text output from the model.
-
- - `text: string`
-
- The text output from the model.
-
- - `type: "output_text"`
-
- The type of the output text. Always `output_text`.
-
- - `"output_text"`
-
- - `InputImage = object { image_url, type, detail }`
-
- An image input block used within EvalItem content arrays.
-
- - `image_url: string`
-
- The URL of the image input.
-
- - `type: "input_image"`
+ - `type: "input_image"`
The type of the image input. Always `input_image`.
@@ -3451,7 +2955,7 @@ Get a list of runs for an evaluation.
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
@@ -3485,21 +2989,11 @@ Get a list of runs for an evaluation.
A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
- - `text: string`
-
- The text input to the model.
-
- - `type: "input_text"`
-
- The type of the input item. Always `input_text`.
-
- - `"input_text"`
-
- - `OutputText = object { text, type }`
+ - `OutputText object { text, type }`
A text output from the model.
@@ -3513,7 +3007,7 @@ Get a list of runs for an evaluation.
- `"output_text"`
- - `InputImage = object { image_url, type, detail }`
+ - `InputImage object { image_url, type, detail }`
An image input block used within EvalItem content arrays.
@@ -3531,31 +3025,10 @@ Get a list of runs for an evaluation.
The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ResponseInputAudio object { input_audio, type }`
An audio input to the model.
- - `input_audio: object { data, format }`
-
- - `data: string`
-
- Base64-encoded audio data.
-
- - `format: "mp3" or "wav"`
-
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
-
- - `"mp3"`
-
- - `"wav"`
-
- - `type: "input_audio"`
-
- The type of the input item. Always `input_audio`.
-
- - `"input_audio"`
-
- `role: "user" or "assistant" or "system" or "developer"`
The role of the message input. One of `user`, `assistant`, `system`, or
@@ -3575,1568 +3048,1441 @@ Get a list of runs for an evaluation.
- `"message"`
- - `type: "template"`
+ - `labels: array of string`
- The type of input messages. Always `template`.
+ The labels to assign to each item in the evaluation.
- - `"template"`
+ - `model: string`
- - `InputMessagesItemReference = object { item_reference, type }`
+ The model to use for the evaluation. Must support structured outputs.
- - `item_reference: string`
+ - `name: string`
- A reference to a variable in the `item` namespace. Ie, "item.name"
+ The name of the grader.
- - `type: "item_reference"`
+ - `passing_labels: array of string`
- The type of input messages. Always `item_reference`.
+ The labels that indicate a passing result. Must be a subset of labels.
- - `"item_reference"`
+ - `type: "label_model"`
- - `model: optional string`
+ The object type, which is always `label_model`.
- The name of the model to use for generating completions (e.g. "o3-mini").
+ - `"label_model"`
- - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
+ - `StringCheckGrader object { input, name, operation, 2 more }`
- - `max_completion_tokens: optional number`
+ A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
- The maximum number of tokens in the generated output.
+ - `input: string`
- - `reasoning_effort: optional ReasoningEffort`
+ The input text. This may include template strings.
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ - `name: string`
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ The name of the grader.
- - `"none"`
+ - `operation: "eq" or "ne" or "like" or "ilike"`
- - `"minimal"`
+ The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
- - `"low"`
+ - `"eq"`
- - `"medium"`
+ - `"ne"`
- - `"high"`
+ - `"like"`
- - `"xhigh"`
+ - `"ilike"`
- - `seed: optional number`
+ - `reference: string`
- A seed value to initialize the randomness, during sampling.
+ The reference text. This may include template strings.
- - `temperature: optional number`
+ - `type: "string_check"`
- A higher temperature increases randomness in the outputs.
+ The object type, which is always `string_check`.
- - `text: optional object { format }`
+ - `"string_check"`
- Configuration options for a text response from the model. Can be plain
- text or structured JSON data. Learn more:
+ - `TextSimilarityGrader = TextSimilarityGrader`
- - [Text inputs and outputs](/docs/guides/text)
- - [Structured Outputs](/docs/guides/structured-outputs)
+ A TextSimilarityGrader object which grades text based on similarity metrics.
- - `format: optional ResponseFormatTextConfig`
+ - `pass_threshold: number`
- An object specifying the format that the model must output.
+ The threshold for the score.
- Configuring `{ "type": "json_schema" }` enables Structured Outputs,
- which ensures the model will match your supplied JSON schema. Learn more in the
- [Structured Outputs guide](/docs/guides/structured-outputs).
+ - `PythonGrader = PythonGrader`
- The default format is `{ "type": "text" }` with no additional options.
+ A PythonGrader object that runs a python script on the input.
- **Not recommended for gpt-4o and newer models:**
+ - `pass_threshold: optional number`
- Setting to `{ "type": "json_object" }` enables the older JSON mode, which
- ensures the message the model generates is valid JSON. Using `json_schema`
- is preferred for models that support it.
+ The threshold for the score.
- - `ResponseFormatText = object { type }`
+ - `ScoreModelGrader = ScoreModelGrader`
- Default response format. Used to generate text responses.
+ A ScoreModelGrader object that uses a model to assign a score to the input.
- - `type: "text"`
+ - `pass_threshold: optional number`
- The type of response format being defined. Always `text`.
+ The threshold for the score.
- - `"text"`
+### Eval Retrieve Response
- - `ResponseFormatTextJSONSchemaConfig = object { name, schema, type, 2 more }`
+- `EvalRetrieveResponse object { id, created_at, data_source_config, 4 more }`
- JSON Schema response format. Used to generate structured JSON responses.
- Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+ An Eval object with a data source config and testing criteria.
+ An Eval represents a task to be done for your LLM integration.
+ Like:
- - `name: string`
+ - Improve the quality of my chatbot
+ - See how well my chatbot handles customer support
+ - Check if o4-mini is better at my usecase than gpt-4o
- The name of the response format. Must be a-z, A-Z, 0-9, or contain
- underscores and dashes, with a maximum length of 64.
+ - `id: string`
- - `schema: map[unknown]`
+ Unique identifier for the evaluation.
- The schema for the response format, described as a JSON Schema object.
- Learn how to build JSON schemas [here](https://json-schema.org/).
+ - `created_at: number`
- - `type: "json_schema"`
+ The Unix timestamp (in seconds) for when the eval was created.
- The type of response format being defined. Always `json_schema`.
+ - `data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig`
- - `"json_schema"`
+ Configuration of data sources used in runs of the evaluation.
- - `description: optional string`
+ - `EvalCustomDataSourceConfig object { schema, type }`
- A description of what the response format is for, used by the model to
- determine how to respond in the format.
+ A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
+ The response schema defines the shape of the data that will be:
- - `strict: optional boolean`
+ - Used to define your testing criteria and
+ - What data is required when creating a run
- Whether to enable strict schema adherence when generating the output.
- If set to true, the model will always follow the exact schema defined
- in the `schema` field. Only a subset of JSON Schema is supported when
- `strict` is `true`. To learn more, read the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ - `schema: map[unknown]`
- - `ResponseFormatJSONObject = object { type }`
+ The json schema for the run data source items.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- JSON object response format. An older method of generating JSON responses.
- Using `json_schema` is recommended for models that support it. Note that the
- model will not generate JSON without a system or user message instructing it
- to do so.
+ - `type: "custom"`
- - `type: "json_object"`
+ The type of data source. Always `custom`.
- The type of response format being defined. Always `json_object`.
+ - `"custom"`
- - `"json_object"`
+ - `LogsDataSourceConfig object { schema, type, metadata }`
- - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
+ A LogsDataSourceConfig which specifies the metadata property of your logs query.
+ This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
+ The schema returned by this data source config is used to defined what variables are available in your evals.
+ `item` and `sample` are both defined when using this data source config.
- An array of tools the model may call while generating a response. You
- can specify which tool to use by setting the `tool_choice` parameter.
+ - `schema: map[unknown]`
- The two categories of tools you can provide the model are:
+ The json schema for the run data source items.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- - **Built-in tools**: Tools that are provided by OpenAI that extend the
- model's capabilities, like [web search](/docs/guides/tools-web-search)
- or [file search](/docs/guides/tools-file-search). Learn more about
- [built-in tools](/docs/guides/tools).
- - **Function calls (custom tools)**: Functions that are defined by you,
- enabling the model to call your own code. Learn more about
- [function calling](/docs/guides/function-calling).
+ - `type: "logs"`
- - `Function = object { name, parameters, strict, 3 more }`
+ The type of data source. Always `logs`.
- Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
+ - `"logs"`
- - `name: string`
+ - `metadata: optional Metadata`
- The name of the function to call.
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - `parameters: map[unknown]`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- A JSON schema object describing the parameters of the function.
+ - `EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }`
- - `strict: boolean`
+ Deprecated in favor of LogsDataSourceConfig.
- Whether to enforce strict parameter validation. Default `true`.
+ - `schema: map[unknown]`
- - `type: "function"`
+ The json schema for the run data source items.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- The type of the function tool. Always `function`.
+ - `type: "stored_completions"`
- - `"function"`
+ The type of data source. Always `stored_completions`.
- - `defer_loading: optional boolean`
+ - `"stored_completions"`
- Whether this function is deferred and loaded via tool search.
+ - `metadata: optional Metadata`
- - `description: optional string`
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- A description of the function. Used by the model to determine whether or not to call the function.
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `FileSearch = object { type, vector_store_ids, filters, 2 more }`
+ - `metadata: Metadata`
- A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - `type: "file_search"`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- The type of the file search tool. Always `file_search`.
+ - `name: string`
- - `"file_search"`
+ The name of the evaluation.
- - `vector_store_ids: array of string`
+ - `object: "eval"`
- The IDs of the vector stores to search.
+ The object type.
- - `filters: optional ComparisonFilter or CompoundFilter`
+ - `"eval"`
- A filter to apply.
+ - `testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more`
- - `ComparisonFilter = object { key, type, value }`
+ A list of testing criteria.
- A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+ - `LabelModelGrader object { input, labels, model, 3 more }`
- - `key: string`
+ A LabelModelGrader object which uses a model to assign labels to each item
+ in the evaluation.
- The key to compare against the value.
+ - `input: array of object { content, role, type }`
- - `type: "eq" or "ne" or "gt" or 5 more`
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- - `eq`: equals
- - `ne`: not equal
- - `gt`: greater than
- - `gte`: greater than or equal
- - `lt`: less than
- - `lte`: less than or equal
- - `in`: in
- - `nin`: not in
+ - `TextInput = string`
- - `"eq"`
+ A text input to the model.
- - `"ne"`
+ - `ResponseInputText object { text, type }`
- - `"gt"`
+ A text input to the model.
- - `"gte"`
+ - `text: string`
- - `"lt"`
+ The text input to the model.
- - `"lte"`
+ - `type: "input_text"`
- - `"in"`
+ The type of the input item. Always `input_text`.
- - `"nin"`
+ - `"input_text"`
- - `value: string or number or boolean or array of string or number`
+ - `OutputText object { text, type }`
- The value to compare against the attribute key; supports string, number, or boolean types.
+ A text output from the model.
- - `string`
+ - `text: string`
- - `number`
+ The text output from the model.
- - `boolean`
+ - `type: "output_text"`
- - `array of string or number`
+ The type of the output text. Always `output_text`.
- - `string`
+ - `"output_text"`
- - `number`
+ - `InputImage object { image_url, type, detail }`
- - `CompoundFilter = object { filters, type }`
+ An image input block used within EvalItem content arrays.
- Combine multiple filters using `and` or `or`.
+ - `image_url: string`
- - `filters: array of ComparisonFilter or unknown`
+ The URL of the image input.
- Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
+ - `type: "input_image"`
- - `ComparisonFilter = object { key, type, value }`
+ The type of the image input. Always `input_image`.
- A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+ - `"input_image"`
- - `key: string`
+ - `detail: optional string`
- The key to compare against the value.
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `type: "eq" or "ne" or "gt" or 5 more`
+ - `ResponseInputAudio object { input_audio, type }`
- Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+ An audio input to the model.
- - `eq`: equals
- - `ne`: not equal
- - `gt`: greater than
- - `gte`: greater than or equal
- - `lt`: less than
- - `lte`: less than or equal
- - `in`: in
- - `nin`: not in
+ - `input_audio: object { data, format }`
- - `"eq"`
+ - `data: string`
- - `"ne"`
+ Base64-encoded audio data.
- - `"gt"`
+ - `format: "mp3" or "wav"`
- - `"gte"`
+ The format of the audio data. Currently supported formats are `mp3` and
+ `wav`.
- - `"lt"`
+ - `"mp3"`
- - `"lte"`
+ - `"wav"`
- - `"in"`
+ - `type: "input_audio"`
- - `"nin"`
+ The type of the input item. Always `input_audio`.
- - `value: string or number or boolean or array of string or number`
+ - `"input_audio"`
- The value to compare against the attribute key; supports string, number, or boolean types.
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
- - `string`
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
- - `number`
+ - `TextInput = string`
- - `boolean`
+ A text input to the model.
- - `array of string or number`
+ - `ResponseInputText object { text, type }`
- - `string`
+ A text input to the model.
- - `number`
+ - `OutputText object { text, type }`
- - `unknown`
+ A text output from the model.
- - `type: "and" or "or"`
+ - `text: string`
- Type of operation: `and` or `or`.
+ The text output from the model.
- - `"and"`
+ - `type: "output_text"`
- - `"or"`
+ The type of the output text. Always `output_text`.
- - `max_num_results: optional number`
+ - `"output_text"`
- The maximum number of results to return. This number should be between 1 and 50 inclusive.
+ - `InputImage object { image_url, type, detail }`
- - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
+ An image input block used within EvalItem content arrays.
- Ranking options for search.
+ - `image_url: string`
- - `hybrid_search: optional object { embedding_weight, text_weight }`
+ The URL of the image input.
- Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
+ - `type: "input_image"`
- - `embedding_weight: number`
+ The type of the image input. Always `input_image`.
- The weight of the embedding in the reciprocal ranking fusion.
+ - `"input_image"`
- - `text_weight: number`
+ - `detail: optional string`
- The weight of the text in the reciprocal ranking fusion.
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `ranker: optional "auto" or "default-2024-11-15"`
+ - `ResponseInputAudio object { input_audio, type }`
- The ranker to use for the file search.
+ An audio input to the model.
- - `"auto"`
+ - `role: "user" or "assistant" or "system" or "developer"`
- - `"default-2024-11-15"`
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- - `score_threshold: optional number`
+ - `"user"`
- The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
+ - `"assistant"`
- - `Computer = object { type }`
+ - `"system"`
- A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+ - `"developer"`
- - `type: "computer"`
+ - `type: optional "message"`
- The type of the computer tool. Always `computer`.
+ The type of the message input. Always `message`.
- - `"computer"`
+ - `"message"`
- - `ComputerUsePreview = object { display_height, display_width, environment, type }`
+ - `labels: array of string`
- A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+ The labels to assign to each item in the evaluation.
- - `display_height: number`
+ - `model: string`
- The height of the computer display.
+ The model to use for the evaluation. Must support structured outputs.
- - `display_width: number`
+ - `name: string`
- The width of the computer display.
+ The name of the grader.
- - `environment: "windows" or "mac" or "linux" or 2 more`
+ - `passing_labels: array of string`
- The type of computer environment to control.
+ The labels that indicate a passing result. Must be a subset of labels.
- - `"windows"`
+ - `type: "label_model"`
- - `"mac"`
+ The object type, which is always `label_model`.
- - `"linux"`
+ - `"label_model"`
- - `"ubuntu"`
+ - `StringCheckGrader object { input, name, operation, 2 more }`
- - `"browser"`
+ A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
- - `type: "computer_use_preview"`
+ - `input: string`
- The type of the computer use tool. Always `computer_use_preview`.
+ The input text. This may include template strings.
- - `"computer_use_preview"`
+ - `name: string`
- - `WebSearch = object { type, filters, search_context_size, user_location }`
+ The name of the grader.
- Search the Internet for sources related to the prompt. Learn more about the
- [web search tool](/docs/guides/tools-web-search).
+ - `operation: "eq" or "ne" or "like" or "ilike"`
- - `type: "web_search" or "web_search_2025_08_26"`
+ The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
- The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
+ - `"eq"`
- - `"web_search"`
+ - `"ne"`
- - `"web_search_2025_08_26"`
+ - `"like"`
- - `filters: optional object { allowed_domains }`
+ - `"ilike"`
- Filters for the search.
+ - `reference: string`
- - `allowed_domains: optional array of string`
+ The reference text. This may include template strings.
- Allowed domains for the search. If not provided, all domains are allowed.
- Subdomains of the provided domains are allowed as well.
+ - `type: "string_check"`
- Example: `["pubmed.ncbi.nlm.nih.gov"]`
+ The object type, which is always `string_check`.
- - `search_context_size: optional "low" or "medium" or "high"`
+ - `"string_check"`
- High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+ - `TextSimilarityGrader = TextSimilarityGrader`
- - `"low"`
+ A TextSimilarityGrader object which grades text based on similarity metrics.
- - `"medium"`
+ - `pass_threshold: number`
- - `"high"`
+ The threshold for the score.
- - `user_location: optional object { city, country, region, 2 more }`
+ - `PythonGrader = PythonGrader`
- The approximate location of the user.
+ A PythonGrader object that runs a python script on the input.
- - `city: optional string`
+ - `pass_threshold: optional number`
- Free text input for the city of the user, e.g. `San Francisco`.
+ The threshold for the score.
- - `country: optional string`
+ - `ScoreModelGrader = ScoreModelGrader`
- The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+ A ScoreModelGrader object that uses a model to assign a score to the input.
- - `region: optional string`
+ - `pass_threshold: optional number`
- Free text input for the region of the user, e.g. `California`.
+ The threshold for the score.
- - `timezone: optional string`
+### Eval Update Response
- The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+- `EvalUpdateResponse object { id, created_at, data_source_config, 4 more }`
- - `type: optional "approximate"`
+ An Eval object with a data source config and testing criteria.
+ An Eval represents a task to be done for your LLM integration.
+ Like:
- The type of location approximation. Always `approximate`.
+ - Improve the quality of my chatbot
+ - See how well my chatbot handles customer support
+ - Check if o4-mini is better at my usecase than gpt-4o
- - `"approximate"`
+ - `id: string`
- - `Mcp = object { server_label, type, allowed_tools, 7 more }`
+ Unique identifier for the evaluation.
- Give the model access to additional tools via remote Model Context Protocol
- (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
+ - `created_at: number`
- - `server_label: string`
+ The Unix timestamp (in seconds) for when the eval was created.
- A label for this MCP server, used to identify it in tool calls.
+ - `data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig`
- - `type: "mcp"`
+ Configuration of data sources used in runs of the evaluation.
- The type of the MCP tool. Always `mcp`.
+ - `EvalCustomDataSourceConfig object { schema, type }`
- - `"mcp"`
+ A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
+ The response schema defines the shape of the data that will be:
- - `allowed_tools: optional array of string or object { read_only, tool_names }`
+ - Used to define your testing criteria and
+ - What data is required when creating a run
- List of allowed tool names or a filter object.
+ - `schema: map[unknown]`
- - `McpAllowedTools = array of string`
+ The json schema for the run data source items.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- A string array of allowed tool names
+ - `type: "custom"`
- - `McpToolFilter = object { read_only, tool_names }`
+ The type of data source. Always `custom`.
- A filter object to specify which tools are allowed.
+ - `"custom"`
- - `read_only: optional boolean`
+ - `LogsDataSourceConfig object { schema, type, metadata }`
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ A LogsDataSourceConfig which specifies the metadata property of your logs query.
+ This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
+ The schema returned by this data source config is used to defined what variables are available in your evals.
+ `item` and `sample` are both defined when using this data source config.
- - `tool_names: optional array of string`
+ - `schema: map[unknown]`
- List of allowed tool names.
+ The json schema for the run data source items.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- - `authorization: optional string`
+ - `type: "logs"`
- An OAuth access token that can be used with a remote MCP server, either
- with a custom MCP server URL or a service connector. Your application
- must handle the OAuth authorization flow and provide the token here.
+ The type of data source. Always `logs`.
- - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
+ - `"logs"`
- Identifier for service connectors, like those available in ChatGPT. One of
- `server_url` or `connector_id` must be provided. Learn more about service
- connectors [here](/docs/guides/tools-remote-mcp#connectors).
+ - `metadata: optional Metadata`
- Currently supported `connector_id` values are:
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - Dropbox: `connector_dropbox`
- - Gmail: `connector_gmail`
- - Google Calendar: `connector_googlecalendar`
- - Google Drive: `connector_googledrive`
- - Microsoft Teams: `connector_microsoftteams`
- - Outlook Calendar: `connector_outlookcalendar`
- - Outlook Email: `connector_outlookemail`
- - SharePoint: `connector_sharepoint`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `"connector_dropbox"`
+ - `EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }`
- - `"connector_gmail"`
+ Deprecated in favor of LogsDataSourceConfig.
- - `"connector_googlecalendar"`
+ - `schema: map[unknown]`
- - `"connector_googledrive"`
+ The json schema for the run data source items.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- - `"connector_microsoftteams"`
+ - `type: "stored_completions"`
- - `"connector_outlookcalendar"`
+ The type of data source. Always `stored_completions`.
- - `"connector_outlookemail"`
+ - `"stored_completions"`
- - `"connector_sharepoint"`
+ - `metadata: optional Metadata`
- - `defer_loading: optional boolean`
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- Whether this MCP tool is deferred and discovered via tool search.
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `headers: optional map[string]`
+ - `metadata: Metadata`
- Optional HTTP headers to send to the MCP server. Use for authentication
- or other purposes.
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - `require_approval: optional object { always, never } or "always" or "never"`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- Specify which of the MCP server's tools require approval.
+ - `name: string`
- - `McpToolApprovalFilter = object { always, never }`
+ The name of the evaluation.
- Specify which of the MCP server's tools require approval. Can be
- `always`, `never`, or a filter object associated with tools
- that require approval.
+ - `object: "eval"`
- - `always: optional object { read_only, tool_names }`
+ The object type.
- A filter object to specify which tools are allowed.
+ - `"eval"`
- - `read_only: optional boolean`
+ - `testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more`
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ A list of testing criteria.
- - `tool_names: optional array of string`
+ - `LabelModelGrader object { input, labels, model, 3 more }`
- List of allowed tool names.
+ A LabelModelGrader object which uses a model to assign labels to each item
+ in the evaluation.
- - `never: optional object { read_only, tool_names }`
+ - `input: array of object { content, role, type }`
- A filter object to specify which tools are allowed.
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- - `read_only: optional boolean`
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ - `TextInput = string`
- - `tool_names: optional array of string`
+ A text input to the model.
- List of allowed tool names.
+ - `ResponseInputText object { text, type }`
- - `McpToolApprovalSetting = "always" or "never"`
+ A text input to the model.
- Specify a single approval policy for all tools. One of `always` or
- `never`. When set to `always`, all tools will require approval. When
- set to `never`, all tools will not require approval.
+ - `text: string`
- - `"always"`
+ The text input to the model.
- - `"never"`
+ - `type: "input_text"`
- - `server_description: optional string`
+ The type of the input item. Always `input_text`.
- Optional description of the MCP server, used to provide more context.
+ - `"input_text"`
- - `server_url: optional string`
+ - `OutputText object { text, type }`
- The URL for the MCP server. One of `server_url` or `connector_id` must be
- provided.
+ A text output from the model.
- - `CodeInterpreter = object { container, type }`
+ - `text: string`
- A tool that runs Python code to help generate a response to a prompt.
+ The text output from the model.
- - `container: string or object { type, file_ids, memory_limit, network_policy }`
+ - `type: "output_text"`
- The code interpreter container. Can be a container ID or an object that
- specifies uploaded file IDs to make available to your code, along with an
- optional `memory_limit` setting.
+ The type of the output text. Always `output_text`.
- - `string`
+ - `"output_text"`
- The container ID.
+ - `InputImage object { image_url, type, detail }`
- - `CodeInterpreterToolAuto = object { type, file_ids, memory_limit, network_policy }`
+ An image input block used within EvalItem content arrays.
- Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
+ - `image_url: string`
- - `type: "auto"`
+ The URL of the image input.
- Always `auto`.
+ - `type: "input_image"`
- - `"auto"`
+ The type of the image input. Always `input_image`.
- - `file_ids: optional array of string`
+ - `"input_image"`
- An optional list of uploaded files to make available to your code.
+ - `detail: optional string`
- - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- The memory limit for the code interpreter container.
+ - `ResponseInputAudio object { input_audio, type }`
- - `"1g"`
+ An audio input to the model.
- - `"4g"`
+ - `input_audio: object { data, format }`
- - `"16g"`
+ - `data: string`
- - `"64g"`
+ Base64-encoded audio data.
- - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+ - `format: "mp3" or "wav"`
- Network access policy for the container.
+ The format of the audio data. Currently supported formats are `mp3` and
+ `wav`.
- - `ContainerNetworkPolicyDisabled = object { type }`
+ - `"mp3"`
- - `type: "disabled"`
+ - `"wav"`
- Disable outbound network access. Always `disabled`.
+ - `type: "input_audio"`
- - `"disabled"`
+ The type of the input item. Always `input_audio`.
- - `ContainerNetworkPolicyAllowlist = object { allowed_domains, type, domain_secrets }`
+ - `"input_audio"`
- - `allowed_domains: array of string`
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
- A list of allowed domains when type is `allowlist`.
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
- - `type: "allowlist"`
+ - `TextInput = string`
- Allow outbound network access only to specified domains. Always `allowlist`.
+ A text input to the model.
- - `"allowlist"`
+ - `ResponseInputText object { text, type }`
- - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+ A text input to the model.
- Optional domain-scoped secrets for allowlisted domains.
+ - `OutputText object { text, type }`
- - `domain: string`
+ A text output from the model.
- The domain associated with the secret.
+ - `text: string`
- - `name: string`
+ The text output from the model.
- The name of the secret to inject for the domain.
+ - `type: "output_text"`
- - `value: string`
+ The type of the output text. Always `output_text`.
- The secret value to inject for the domain.
+ - `"output_text"`
- - `type: "code_interpreter"`
+ - `InputImage object { image_url, type, detail }`
- The type of the code interpreter tool. Always `code_interpreter`.
+ An image input block used within EvalItem content arrays.
- - `"code_interpreter"`
+ - `image_url: string`
- - `ImageGeneration = object { type, action, background, 9 more }`
+ The URL of the image input.
- A tool that generates images using the GPT image models.
+ - `type: "input_image"`
- - `type: "image_generation"`
+ The type of the image input. Always `input_image`.
- The type of the image generation tool. Always `image_generation`.
+ - `"input_image"`
- - `"image_generation"`
+ - `detail: optional string`
- - `action: optional "generate" or "edit" or "auto"`
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- Whether to generate a new image or edit an existing image. Default: `auto`.
+ - `ResponseInputAudio object { input_audio, type }`
- - `"generate"`
+ An audio input to the model.
- - `"edit"`
+ - `role: "user" or "assistant" or "system" or "developer"`
- - `"auto"`
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- - `background: optional "transparent" or "opaque" or "auto"`
+ - `"user"`
- Background type for the generated image. One of `transparent`,
- `opaque`, or `auto`. Default: `auto`.
+ - `"assistant"`
- - `"transparent"`
+ - `"system"`
- - `"opaque"`
+ - `"developer"`
- - `"auto"`
+ - `type: optional "message"`
- - `input_fidelity: optional "high" or "low"`
+ The type of the message input. Always `message`.
- Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
+ - `"message"`
- - `"high"`
+ - `labels: array of string`
- - `"low"`
+ The labels to assign to each item in the evaluation.
- - `input_image_mask: optional object { file_id, image_url }`
+ - `model: string`
- Optional mask for inpainting. Contains `image_url`
- (string, optional) and `file_id` (string, optional).
+ The model to use for the evaluation. Must support structured outputs.
- - `file_id: optional string`
+ - `name: string`
- File ID for the mask image.
+ The name of the grader.
- - `image_url: optional string`
+ - `passing_labels: array of string`
- Base64-encoded mask image.
+ The labels that indicate a passing result. Must be a subset of labels.
- - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+ - `type: "label_model"`
- The image generation model to use. Default: `gpt-image-1`.
+ The object type, which is always `label_model`.
- - `string`
+ - `"label_model"`
- - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+ - `StringCheckGrader object { input, name, operation, 2 more }`
- The image generation model to use. Default: `gpt-image-1`.
+ A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
- - `"gpt-image-1"`
+ - `input: string`
- - `"gpt-image-1-mini"`
+ The input text. This may include template strings.
- - `"gpt-image-1.5"`
+ - `name: string`
- - `moderation: optional "auto" or "low"`
+ The name of the grader.
- Moderation level for the generated image. Default: `auto`.
+ - `operation: "eq" or "ne" or "like" or "ilike"`
- - `"auto"`
+ The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
- - `"low"`
+ - `"eq"`
- - `output_compression: optional number`
+ - `"ne"`
- Compression level for the output image. Default: 100.
+ - `"like"`
- - `output_format: optional "png" or "webp" or "jpeg"`
+ - `"ilike"`
- The output format of the generated image. One of `png`, `webp`, or
- `jpeg`. Default: `png`.
+ - `reference: string`
- - `"png"`
+ The reference text. This may include template strings.
- - `"webp"`
+ - `type: "string_check"`
- - `"jpeg"`
+ The object type, which is always `string_check`.
- - `partial_images: optional number`
+ - `"string_check"`
- Number of partial images to generate in streaming mode, from 0 (default value) to 3.
+ - `TextSimilarityGrader = TextSimilarityGrader`
- - `quality: optional "low" or "medium" or "high" or "auto"`
+ A TextSimilarityGrader object which grades text based on similarity metrics.
- The quality of the generated image. One of `low`, `medium`, `high`,
- or `auto`. Default: `auto`.
+ - `pass_threshold: number`
- - `"low"`
+ The threshold for the score.
- - `"medium"`
+ - `PythonGrader = PythonGrader`
- - `"high"`
+ A PythonGrader object that runs a python script on the input.
- - `"auto"`
+ - `pass_threshold: optional number`
- - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
+ The threshold for the score.
- The size of the generated image. One of `1024x1024`, `1024x1536`,
- `1536x1024`, or `auto`. Default: `auto`.
+ - `ScoreModelGrader = ScoreModelGrader`
- - `"1024x1024"`
+ A ScoreModelGrader object that uses a model to assign a score to the input.
- - `"1024x1536"`
+ - `pass_threshold: optional number`
- - `"1536x1024"`
+ The threshold for the score.
- - `"auto"`
+### Eval Delete Response
- - `LocalShell = object { type }`
+- `EvalDeleteResponse object { deleted, eval_id, object }`
- A tool that allows the model to execute shell commands in a local environment.
+ - `deleted: boolean`
- - `type: "local_shell"`
+ - `eval_id: string`
- The type of the local shell tool. Always `local_shell`.
+ - `object: string`
- - `"local_shell"`
+# Runs
- - `Shell = object { type, environment }`
+## Get eval runs
- A tool that allows the model to execute shell commands.
+**get** `/evals/{eval_id}/runs`
- - `type: "shell"`
+Get a list of runs for an evaluation.
- The type of the shell tool. Always `shell`.
+### Path Parameters
- - `"shell"`
+- `eval_id: string`
- - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
+### Query Parameters
- - `ContainerAuto = object { type, file_ids, memory_limit, 2 more }`
+- `after: optional string`
- - `type: "container_auto"`
+ Identifier for the last run from the previous pagination request.
- Automatically creates a container for this request
+- `limit: optional number`
- - `"container_auto"`
+ Number of runs to retrieve.
- - `file_ids: optional array of string`
+- `order: optional "asc" or "desc"`
- An optional list of uploaded files to make available to your code.
+ Sort order for runs by timestamp. Use `asc` for ascending order or `desc` for descending order. Defaults to `asc`.
- - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+ - `"asc"`
- The memory limit for the container.
+ - `"desc"`
- - `"1g"`
+- `status: optional "queued" or "in_progress" or "completed" or 2 more`
- - `"4g"`
+ Filter runs by status. One of `queued` | `in_progress` | `failed` | `completed` | `canceled`.
- - `"16g"`
+ - `"queued"`
- - `"64g"`
+ - `"in_progress"`
- - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+ - `"completed"`
- Network access policy for the container.
+ - `"canceled"`
- - `ContainerNetworkPolicyDisabled = object { type }`
+ - `"failed"`
- - `type: "disabled"`
+### Returns
- Disable outbound network access. Always `disabled`.
+- `data: array of object { id, created_at, data_source, 11 more }`
- - `"disabled"`
+ An array of eval run objects.
- - `ContainerNetworkPolicyAllowlist = object { allowed_domains, type, domain_secrets }`
+ - `id: string`
- - `allowed_domains: array of string`
+ Unique identifier for the evaluation run.
- A list of allowed domains when type is `allowlist`.
+ - `created_at: number`
- - `type: "allowlist"`
+ Unix timestamp (in seconds) when the evaluation run was created.
- Allow outbound network access only to specified domains. Always `allowlist`.
+ - `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
- - `"allowlist"`
+ Information about the run's data source.
- - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+ - `CreateEvalJSONLRunDataSource object { source, type }`
- Optional domain-scoped secrets for allowlisted domains.
+ A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
- - `domain: string`
+ - `source: object { content, type } or object { id, type }`
- The domain associated with the secret.
+ Determines what populates the `item` namespace in the data source.
- - `name: string`
+ - `EvalJSONLFileContentSource object { content, type }`
- The name of the secret to inject for the domain.
+ - `content: array of object { item, sample }`
- - `value: string`
+ The content of the jsonl file.
- The secret value to inject for the domain.
+ - `item: map[unknown]`
- - `skills: optional array of SkillReference or InlineSkill`
+ - `sample: optional map[unknown]`
- An optional list of skills referenced by id or inline data.
+ - `type: "file_content"`
- - `SkillReference = object { skill_id, type, version }`
+ The type of jsonl source. Always `file_content`.
- - `skill_id: string`
+ - `"file_content"`
- The ID of the referenced skill.
+ - `EvalJSONLFileIDSource object { id, type }`
- - `type: "skill_reference"`
+ - `id: string`
- References a skill created with the /v1/skills endpoint.
+ The identifier of the file.
- - `"skill_reference"`
+ - `type: "file_id"`
- - `version: optional string`
+ The type of jsonl source. Always `file_id`.
- Optional skill version. Use a positive integer or 'latest'. Omit for default.
+ - `"file_id"`
- - `InlineSkill = object { description, name, source, type }`
+ - `type: "jsonl"`
- - `description: string`
+ The type of data source. Always `jsonl`.
- The description of the skill.
+ - `"jsonl"`
- - `name: string`
+ - `CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }`
- The name of the skill.
+ A CompletionsRunDataSource object describing a model sampling configuration.
- - `source: InlineSkillSource`
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
- Inline skill payload
+ Determines what populates the `item` namespace in this run's data source.
- - `data: string`
+ - `EvalJSONLFileContentSource object { content, type }`
- Base64-encoded skill zip bundle.
+ - `content: array of object { item, sample }`
- - `media_type: "application/zip"`
+ The content of the jsonl file.
- The media type of the inline skill payload. Must be `application/zip`.
+ - `item: map[unknown]`
- - `"application/zip"`
+ - `sample: optional map[unknown]`
- - `type: "base64"`
+ - `type: "file_content"`
- The type of the inline skill source. Must be `base64`.
+ The type of jsonl source. Always `file_content`.
- - `"base64"`
+ - `"file_content"`
- - `type: "inline"`
+ - `EvalJSONLFileIDSource object { id, type }`
- Defines an inline skill for this request.
+ - `id: string`
- - `"inline"`
+ The identifier of the file.
- - `LocalEnvironment = object { type, skills }`
+ - `type: "file_id"`
- - `type: "local"`
+ The type of jsonl source. Always `file_id`.
- Use a local computer environment.
+ - `"file_id"`
- - `"local"`
+ - `StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more }`
- - `skills: optional array of LocalSkill`
+ A StoredCompletionsRunDataSource configuration describing a set of filters
- An optional list of skills.
+ - `type: "stored_completions"`
- - `description: string`
+ The type of source. Always `stored_completions`.
- The description of the skill.
+ - `"stored_completions"`
- - `name: string`
+ - `created_after: optional number`
- The name of the skill.
+ An optional Unix timestamp to filter items created after this time.
- - `path: string`
+ - `created_before: optional number`
- The path to the directory containing the skill.
+ An optional Unix timestamp to filter items created before this time.
- - `ContainerReference = object { container_id, type }`
+ - `limit: optional number`
- - `container_id: string`
+ An optional maximum number of items to return.
- The ID of the referenced container.
+ - `metadata: optional Metadata`
- - `type: "container_reference"`
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- References a container created with the /v1/containers endpoint
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `"container_reference"`
+ - `model: optional string`
- - `Custom = object { name, type, defer_loading, 2 more }`
+ An optional model to filter by (e.g., 'gpt-4o').
- A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+ - `type: "completions"`
- - `name: string`
+ The type of run data source. Always `completions`.
- The name of the custom tool, used to identify it in tool calls.
+ - `"completions"`
- - `type: "custom"`
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
- The type of the custom tool. Always `custom`.
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
- - `"custom"`
+ - `TemplateInputMessages object { template, type }`
- - `defer_loading: optional boolean`
+ - `template: array of EasyInputMessage or object { content, role, type }`
- Whether this tool should be deferred and discovered via tool search.
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
- - `description: optional string`
+ - `EasyInputMessage object { content, role, phase, type }`
- Optional description of the custom tool, used to provide more context.
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- - `format: optional CustomToolInputFormat`
+ - `content: string or ResponseInputMessageContentList`
- The input format for the custom tool. Default is unconstrained text.
+ Text, image, or audio input to the model, used to generate a response.
+ Can also contain previous assistant responses.
- - `Text = object { type }`
+ - `TextInput = string`
- Unconstrained free-form text.
+ A text input to the model.
- - `type: "text"`
+ - `ResponseInputMessageContentList = array of ResponseInputContent`
- Unconstrained text format. Always `text`.
+ A list of one or many input items to the model, containing different content
+ types.
- - `"text"`
+ - `ResponseInputText object { text, type }`
- - `Grammar = object { definition, syntax, type }`
+ A text input to the model.
- A grammar defined by the user.
+ - `text: string`
- - `definition: string`
+ The text input to the model.
- The grammar definition.
+ - `type: "input_text"`
- - `syntax: "lark" or "regex"`
+ The type of the input item. Always `input_text`.
- The syntax of the grammar definition. One of `lark` or `regex`.
+ - `"input_text"`
- - `"lark"`
+ - `ResponseInputImage object { detail, type, file_id, image_url }`
- - `"regex"`
+ An image input to the model. Learn about [image inputs](/docs/guides/vision).
- - `type: "grammar"`
+ - `detail: "low" or "high" or "auto" or "original"`
- Grammar format. Always `grammar`.
+ The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
- - `"grammar"`
+ - `"low"`
- - `Namespace = object { description, name, tools, type }`
+ - `"high"`
- Groups function/custom tools under a shared namespace.
+ - `"auto"`
- - `description: string`
+ - `"original"`
- A description of the namespace shown to the model.
+ - `type: "input_image"`
- - `name: string`
+ The type of the input item. Always `input_image`.
- The namespace name used in tool calls (for example, `crm`).
+ - `"input_image"`
- - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
+ - `file_id: optional string`
- The function/custom tools available inside this namespace.
+ The ID of the file to be sent to the model.
- - `Function = object { name, type, defer_loading, 3 more }`
+ - `image_url: optional string`
- - `name: string`
+ The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
- - `type: "function"`
+ - `ResponseInputFile object { type, detail, file_data, 3 more }`
- - `"function"`
+ A file input to the model.
- - `defer_loading: optional boolean`
+ - `type: "input_file"`
- Whether this function should be deferred and discovered via tool search.
+ The type of the input item. Always `input_file`.
- - `description: optional string`
+ - `"input_file"`
- - `parameters: optional unknown`
+ - `detail: optional "low" or "high"`
- - `strict: optional boolean`
+ The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
- - `Custom = object { name, type, defer_loading, 2 more }`
+ - `"low"`
- A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+ - `"high"`
- - `name: string`
+ - `file_data: optional string`
- The name of the custom tool, used to identify it in tool calls.
+ The content of the file to be sent to the model.
- - `type: "custom"`
+ - `file_id: optional string`
- The type of the custom tool. Always `custom`.
+ The ID of the file to be sent to the model.
- - `"custom"`
+ - `file_url: optional string`
- - `defer_loading: optional boolean`
+ The URL of the file to be sent to the model.
- Whether this tool should be deferred and discovered via tool search.
+ - `filename: optional string`
- - `description: optional string`
+ The name of the file to be sent to the model.
- Optional description of the custom tool, used to provide more context.
+ - `role: "user" or "assistant" or "system" or "developer"`
- - `format: optional CustomToolInputFormat`
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- The input format for the custom tool. Default is unconstrained text.
+ - `"user"`
- - `Text = object { type }`
+ - `"assistant"`
- Unconstrained free-form text.
+ - `"system"`
- - `type: "text"`
+ - `"developer"`
- Unconstrained text format. Always `text`.
+ - `phase: optional "commentary" or "final_answer"`
- - `"text"`
+ Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
+ For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
+ phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
- - `Grammar = object { definition, syntax, type }`
+ - `"commentary"`
- A grammar defined by the user.
+ - `"final_answer"`
- - `definition: string`
+ - `type: optional "message"`
- The grammar definition.
+ The type of the message input. Always `message`.
- - `syntax: "lark" or "regex"`
+ - `"message"`
- The syntax of the grammar definition. One of `lark` or `regex`.
+ - `EvalMessageObject object { content, role, type }`
- - `"lark"`
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- - `"regex"`
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- - `type: "grammar"`
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- Grammar format. Always `grammar`.
+ - `TextInput = string`
- - `"grammar"`
+ A text input to the model.
- - `type: "namespace"`
+ - `ResponseInputText object { text, type }`
- The type of the tool. Always `namespace`.
+ A text input to the model.
- - `"namespace"`
+ - `OutputText object { text, type }`
- - `ToolSearch = object { type, description, execution, parameters }`
+ A text output from the model.
- Hosted or BYOT tool search configuration for deferred tools.
+ - `text: string`
- - `type: "tool_search"`
+ The text output from the model.
- The type of the tool. Always `tool_search`.
+ - `type: "output_text"`
- - `"tool_search"`
+ The type of the output text. Always `output_text`.
- - `description: optional string`
+ - `"output_text"`
- Description shown to the model for a client-executed tool search tool.
+ - `InputImage object { image_url, type, detail }`
- - `execution: optional "server" or "client"`
+ An image input block used within EvalItem content arrays.
- Whether tool search is executed by the server or by the client.
+ - `image_url: string`
- - `"server"`
+ The URL of the image input.
- - `"client"`
+ - `type: "input_image"`
- - `parameters: optional unknown`
+ The type of the image input. Always `input_image`.
- Parameter schema for a client-executed tool search tool.
+ - `"input_image"`
- - `WebSearchPreview = object { type, search_content_types, search_context_size, user_location }`
+ - `detail: optional string`
- This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
+ - `ResponseInputAudio object { input_audio, type }`
- The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
+ An audio input to the model.
- - `"web_search_preview"`
+ - `input_audio: object { data, format }`
- - `"web_search_preview_2025_03_11"`
+ - `data: string`
- - `search_content_types: optional array of "text" or "image"`
+ Base64-encoded audio data.
- - `"text"`
+ - `format: "mp3" or "wav"`
- - `"image"`
+ The format of the audio data. Currently supported formats are `mp3` and
+ `wav`.
- - `search_context_size: optional "low" or "medium" or "high"`
+ - `"mp3"`
- High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+ - `"wav"`
- - `"low"`
+ - `type: "input_audio"`
- - `"medium"`
+ The type of the input item. Always `input_audio`.
- - `"high"`
+ - `"input_audio"`
- - `user_location: optional object { type, city, country, 2 more }`
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
- The user's location.
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
- - `type: "approximate"`
+ - `TextInput = string`
- The type of location approximation. Always `approximate`.
+ A text input to the model.
- - `"approximate"`
+ - `ResponseInputText object { text, type }`
- - `city: optional string`
+ A text input to the model.
- Free text input for the city of the user, e.g. `San Francisco`.
+ - `OutputText object { text, type }`
- - `country: optional string`
+ A text output from the model.
- The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+ - `text: string`
- - `region: optional string`
+ The text output from the model.
- Free text input for the region of the user, e.g. `California`.
+ - `type: "output_text"`
- - `timezone: optional string`
+ The type of the output text. Always `output_text`.
- The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+ - `"output_text"`
- - `ApplyPatch = object { type }`
+ - `InputImage object { image_url, type, detail }`
- Allows the assistant to create, delete, or update files using unified diffs.
+ An image input block used within EvalItem content arrays.
- - `type: "apply_patch"`
+ - `image_url: string`
- The type of the tool. Always `apply_patch`.
+ The URL of the image input.
- - `"apply_patch"`
+ - `type: "input_image"`
- - `top_p: optional number`
+ The type of the image input. Always `input_image`.
- An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+ - `"input_image"`
- - `error: EvalAPIError`
+ - `detail: optional string`
- An object representing an error response from the Eval API.
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `code: string`
+ - `ResponseInputAudio object { input_audio, type }`
- The error code.
+ An audio input to the model.
- - `message: string`
+ - `role: "user" or "assistant" or "system" or "developer"`
- The error message.
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- - `eval_id: string`
+ - `"user"`
- The identifier of the associated evaluation.
+ - `"assistant"`
- - `metadata: Metadata`
+ - `"system"`
- Set of 16 key-value pairs that can be attached to an object. This can be
- useful for storing additional information about the object in a structured
- format, and querying for objects via API or the dashboard.
+ - `"developer"`
- Keys are strings with a maximum length of 64 characters. Values are strings
- with a maximum length of 512 characters.
+ - `type: optional "message"`
- - `model: string`
+ The type of the message input. Always `message`.
- The model that is evaluated, if applicable.
+ - `"message"`
- - `name: string`
+ - `type: "template"`
- The name of the evaluation run.
+ The type of input messages. Always `template`.
- - `object: "eval.run"`
+ - `"template"`
- The type of the object. Always "eval.run".
+ - `ItemReferenceInputMessages object { item_reference, type }`
- - `"eval.run"`
+ - `item_reference: string`
- - `per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }`
+ A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
- Usage statistics for each model during the evaluation run.
+ - `type: "item_reference"`
- - `cached_tokens: number`
+ The type of input messages. Always `item_reference`.
- The number of tokens retrieved from cache.
+ - `"item_reference"`
- - `completion_tokens: number`
+ - `model: optional string`
- The number of completion tokens generated.
+ The name of the model to use for generating completions (e.g. "o3-mini").
- - `invocation_count: number`
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
- The number of invocations.
+ - `max_completion_tokens: optional number`
- - `model_name: string`
+ The maximum number of tokens in the generated output.
- The name of the model.
+ - `reasoning_effort: optional ReasoningEffort`
- - `prompt_tokens: number`
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- The number of prompt tokens used.
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- - `total_tokens: number`
+ - `"none"`
- The total number of tokens used.
+ - `"minimal"`
- - `per_testing_criteria_results: array of object { failed, passed, testing_criteria }`
+ - `"low"`
- Results per testing criteria applied during the evaluation run.
+ - `"medium"`
- - `failed: number`
+ - `"high"`
- Number of tests failed for this criteria.
+ - `"xhigh"`
- - `passed: number`
+ - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
- Number of tests passed for this criteria.
+ An object specifying the format that the model must output.
- - `testing_criteria: string`
+ Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
+ Structured Outputs which ensures the model will match your supplied JSON
+ schema. Learn more in the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- A description of the testing criteria.
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
- - `report_url: string`
+ - `ResponseFormatText object { type }`
- The URL to the rendered evaluation run report on the UI dashboard.
+ Default response format. Used to generate text responses.
- - `result_counts: object { errored, failed, passed, total }`
+ - `type: "text"`
- Counters summarizing the outcomes of the evaluation run.
+ The type of response format being defined. Always `text`.
- - `errored: number`
+ - `"text"`
- Number of output items that resulted in an error.
+ - `ResponseFormatJSONSchema object { json_schema, type }`
- - `failed: number`
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
- Number of output items that failed to pass the evaluation.
+ - `json_schema: object { name, description, schema, strict }`
- - `passed: number`
+ Structured Outputs configuration options, including a JSON Schema.
- Number of output items that passed the evaluation.
+ - `name: string`
- - `total: number`
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
- Total number of executed output items.
+ - `description: optional string`
- - `status: string`
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
- The status of the evaluation run.
+ - `schema: optional map[unknown]`
-- `first_id: string`
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- The identifier of the first eval run in the data array.
+ - `strict: optional boolean`
-- `has_more: boolean`
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- Indicates whether there are more evals available.
+ - `type: "json_schema"`
-- `last_id: string`
+ The type of response format being defined. Always `json_schema`.
- The identifier of the last eval run in the data array.
+ - `"json_schema"`
-- `object: "list"`
+ - `ResponseFormatJSONObject object { type }`
- The type of this object. It is always set to "list".
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
- - `"list"`
+ - `type: "json_object"`
-### Example
+ The type of response format being defined. Always `json_object`.
-```http
-curl https://api.openai.com/v1/evals/$EVAL_ID/runs \
- -H "Authorization: Bearer $OPENAI_API_KEY"
-```
+ - `"json_object"`
-#### Response
+ - `seed: optional number`
-```json
-{
- "data": [
- {
- "id": "id",
- "created_at": 0,
- "data_source": {
- "source": {
- "content": [
- {
- "item": {
- "foo": "bar"
- },
- "sample": {
- "foo": "bar"
- }
- }
- ],
- "type": "file_content"
- },
- "type": "jsonl"
- },
- "error": {
- "code": "code",
- "message": "message"
- },
- "eval_id": "eval_id",
- "metadata": {
- "foo": "string"
- },
- "model": "model",
- "name": "name",
- "object": "eval.run",
- "per_model_usage": [
- {
- "cached_tokens": 0,
- "completion_tokens": 0,
- "invocation_count": 0,
- "model_name": "model_name",
- "prompt_tokens": 0,
- "total_tokens": 0
- }
- ],
- "per_testing_criteria_results": [
- {
- "failed": 0,
- "passed": 0,
- "testing_criteria": "testing_criteria"
- }
- ],
- "report_url": "report_url",
- "result_counts": {
- "errored": 0,
- "failed": 0,
- "passed": 0,
- "total": 0
- },
- "status": "status"
- }
- ],
- "first_id": "first_id",
- "has_more": true,
- "last_id": "last_id",
- "object": "list"
-}
-```
+ A seed value to initialize the randomness, during sampling.
-### Example
+ - `temperature: optional number`
-```http
-curl https://api.openai.com/v1/evals/egroup_67abd54d9b0081909a86353f6fb9317a/runs \
- -H "Authorization: Bearer $OPENAI_API_KEY" \
- -H "Content-Type: application/json"
-```
+ A higher temperature increases randomness in the outputs.
-#### Response
+ - `tools: optional array of ChatCompletionFunctionTool`
-```json
-{
- "object": "list",
- "data": [
- {
- "object": "eval.run",
- "id": "evalrun_67e0c7d31560819090d60c0780591042",
- "eval_id": "eval_67e0c726d560819083f19a957c4c640b",
- "report_url": "https://platform.openai.com/evaluations/eval_67e0c726d560819083f19a957c4c640b",
- "status": "completed",
- "model": "o3-mini",
- "name": "bulk_with_negative_examples_o3-mini",
- "created_at": 1742784467,
- "result_counts": {
- "total": 1,
- "errored": 0,
- "failed": 0,
- "passed": 1
- },
- "per_model_usage": [
- {
- "model_name": "o3-mini",
- "invocation_count": 1,
- "prompt_tokens": 563,
- "completion_tokens": 874,
- "total_tokens": 1437,
- "cached_tokens": 0
- }
- ],
- "per_testing_criteria_results": [
- {
- "testing_criteria": "Push Notification Summary Grader-1808cd0b-eeec-4e0b-a519-337e79f4f5d1",
- "passed": 1,
- "failed": 0
- }
- ],
- "data_source": {
- "type": "completions",
- "source": {
- "type": "file_content",
- "content": [
- {
- "item": {
- "notifications": "\n- New message from Sarah: \"Can you call me later?\"\n- Your package has been delivered!\n- Flash sale: 20% off electronics for the next 2 hours!\n"
- }
- }
- ]
- },
- "input_messages": {
- "type": "template",
- "template": [
- {
- "type": "message",
- "role": "developer",
- "content": {
- "type": "input_text",
- "text": "\n\n\n\nYou are a helpful assistant that takes in an array of push notifications and returns a collapsed summary of them.\nThe push notification will be provided as follows:\n\n...notificationlist...\n\n\nYou should return just the summary and nothing else.\n\n\nYou should return a summary that is concise and snappy.\n\n\nHere is an example of a good summary:\n\n- Traffic alert: Accident reported on Main Street.- Package out for delivery: Expected by 5 PM.- New friend suggestion: Connect with Emma.\n\n\nTraffic alert, package expected by 5pm, suggestion for new friend (Emily).\n\n\n\nHere is an example of a bad summary:\n\n- Traffic alert: Accident reported on Main Street.- Package out for delivery: Expected by 5 PM.- New friend suggestion: Connect with Emma.\n\n\nTraffic alert reported on main street. You have a package that will arrive by 5pm, Emily is a new friend suggested for you.\n\n"
- }
- },
- {
- "type": "message",
- "role": "user",
- "content": {
- "type": "input_text",
- "text": "{{item.notifications}}"
- }
- }
- ]
- },
- "model": "o3-mini",
- "sampling_params": null
- },
- "error": null,
- "metadata": {}
- }
- ],
- "first_id": "evalrun_67e0c7d31560819090d60c0780591042",
- "last_id": "evalrun_67e0c7d31560819090d60c0780591042",
- "has_more": true
-}
-```
+ A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
-## Create eval run
+ - `function: FunctionDefinition`
-**post** `/evals/{eval_id}/runs`
+ - `name: string`
-Kicks off a new run for a given evaluation, specifying the data source, and what model configuration to use to test. The datasource will be validated against the schema specified in the config of the evaluation.
+ The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
-### Path Parameters
+ - `description: optional string`
-- `eval_id: string`
+ A description of what the function does, used by the model to choose when and how to call the function.
-### Body Parameters
+ - `parameters: optional FunctionParameters`
-- `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
+ The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
- Details about the run's data source.
+ Omitting `parameters` defines a function with an empty parameter list.
+
+ - `strict: optional boolean`
- - `CreateEvalJSONLRunDataSource = object { source, type }`
+ Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
- A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
+ - `type: "function"`
- - `source: object { content, type } or object { id, type }`
+ The type of the tool. Currently, only `function` is supported.
- Determines what populates the `item` namespace in the data source.
+ - `"function"`
- - `EvalJSONLFileContentSource = object { content, type }`
+ - `top_p: optional number`
+
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+
+ - `ResponsesRunDataSource object { source, type, input_messages, 2 more }`
+
+ A ResponsesRunDataSource object describing a model sampling configuration.
+
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
+
+ Determines what populates the `item` namespace in this run's data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
- `content: array of object { item, sample }`
@@ -5152,7 +4498,7 @@ Kicks off a new run for a given evaluation, specifying the data source, and what
- `"file_content"`
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `EvalJSONLFileIDSource object { id, type }`
- `id: string`
@@ -5164,100 +4510,92 @@ Kicks off a new run for a given evaluation, specifying the data source, and what
- `"file_id"`
- - `type: "jsonl"`
-
- The type of data source. Always `jsonl`.
-
- - `"jsonl"`
-
- - `CreateEvalCompletionsRunDataSource = object { source, type, input_messages, 2 more }`
-
- A CompletionsRunDataSource object describing a model sampling configuration.
-
- - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
-
- Determines what populates the `item` namespace in this run's data source.
+ - `EvalResponsesSource object { type, created_after, created_before, 8 more }`
- - `EvalJSONLFileContentSource = object { content, type }`
+ A EvalResponsesSource object describing a run data source configuration.
- - `content: array of object { item, sample }`
+ - `type: "responses"`
- The content of the jsonl file.
+ The type of run data source. Always `responses`.
- - `item: map[unknown]`
+ - `"responses"`
- - `sample: optional map[unknown]`
+ - `created_after: optional number`
- - `type: "file_content"`
+ Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
- The type of jsonl source. Always `file_content`.
+ - `created_before: optional number`
- - `"file_content"`
+ Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `instructions_search: optional string`
- - `id: string`
+ Optional string to search the 'instructions' field. This is a query parameter used to select responses.
- The identifier of the file.
+ - `metadata: optional unknown`
- - `type: "file_id"`
+ Metadata filter for the responses. This is a query parameter used to select responses.
- The type of jsonl source. Always `file_id`.
+ - `model: optional string`
- - `"file_id"`
+ The name of the model to find responses for. This is a query parameter used to select responses.
- - `StoredCompletionsRunDataSource = object { type, created_after, created_before, 3 more }`
+ - `reasoning_effort: optional ReasoningEffort`
- A StoredCompletionsRunDataSource configuration describing a set of filters
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- - `type: "stored_completions"`
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- The type of source. Always `stored_completions`.
+ - `temperature: optional number`
- - `"stored_completions"`
+ Sampling temperature. This is a query parameter used to select responses.
- - `created_after: optional number`
+ - `tools: optional array of string`
- An optional Unix timestamp to filter items created after this time.
+ List of tool names. This is a query parameter used to select responses.
- - `created_before: optional number`
+ - `top_p: optional number`
- An optional Unix timestamp to filter items created before this time.
+ Nucleus sampling parameter. This is a query parameter used to select responses.
- - `limit: optional number`
+ - `users: optional array of string`
- An optional maximum number of items to return.
+ List of user identifiers. This is a query parameter used to select responses.
- - `metadata: optional Metadata`
+ - `type: "responses"`
- Set of 16 key-value pairs that can be attached to an object. This can be
- useful for storing additional information about the object in a structured
- format, and querying for objects via API or the dashboard.
+ The type of run data source. Always `responses`.
- Keys are strings with a maximum length of 64 characters. Values are strings
- with a maximum length of 512 characters.
+ - `"responses"`
- - `model: optional string`
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
- An optional model to filter by (e.g., 'gpt-4o').
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
- - `type: "completions"`
+ - `InputMessagesTemplate object { template, type }`
- The type of run data source. Always `completions`.
+ - `template: array of object { content, role } or object { content, role, type }`
- - `"completions"`
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
- - `input_messages: optional object { template, type } or object { item_reference, type }`
+ - `ChatMessage object { content, role }`
- Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+ - `content: string`
- - `TemplateInputMessages = object { template, type }`
+ The content of the message.
- - `template: array of EasyInputMessage or object { content, role, type }`
+ - `role: string`
- A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+ The role of the message (e.g. "system", "assistant", "user").
- - `EasyInputMessage = object { content, role, phase, type }`
+ - `EvalMessageObject object { content, role, type }`
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the `developer` or `system` role take
@@ -5265,97 +4603,58 @@ Kicks off a new run for a given evaluation, specifying the data source, and what
`assistant` role are presumed to have been generated by the model in previous
interactions.
- - `content: string or ResponseInputMessageContentList`
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- Text, image, or audio input to the model, used to generate a response.
- Can also contain previous assistant responses.
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- `TextInput = string`
A text input to the model.
- - `ResponseInputMessageContentList = array of ResponseInputContent`
+ - `ResponseInputText object { text, type }`
- A list of one or many input items to the model, containing different content
- types.
+ A text input to the model.
- - `ResponseInputText = object { text, type }`
+ - `OutputText object { text, type }`
- A text input to the model.
+ A text output from the model.
- `text: string`
- The text input to the model.
+ The text output from the model.
- - `type: "input_text"`
+ - `type: "output_text"`
- The type of the input item. Always `input_text`.
+ The type of the output text. Always `output_text`.
- - `"input_text"`
+ - `"output_text"`
- - `ResponseInputImage = object { detail, type, file_id, image_url }`
+ - `InputImage object { image_url, type, detail }`
- An image input to the model. Learn about [image inputs](/docs/guides/vision).
+ An image input block used within EvalItem content arrays.
- - `detail: "low" or "high" or "auto" or "original"`
+ - `image_url: string`
- The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
-
- - `"low"`
-
- - `"high"`
-
- - `"auto"`
-
- - `"original"`
+ The URL of the image input.
- `type: "input_image"`
- The type of the input item. Always `input_image`.
+ The type of the image input. Always `input_image`.
- `"input_image"`
- - `file_id: optional string`
-
- The ID of the file to be sent to the model.
-
- - `image_url: optional string`
-
- The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
-
- - `ResponseInputFile = object { type, detail, file_data, 3 more }`
-
- A file input to the model.
-
- - `type: "input_file"`
-
- The type of the input item. Always `input_file`.
-
- - `"input_file"`
-
- - `detail: optional "low" or "high"`
-
- The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
-
- - `"low"`
-
- - `"high"`
-
- - `file_data: optional string`
-
- The content of the file to be sent to the model.
-
- - `file_id: optional string`
+ - `detail: optional string`
- The ID of the file to be sent to the model.
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `file_url: optional string`
+ - `ResponseInputAudio object { input_audio, type }`
- The URL of the file to be sent to the model.
+ An audio input to the model.
- - `filename: optional string`
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
- The name of the file to be sent to the model.
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
- `role: "user" or "assistant" or "system" or "developer"`
@@ -5370,1122 +4669,1064 @@ Kicks off a new run for a given evaluation, specifying the data source, and what
- `"developer"`
- - `phase: optional "commentary" or "final_answer"`
-
- Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
- For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
- phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
-
- - `"commentary"`
-
- - `"final_answer"`
-
- `type: optional "message"`
The type of the message input. Always `message`.
- `"message"`
- - `EvalMessageObject = object { content, role, type }`
+ - `type: "template"`
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ The type of input messages. Always `template`.
- - `content: string or ResponseInputText or object { text, type } or 3 more`
+ - `"template"`
- Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+ - `InputMessagesItemReference object { item_reference, type }`
- - `TextInput = string`
+ - `item_reference: string`
- A text input to the model.
+ A reference to a variable in the `item` namespace. Ie, "item.name"
- - `ResponseInputText = object { text, type }`
+ - `type: "item_reference"`
- A text input to the model.
+ The type of input messages. Always `item_reference`.
- - `text: string`
+ - `"item_reference"`
- The text input to the model.
+ - `model: optional string`
- - `type: "input_text"`
+ The name of the model to use for generating completions (e.g. "o3-mini").
- The type of the input item. Always `input_text`.
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
- - `"input_text"`
+ - `max_completion_tokens: optional number`
- - `OutputText = object { text, type }`
+ The maximum number of tokens in the generated output.
- A text output from the model.
+ - `reasoning_effort: optional ReasoningEffort`
- - `text: string`
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- The text output from the model.
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- - `type: "output_text"`
+ - `seed: optional number`
- The type of the output text. Always `output_text`.
+ A seed value to initialize the randomness, during sampling.
- - `"output_text"`
+ - `temperature: optional number`
- - `InputImage = object { image_url, type, detail }`
+ A higher temperature increases randomness in the outputs.
- An image input block used within EvalItem content arrays.
+ - `text: optional object { format }`
- - `image_url: string`
+ Configuration options for a text response from the model. Can be plain
+ text or structured JSON data. Learn more:
- The URL of the image input.
+ - [Text inputs and outputs](/docs/guides/text)
+ - [Structured Outputs](/docs/guides/structured-outputs)
- - `type: "input_image"`
+ - `format: optional ResponseFormatTextConfig`
- The type of the image input. Always `input_image`.
+ An object specifying the format that the model must output.
- - `"input_image"`
+ Configuring `{ "type": "json_schema" }` enables Structured Outputs,
+ which ensures the model will match your supplied JSON schema. Learn more in the
+ [Structured Outputs guide](/docs/guides/structured-outputs).
- - `detail: optional string`
+ The default format is `{ "type": "text" }` with no additional options.
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ **Not recommended for gpt-4o and newer models:**
- - `ResponseInputAudio = object { input_audio, type }`
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
- An audio input to the model.
+ - `ResponseFormatText object { type }`
- - `input_audio: object { data, format }`
+ Default response format. Used to generate text responses.
- - `data: string`
+ - `ResponseFormatTextJSONSchemaConfig object { name, schema, type, 2 more }`
- Base64-encoded audio data.
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
- - `format: "mp3" or "wav"`
+ - `name: string`
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
- - `"mp3"`
+ - `schema: map[unknown]`
- - `"wav"`
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- - `type: "input_audio"`
+ - `type: "json_schema"`
- The type of the input item. Always `input_audio`.
+ The type of response format being defined. Always `json_schema`.
- - `"input_audio"`
+ - `"json_schema"`
- - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+ - `description: optional string`
- A list of inputs, each of which may be either an input text, output text, input
- image, or input audio object.
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
- - `TextInput = string`
+ - `strict: optional boolean`
- A text input to the model.
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- - `ResponseInputText = object { text, type }`
+ - `ResponseFormatJSONObject object { type }`
- A text input to the model.
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
- - `text: string`
+ - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
- The text input to the model.
+ An array of tools the model may call while generating a response. You
+ can specify which tool to use by setting the `tool_choice` parameter.
- - `type: "input_text"`
+ The two categories of tools you can provide the model are:
- The type of the input item. Always `input_text`.
+ - **Built-in tools**: Tools that are provided by OpenAI that extend the
+ model's capabilities, like [web search](/docs/guides/tools-web-search)
+ or [file search](/docs/guides/tools-file-search). Learn more about
+ [built-in tools](/docs/guides/tools).
+ - **Function calls (custom tools)**: Functions that are defined by you,
+ enabling the model to call your own code. Learn more about
+ [function calling](/docs/guides/function-calling).
- - `"input_text"`
+ - `Function object { name, parameters, strict, 3 more }`
- - `OutputText = object { text, type }`
+ Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
- A text output from the model.
+ - `name: string`
- - `text: string`
+ The name of the function to call.
- The text output from the model.
+ - `parameters: map[unknown]`
- - `type: "output_text"`
+ A JSON schema object describing the parameters of the function.
- The type of the output text. Always `output_text`.
+ - `strict: boolean`
- - `"output_text"`
+ Whether to enforce strict parameter validation. Default `true`.
- - `InputImage = object { image_url, type, detail }`
+ - `type: "function"`
- An image input block used within EvalItem content arrays.
+ The type of the function tool. Always `function`.
- - `image_url: string`
+ - `"function"`
- The URL of the image input.
+ - `defer_loading: optional boolean`
- - `type: "input_image"`
+ Whether this function is deferred and loaded via tool search.
- The type of the image input. Always `input_image`.
+ - `description: optional string`
- - `"input_image"`
+ A description of the function. Used by the model to determine whether or not to call the function.
- - `detail: optional string`
+ - `FileSearch object { type, vector_store_ids, filters, 2 more }`
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
- - `ResponseInputAudio = object { input_audio, type }`
+ - `type: "file_search"`
- An audio input to the model.
+ The type of the file search tool. Always `file_search`.
- - `input_audio: object { data, format }`
+ - `"file_search"`
- - `data: string`
+ - `vector_store_ids: array of string`
- Base64-encoded audio data.
+ The IDs of the vector stores to search.
- - `format: "mp3" or "wav"`
+ - `filters: optional ComparisonFilter or CompoundFilter`
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ A filter to apply.
- - `"mp3"`
+ - `ComparisonFilter object { key, type, value }`
- - `"wav"`
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
- - `type: "input_audio"`
+ - `key: string`
- The type of the input item. Always `input_audio`.
+ The key to compare against the value.
- - `"input_audio"`
+ - `type: "eq" or "ne" or "gt" or 5 more`
- - `role: "user" or "assistant" or "system" or "developer"`
+ Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
+ - `eq`: equals
+ - `ne`: not equal
+ - `gt`: greater than
+ - `gte`: greater than or equal
+ - `lt`: less than
+ - `lte`: less than or equal
+ - `in`: in
+ - `nin`: not in
- - `"user"`
-
- - `"assistant"`
+ - `"eq"`
- - `"system"`
+ - `"ne"`
- - `"developer"`
+ - `"gt"`
- - `type: optional "message"`
+ - `"gte"`
- The type of the message input. Always `message`.
+ - `"lt"`
- - `"message"`
+ - `"lte"`
- - `type: "template"`
+ - `"in"`
- The type of input messages. Always `template`.
+ - `"nin"`
- - `"template"`
+ - `value: string or number or boolean or array of string or number`
- - `ItemReferenceInputMessages = object { item_reference, type }`
+ The value to compare against the attribute key; supports string, number, or boolean types.
- - `item_reference: string`
+ - `string`
- A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
+ - `number`
- - `type: "item_reference"`
+ - `boolean`
- The type of input messages. Always `item_reference`.
+ - `array of string or number`
- - `"item_reference"`
+ - `string`
- - `model: optional string`
+ - `number`
- The name of the model to use for generating completions (e.g. "o3-mini").
+ - `CompoundFilter object { filters, type }`
- - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
+ Combine multiple filters using `and` or `or`.
- - `max_completion_tokens: optional number`
+ - `filters: array of ComparisonFilter or unknown`
- The maximum number of tokens in the generated output.
+ Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
- - `reasoning_effort: optional ReasoningEffort`
+ - `ComparisonFilter object { key, type, value }`
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ - `unknown`
- - `"none"`
+ - `type: "and" or "or"`
- - `"minimal"`
+ Type of operation: `and` or `or`.
- - `"low"`
+ - `"and"`
- - `"medium"`
+ - `"or"`
- - `"high"`
+ - `max_num_results: optional number`
- - `"xhigh"`
+ The maximum number of results to return. This number should be between 1 and 50 inclusive.
- - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
+ - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
- An object specifying the format that the model must output.
+ Ranking options for search.
- Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
- Structured Outputs which ensures the model will match your supplied JSON
- schema. Learn more in the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ - `hybrid_search: optional object { embedding_weight, text_weight }`
- Setting to `{ "type": "json_object" }` enables the older JSON mode, which
- ensures the message the model generates is valid JSON. Using `json_schema`
- is preferred for models that support it.
+ Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
- - `ResponseFormatText = object { type }`
+ - `embedding_weight: number`
- Default response format. Used to generate text responses.
+ The weight of the embedding in the reciprocal ranking fusion.
- - `type: "text"`
+ - `text_weight: number`
- The type of response format being defined. Always `text`.
+ The weight of the text in the reciprocal ranking fusion.
- - `"text"`
+ - `ranker: optional "auto" or "default-2024-11-15"`
- - `ResponseFormatJSONSchema = object { json_schema, type }`
+ The ranker to use for the file search.
- JSON Schema response format. Used to generate structured JSON responses.
- Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+ - `"auto"`
- - `json_schema: object { name, description, schema, strict }`
+ - `"default-2024-11-15"`
- Structured Outputs configuration options, including a JSON Schema.
+ - `score_threshold: optional number`
- - `name: string`
+ The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
- The name of the response format. Must be a-z, A-Z, 0-9, or contain
- underscores and dashes, with a maximum length of 64.
+ - `Computer object { type }`
- - `description: optional string`
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
- A description of what the response format is for, used by the model to
- determine how to respond in the format.
+ - `type: "computer"`
- - `schema: optional map[unknown]`
+ The type of the computer tool. Always `computer`.
- The schema for the response format, described as a JSON Schema object.
- Learn how to build JSON schemas [here](https://json-schema.org/).
+ - `"computer"`
- - `strict: optional boolean`
+ - `ComputerUsePreview object { display_height, display_width, environment, type }`
- Whether to enable strict schema adherence when generating the output.
- If set to true, the model will always follow the exact schema defined
- in the `schema` field. Only a subset of JSON Schema is supported when
- `strict` is `true`. To learn more, read the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
- - `type: "json_schema"`
+ - `display_height: number`
- The type of response format being defined. Always `json_schema`.
+ The height of the computer display.
- - `"json_schema"`
+ - `display_width: number`
- - `ResponseFormatJSONObject = object { type }`
+ The width of the computer display.
- JSON object response format. An older method of generating JSON responses.
- Using `json_schema` is recommended for models that support it. Note that the
- model will not generate JSON without a system or user message instructing it
- to do so.
+ - `environment: "windows" or "mac" or "linux" or 2 more`
- - `type: "json_object"`
+ The type of computer environment to control.
- The type of response format being defined. Always `json_object`.
+ - `"windows"`
- - `"json_object"`
+ - `"mac"`
- - `seed: optional number`
+ - `"linux"`
- A seed value to initialize the randomness, during sampling.
+ - `"ubuntu"`
- - `temperature: optional number`
+ - `"browser"`
- A higher temperature increases randomness in the outputs.
+ - `type: "computer_use_preview"`
- - `tools: optional array of ChatCompletionFunctionTool`
+ The type of the computer use tool. Always `computer_use_preview`.
- A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
+ - `"computer_use_preview"`
- - `function: FunctionDefinition`
+ - `WebSearch object { type, filters, search_context_size, user_location }`
- - `name: string`
+ Search the Internet for sources related to the prompt. Learn more about the
+ [web search tool](/docs/guides/tools-web-search).
- The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
+ - `type: "web_search" or "web_search_2025_08_26"`
- - `description: optional string`
+ The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
- A description of what the function does, used by the model to choose when and how to call the function.
+ - `"web_search"`
- - `parameters: optional FunctionParameters`
+ - `"web_search_2025_08_26"`
- The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
+ - `filters: optional object { allowed_domains }`
- Omitting `parameters` defines a function with an empty parameter list.
+ Filters for the search.
- - `strict: optional boolean`
+ - `allowed_domains: optional array of string`
- Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
+ Allowed domains for the search. If not provided, all domains are allowed.
+ Subdomains of the provided domains are allowed as well.
- - `type: "function"`
+ Example: `["pubmed.ncbi.nlm.nih.gov"]`
- The type of the tool. Currently, only `function` is supported.
+ - `search_context_size: optional "low" or "medium" or "high"`
- - `"function"`
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
- - `top_p: optional number`
+ - `"low"`
- An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+ - `"medium"`
- - `ResponsesRunDataSource = object { source, type, input_messages, 2 more }`
+ - `"high"`
- A ResponsesRunDataSource object describing a model sampling configuration.
+ - `user_location: optional object { city, country, region, 2 more }`
- - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
+ The approximate location of the user.
- Determines what populates the `item` namespace in this run's data source.
+ - `city: optional string`
- - `EvalJSONLFileContentSource = object { content, type }`
+ Free text input for the city of the user, e.g. `San Francisco`.
- - `content: array of object { item, sample }`
+ - `country: optional string`
- The content of the jsonl file.
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
- - `item: map[unknown]`
+ - `region: optional string`
- - `sample: optional map[unknown]`
+ Free text input for the region of the user, e.g. `California`.
- - `type: "file_content"`
+ - `timezone: optional string`
- The type of jsonl source. Always `file_content`.
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
- - `"file_content"`
+ - `type: optional "approximate"`
- - `EvalJSONLFileIDSource = object { id, type }`
+ The type of location approximation. Always `approximate`.
- - `id: string`
+ - `"approximate"`
- The identifier of the file.
+ - `Mcp object { server_label, type, allowed_tools, 7 more }`
- - `type: "file_id"`
+ Give the model access to additional tools via remote Model Context Protocol
+ (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
- The type of jsonl source. Always `file_id`.
+ - `server_label: string`
- - `"file_id"`
+ A label for this MCP server, used to identify it in tool calls.
- - `EvalResponsesSource = object { type, created_after, created_before, 8 more }`
+ - `type: "mcp"`
- A EvalResponsesSource object describing a run data source configuration.
+ The type of the MCP tool. Always `mcp`.
- - `type: "responses"`
+ - `"mcp"`
- The type of run data source. Always `responses`.
+ - `allowed_tools: optional array of string or object { read_only, tool_names }`
- - `"responses"`
+ List of allowed tool names or a filter object.
- - `created_after: optional number`
+ - `McpAllowedTools = array of string`
- Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
+ A string array of allowed tool names
- - `created_before: optional number`
+ - `McpToolFilter object { read_only, tool_names }`
- Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
+ A filter object to specify which tools are allowed.
- - `instructions_search: optional string`
+ - `read_only: optional boolean`
- Optional string to search the 'instructions' field. This is a query parameter used to select responses.
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- - `metadata: optional unknown`
+ - `tool_names: optional array of string`
- Metadata filter for the responses. This is a query parameter used to select responses.
+ List of allowed tool names.
- - `model: optional string`
+ - `authorization: optional string`
- The name of the model to find responses for. This is a query parameter used to select responses.
+ An OAuth access token that can be used with a remote MCP server, either
+ with a custom MCP server URL or a service connector. Your application
+ must handle the OAuth authorization flow and provide the token here.
- - `reasoning_effort: optional ReasoningEffort`
+ - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ Identifier for service connectors, like those available in ChatGPT. One of
+ `server_url` or `connector_id` must be provided. Learn more about service
+ connectors [here](/docs/guides/tools-remote-mcp#connectors).
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ Currently supported `connector_id` values are:
- - `"none"`
+ - Dropbox: `connector_dropbox`
+ - Gmail: `connector_gmail`
+ - Google Calendar: `connector_googlecalendar`
+ - Google Drive: `connector_googledrive`
+ - Microsoft Teams: `connector_microsoftteams`
+ - Outlook Calendar: `connector_outlookcalendar`
+ - Outlook Email: `connector_outlookemail`
+ - SharePoint: `connector_sharepoint`
- - `"minimal"`
+ - `"connector_dropbox"`
- - `"low"`
+ - `"connector_gmail"`
- - `"medium"`
+ - `"connector_googlecalendar"`
- - `"high"`
+ - `"connector_googledrive"`
- - `"xhigh"`
+ - `"connector_microsoftteams"`
- - `temperature: optional number`
+ - `"connector_outlookcalendar"`
- Sampling temperature. This is a query parameter used to select responses.
+ - `"connector_outlookemail"`
- - `tools: optional array of string`
+ - `"connector_sharepoint"`
- List of tool names. This is a query parameter used to select responses.
+ - `defer_loading: optional boolean`
- - `top_p: optional number`
+ Whether this MCP tool is deferred and discovered via tool search.
- Nucleus sampling parameter. This is a query parameter used to select responses.
+ - `headers: optional map[string]`
- - `users: optional array of string`
+ Optional HTTP headers to send to the MCP server. Use for authentication
+ or other purposes.
- List of user identifiers. This is a query parameter used to select responses.
+ - `require_approval: optional object { always, never } or "always" or "never"`
- - `type: "responses"`
+ Specify which of the MCP server's tools require approval.
- The type of run data source. Always `responses`.
+ - `McpToolApprovalFilter object { always, never }`
- - `"responses"`
+ Specify which of the MCP server's tools require approval. Can be
+ `always`, `never`, or a filter object associated with tools
+ that require approval.
- - `input_messages: optional object { template, type } or object { item_reference, type }`
+ - `always: optional object { read_only, tool_names }`
- Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+ A filter object to specify which tools are allowed.
- - `InputMessagesTemplate = object { template, type }`
+ - `read_only: optional boolean`
- - `template: array of object { content, role } or object { content, role, type }`
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+ - `tool_names: optional array of string`
- - `ChatMessage = object { content, role }`
+ List of allowed tool names.
- - `content: string`
+ - `never: optional object { read_only, tool_names }`
- The content of the message.
+ A filter object to specify which tools are allowed.
- - `role: string`
+ - `read_only: optional boolean`
- The role of the message (e.g. "system", "assistant", "user").
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- - `EvalMessageObject = object { content, role, type }`
+ - `tool_names: optional array of string`
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ List of allowed tool names.
- - `content: string or ResponseInputText or object { text, type } or 3 more`
+ - `McpToolApprovalSetting = "always" or "never"`
- Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+ Specify a single approval policy for all tools. One of `always` or
+ `never`. When set to `always`, all tools will require approval. When
+ set to `never`, all tools will not require approval.
- - `TextInput = string`
+ - `"always"`
- A text input to the model.
+ - `"never"`
- - `ResponseInputText = object { text, type }`
+ - `server_description: optional string`
- A text input to the model.
+ Optional description of the MCP server, used to provide more context.
- - `text: string`
+ - `server_url: optional string`
- The text input to the model.
+ The URL for the MCP server. One of `server_url` or `connector_id` must be
+ provided.
- - `type: "input_text"`
+ - `CodeInterpreter object { container, type }`
- The type of the input item. Always `input_text`.
+ A tool that runs Python code to help generate a response to a prompt.
- - `"input_text"`
+ - `container: string or object { type, file_ids, memory_limit, network_policy }`
- - `OutputText = object { text, type }`
+ The code interpreter container. Can be a container ID or an object that
+ specifies uploaded file IDs to make available to your code, along with an
+ optional `memory_limit` setting.
- A text output from the model.
+ - `string`
- - `text: string`
+ The container ID.
- The text output from the model.
+ - `CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy }`
- - `type: "output_text"`
+ Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
- The type of the output text. Always `output_text`.
+ - `type: "auto"`
- - `"output_text"`
+ Always `auto`.
- - `InputImage = object { image_url, type, detail }`
+ - `"auto"`
- An image input block used within EvalItem content arrays.
+ - `file_ids: optional array of string`
- - `image_url: string`
+ An optional list of uploaded files to make available to your code.
- The URL of the image input.
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
- - `type: "input_image"`
+ The memory limit for the code interpreter container.
- The type of the image input. Always `input_image`.
+ - `"1g"`
- - `"input_image"`
+ - `"4g"`
- - `detail: optional string`
+ - `"16g"`
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ - `"64g"`
- - `ResponseInputAudio = object { input_audio, type }`
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
- An audio input to the model.
+ Network access policy for the container.
- - `input_audio: object { data, format }`
+ - `ContainerNetworkPolicyDisabled object { type }`
- - `data: string`
+ - `type: "disabled"`
- Base64-encoded audio data.
+ Disable outbound network access. Always `disabled`.
- - `format: "mp3" or "wav"`
+ - `"disabled"`
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
- - `"mp3"`
+ - `allowed_domains: array of string`
- - `"wav"`
+ A list of allowed domains when type is `allowlist`.
- - `type: "input_audio"`
+ - `type: "allowlist"`
- The type of the input item. Always `input_audio`.
+ Allow outbound network access only to specified domains. Always `allowlist`.
- - `"input_audio"`
+ - `"allowlist"`
- - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+ - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
- A list of inputs, each of which may be either an input text, output text, input
- image, or input audio object.
+ Optional domain-scoped secrets for allowlisted domains.
- - `TextInput = string`
+ - `domain: string`
- A text input to the model.
+ The domain associated with the secret.
- - `ResponseInputText = object { text, type }`
+ - `name: string`
- A text input to the model.
+ The name of the secret to inject for the domain.
- - `text: string`
+ - `value: string`
- The text input to the model.
+ The secret value to inject for the domain.
- - `type: "input_text"`
+ - `type: "code_interpreter"`
- The type of the input item. Always `input_text`.
+ The type of the code interpreter tool. Always `code_interpreter`.
- - `"input_text"`
+ - `"code_interpreter"`
- - `OutputText = object { text, type }`
+ - `ImageGeneration object { type, action, background, 9 more }`
- A text output from the model.
+ A tool that generates images using the GPT image models.
- - `text: string`
+ - `type: "image_generation"`
- The text output from the model.
+ The type of the image generation tool. Always `image_generation`.
- - `type: "output_text"`
+ - `"image_generation"`
- The type of the output text. Always `output_text`.
+ - `action: optional "generate" or "edit" or "auto"`
- - `"output_text"`
+ Whether to generate a new image or edit an existing image. Default: `auto`.
- - `InputImage = object { image_url, type, detail }`
+ - `"generate"`
- An image input block used within EvalItem content arrays.
+ - `"edit"`
- - `image_url: string`
+ - `"auto"`
- The URL of the image input.
+ - `background: optional "transparent" or "opaque" or "auto"`
- - `type: "input_image"`
+ Background type for the generated image. One of `transparent`,
+ `opaque`, or `auto`. Default: `auto`.
- The type of the image input. Always `input_image`.
+ - `"transparent"`
- - `"input_image"`
+ - `"opaque"`
- - `detail: optional string`
+ - `"auto"`
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ - `input_fidelity: optional "high" or "low"`
- - `ResponseInputAudio = object { input_audio, type }`
+ Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
- An audio input to the model.
+ - `"high"`
- - `input_audio: object { data, format }`
+ - `"low"`
- - `data: string`
+ - `input_image_mask: optional object { file_id, image_url }`
- Base64-encoded audio data.
+ Optional mask for inpainting. Contains `image_url`
+ (string, optional) and `file_id` (string, optional).
- - `format: "mp3" or "wav"`
+ - `file_id: optional string`
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ File ID for the mask image.
- - `"mp3"`
+ - `image_url: optional string`
- - `"wav"`
+ Base64-encoded mask image.
- - `type: "input_audio"`
+ - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
- The type of the input item. Always `input_audio`.
+ The image generation model to use. Default: `gpt-image-1`.
- - `"input_audio"`
+ - `string`
- - `role: "user" or "assistant" or "system" or "developer"`
+ - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
+ The image generation model to use. Default: `gpt-image-1`.
- - `"user"`
+ - `"gpt-image-1"`
- - `"assistant"`
+ - `"gpt-image-1-mini"`
- - `"system"`
+ - `"gpt-image-1.5"`
- - `"developer"`
+ - `moderation: optional "auto" or "low"`
- - `type: optional "message"`
+ Moderation level for the generated image. Default: `auto`.
- The type of the message input. Always `message`.
+ - `"auto"`
- - `"message"`
+ - `"low"`
- - `type: "template"`
+ - `output_compression: optional number`
- The type of input messages. Always `template`.
+ Compression level for the output image. Default: 100.
- - `"template"`
+ - `output_format: optional "png" or "webp" or "jpeg"`
- - `InputMessagesItemReference = object { item_reference, type }`
+ The output format of the generated image. One of `png`, `webp`, or
+ `jpeg`. Default: `png`.
- - `item_reference: string`
+ - `"png"`
- A reference to a variable in the `item` namespace. Ie, "item.name"
+ - `"webp"`
- - `type: "item_reference"`
+ - `"jpeg"`
- The type of input messages. Always `item_reference`.
+ - `partial_images: optional number`
- - `"item_reference"`
+ Number of partial images to generate in streaming mode, from 0 (default value) to 3.
- - `model: optional string`
+ - `quality: optional "low" or "medium" or "high" or "auto"`
- The name of the model to use for generating completions (e.g. "o3-mini").
+ The quality of the generated image. One of `low`, `medium`, `high`,
+ or `auto`. Default: `auto`.
- - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
+ - `"low"`
- - `max_completion_tokens: optional number`
+ - `"medium"`
- The maximum number of tokens in the generated output.
+ - `"high"`
- - `reasoning_effort: optional ReasoningEffort`
+ - `"auto"`
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ The size of the generated image. One of `1024x1024`, `1024x1536`,
+ `1536x1024`, or `auto`. Default: `auto`.
- - `"none"`
+ - `"1024x1024"`
- - `"minimal"`
+ - `"1024x1536"`
- - `"low"`
+ - `"1536x1024"`
- - `"medium"`
+ - `"auto"`
- - `"high"`
+ - `LocalShell object { type }`
- - `"xhigh"`
+ A tool that allows the model to execute shell commands in a local environment.
- - `seed: optional number`
+ - `type: "local_shell"`
- A seed value to initialize the randomness, during sampling.
+ The type of the local shell tool. Always `local_shell`.
- - `temperature: optional number`
+ - `"local_shell"`
- A higher temperature increases randomness in the outputs.
+ - `Shell object { type, environment }`
- - `text: optional object { format }`
+ A tool that allows the model to execute shell commands.
- Configuration options for a text response from the model. Can be plain
- text or structured JSON data. Learn more:
+ - `type: "shell"`
- - [Text inputs and outputs](/docs/guides/text)
- - [Structured Outputs](/docs/guides/structured-outputs)
+ The type of the shell tool. Always `shell`.
- - `format: optional ResponseFormatTextConfig`
+ - `"shell"`
- An object specifying the format that the model must output.
+ - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
- Configuring `{ "type": "json_schema" }` enables Structured Outputs,
- which ensures the model will match your supplied JSON schema. Learn more in the
- [Structured Outputs guide](/docs/guides/structured-outputs).
+ - `ContainerAuto object { type, file_ids, memory_limit, 2 more }`
- The default format is `{ "type": "text" }` with no additional options.
+ - `type: "container_auto"`
- **Not recommended for gpt-4o and newer models:**
+ Automatically creates a container for this request
- Setting to `{ "type": "json_object" }` enables the older JSON mode, which
- ensures the message the model generates is valid JSON. Using `json_schema`
- is preferred for models that support it.
+ - `"container_auto"`
- - `ResponseFormatText = object { type }`
+ - `file_ids: optional array of string`
- Default response format. Used to generate text responses.
+ An optional list of uploaded files to make available to your code.
- - `type: "text"`
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
- The type of response format being defined. Always `text`.
+ The memory limit for the container.
- - `"text"`
+ - `"1g"`
- - `ResponseFormatTextJSONSchemaConfig = object { name, schema, type, 2 more }`
+ - `"4g"`
- JSON Schema response format. Used to generate structured JSON responses.
- Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+ - `"16g"`
- - `name: string`
+ - `"64g"`
- The name of the response format. Must be a-z, A-Z, 0-9, or contain
- underscores and dashes, with a maximum length of 64.
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
- - `schema: map[unknown]`
+ Network access policy for the container.
- The schema for the response format, described as a JSON Schema object.
- Learn how to build JSON schemas [here](https://json-schema.org/).
+ - `ContainerNetworkPolicyDisabled object { type }`
- - `type: "json_schema"`
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
- The type of response format being defined. Always `json_schema`.
+ - `skills: optional array of SkillReference or InlineSkill`
- - `"json_schema"`
+ An optional list of skills referenced by id or inline data.
- - `description: optional string`
+ - `SkillReference object { skill_id, type, version }`
- A description of what the response format is for, used by the model to
- determine how to respond in the format.
+ - `skill_id: string`
- - `strict: optional boolean`
+ The ID of the referenced skill.
- Whether to enable strict schema adherence when generating the output.
- If set to true, the model will always follow the exact schema defined
- in the `schema` field. Only a subset of JSON Schema is supported when
- `strict` is `true`. To learn more, read the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ - `type: "skill_reference"`
- - `ResponseFormatJSONObject = object { type }`
+ References a skill created with the /v1/skills endpoint.
- JSON object response format. An older method of generating JSON responses.
- Using `json_schema` is recommended for models that support it. Note that the
- model will not generate JSON without a system or user message instructing it
- to do so.
+ - `"skill_reference"`
- - `type: "json_object"`
+ - `version: optional string`
- The type of response format being defined. Always `json_object`.
+ Optional skill version. Use a positive integer or 'latest'. Omit for default.
- - `"json_object"`
+ - `InlineSkill object { description, name, source, type }`
- - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
+ - `description: string`
- An array of tools the model may call while generating a response. You
- can specify which tool to use by setting the `tool_choice` parameter.
+ The description of the skill.
- The two categories of tools you can provide the model are:
+ - `name: string`
- - **Built-in tools**: Tools that are provided by OpenAI that extend the
- model's capabilities, like [web search](/docs/guides/tools-web-search)
- or [file search](/docs/guides/tools-file-search). Learn more about
- [built-in tools](/docs/guides/tools).
- - **Function calls (custom tools)**: Functions that are defined by you,
- enabling the model to call your own code. Learn more about
- [function calling](/docs/guides/function-calling).
+ The name of the skill.
- - `Function = object { name, parameters, strict, 3 more }`
+ - `source: InlineSkillSource`
- Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
+ Inline skill payload
- - `name: string`
+ - `data: string`
- The name of the function to call.
+ Base64-encoded skill zip bundle.
- - `parameters: map[unknown]`
+ - `media_type: "application/zip"`
- A JSON schema object describing the parameters of the function.
+ The media type of the inline skill payload. Must be `application/zip`.
- - `strict: boolean`
+ - `"application/zip"`
- Whether to enforce strict parameter validation. Default `true`.
+ - `type: "base64"`
- - `type: "function"`
+ The type of the inline skill source. Must be `base64`.
- The type of the function tool. Always `function`.
+ - `"base64"`
- - `"function"`
+ - `type: "inline"`
- - `defer_loading: optional boolean`
+ Defines an inline skill for this request.
- Whether this function is deferred and loaded via tool search.
+ - `"inline"`
- - `description: optional string`
+ - `LocalEnvironment object { type, skills }`
- A description of the function. Used by the model to determine whether or not to call the function.
+ - `type: "local"`
- - `FileSearch = object { type, vector_store_ids, filters, 2 more }`
+ Use a local computer environment.
- A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
+ - `"local"`
- - `type: "file_search"`
+ - `skills: optional array of LocalSkill`
- The type of the file search tool. Always `file_search`.
+ An optional list of skills.
- - `"file_search"`
+ - `description: string`
- - `vector_store_ids: array of string`
+ The description of the skill.
- The IDs of the vector stores to search.
+ - `name: string`
- - `filters: optional ComparisonFilter or CompoundFilter`
+ The name of the skill.
- A filter to apply.
+ - `path: string`
- - `ComparisonFilter = object { key, type, value }`
+ The path to the directory containing the skill.
- A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+ - `ContainerReference object { container_id, type }`
- - `key: string`
+ - `container_id: string`
- The key to compare against the value.
+ The ID of the referenced container.
- - `type: "eq" or "ne" or "gt" or 5 more`
+ - `type: "container_reference"`
- Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+ References a container created with the /v1/containers endpoint
- - `eq`: equals
- - `ne`: not equal
- - `gt`: greater than
- - `gte`: greater than or equal
- - `lt`: less than
- - `lte`: less than or equal
- - `in`: in
- - `nin`: not in
+ - `"container_reference"`
- - `"eq"`
+ - `Custom object { name, type, defer_loading, 2 more }`
- - `"ne"`
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
- - `"gt"`
+ - `name: string`
- - `"gte"`
+ The name of the custom tool, used to identify it in tool calls.
- - `"lt"`
+ - `type: "custom"`
- - `"lte"`
+ The type of the custom tool. Always `custom`.
- - `"in"`
+ - `"custom"`
- - `"nin"`
+ - `defer_loading: optional boolean`
- - `value: string or number or boolean or array of string or number`
+ Whether this tool should be deferred and discovered via tool search.
- The value to compare against the attribute key; supports string, number, or boolean types.
+ - `description: optional string`
- - `string`
+ Optional description of the custom tool, used to provide more context.
- - `number`
+ - `format: optional CustomToolInputFormat`
- - `boolean`
+ The input format for the custom tool. Default is unconstrained text.
- - `array of string or number`
+ - `Text object { type }`
- - `string`
+ Unconstrained free-form text.
- - `number`
+ - `type: "text"`
- - `CompoundFilter = object { filters, type }`
+ Unconstrained text format. Always `text`.
- Combine multiple filters using `and` or `or`.
+ - `"text"`
- - `filters: array of ComparisonFilter or unknown`
+ - `Grammar object { definition, syntax, type }`
- Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
+ A grammar defined by the user.
- - `ComparisonFilter = object { key, type, value }`
+ - `definition: string`
- A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+ The grammar definition.
- - `key: string`
+ - `syntax: "lark" or "regex"`
- The key to compare against the value.
+ The syntax of the grammar definition. One of `lark` or `regex`.
- - `type: "eq" or "ne" or "gt" or 5 more`
+ - `"lark"`
- Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+ - `"regex"`
- - `eq`: equals
- - `ne`: not equal
- - `gt`: greater than
- - `gte`: greater than or equal
- - `lt`: less than
- - `lte`: less than or equal
- - `in`: in
- - `nin`: not in
+ - `type: "grammar"`
- - `"eq"`
-
- - `"ne"`
-
- - `"gt"`
-
- - `"gte"`
-
- - `"lt"`
-
- - `"lte"`
-
- - `"in"`
-
- - `"nin"`
-
- - `value: string or number or boolean or array of string or number`
-
- The value to compare against the attribute key; supports string, number, or boolean types.
-
- - `string`
-
- - `number`
-
- - `boolean`
+ Grammar format. Always `grammar`.
- - `array of string or number`
+ - `"grammar"`
- - `string`
+ - `Namespace object { description, name, tools, type }`
- - `number`
+ Groups function/custom tools under a shared namespace.
- - `unknown`
+ - `description: string`
- - `type: "and" or "or"`
+ A description of the namespace shown to the model.
- Type of operation: `and` or `or`.
+ - `name: string`
- - `"and"`
+ The namespace name used in tool calls (for example, `crm`).
- - `"or"`
+ - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
- - `max_num_results: optional number`
+ The function/custom tools available inside this namespace.
- The maximum number of results to return. This number should be between 1 and 50 inclusive.
+ - `Function object { name, type, defer_loading, 3 more }`
- - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
+ - `name: string`
- Ranking options for search.
+ - `type: "function"`
- - `hybrid_search: optional object { embedding_weight, text_weight }`
+ - `"function"`
- Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
+ - `defer_loading: optional boolean`
- - `embedding_weight: number`
+ Whether this function should be deferred and discovered via tool search.
- The weight of the embedding in the reciprocal ranking fusion.
+ - `description: optional string`
- - `text_weight: number`
+ - `parameters: optional unknown`
- The weight of the text in the reciprocal ranking fusion.
+ - `strict: optional boolean`
- - `ranker: optional "auto" or "default-2024-11-15"`
+ - `Custom object { name, type, defer_loading, 2 more }`
- The ranker to use for the file search.
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
- - `"auto"`
+ - `name: string`
- - `"default-2024-11-15"`
+ The name of the custom tool, used to identify it in tool calls.
- - `score_threshold: optional number`
+ - `type: "custom"`
- The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
+ The type of the custom tool. Always `custom`.
- - `Computer = object { type }`
+ - `"custom"`
- A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+ - `defer_loading: optional boolean`
- - `type: "computer"`
+ Whether this tool should be deferred and discovered via tool search.
- The type of the computer tool. Always `computer`.
+ - `description: optional string`
- - `"computer"`
+ Optional description of the custom tool, used to provide more context.
- - `ComputerUsePreview = object { display_height, display_width, environment, type }`
+ - `format: optional CustomToolInputFormat`
- A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+ The input format for the custom tool. Default is unconstrained text.
- - `display_height: number`
+ - `type: "namespace"`
- The height of the computer display.
+ The type of the tool. Always `namespace`.
- - `display_width: number`
+ - `"namespace"`
- The width of the computer display.
+ - `ToolSearch object { type, description, execution, parameters }`
- - `environment: "windows" or "mac" or "linux" or 2 more`
+ Hosted or BYOT tool search configuration for deferred tools.
- The type of computer environment to control.
+ - `type: "tool_search"`
- - `"windows"`
+ The type of the tool. Always `tool_search`.
- - `"mac"`
+ - `"tool_search"`
- - `"linux"`
+ - `description: optional string`
- - `"ubuntu"`
+ Description shown to the model for a client-executed tool search tool.
- - `"browser"`
+ - `execution: optional "server" or "client"`
- - `type: "computer_use_preview"`
+ Whether tool search is executed by the server or by the client.
- The type of the computer use tool. Always `computer_use_preview`.
+ - `"server"`
- - `"computer_use_preview"`
+ - `"client"`
- - `WebSearch = object { type, filters, search_context_size, user_location }`
+ - `parameters: optional unknown`
- Search the Internet for sources related to the prompt. Learn more about the
- [web search tool](/docs/guides/tools-web-search).
+ Parameter schema for a client-executed tool search tool.
- - `type: "web_search" or "web_search_2025_08_26"`
+ - `WebSearchPreview object { type, search_content_types, search_context_size, user_location }`
- The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
+ This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
- - `"web_search"`
+ - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
- - `"web_search_2025_08_26"`
+ The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
- - `filters: optional object { allowed_domains }`
+ - `"web_search_preview"`
- Filters for the search.
+ - `"web_search_preview_2025_03_11"`
- - `allowed_domains: optional array of string`
+ - `search_content_types: optional array of "text" or "image"`
- Allowed domains for the search. If not provided, all domains are allowed.
- Subdomains of the provided domains are allowed as well.
+ - `"text"`
- Example: `["pubmed.ncbi.nlm.nih.gov"]`
+ - `"image"`
- `search_context_size: optional "low" or "medium" or "high"`
@@ -6497,9 +5738,15 @@ Kicks off a new run for a given evaluation, specifying the data source, and what
- `"high"`
- - `user_location: optional object { city, country, region, 2 more }`
+ - `user_location: optional object { type, city, country, 2 more }`
- The approximate location of the user.
+ The user's location.
+
+ - `type: "approximate"`
+
+ The type of location approximation. Always `approximate`.
+
+ - `"approximate"`
- `city: optional string`
@@ -6517,825 +5764,914 @@ Kicks off a new run for a given evaluation, specifying the data source, and what
The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
- - `type: optional "approximate"`
+ - `ApplyPatch object { type }`
- The type of location approximation. Always `approximate`.
+ Allows the assistant to create, delete, or update files using unified diffs.
- - `"approximate"`
+ - `type: "apply_patch"`
- - `Mcp = object { server_label, type, allowed_tools, 7 more }`
+ The type of the tool. Always `apply_patch`.
- Give the model access to additional tools via remote Model Context Protocol
- (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
+ - `"apply_patch"`
- - `server_label: string`
+ - `top_p: optional number`
- A label for this MCP server, used to identify it in tool calls.
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- - `type: "mcp"`
+ - `error: EvalAPIError`
- The type of the MCP tool. Always `mcp`.
+ An object representing an error response from the Eval API.
- - `"mcp"`
+ - `code: string`
- - `allowed_tools: optional array of string or object { read_only, tool_names }`
+ The error code.
- List of allowed tool names or a filter object.
+ - `message: string`
- - `McpAllowedTools = array of string`
+ The error message.
- A string array of allowed tool names
+ - `eval_id: string`
- - `McpToolFilter = object { read_only, tool_names }`
+ The identifier of the associated evaluation.
- A filter object to specify which tools are allowed.
+ - `metadata: Metadata`
- - `read_only: optional boolean`
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `tool_names: optional array of string`
+ - `model: string`
- List of allowed tool names.
+ The model that is evaluated, if applicable.
- - `authorization: optional string`
+ - `name: string`
- An OAuth access token that can be used with a remote MCP server, either
- with a custom MCP server URL or a service connector. Your application
- must handle the OAuth authorization flow and provide the token here.
+ The name of the evaluation run.
- - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
+ - `object: "eval.run"`
- Identifier for service connectors, like those available in ChatGPT. One of
- `server_url` or `connector_id` must be provided. Learn more about service
- connectors [here](/docs/guides/tools-remote-mcp#connectors).
+ The type of the object. Always "eval.run".
- Currently supported `connector_id` values are:
+ - `"eval.run"`
- - Dropbox: `connector_dropbox`
- - Gmail: `connector_gmail`
- - Google Calendar: `connector_googlecalendar`
- - Google Drive: `connector_googledrive`
- - Microsoft Teams: `connector_microsoftteams`
- - Outlook Calendar: `connector_outlookcalendar`
- - Outlook Email: `connector_outlookemail`
- - SharePoint: `connector_sharepoint`
+ - `per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }`
- - `"connector_dropbox"`
+ Usage statistics for each model during the evaluation run.
- - `"connector_gmail"`
+ - `cached_tokens: number`
- - `"connector_googlecalendar"`
+ The number of tokens retrieved from cache.
- - `"connector_googledrive"`
+ - `completion_tokens: number`
- - `"connector_microsoftteams"`
+ The number of completion tokens generated.
- - `"connector_outlookcalendar"`
+ - `invocation_count: number`
- - `"connector_outlookemail"`
+ The number of invocations.
- - `"connector_sharepoint"`
+ - `model_name: string`
- - `defer_loading: optional boolean`
+ The name of the model.
- Whether this MCP tool is deferred and discovered via tool search.
+ - `prompt_tokens: number`
- - `headers: optional map[string]`
+ The number of prompt tokens used.
- Optional HTTP headers to send to the MCP server. Use for authentication
- or other purposes.
+ - `total_tokens: number`
- - `require_approval: optional object { always, never } or "always" or "never"`
+ The total number of tokens used.
- Specify which of the MCP server's tools require approval.
+ - `per_testing_criteria_results: array of object { failed, passed, testing_criteria }`
- - `McpToolApprovalFilter = object { always, never }`
+ Results per testing criteria applied during the evaluation run.
- Specify which of the MCP server's tools require approval. Can be
- `always`, `never`, or a filter object associated with tools
- that require approval.
+ - `failed: number`
- - `always: optional object { read_only, tool_names }`
+ Number of tests failed for this criteria.
- A filter object to specify which tools are allowed.
+ - `passed: number`
- - `read_only: optional boolean`
+ Number of tests passed for this criteria.
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ - `testing_criteria: string`
- - `tool_names: optional array of string`
+ A description of the testing criteria.
- List of allowed tool names.
+ - `report_url: string`
- - `never: optional object { read_only, tool_names }`
+ The URL to the rendered evaluation run report on the UI dashboard.
- A filter object to specify which tools are allowed.
+ - `result_counts: object { errored, failed, passed, total }`
- - `read_only: optional boolean`
+ Counters summarizing the outcomes of the evaluation run.
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ - `errored: number`
- - `tool_names: optional array of string`
+ Number of output items that resulted in an error.
- List of allowed tool names.
+ - `failed: number`
- - `McpToolApprovalSetting = "always" or "never"`
+ Number of output items that failed to pass the evaluation.
- Specify a single approval policy for all tools. One of `always` or
- `never`. When set to `always`, all tools will require approval. When
- set to `never`, all tools will not require approval.
+ - `passed: number`
- - `"always"`
+ Number of output items that passed the evaluation.
- - `"never"`
+ - `total: number`
- - `server_description: optional string`
+ Total number of executed output items.
- Optional description of the MCP server, used to provide more context.
+ - `status: string`
- - `server_url: optional string`
+ The status of the evaluation run.
- The URL for the MCP server. One of `server_url` or `connector_id` must be
- provided.
+- `first_id: string`
- - `CodeInterpreter = object { container, type }`
+ The identifier of the first eval run in the data array.
- A tool that runs Python code to help generate a response to a prompt.
+- `has_more: boolean`
- - `container: string or object { type, file_ids, memory_limit, network_policy }`
+ Indicates whether there are more evals available.
- The code interpreter container. Can be a container ID or an object that
- specifies uploaded file IDs to make available to your code, along with an
- optional `memory_limit` setting.
+- `last_id: string`
- - `string`
+ The identifier of the last eval run in the data array.
- The container ID.
+- `object: "list"`
- - `CodeInterpreterToolAuto = object { type, file_ids, memory_limit, network_policy }`
+ The type of this object. It is always set to "list".
- Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
+ - `"list"`
- - `type: "auto"`
+### Example
- Always `auto`.
+```http
+curl https://api.openai.com/v1/evals/$EVAL_ID/runs \
+ -H "Authorization: Bearer $OPENAI_API_KEY"
+```
- - `"auto"`
+#### Response
- - `file_ids: optional array of string`
+```json
+{
+ "data": [
+ {
+ "id": "id",
+ "created_at": 0,
+ "data_source": {
+ "source": {
+ "content": [
+ {
+ "item": {
+ "foo": "bar"
+ },
+ "sample": {
+ "foo": "bar"
+ }
+ }
+ ],
+ "type": "file_content"
+ },
+ "type": "jsonl"
+ },
+ "error": {
+ "code": "code",
+ "message": "message"
+ },
+ "eval_id": "eval_id",
+ "metadata": {
+ "foo": "string"
+ },
+ "model": "model",
+ "name": "name",
+ "object": "eval.run",
+ "per_model_usage": [
+ {
+ "cached_tokens": 0,
+ "completion_tokens": 0,
+ "invocation_count": 0,
+ "model_name": "model_name",
+ "prompt_tokens": 0,
+ "total_tokens": 0
+ }
+ ],
+ "per_testing_criteria_results": [
+ {
+ "failed": 0,
+ "passed": 0,
+ "testing_criteria": "testing_criteria"
+ }
+ ],
+ "report_url": "report_url",
+ "result_counts": {
+ "errored": 0,
+ "failed": 0,
+ "passed": 0,
+ "total": 0
+ },
+ "status": "status"
+ }
+ ],
+ "first_id": "first_id",
+ "has_more": true,
+ "last_id": "last_id",
+ "object": "list"
+}
+```
- An optional list of uploaded files to make available to your code.
+### Example
- - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+```http
+curl https://api.openai.com/v1/evals/egroup_67abd54d9b0081909a86353f6fb9317a/runs \
+ -H "Authorization: Bearer $OPENAI_API_KEY" \
+ -H "Content-Type: application/json"
+```
- The memory limit for the code interpreter container.
+#### Response
- - `"1g"`
+```json
+{
+ "object": "list",
+ "data": [
+ {
+ "object": "eval.run",
+ "id": "evalrun_67e0c7d31560819090d60c0780591042",
+ "eval_id": "eval_67e0c726d560819083f19a957c4c640b",
+ "report_url": "https://platform.openai.com/evaluations/eval_67e0c726d560819083f19a957c4c640b",
+ "status": "completed",
+ "model": "o3-mini",
+ "name": "bulk_with_negative_examples_o3-mini",
+ "created_at": 1742784467,
+ "result_counts": {
+ "total": 1,
+ "errored": 0,
+ "failed": 0,
+ "passed": 1
+ },
+ "per_model_usage": [
+ {
+ "model_name": "o3-mini",
+ "invocation_count": 1,
+ "prompt_tokens": 563,
+ "completion_tokens": 874,
+ "total_tokens": 1437,
+ "cached_tokens": 0
+ }
+ ],
+ "per_testing_criteria_results": [
+ {
+ "testing_criteria": "Push Notification Summary Grader-1808cd0b-eeec-4e0b-a519-337e79f4f5d1",
+ "passed": 1,
+ "failed": 0
+ }
+ ],
+ "data_source": {
+ "type": "completions",
+ "source": {
+ "type": "file_content",
+ "content": [
+ {
+ "item": {
+ "notifications": "\n- New message from Sarah: \"Can you call me later?\"\n- Your package has been delivered!\n- Flash sale: 20% off electronics for the next 2 hours!\n"
+ }
+ }
+ ]
+ },
+ "input_messages": {
+ "type": "template",
+ "template": [
+ {
+ "type": "message",
+ "role": "developer",
+ "content": {
+ "type": "input_text",
+ "text": "\n\n\n\nYou are a helpful assistant that takes in an array of push notifications and returns a collapsed summary of them.\nThe push notification will be provided as follows:\n\n...notificationlist...\n\n\nYou should return just the summary and nothing else.\n\n\nYou should return a summary that is concise and snappy.\n\n\nHere is an example of a good summary:\n\n- Traffic alert: Accident reported on Main Street.- Package out for delivery: Expected by 5 PM.- New friend suggestion: Connect with Emma.\n\n\nTraffic alert, package expected by 5pm, suggestion for new friend (Emily).\n\n\n\nHere is an example of a bad summary:\n\n- Traffic alert: Accident reported on Main Street.- Package out for delivery: Expected by 5 PM.- New friend suggestion: Connect with Emma.\n\n\nTraffic alert reported on main street. You have a package that will arrive by 5pm, Emily is a new friend suggested for you.\n\n"
+ }
+ },
+ {
+ "type": "message",
+ "role": "user",
+ "content": {
+ "type": "input_text",
+ "text": "{{item.notifications}}"
+ }
+ }
+ ]
+ },
+ "model": "o3-mini",
+ "sampling_params": null
+ },
+ "error": null,
+ "metadata": {}
+ }
+ ],
+ "first_id": "evalrun_67e0c7d31560819090d60c0780591042",
+ "last_id": "evalrun_67e0c7d31560819090d60c0780591042",
+ "has_more": true
+}
+```
- - `"4g"`
+## Create eval run
- - `"16g"`
+**post** `/evals/{eval_id}/runs`
- - `"64g"`
+Kicks off a new run for a given evaluation, specifying the data source, and what model configuration to use to test. The datasource will be validated against the schema specified in the config of the evaluation.
- - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+### Path Parameters
- Network access policy for the container.
+- `eval_id: string`
- - `ContainerNetworkPolicyDisabled = object { type }`
+### Body Parameters
- - `type: "disabled"`
+- `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
- Disable outbound network access. Always `disabled`.
+ Details about the run's data source.
- - `"disabled"`
+ - `CreateEvalJSONLRunDataSource object { source, type }`
- - `ContainerNetworkPolicyAllowlist = object { allowed_domains, type, domain_secrets }`
+ A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
- - `allowed_domains: array of string`
+ - `source: object { content, type } or object { id, type }`
- A list of allowed domains when type is `allowlist`.
+ Determines what populates the `item` namespace in the data source.
- - `type: "allowlist"`
+ - `EvalJSONLFileContentSource object { content, type }`
- Allow outbound network access only to specified domains. Always `allowlist`.
+ - `content: array of object { item, sample }`
- - `"allowlist"`
+ The content of the jsonl file.
- - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+ - `item: map[unknown]`
- Optional domain-scoped secrets for allowlisted domains.
+ - `sample: optional map[unknown]`
- - `domain: string`
+ - `type: "file_content"`
- The domain associated with the secret.
+ The type of jsonl source. Always `file_content`.
- - `name: string`
+ - `"file_content"`
- The name of the secret to inject for the domain.
+ - `EvalJSONLFileIDSource object { id, type }`
- - `value: string`
+ - `id: string`
- The secret value to inject for the domain.
+ The identifier of the file.
- - `type: "code_interpreter"`
+ - `type: "file_id"`
- The type of the code interpreter tool. Always `code_interpreter`.
+ The type of jsonl source. Always `file_id`.
- - `"code_interpreter"`
+ - `"file_id"`
- - `ImageGeneration = object { type, action, background, 9 more }`
+ - `type: "jsonl"`
- A tool that generates images using the GPT image models.
+ The type of data source. Always `jsonl`.
- - `type: "image_generation"`
+ - `"jsonl"`
- The type of the image generation tool. Always `image_generation`.
+ - `CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }`
- - `"image_generation"`
+ A CompletionsRunDataSource object describing a model sampling configuration.
- - `action: optional "generate" or "edit" or "auto"`
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
- Whether to generate a new image or edit an existing image. Default: `auto`.
+ Determines what populates the `item` namespace in this run's data source.
- - `"generate"`
+ - `EvalJSONLFileContentSource object { content, type }`
- - `"edit"`
+ - `content: array of object { item, sample }`
- - `"auto"`
+ The content of the jsonl file.
- - `background: optional "transparent" or "opaque" or "auto"`
+ - `item: map[unknown]`
- Background type for the generated image. One of `transparent`,
- `opaque`, or `auto`. Default: `auto`.
+ - `sample: optional map[unknown]`
- - `"transparent"`
+ - `type: "file_content"`
- - `"opaque"`
+ The type of jsonl source. Always `file_content`.
- - `"auto"`
+ - `"file_content"`
- - `input_fidelity: optional "high" or "low"`
+ - `EvalJSONLFileIDSource object { id, type }`
- Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
+ - `id: string`
- - `"high"`
+ The identifier of the file.
- - `"low"`
+ - `type: "file_id"`
- - `input_image_mask: optional object { file_id, image_url }`
+ The type of jsonl source. Always `file_id`.
- Optional mask for inpainting. Contains `image_url`
- (string, optional) and `file_id` (string, optional).
+ - `"file_id"`
- - `file_id: optional string`
+ - `StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more }`
- File ID for the mask image.
+ A StoredCompletionsRunDataSource configuration describing a set of filters
- - `image_url: optional string`
+ - `type: "stored_completions"`
- Base64-encoded mask image.
+ The type of source. Always `stored_completions`.
- - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+ - `"stored_completions"`
- The image generation model to use. Default: `gpt-image-1`.
+ - `created_after: optional number`
- - `string`
+ An optional Unix timestamp to filter items created after this time.
- - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+ - `created_before: optional number`
- The image generation model to use. Default: `gpt-image-1`.
+ An optional Unix timestamp to filter items created before this time.
- - `"gpt-image-1"`
+ - `limit: optional number`
- - `"gpt-image-1-mini"`
+ An optional maximum number of items to return.
- - `"gpt-image-1.5"`
+ - `metadata: optional Metadata`
- - `moderation: optional "auto" or "low"`
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- Moderation level for the generated image. Default: `auto`.
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `"auto"`
+ - `model: optional string`
- - `"low"`
+ An optional model to filter by (e.g., 'gpt-4o').
- - `output_compression: optional number`
+ - `type: "completions"`
- Compression level for the output image. Default: 100.
+ The type of run data source. Always `completions`.
- - `output_format: optional "png" or "webp" or "jpeg"`
+ - `"completions"`
- The output format of the generated image. One of `png`, `webp`, or
- `jpeg`. Default: `png`.
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
- - `"png"`
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
- - `"webp"`
+ - `TemplateInputMessages object { template, type }`
- - `"jpeg"`
+ - `template: array of EasyInputMessage or object { content, role, type }`
- - `partial_images: optional number`
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
- Number of partial images to generate in streaming mode, from 0 (default value) to 3.
+ - `EasyInputMessage object { content, role, phase, type }`
- - `quality: optional "low" or "medium" or "high" or "auto"`
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- The quality of the generated image. One of `low`, `medium`, `high`,
- or `auto`. Default: `auto`.
+ - `content: string or ResponseInputMessageContentList`
- - `"low"`
+ Text, image, or audio input to the model, used to generate a response.
+ Can also contain previous assistant responses.
- - `"medium"`
+ - `TextInput = string`
- - `"high"`
+ A text input to the model.
- - `"auto"`
+ - `ResponseInputMessageContentList = array of ResponseInputContent`
- - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
+ A list of one or many input items to the model, containing different content
+ types.
- The size of the generated image. One of `1024x1024`, `1024x1536`,
- `1536x1024`, or `auto`. Default: `auto`.
+ - `ResponseInputText object { text, type }`
- - `"1024x1024"`
+ A text input to the model.
- - `"1024x1536"`
+ - `text: string`
- - `"1536x1024"`
+ The text input to the model.
- - `"auto"`
+ - `type: "input_text"`
- - `LocalShell = object { type }`
+ The type of the input item. Always `input_text`.
- A tool that allows the model to execute shell commands in a local environment.
+ - `"input_text"`
- - `type: "local_shell"`
+ - `ResponseInputImage object { detail, type, file_id, image_url }`
- The type of the local shell tool. Always `local_shell`.
+ An image input to the model. Learn about [image inputs](/docs/guides/vision).
- - `"local_shell"`
+ - `detail: "low" or "high" or "auto" or "original"`
- - `Shell = object { type, environment }`
+ The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
- A tool that allows the model to execute shell commands.
+ - `"low"`
- - `type: "shell"`
+ - `"high"`
- The type of the shell tool. Always `shell`.
+ - `"auto"`
- - `"shell"`
+ - `"original"`
- - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
+ - `type: "input_image"`
- - `ContainerAuto = object { type, file_ids, memory_limit, 2 more }`
+ The type of the input item. Always `input_image`.
- - `type: "container_auto"`
+ - `"input_image"`
- Automatically creates a container for this request
+ - `file_id: optional string`
- - `"container_auto"`
+ The ID of the file to be sent to the model.
- - `file_ids: optional array of string`
+ - `image_url: optional string`
- An optional list of uploaded files to make available to your code.
+ The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
- - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+ - `ResponseInputFile object { type, detail, file_data, 3 more }`
- The memory limit for the container.
+ A file input to the model.
- - `"1g"`
+ - `type: "input_file"`
- - `"4g"`
+ The type of the input item. Always `input_file`.
- - `"16g"`
+ - `"input_file"`
- - `"64g"`
+ - `detail: optional "low" or "high"`
- - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+ The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
- Network access policy for the container.
+ - `"low"`
- - `ContainerNetworkPolicyDisabled = object { type }`
+ - `"high"`
- - `type: "disabled"`
+ - `file_data: optional string`
- Disable outbound network access. Always `disabled`.
+ The content of the file to be sent to the model.
- - `"disabled"`
+ - `file_id: optional string`
- - `ContainerNetworkPolicyAllowlist = object { allowed_domains, type, domain_secrets }`
+ The ID of the file to be sent to the model.
- - `allowed_domains: array of string`
+ - `file_url: optional string`
- A list of allowed domains when type is `allowlist`.
+ The URL of the file to be sent to the model.
- - `type: "allowlist"`
+ - `filename: optional string`
- Allow outbound network access only to specified domains. Always `allowlist`.
+ The name of the file to be sent to the model.
- - `"allowlist"`
+ - `role: "user" or "assistant" or "system" or "developer"`
- - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- Optional domain-scoped secrets for allowlisted domains.
+ - `"user"`
- - `domain: string`
+ - `"assistant"`
- The domain associated with the secret.
+ - `"system"`
- - `name: string`
+ - `"developer"`
- The name of the secret to inject for the domain.
+ - `phase: optional "commentary" or "final_answer"`
- - `value: string`
+ Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
+ For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
+ phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
- The secret value to inject for the domain.
+ - `"commentary"`
- - `skills: optional array of SkillReference or InlineSkill`
+ - `"final_answer"`
- An optional list of skills referenced by id or inline data.
+ - `type: optional "message"`
- - `SkillReference = object { skill_id, type, version }`
+ The type of the message input. Always `message`.
- - `skill_id: string`
+ - `"message"`
- The ID of the referenced skill.
+ - `EvalMessageObject object { content, role, type }`
- - `type: "skill_reference"`
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- References a skill created with the /v1/skills endpoint.
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- - `"skill_reference"`
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- - `version: optional string`
+ - `TextInput = string`
- Optional skill version. Use a positive integer or 'latest'. Omit for default.
+ A text input to the model.
- - `InlineSkill = object { description, name, source, type }`
+ - `ResponseInputText object { text, type }`
- - `description: string`
+ A text input to the model.
- The description of the skill.
+ - `OutputText object { text, type }`
- - `name: string`
+ A text output from the model.
- The name of the skill.
+ - `text: string`
- - `source: InlineSkillSource`
+ The text output from the model.
- Inline skill payload
+ - `type: "output_text"`
- - `data: string`
+ The type of the output text. Always `output_text`.
- Base64-encoded skill zip bundle.
+ - `"output_text"`
- - `media_type: "application/zip"`
+ - `InputImage object { image_url, type, detail }`
- The media type of the inline skill payload. Must be `application/zip`.
+ An image input block used within EvalItem content arrays.
- - `"application/zip"`
+ - `image_url: string`
- - `type: "base64"`
+ The URL of the image input.
- The type of the inline skill source. Must be `base64`.
+ - `type: "input_image"`
- - `"base64"`
+ The type of the image input. Always `input_image`.
- - `type: "inline"`
+ - `"input_image"`
- Defines an inline skill for this request.
+ - `detail: optional string`
- - `"inline"`
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `LocalEnvironment = object { type, skills }`
+ - `ResponseInputAudio object { input_audio, type }`
- - `type: "local"`
+ An audio input to the model.
- Use a local computer environment.
+ - `input_audio: object { data, format }`
- - `"local"`
+ - `data: string`
- - `skills: optional array of LocalSkill`
+ Base64-encoded audio data.
- An optional list of skills.
+ - `format: "mp3" or "wav"`
- - `description: string`
+ The format of the audio data. Currently supported formats are `mp3` and
+ `wav`.
- The description of the skill.
+ - `"mp3"`
- - `name: string`
+ - `"wav"`
- The name of the skill.
+ - `type: "input_audio"`
- - `path: string`
+ The type of the input item. Always `input_audio`.
- The path to the directory containing the skill.
+ - `"input_audio"`
- - `ContainerReference = object { container_id, type }`
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
- - `container_id: string`
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
- The ID of the referenced container.
+ - `TextInput = string`
- - `type: "container_reference"`
+ A text input to the model.
- References a container created with the /v1/containers endpoint
+ - `ResponseInputText object { text, type }`
- - `"container_reference"`
+ A text input to the model.
- - `Custom = object { name, type, defer_loading, 2 more }`
+ - `OutputText object { text, type }`
- A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+ A text output from the model.
- - `name: string`
+ - `text: string`
- The name of the custom tool, used to identify it in tool calls.
+ The text output from the model.
- - `type: "custom"`
+ - `type: "output_text"`
- The type of the custom tool. Always `custom`.
+ The type of the output text. Always `output_text`.
- - `"custom"`
+ - `"output_text"`
- - `defer_loading: optional boolean`
+ - `InputImage object { image_url, type, detail }`
- Whether this tool should be deferred and discovered via tool search.
+ An image input block used within EvalItem content arrays.
- - `description: optional string`
+ - `image_url: string`
- Optional description of the custom tool, used to provide more context.
+ The URL of the image input.
- - `format: optional CustomToolInputFormat`
+ - `type: "input_image"`
- The input format for the custom tool. Default is unconstrained text.
+ The type of the image input. Always `input_image`.
- - `Text = object { type }`
+ - `"input_image"`
- Unconstrained free-form text.
+ - `detail: optional string`
- - `type: "text"`
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- Unconstrained text format. Always `text`.
+ - `ResponseInputAudio object { input_audio, type }`
- - `"text"`
+ An audio input to the model.
- - `Grammar = object { definition, syntax, type }`
+ - `role: "user" or "assistant" or "system" or "developer"`
- A grammar defined by the user.
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- - `definition: string`
+ - `"user"`
- The grammar definition.
+ - `"assistant"`
- - `syntax: "lark" or "regex"`
+ - `"system"`
- The syntax of the grammar definition. One of `lark` or `regex`.
+ - `"developer"`
- - `"lark"`
+ - `type: optional "message"`
- - `"regex"`
+ The type of the message input. Always `message`.
- - `type: "grammar"`
+ - `"message"`
- Grammar format. Always `grammar`.
+ - `type: "template"`
- - `"grammar"`
+ The type of input messages. Always `template`.
- - `Namespace = object { description, name, tools, type }`
+ - `"template"`
- Groups function/custom tools under a shared namespace.
+ - `ItemReferenceInputMessages object { item_reference, type }`
- - `description: string`
+ - `item_reference: string`
- A description of the namespace shown to the model.
+ A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
- - `name: string`
+ - `type: "item_reference"`
- The namespace name used in tool calls (for example, `crm`).
+ The type of input messages. Always `item_reference`.
- - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
+ - `"item_reference"`
- The function/custom tools available inside this namespace.
+ - `model: optional string`
- - `Function = object { name, type, defer_loading, 3 more }`
+ The name of the model to use for generating completions (e.g. "o3-mini").
- - `name: string`
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
- - `type: "function"`
+ - `max_completion_tokens: optional number`
- - `"function"`
+ The maximum number of tokens in the generated output.
- - `defer_loading: optional boolean`
+ - `reasoning_effort: optional ReasoningEffort`
- Whether this function should be deferred and discovered via tool search.
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- - `description: optional string`
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- - `parameters: optional unknown`
+ - `"none"`
- - `strict: optional boolean`
+ - `"minimal"`
- - `Custom = object { name, type, defer_loading, 2 more }`
+ - `"low"`
- A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+ - `"medium"`
- - `name: string`
+ - `"high"`
- The name of the custom tool, used to identify it in tool calls.
+ - `"xhigh"`
- - `type: "custom"`
+ - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
- The type of the custom tool. Always `custom`.
+ An object specifying the format that the model must output.
- - `"custom"`
+ Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
+ Structured Outputs which ensures the model will match your supplied JSON
+ schema. Learn more in the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- - `defer_loading: optional boolean`
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
- Whether this tool should be deferred and discovered via tool search.
+ - `ResponseFormatText object { type }`
- - `description: optional string`
+ Default response format. Used to generate text responses.
- Optional description of the custom tool, used to provide more context.
+ - `type: "text"`
- - `format: optional CustomToolInputFormat`
+ The type of response format being defined. Always `text`.
- The input format for the custom tool. Default is unconstrained text.
+ - `"text"`
- - `Text = object { type }`
+ - `ResponseFormatJSONSchema object { json_schema, type }`
- Unconstrained free-form text.
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
- - `type: "text"`
+ - `json_schema: object { name, description, schema, strict }`
- Unconstrained text format. Always `text`.
+ Structured Outputs configuration options, including a JSON Schema.
- - `"text"`
+ - `name: string`
- - `Grammar = object { definition, syntax, type }`
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
- A grammar defined by the user.
+ - `description: optional string`
- - `definition: string`
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
- The grammar definition.
+ - `schema: optional map[unknown]`
- - `syntax: "lark" or "regex"`
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- The syntax of the grammar definition. One of `lark` or `regex`.
+ - `strict: optional boolean`
- - `"lark"`
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- - `"regex"`
+ - `type: "json_schema"`
- - `type: "grammar"`
+ The type of response format being defined. Always `json_schema`.
- Grammar format. Always `grammar`.
+ - `"json_schema"`
- - `"grammar"`
+ - `ResponseFormatJSONObject object { type }`
- - `type: "namespace"`
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
- The type of the tool. Always `namespace`.
+ - `type: "json_object"`
- - `"namespace"`
+ The type of response format being defined. Always `json_object`.
- - `ToolSearch = object { type, description, execution, parameters }`
+ - `"json_object"`
- Hosted or BYOT tool search configuration for deferred tools.
+ - `seed: optional number`
- - `type: "tool_search"`
+ A seed value to initialize the randomness, during sampling.
- The type of the tool. Always `tool_search`.
+ - `temperature: optional number`
- - `"tool_search"`
+ A higher temperature increases randomness in the outputs.
- - `description: optional string`
+ - `tools: optional array of ChatCompletionFunctionTool`
- Description shown to the model for a client-executed tool search tool.
+ A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
- - `execution: optional "server" or "client"`
+ - `function: FunctionDefinition`
- Whether tool search is executed by the server or by the client.
+ - `name: string`
- - `"server"`
+ The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
- - `"client"`
+ - `description: optional string`
- - `parameters: optional unknown`
+ A description of what the function does, used by the model to choose when and how to call the function.
- Parameter schema for a client-executed tool search tool.
+ - `parameters: optional FunctionParameters`
- - `WebSearchPreview = object { type, search_content_types, search_context_size, user_location }`
+ The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
- This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
+ Omitting `parameters` defines a function with an empty parameter list.
- - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
+ - `strict: optional boolean`
- The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
+ Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
- - `"web_search_preview"`
+ - `type: "function"`
- - `"web_search_preview_2025_03_11"`
+ The type of the tool. Currently, only `function` is supported.
- - `search_content_types: optional array of "text" or "image"`
+ - `"function"`
- - `"text"`
+ - `top_p: optional number`
- - `"image"`
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- - `search_context_size: optional "low" or "medium" or "high"`
+ - `ResponsesRunDataSource object { source, type, input_messages, 2 more }`
- High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+ A ResponsesRunDataSource object describing a model sampling configuration.
- - `"low"`
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
- - `"medium"`
-
- - `"high"`
-
- - `user_location: optional object { type, city, country, 2 more }`
-
- The user's location.
-
- - `type: "approximate"`
-
- The type of location approximation. Always `approximate`.
-
- - `"approximate"`
-
- - `city: optional string`
-
- Free text input for the city of the user, e.g. `San Francisco`.
-
- - `country: optional string`
-
- The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
-
- - `region: optional string`
-
- Free text input for the region of the user, e.g. `California`.
-
- - `timezone: optional string`
-
- The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
-
- - `ApplyPatch = object { type }`
-
- Allows the assistant to create, delete, or update files using unified diffs.
-
- - `type: "apply_patch"`
-
- The type of the tool. Always `apply_patch`.
-
- - `"apply_patch"`
-
- - `top_p: optional number`
-
- An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
-
-- `metadata: optional Metadata`
-
- Set of 16 key-value pairs that can be attached to an object. This can be
- useful for storing additional information about the object in a structured
- format, and querying for objects via API or the dashboard.
-
- Keys are strings with a maximum length of 64 characters. Values are strings
- with a maximum length of 512 characters.
-
-- `name: optional string`
-
- The name of the run.
-
-### Returns
-
-- `id: string`
-
- Unique identifier for the evaluation run.
-
-- `created_at: number`
-
- Unix timestamp (in seconds) when the evaluation run was created.
-
-- `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
-
- Information about the run's data source.
-
- - `CreateEvalJSONLRunDataSource = object { source, type }`
-
- A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
-
- - `source: object { content, type } or object { id, type }`
-
- Determines what populates the `item` namespace in the data source.
+ Determines what populates the `item` namespace in this run's data source.
- - `EvalJSONLFileContentSource = object { content, type }`
+ - `EvalJSONLFileContentSource object { content, type }`
- `content: array of object { item, sample }`
@@ -7351,7 +6687,7 @@ Kicks off a new run for a given evaluation, specifying the data source, and what
- `"file_content"`
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `EvalJSONLFileIDSource object { id, type }`
- `id: string`
@@ -7363,100 +6699,92 @@ Kicks off a new run for a given evaluation, specifying the data source, and what
- `"file_id"`
- - `type: "jsonl"`
-
- The type of data source. Always `jsonl`.
-
- - `"jsonl"`
-
- - `CreateEvalCompletionsRunDataSource = object { source, type, input_messages, 2 more }`
-
- A CompletionsRunDataSource object describing a model sampling configuration.
-
- - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
-
- Determines what populates the `item` namespace in this run's data source.
+ - `EvalResponsesSource object { type, created_after, created_before, 8 more }`
- - `EvalJSONLFileContentSource = object { content, type }`
+ A EvalResponsesSource object describing a run data source configuration.
- - `content: array of object { item, sample }`
+ - `type: "responses"`
- The content of the jsonl file.
+ The type of run data source. Always `responses`.
- - `item: map[unknown]`
+ - `"responses"`
- - `sample: optional map[unknown]`
+ - `created_after: optional number`
- - `type: "file_content"`
+ Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
- The type of jsonl source. Always `file_content`.
+ - `created_before: optional number`
- - `"file_content"`
+ Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `instructions_search: optional string`
- - `id: string`
+ Optional string to search the 'instructions' field. This is a query parameter used to select responses.
- The identifier of the file.
+ - `metadata: optional unknown`
- - `type: "file_id"`
+ Metadata filter for the responses. This is a query parameter used to select responses.
- The type of jsonl source. Always `file_id`.
+ - `model: optional string`
- - `"file_id"`
+ The name of the model to find responses for. This is a query parameter used to select responses.
- - `StoredCompletionsRunDataSource = object { type, created_after, created_before, 3 more }`
+ - `reasoning_effort: optional ReasoningEffort`
- A StoredCompletionsRunDataSource configuration describing a set of filters
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- - `type: "stored_completions"`
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- The type of source. Always `stored_completions`.
+ - `temperature: optional number`
- - `"stored_completions"`
+ Sampling temperature. This is a query parameter used to select responses.
- - `created_after: optional number`
+ - `tools: optional array of string`
- An optional Unix timestamp to filter items created after this time.
+ List of tool names. This is a query parameter used to select responses.
- - `created_before: optional number`
+ - `top_p: optional number`
- An optional Unix timestamp to filter items created before this time.
+ Nucleus sampling parameter. This is a query parameter used to select responses.
- - `limit: optional number`
+ - `users: optional array of string`
- An optional maximum number of items to return.
+ List of user identifiers. This is a query parameter used to select responses.
- - `metadata: optional Metadata`
+ - `type: "responses"`
- Set of 16 key-value pairs that can be attached to an object. This can be
- useful for storing additional information about the object in a structured
- format, and querying for objects via API or the dashboard.
+ The type of run data source. Always `responses`.
- Keys are strings with a maximum length of 64 characters. Values are strings
- with a maximum length of 512 characters.
+ - `"responses"`
- - `model: optional string`
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
- An optional model to filter by (e.g., 'gpt-4o').
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
- - `type: "completions"`
+ - `InputMessagesTemplate object { template, type }`
- The type of run data source. Always `completions`.
+ - `template: array of object { content, role } or object { content, role, type }`
- - `"completions"`
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
- - `input_messages: optional object { template, type } or object { item_reference, type }`
+ - `ChatMessage object { content, role }`
- Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+ - `content: string`
- - `TemplateInputMessages = object { template, type }`
+ The content of the message.
- - `template: array of EasyInputMessage or object { content, role, type }`
+ - `role: string`
- A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+ The role of the message (e.g. "system", "assistant", "user").
- - `EasyInputMessage = object { content, role, phase, type }`
+ - `EvalMessageObject object { content, role, type }`
A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the `developer` or `system` role take
@@ -7464,97 +6792,58 @@ Kicks off a new run for a given evaluation, specifying the data source, and what
`assistant` role are presumed to have been generated by the model in previous
interactions.
- - `content: string or ResponseInputMessageContentList`
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- Text, image, or audio input to the model, used to generate a response.
- Can also contain previous assistant responses.
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- `TextInput = string`
A text input to the model.
- - `ResponseInputMessageContentList = array of ResponseInputContent`
-
- A list of one or many input items to the model, containing different content
- types.
-
- - `ResponseInputText = object { text, type }`
+ - `ResponseInputText object { text, type }`
A text input to the model.
- - `text: string`
-
- The text input to the model.
-
- - `type: "input_text"`
+ - `OutputText object { text, type }`
- The type of the input item. Always `input_text`.
+ A text output from the model.
- - `"input_text"`
+ - `text: string`
- - `ResponseInputImage = object { detail, type, file_id, image_url }`
+ The text output from the model.
- An image input to the model. Learn about [image inputs](/docs/guides/vision).
+ - `type: "output_text"`
- - `detail: "low" or "high" or "auto" or "original"`
+ The type of the output text. Always `output_text`.
- The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
+ - `"output_text"`
- - `"low"`
+ - `InputImage object { image_url, type, detail }`
- - `"high"`
+ An image input block used within EvalItem content arrays.
- - `"auto"`
+ - `image_url: string`
- - `"original"`
+ The URL of the image input.
- `type: "input_image"`
- The type of the input item. Always `input_image`.
+ The type of the image input. Always `input_image`.
- `"input_image"`
- - `file_id: optional string`
-
- The ID of the file to be sent to the model.
-
- - `image_url: optional string`
-
- The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
-
- - `ResponseInputFile = object { type, detail, file_data, 3 more }`
-
- A file input to the model.
-
- - `type: "input_file"`
-
- The type of the input item. Always `input_file`.
-
- - `"input_file"`
-
- - `detail: optional "low" or "high"`
-
- The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
-
- - `"low"`
-
- - `"high"`
-
- - `file_data: optional string`
-
- The content of the file to be sent to the model.
-
- - `file_id: optional string`
+ - `detail: optional string`
- The ID of the file to be sent to the model.
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `file_url: optional string`
+ - `ResponseInputAudio object { input_audio, type }`
- The URL of the file to be sent to the model.
+ An audio input to the model.
- - `filename: optional string`
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
- The name of the file to be sent to the model.
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
- `role: "user" or "assistant" or "system" or "developer"`
@@ -7569,1122 +6858,1064 @@ Kicks off a new run for a given evaluation, specifying the data source, and what
- `"developer"`
- - `phase: optional "commentary" or "final_answer"`
-
- Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
- For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
- phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
-
- - `"commentary"`
-
- - `"final_answer"`
-
- `type: optional "message"`
The type of the message input. Always `message`.
- `"message"`
- - `EvalMessageObject = object { content, role, type }`
+ - `type: "template"`
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ The type of input messages. Always `template`.
- - `content: string or ResponseInputText or object { text, type } or 3 more`
+ - `"template"`
- Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+ - `InputMessagesItemReference object { item_reference, type }`
- - `TextInput = string`
+ - `item_reference: string`
- A text input to the model.
+ A reference to a variable in the `item` namespace. Ie, "item.name"
- - `ResponseInputText = object { text, type }`
+ - `type: "item_reference"`
- A text input to the model.
+ The type of input messages. Always `item_reference`.
- - `text: string`
+ - `"item_reference"`
- The text input to the model.
+ - `model: optional string`
- - `type: "input_text"`
+ The name of the model to use for generating completions (e.g. "o3-mini").
- The type of the input item. Always `input_text`.
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
- - `"input_text"`
+ - `max_completion_tokens: optional number`
- - `OutputText = object { text, type }`
+ The maximum number of tokens in the generated output.
- A text output from the model.
+ - `reasoning_effort: optional ReasoningEffort`
- - `text: string`
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- The text output from the model.
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- - `type: "output_text"`
+ - `seed: optional number`
- The type of the output text. Always `output_text`.
+ A seed value to initialize the randomness, during sampling.
- - `"output_text"`
+ - `temperature: optional number`
- - `InputImage = object { image_url, type, detail }`
+ A higher temperature increases randomness in the outputs.
- An image input block used within EvalItem content arrays.
+ - `text: optional object { format }`
- - `image_url: string`
+ Configuration options for a text response from the model. Can be plain
+ text or structured JSON data. Learn more:
- The URL of the image input.
+ - [Text inputs and outputs](/docs/guides/text)
+ - [Structured Outputs](/docs/guides/structured-outputs)
- - `type: "input_image"`
+ - `format: optional ResponseFormatTextConfig`
- The type of the image input. Always `input_image`.
+ An object specifying the format that the model must output.
- - `"input_image"`
+ Configuring `{ "type": "json_schema" }` enables Structured Outputs,
+ which ensures the model will match your supplied JSON schema. Learn more in the
+ [Structured Outputs guide](/docs/guides/structured-outputs).
- - `detail: optional string`
+ The default format is `{ "type": "text" }` with no additional options.
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ **Not recommended for gpt-4o and newer models:**
- - `ResponseInputAudio = object { input_audio, type }`
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
- An audio input to the model.
+ - `ResponseFormatText object { type }`
- - `input_audio: object { data, format }`
+ Default response format. Used to generate text responses.
- - `data: string`
+ - `ResponseFormatTextJSONSchemaConfig object { name, schema, type, 2 more }`
- Base64-encoded audio data.
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
- - `format: "mp3" or "wav"`
+ - `name: string`
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
- - `"mp3"`
+ - `schema: map[unknown]`
- - `"wav"`
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- - `type: "input_audio"`
+ - `type: "json_schema"`
- The type of the input item. Always `input_audio`.
+ The type of response format being defined. Always `json_schema`.
- - `"input_audio"`
+ - `"json_schema"`
- - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+ - `description: optional string`
- A list of inputs, each of which may be either an input text, output text, input
- image, or input audio object.
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
- - `TextInput = string`
+ - `strict: optional boolean`
- A text input to the model.
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- - `ResponseInputText = object { text, type }`
+ - `ResponseFormatJSONObject object { type }`
- A text input to the model.
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
- - `text: string`
+ - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
- The text input to the model.
+ An array of tools the model may call while generating a response. You
+ can specify which tool to use by setting the `tool_choice` parameter.
- - `type: "input_text"`
+ The two categories of tools you can provide the model are:
- The type of the input item. Always `input_text`.
+ - **Built-in tools**: Tools that are provided by OpenAI that extend the
+ model's capabilities, like [web search](/docs/guides/tools-web-search)
+ or [file search](/docs/guides/tools-file-search). Learn more about
+ [built-in tools](/docs/guides/tools).
+ - **Function calls (custom tools)**: Functions that are defined by you,
+ enabling the model to call your own code. Learn more about
+ [function calling](/docs/guides/function-calling).
- - `"input_text"`
+ - `Function object { name, parameters, strict, 3 more }`
- - `OutputText = object { text, type }`
+ Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
- A text output from the model.
+ - `name: string`
- - `text: string`
+ The name of the function to call.
- The text output from the model.
+ - `parameters: map[unknown]`
- - `type: "output_text"`
+ A JSON schema object describing the parameters of the function.
- The type of the output text. Always `output_text`.
+ - `strict: boolean`
- - `"output_text"`
+ Whether to enforce strict parameter validation. Default `true`.
- - `InputImage = object { image_url, type, detail }`
+ - `type: "function"`
- An image input block used within EvalItem content arrays.
+ The type of the function tool. Always `function`.
- - `image_url: string`
+ - `"function"`
- The URL of the image input.
+ - `defer_loading: optional boolean`
- - `type: "input_image"`
+ Whether this function is deferred and loaded via tool search.
- The type of the image input. Always `input_image`.
+ - `description: optional string`
- - `"input_image"`
+ A description of the function. Used by the model to determine whether or not to call the function.
- - `detail: optional string`
+ - `FileSearch object { type, vector_store_ids, filters, 2 more }`
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
- - `ResponseInputAudio = object { input_audio, type }`
+ - `type: "file_search"`
- An audio input to the model.
+ The type of the file search tool. Always `file_search`.
- - `input_audio: object { data, format }`
+ - `"file_search"`
- - `data: string`
+ - `vector_store_ids: array of string`
- Base64-encoded audio data.
+ The IDs of the vector stores to search.
- - `format: "mp3" or "wav"`
+ - `filters: optional ComparisonFilter or CompoundFilter`
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ A filter to apply.
- - `"mp3"`
+ - `ComparisonFilter object { key, type, value }`
- - `"wav"`
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
- - `type: "input_audio"`
+ - `key: string`
- The type of the input item. Always `input_audio`.
+ The key to compare against the value.
- - `"input_audio"`
+ - `type: "eq" or "ne" or "gt" or 5 more`
- - `role: "user" or "assistant" or "system" or "developer"`
+ Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
+ - `eq`: equals
+ - `ne`: not equal
+ - `gt`: greater than
+ - `gte`: greater than or equal
+ - `lt`: less than
+ - `lte`: less than or equal
+ - `in`: in
+ - `nin`: not in
- - `"user"`
+ - `"eq"`
- - `"assistant"`
+ - `"ne"`
- - `"system"`
+ - `"gt"`
- - `"developer"`
+ - `"gte"`
- - `type: optional "message"`
+ - `"lt"`
- The type of the message input. Always `message`.
+ - `"lte"`
- - `"message"`
+ - `"in"`
- - `type: "template"`
+ - `"nin"`
- The type of input messages. Always `template`.
+ - `value: string or number or boolean or array of string or number`
- - `"template"`
+ The value to compare against the attribute key; supports string, number, or boolean types.
- - `ItemReferenceInputMessages = object { item_reference, type }`
+ - `string`
- - `item_reference: string`
+ - `number`
- A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
+ - `boolean`
- - `type: "item_reference"`
+ - `array of string or number`
- The type of input messages. Always `item_reference`.
+ - `string`
- - `"item_reference"`
+ - `number`
- - `model: optional string`
+ - `CompoundFilter object { filters, type }`
- The name of the model to use for generating completions (e.g. "o3-mini").
+ Combine multiple filters using `and` or `or`.
- - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
+ - `filters: array of ComparisonFilter or unknown`
- - `max_completion_tokens: optional number`
+ Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
- The maximum number of tokens in the generated output.
+ - `ComparisonFilter object { key, type, value }`
- - `reasoning_effort: optional ReasoningEffort`
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ - `unknown`
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ - `type: "and" or "or"`
- - `"none"`
+ Type of operation: `and` or `or`.
- - `"minimal"`
+ - `"and"`
- - `"low"`
+ - `"or"`
- - `"medium"`
+ - `max_num_results: optional number`
- - `"high"`
+ The maximum number of results to return. This number should be between 1 and 50 inclusive.
- - `"xhigh"`
+ - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
- - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
+ Ranking options for search.
- An object specifying the format that the model must output.
+ - `hybrid_search: optional object { embedding_weight, text_weight }`
- Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
- Structured Outputs which ensures the model will match your supplied JSON
- schema. Learn more in the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
- Setting to `{ "type": "json_object" }` enables the older JSON mode, which
- ensures the message the model generates is valid JSON. Using `json_schema`
- is preferred for models that support it.
+ - `embedding_weight: number`
- - `ResponseFormatText = object { type }`
+ The weight of the embedding in the reciprocal ranking fusion.
- Default response format. Used to generate text responses.
+ - `text_weight: number`
- - `type: "text"`
+ The weight of the text in the reciprocal ranking fusion.
- The type of response format being defined. Always `text`.
+ - `ranker: optional "auto" or "default-2024-11-15"`
- - `"text"`
+ The ranker to use for the file search.
- - `ResponseFormatJSONSchema = object { json_schema, type }`
+ - `"auto"`
- JSON Schema response format. Used to generate structured JSON responses.
- Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+ - `"default-2024-11-15"`
- - `json_schema: object { name, description, schema, strict }`
+ - `score_threshold: optional number`
- Structured Outputs configuration options, including a JSON Schema.
+ The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
- - `name: string`
+ - `Computer object { type }`
- The name of the response format. Must be a-z, A-Z, 0-9, or contain
- underscores and dashes, with a maximum length of 64.
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
- - `description: optional string`
+ - `type: "computer"`
- A description of what the response format is for, used by the model to
- determine how to respond in the format.
+ The type of the computer tool. Always `computer`.
- - `schema: optional map[unknown]`
+ - `"computer"`
- The schema for the response format, described as a JSON Schema object.
- Learn how to build JSON schemas [here](https://json-schema.org/).
+ - `ComputerUsePreview object { display_height, display_width, environment, type }`
- - `strict: optional boolean`
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
- Whether to enable strict schema adherence when generating the output.
- If set to true, the model will always follow the exact schema defined
- in the `schema` field. Only a subset of JSON Schema is supported when
- `strict` is `true`. To learn more, read the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ - `display_height: number`
- - `type: "json_schema"`
+ The height of the computer display.
- The type of response format being defined. Always `json_schema`.
+ - `display_width: number`
- - `"json_schema"`
+ The width of the computer display.
- - `ResponseFormatJSONObject = object { type }`
+ - `environment: "windows" or "mac" or "linux" or 2 more`
- JSON object response format. An older method of generating JSON responses.
- Using `json_schema` is recommended for models that support it. Note that the
- model will not generate JSON without a system or user message instructing it
- to do so.
+ The type of computer environment to control.
- - `type: "json_object"`
+ - `"windows"`
- The type of response format being defined. Always `json_object`.
+ - `"mac"`
- - `"json_object"`
+ - `"linux"`
- - `seed: optional number`
+ - `"ubuntu"`
- A seed value to initialize the randomness, during sampling.
+ - `"browser"`
- - `temperature: optional number`
+ - `type: "computer_use_preview"`
- A higher temperature increases randomness in the outputs.
+ The type of the computer use tool. Always `computer_use_preview`.
- - `tools: optional array of ChatCompletionFunctionTool`
+ - `"computer_use_preview"`
- A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
+ - `WebSearch object { type, filters, search_context_size, user_location }`
- - `function: FunctionDefinition`
+ Search the Internet for sources related to the prompt. Learn more about the
+ [web search tool](/docs/guides/tools-web-search).
- - `name: string`
+ - `type: "web_search" or "web_search_2025_08_26"`
- The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
+ The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
- - `description: optional string`
+ - `"web_search"`
- A description of what the function does, used by the model to choose when and how to call the function.
+ - `"web_search_2025_08_26"`
- - `parameters: optional FunctionParameters`
+ - `filters: optional object { allowed_domains }`
- The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
+ Filters for the search.
- Omitting `parameters` defines a function with an empty parameter list.
+ - `allowed_domains: optional array of string`
- - `strict: optional boolean`
+ Allowed domains for the search. If not provided, all domains are allowed.
+ Subdomains of the provided domains are allowed as well.
- Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
+ Example: `["pubmed.ncbi.nlm.nih.gov"]`
- - `type: "function"`
+ - `search_context_size: optional "low" or "medium" or "high"`
- The type of the tool. Currently, only `function` is supported.
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
- - `"function"`
+ - `"low"`
- - `top_p: optional number`
+ - `"medium"`
- An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+ - `"high"`
- - `ResponsesRunDataSource = object { source, type, input_messages, 2 more }`
+ - `user_location: optional object { city, country, region, 2 more }`
- A ResponsesRunDataSource object describing a model sampling configuration.
+ The approximate location of the user.
- - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
+ - `city: optional string`
- Determines what populates the `item` namespace in this run's data source.
+ Free text input for the city of the user, e.g. `San Francisco`.
- - `EvalJSONLFileContentSource = object { content, type }`
+ - `country: optional string`
- - `content: array of object { item, sample }`
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
- The content of the jsonl file.
+ - `region: optional string`
- - `item: map[unknown]`
+ Free text input for the region of the user, e.g. `California`.
- - `sample: optional map[unknown]`
+ - `timezone: optional string`
- - `type: "file_content"`
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
- The type of jsonl source. Always `file_content`.
+ - `type: optional "approximate"`
- - `"file_content"`
+ The type of location approximation. Always `approximate`.
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `"approximate"`
- - `id: string`
+ - `Mcp object { server_label, type, allowed_tools, 7 more }`
- The identifier of the file.
+ Give the model access to additional tools via remote Model Context Protocol
+ (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
- - `type: "file_id"`
+ - `server_label: string`
- The type of jsonl source. Always `file_id`.
+ A label for this MCP server, used to identify it in tool calls.
- - `"file_id"`
+ - `type: "mcp"`
- - `EvalResponsesSource = object { type, created_after, created_before, 8 more }`
+ The type of the MCP tool. Always `mcp`.
- A EvalResponsesSource object describing a run data source configuration.
+ - `"mcp"`
- - `type: "responses"`
+ - `allowed_tools: optional array of string or object { read_only, tool_names }`
- The type of run data source. Always `responses`.
+ List of allowed tool names or a filter object.
- - `"responses"`
+ - `McpAllowedTools = array of string`
- - `created_after: optional number`
+ A string array of allowed tool names
- Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
+ - `McpToolFilter object { read_only, tool_names }`
- - `created_before: optional number`
+ A filter object to specify which tools are allowed.
- Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
+ - `read_only: optional boolean`
- - `instructions_search: optional string`
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- Optional string to search the 'instructions' field. This is a query parameter used to select responses.
+ - `tool_names: optional array of string`
- - `metadata: optional unknown`
+ List of allowed tool names.
- Metadata filter for the responses. This is a query parameter used to select responses.
+ - `authorization: optional string`
- - `model: optional string`
+ An OAuth access token that can be used with a remote MCP server, either
+ with a custom MCP server URL or a service connector. Your application
+ must handle the OAuth authorization flow and provide the token here.
- The name of the model to find responses for. This is a query parameter used to select responses.
+ - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
- - `reasoning_effort: optional ReasoningEffort`
+ Identifier for service connectors, like those available in ChatGPT. One of
+ `server_url` or `connector_id` must be provided. Learn more about service
+ connectors [here](/docs/guides/tools-remote-mcp#connectors).
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ Currently supported `connector_id` values are:
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ - Dropbox: `connector_dropbox`
+ - Gmail: `connector_gmail`
+ - Google Calendar: `connector_googlecalendar`
+ - Google Drive: `connector_googledrive`
+ - Microsoft Teams: `connector_microsoftteams`
+ - Outlook Calendar: `connector_outlookcalendar`
+ - Outlook Email: `connector_outlookemail`
+ - SharePoint: `connector_sharepoint`
- - `"none"`
+ - `"connector_dropbox"`
- - `"minimal"`
+ - `"connector_gmail"`
- - `"low"`
+ - `"connector_googlecalendar"`
- - `"medium"`
+ - `"connector_googledrive"`
- - `"high"`
+ - `"connector_microsoftteams"`
- - `"xhigh"`
+ - `"connector_outlookcalendar"`
- - `temperature: optional number`
+ - `"connector_outlookemail"`
- Sampling temperature. This is a query parameter used to select responses.
+ - `"connector_sharepoint"`
- - `tools: optional array of string`
+ - `defer_loading: optional boolean`
- List of tool names. This is a query parameter used to select responses.
+ Whether this MCP tool is deferred and discovered via tool search.
- - `top_p: optional number`
+ - `headers: optional map[string]`
- Nucleus sampling parameter. This is a query parameter used to select responses.
+ Optional HTTP headers to send to the MCP server. Use for authentication
+ or other purposes.
- - `users: optional array of string`
+ - `require_approval: optional object { always, never } or "always" or "never"`
- List of user identifiers. This is a query parameter used to select responses.
+ Specify which of the MCP server's tools require approval.
- - `type: "responses"`
+ - `McpToolApprovalFilter object { always, never }`
- The type of run data source. Always `responses`.
+ Specify which of the MCP server's tools require approval. Can be
+ `always`, `never`, or a filter object associated with tools
+ that require approval.
- - `"responses"`
+ - `always: optional object { read_only, tool_names }`
- - `input_messages: optional object { template, type } or object { item_reference, type }`
+ A filter object to specify which tools are allowed.
- Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+ - `read_only: optional boolean`
- - `InputMessagesTemplate = object { template, type }`
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- - `template: array of object { content, role } or object { content, role, type }`
+ - `tool_names: optional array of string`
- A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+ List of allowed tool names.
- - `ChatMessage = object { content, role }`
+ - `never: optional object { read_only, tool_names }`
- - `content: string`
+ A filter object to specify which tools are allowed.
- The content of the message.
+ - `read_only: optional boolean`
- - `role: string`
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- The role of the message (e.g. "system", "assistant", "user").
+ - `tool_names: optional array of string`
- - `EvalMessageObject = object { content, role, type }`
+ List of allowed tool names.
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ - `McpToolApprovalSetting = "always" or "never"`
- - `content: string or ResponseInputText or object { text, type } or 3 more`
+ Specify a single approval policy for all tools. One of `always` or
+ `never`. When set to `always`, all tools will require approval. When
+ set to `never`, all tools will not require approval.
- Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+ - `"always"`
- - `TextInput = string`
+ - `"never"`
- A text input to the model.
+ - `server_description: optional string`
- - `ResponseInputText = object { text, type }`
+ Optional description of the MCP server, used to provide more context.
- A text input to the model.
+ - `server_url: optional string`
- - `text: string`
+ The URL for the MCP server. One of `server_url` or `connector_id` must be
+ provided.
- The text input to the model.
+ - `CodeInterpreter object { container, type }`
- - `type: "input_text"`
+ A tool that runs Python code to help generate a response to a prompt.
- The type of the input item. Always `input_text`.
+ - `container: string or object { type, file_ids, memory_limit, network_policy }`
- - `"input_text"`
+ The code interpreter container. Can be a container ID or an object that
+ specifies uploaded file IDs to make available to your code, along with an
+ optional `memory_limit` setting.
- - `OutputText = object { text, type }`
+ - `string`
- A text output from the model.
+ The container ID.
- - `text: string`
+ - `CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy }`
- The text output from the model.
+ Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
- - `type: "output_text"`
+ - `type: "auto"`
- The type of the output text. Always `output_text`.
+ Always `auto`.
- - `"output_text"`
+ - `"auto"`
- - `InputImage = object { image_url, type, detail }`
+ - `file_ids: optional array of string`
- An image input block used within EvalItem content arrays.
+ An optional list of uploaded files to make available to your code.
- - `image_url: string`
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
- The URL of the image input.
+ The memory limit for the code interpreter container.
- - `type: "input_image"`
+ - `"1g"`
- The type of the image input. Always `input_image`.
+ - `"4g"`
- - `"input_image"`
+ - `"16g"`
- - `detail: optional string`
+ - `"64g"`
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
- - `ResponseInputAudio = object { input_audio, type }`
+ Network access policy for the container.
- An audio input to the model.
+ - `ContainerNetworkPolicyDisabled object { type }`
- - `input_audio: object { data, format }`
+ - `type: "disabled"`
- - `data: string`
+ Disable outbound network access. Always `disabled`.
- Base64-encoded audio data.
+ - `"disabled"`
- - `format: "mp3" or "wav"`
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ - `allowed_domains: array of string`
- - `"mp3"`
+ A list of allowed domains when type is `allowlist`.
- - `"wav"`
+ - `type: "allowlist"`
- - `type: "input_audio"`
+ Allow outbound network access only to specified domains. Always `allowlist`.
- The type of the input item. Always `input_audio`.
+ - `"allowlist"`
- - `"input_audio"`
+ - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
- - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+ Optional domain-scoped secrets for allowlisted domains.
- A list of inputs, each of which may be either an input text, output text, input
- image, or input audio object.
+ - `domain: string`
- - `TextInput = string`
+ The domain associated with the secret.
- A text input to the model.
+ - `name: string`
- - `ResponseInputText = object { text, type }`
+ The name of the secret to inject for the domain.
- A text input to the model.
+ - `value: string`
- - `text: string`
+ The secret value to inject for the domain.
- The text input to the model.
+ - `type: "code_interpreter"`
- - `type: "input_text"`
+ The type of the code interpreter tool. Always `code_interpreter`.
- The type of the input item. Always `input_text`.
+ - `"code_interpreter"`
- - `"input_text"`
+ - `ImageGeneration object { type, action, background, 9 more }`
- - `OutputText = object { text, type }`
+ A tool that generates images using the GPT image models.
- A text output from the model.
+ - `type: "image_generation"`
- - `text: string`
+ The type of the image generation tool. Always `image_generation`.
- The text output from the model.
+ - `"image_generation"`
- - `type: "output_text"`
+ - `action: optional "generate" or "edit" or "auto"`
- The type of the output text. Always `output_text`.
+ Whether to generate a new image or edit an existing image. Default: `auto`.
- - `"output_text"`
+ - `"generate"`
- - `InputImage = object { image_url, type, detail }`
+ - `"edit"`
- An image input block used within EvalItem content arrays.
+ - `"auto"`
- - `image_url: string`
+ - `background: optional "transparent" or "opaque" or "auto"`
- The URL of the image input.
+ Background type for the generated image. One of `transparent`,
+ `opaque`, or `auto`. Default: `auto`.
- - `type: "input_image"`
+ - `"transparent"`
- The type of the image input. Always `input_image`.
+ - `"opaque"`
- - `"input_image"`
+ - `"auto"`
- - `detail: optional string`
+ - `input_fidelity: optional "high" or "low"`
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `"high"`
- An audio input to the model.
+ - `"low"`
- - `input_audio: object { data, format }`
+ - `input_image_mask: optional object { file_id, image_url }`
- - `data: string`
+ Optional mask for inpainting. Contains `image_url`
+ (string, optional) and `file_id` (string, optional).
- Base64-encoded audio data.
+ - `file_id: optional string`
- - `format: "mp3" or "wav"`
+ File ID for the mask image.
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ - `image_url: optional string`
- - `"mp3"`
+ Base64-encoded mask image.
- - `"wav"`
+ - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
- - `type: "input_audio"`
+ The image generation model to use. Default: `gpt-image-1`.
- The type of the input item. Always `input_audio`.
+ - `string`
- - `"input_audio"`
+ - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
- - `role: "user" or "assistant" or "system" or "developer"`
+ The image generation model to use. Default: `gpt-image-1`.
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
+ - `"gpt-image-1"`
- - `"user"`
+ - `"gpt-image-1-mini"`
- - `"assistant"`
+ - `"gpt-image-1.5"`
- - `"system"`
+ - `moderation: optional "auto" or "low"`
- - `"developer"`
+ Moderation level for the generated image. Default: `auto`.
- - `type: optional "message"`
+ - `"auto"`
- The type of the message input. Always `message`.
+ - `"low"`
- - `"message"`
+ - `output_compression: optional number`
- - `type: "template"`
+ Compression level for the output image. Default: 100.
- The type of input messages. Always `template`.
+ - `output_format: optional "png" or "webp" or "jpeg"`
- - `"template"`
+ The output format of the generated image. One of `png`, `webp`, or
+ `jpeg`. Default: `png`.
- - `InputMessagesItemReference = object { item_reference, type }`
+ - `"png"`
- - `item_reference: string`
+ - `"webp"`
- A reference to a variable in the `item` namespace. Ie, "item.name"
+ - `"jpeg"`
- - `type: "item_reference"`
+ - `partial_images: optional number`
- The type of input messages. Always `item_reference`.
+ Number of partial images to generate in streaming mode, from 0 (default value) to 3.
- - `"item_reference"`
+ - `quality: optional "low" or "medium" or "high" or "auto"`
- - `model: optional string`
+ The quality of the generated image. One of `low`, `medium`, `high`,
+ or `auto`. Default: `auto`.
- The name of the model to use for generating completions (e.g. "o3-mini").
+ - `"low"`
- - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
+ - `"medium"`
- - `max_completion_tokens: optional number`
+ - `"high"`
- The maximum number of tokens in the generated output.
+ - `"auto"`
- - `reasoning_effort: optional ReasoningEffort`
+ - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ The size of the generated image. One of `1024x1024`, `1024x1536`,
+ `1536x1024`, or `auto`. Default: `auto`.
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ - `"1024x1024"`
- - `"none"`
+ - `"1024x1536"`
- - `"minimal"`
+ - `"1536x1024"`
- - `"low"`
+ - `"auto"`
- - `"medium"`
+ - `LocalShell object { type }`
- - `"high"`
+ A tool that allows the model to execute shell commands in a local environment.
- - `"xhigh"`
+ - `type: "local_shell"`
- - `seed: optional number`
+ The type of the local shell tool. Always `local_shell`.
- A seed value to initialize the randomness, during sampling.
+ - `"local_shell"`
- - `temperature: optional number`
+ - `Shell object { type, environment }`
- A higher temperature increases randomness in the outputs.
+ A tool that allows the model to execute shell commands.
- - `text: optional object { format }`
+ - `type: "shell"`
- Configuration options for a text response from the model. Can be plain
- text or structured JSON data. Learn more:
+ The type of the shell tool. Always `shell`.
- - [Text inputs and outputs](/docs/guides/text)
- - [Structured Outputs](/docs/guides/structured-outputs)
+ - `"shell"`
- - `format: optional ResponseFormatTextConfig`
+ - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
- An object specifying the format that the model must output.
+ - `ContainerAuto object { type, file_ids, memory_limit, 2 more }`
- Configuring `{ "type": "json_schema" }` enables Structured Outputs,
- which ensures the model will match your supplied JSON schema. Learn more in the
- [Structured Outputs guide](/docs/guides/structured-outputs).
+ - `type: "container_auto"`
- The default format is `{ "type": "text" }` with no additional options.
+ Automatically creates a container for this request
- **Not recommended for gpt-4o and newer models:**
+ - `"container_auto"`
- Setting to `{ "type": "json_object" }` enables the older JSON mode, which
- ensures the message the model generates is valid JSON. Using `json_schema`
- is preferred for models that support it.
+ - `file_ids: optional array of string`
- - `ResponseFormatText = object { type }`
+ An optional list of uploaded files to make available to your code.
- Default response format. Used to generate text responses.
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
- - `type: "text"`
+ The memory limit for the container.
- The type of response format being defined. Always `text`.
+ - `"1g"`
- - `"text"`
+ - `"4g"`
- - `ResponseFormatTextJSONSchemaConfig = object { name, schema, type, 2 more }`
+ - `"16g"`
- JSON Schema response format. Used to generate structured JSON responses.
- Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+ - `"64g"`
- - `name: string`
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
- The name of the response format. Must be a-z, A-Z, 0-9, or contain
- underscores and dashes, with a maximum length of 64.
+ Network access policy for the container.
- - `schema: map[unknown]`
+ - `ContainerNetworkPolicyDisabled object { type }`
- The schema for the response format, described as a JSON Schema object.
- Learn how to build JSON schemas [here](https://json-schema.org/).
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
- - `type: "json_schema"`
+ - `skills: optional array of SkillReference or InlineSkill`
- The type of response format being defined. Always `json_schema`.
+ An optional list of skills referenced by id or inline data.
- - `"json_schema"`
+ - `SkillReference object { skill_id, type, version }`
- - `description: optional string`
+ - `skill_id: string`
- A description of what the response format is for, used by the model to
- determine how to respond in the format.
+ The ID of the referenced skill.
- - `strict: optional boolean`
+ - `type: "skill_reference"`
- Whether to enable strict schema adherence when generating the output.
- If set to true, the model will always follow the exact schema defined
- in the `schema` field. Only a subset of JSON Schema is supported when
- `strict` is `true`. To learn more, read the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ References a skill created with the /v1/skills endpoint.
- - `ResponseFormatJSONObject = object { type }`
+ - `"skill_reference"`
- JSON object response format. An older method of generating JSON responses.
- Using `json_schema` is recommended for models that support it. Note that the
- model will not generate JSON without a system or user message instructing it
- to do so.
+ - `version: optional string`
- - `type: "json_object"`
+ Optional skill version. Use a positive integer or 'latest'. Omit for default.
- The type of response format being defined. Always `json_object`.
+ - `InlineSkill object { description, name, source, type }`
- - `"json_object"`
+ - `description: string`
- - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
+ The description of the skill.
- An array of tools the model may call while generating a response. You
- can specify which tool to use by setting the `tool_choice` parameter.
+ - `name: string`
- The two categories of tools you can provide the model are:
+ The name of the skill.
- - **Built-in tools**: Tools that are provided by OpenAI that extend the
- model's capabilities, like [web search](/docs/guides/tools-web-search)
- or [file search](/docs/guides/tools-file-search). Learn more about
- [built-in tools](/docs/guides/tools).
- - **Function calls (custom tools)**: Functions that are defined by you,
- enabling the model to call your own code. Learn more about
- [function calling](/docs/guides/function-calling).
+ - `source: InlineSkillSource`
- - `Function = object { name, parameters, strict, 3 more }`
+ Inline skill payload
- Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
+ - `data: string`
- - `name: string`
+ Base64-encoded skill zip bundle.
- The name of the function to call.
+ - `media_type: "application/zip"`
- - `parameters: map[unknown]`
+ The media type of the inline skill payload. Must be `application/zip`.
- A JSON schema object describing the parameters of the function.
+ - `"application/zip"`
- - `strict: boolean`
+ - `type: "base64"`
- Whether to enforce strict parameter validation. Default `true`.
+ The type of the inline skill source. Must be `base64`.
- - `type: "function"`
+ - `"base64"`
- The type of the function tool. Always `function`.
+ - `type: "inline"`
- - `"function"`
+ Defines an inline skill for this request.
- - `defer_loading: optional boolean`
+ - `"inline"`
- Whether this function is deferred and loaded via tool search.
+ - `LocalEnvironment object { type, skills }`
- - `description: optional string`
+ - `type: "local"`
- A description of the function. Used by the model to determine whether or not to call the function.
+ Use a local computer environment.
- - `FileSearch = object { type, vector_store_ids, filters, 2 more }`
+ - `"local"`
- A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
+ - `skills: optional array of LocalSkill`
- - `type: "file_search"`
+ An optional list of skills.
- The type of the file search tool. Always `file_search`.
+ - `description: string`
- - `"file_search"`
+ The description of the skill.
- - `vector_store_ids: array of string`
+ - `name: string`
- The IDs of the vector stores to search.
+ The name of the skill.
- - `filters: optional ComparisonFilter or CompoundFilter`
+ - `path: string`
- A filter to apply.
+ The path to the directory containing the skill.
- - `ComparisonFilter = object { key, type, value }`
+ - `ContainerReference object { container_id, type }`
- A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+ - `container_id: string`
- - `key: string`
+ The ID of the referenced container.
- The key to compare against the value.
+ - `type: "container_reference"`
- - `type: "eq" or "ne" or "gt" or 5 more`
+ References a container created with the /v1/containers endpoint
- Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+ - `"container_reference"`
- - `eq`: equals
- - `ne`: not equal
- - `gt`: greater than
- - `gte`: greater than or equal
- - `lt`: less than
- - `lte`: less than or equal
- - `in`: in
- - `nin`: not in
+ - `Custom object { name, type, defer_loading, 2 more }`
- - `"eq"`
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
- - `"ne"`
+ - `name: string`
- - `"gt"`
+ The name of the custom tool, used to identify it in tool calls.
- - `"gte"`
+ - `type: "custom"`
- - `"lt"`
+ The type of the custom tool. Always `custom`.
- - `"lte"`
+ - `"custom"`
- - `"in"`
+ - `defer_loading: optional boolean`
- - `"nin"`
+ Whether this tool should be deferred and discovered via tool search.
- - `value: string or number or boolean or array of string or number`
+ - `description: optional string`
- The value to compare against the attribute key; supports string, number, or boolean types.
+ Optional description of the custom tool, used to provide more context.
- - `string`
+ - `format: optional CustomToolInputFormat`
- - `number`
+ The input format for the custom tool. Default is unconstrained text.
- - `boolean`
+ - `Text object { type }`
- - `array of string or number`
+ Unconstrained free-form text.
- - `string`
+ - `type: "text"`
- - `number`
+ Unconstrained text format. Always `text`.
- - `CompoundFilter = object { filters, type }`
+ - `"text"`
- Combine multiple filters using `and` or `or`.
+ - `Grammar object { definition, syntax, type }`
- - `filters: array of ComparisonFilter or unknown`
+ A grammar defined by the user.
- Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
+ - `definition: string`
- - `ComparisonFilter = object { key, type, value }`
+ The grammar definition.
- A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+ - `syntax: "lark" or "regex"`
- - `key: string`
+ The syntax of the grammar definition. One of `lark` or `regex`.
- The key to compare against the value.
+ - `"lark"`
- - `type: "eq" or "ne" or "gt" or 5 more`
+ - `"regex"`
- Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+ - `type: "grammar"`
- - `eq`: equals
- - `ne`: not equal
- - `gt`: greater than
- - `gte`: greater than or equal
- - `lt`: less than
- - `lte`: less than or equal
- - `in`: in
- - `nin`: not in
+ Grammar format. Always `grammar`.
- - `"eq"`
+ - `"grammar"`
- - `"ne"`
+ - `Namespace object { description, name, tools, type }`
- - `"gt"`
+ Groups function/custom tools under a shared namespace.
- - `"gte"`
+ - `description: string`
- - `"lt"`
+ A description of the namespace shown to the model.
- - `"lte"`
+ - `name: string`
- - `"in"`
+ The namespace name used in tool calls (for example, `crm`).
- - `"nin"`
+ - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
- - `value: string or number or boolean or array of string or number`
+ The function/custom tools available inside this namespace.
- The value to compare against the attribute key; supports string, number, or boolean types.
+ - `Function object { name, type, defer_loading, 3 more }`
- - `string`
+ - `name: string`
- - `number`
+ - `type: "function"`
- - `boolean`
+ - `"function"`
- - `array of string or number`
+ - `defer_loading: optional boolean`
- - `string`
+ Whether this function should be deferred and discovered via tool search.
- - `number`
+ - `description: optional string`
- - `unknown`
+ - `parameters: optional unknown`
- - `type: "and" or "or"`
+ - `strict: optional boolean`
- Type of operation: `and` or `or`.
+ - `Custom object { name, type, defer_loading, 2 more }`
- - `"and"`
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
- - `"or"`
+ - `name: string`
- - `max_num_results: optional number`
+ The name of the custom tool, used to identify it in tool calls.
- The maximum number of results to return. This number should be between 1 and 50 inclusive.
+ - `type: "custom"`
- - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
+ The type of the custom tool. Always `custom`.
- Ranking options for search.
+ - `"custom"`
- - `hybrid_search: optional object { embedding_weight, text_weight }`
+ - `defer_loading: optional boolean`
- Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
+ Whether this tool should be deferred and discovered via tool search.
- - `embedding_weight: number`
+ - `description: optional string`
- The weight of the embedding in the reciprocal ranking fusion.
+ Optional description of the custom tool, used to provide more context.
- - `text_weight: number`
+ - `format: optional CustomToolInputFormat`
- The weight of the text in the reciprocal ranking fusion.
+ The input format for the custom tool. Default is unconstrained text.
- - `ranker: optional "auto" or "default-2024-11-15"`
+ - `type: "namespace"`
- The ranker to use for the file search.
-
- - `"auto"`
-
- - `"default-2024-11-15"`
-
- - `score_threshold: optional number`
-
- The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
-
- - `Computer = object { type }`
-
- A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
-
- - `type: "computer"`
-
- The type of the computer tool. Always `computer`.
-
- - `"computer"`
-
- - `ComputerUsePreview = object { display_height, display_width, environment, type }`
-
- A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
-
- - `display_height: number`
-
- The height of the computer display.
+ The type of the tool. Always `namespace`.
- - `display_width: number`
+ - `"namespace"`
- The width of the computer display.
+ - `ToolSearch object { type, description, execution, parameters }`
- - `environment: "windows" or "mac" or "linux" or 2 more`
+ Hosted or BYOT tool search configuration for deferred tools.
- The type of computer environment to control.
+ - `type: "tool_search"`
- - `"windows"`
+ The type of the tool. Always `tool_search`.
- - `"mac"`
+ - `"tool_search"`
- - `"linux"`
+ - `description: optional string`
- - `"ubuntu"`
+ Description shown to the model for a client-executed tool search tool.
- - `"browser"`
+ - `execution: optional "server" or "client"`
- - `type: "computer_use_preview"`
+ Whether tool search is executed by the server or by the client.
- The type of the computer use tool. Always `computer_use_preview`.
+ - `"server"`
- - `"computer_use_preview"`
+ - `"client"`
- - `WebSearch = object { type, filters, search_context_size, user_location }`
+ - `parameters: optional unknown`
- Search the Internet for sources related to the prompt. Learn more about the
- [web search tool](/docs/guides/tools-web-search).
+ Parameter schema for a client-executed tool search tool.
- - `type: "web_search" or "web_search_2025_08_26"`
+ - `WebSearchPreview object { type, search_content_types, search_context_size, user_location }`
- The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
+ This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
- - `"web_search"`
+ - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
- - `"web_search_2025_08_26"`
+ The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
- - `filters: optional object { allowed_domains }`
+ - `"web_search_preview"`
- Filters for the search.
+ - `"web_search_preview_2025_03_11"`
- - `allowed_domains: optional array of string`
+ - `search_content_types: optional array of "text" or "image"`
- Allowed domains for the search. If not provided, all domains are allowed.
- Subdomains of the provided domains are allowed as well.
+ - `"text"`
- Example: `["pubmed.ncbi.nlm.nih.gov"]`
+ - `"image"`
- `search_context_size: optional "low" or "medium" or "high"`
@@ -8696,9 +7927,15 @@ Kicks off a new run for a given evaluation, specifying the data source, and what
- `"high"`
- - `user_location: optional object { city, country, region, 2 more }`
+ - `user_location: optional object { type, city, country, 2 more }`
- The approximate location of the user.
+ The user's location.
+
+ - `type: "approximate"`
+
+ The type of location approximation. Always `approximate`.
+
+ - `"approximate"`
- `city: optional string`
@@ -8716,4318 +7953,4050 @@ Kicks off a new run for a given evaluation, specifying the data source, and what
The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
- - `type: optional "approximate"`
+ - `ApplyPatch object { type }`
- The type of location approximation. Always `approximate`.
+ Allows the assistant to create, delete, or update files using unified diffs.
- - `"approximate"`
+ - `type: "apply_patch"`
- - `Mcp = object { server_label, type, allowed_tools, 7 more }`
+ The type of the tool. Always `apply_patch`.
- Give the model access to additional tools via remote Model Context Protocol
- (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
+ - `"apply_patch"`
- - `server_label: string`
+ - `top_p: optional number`
- A label for this MCP server, used to identify it in tool calls.
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- - `type: "mcp"`
+- `metadata: optional Metadata`
- The type of the MCP tool. Always `mcp`.
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - `"mcp"`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `allowed_tools: optional array of string or object { read_only, tool_names }`
+- `name: optional string`
- List of allowed tool names or a filter object.
+ The name of the run.
- - `McpAllowedTools = array of string`
+### Returns
- A string array of allowed tool names
+- `id: string`
- - `McpToolFilter = object { read_only, tool_names }`
+ Unique identifier for the evaluation run.
- A filter object to specify which tools are allowed.
+- `created_at: number`
- - `read_only: optional boolean`
+ Unix timestamp (in seconds) when the evaluation run was created.
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+- `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
- - `tool_names: optional array of string`
+ Information about the run's data source.
- List of allowed tool names.
+ - `CreateEvalJSONLRunDataSource object { source, type }`
- - `authorization: optional string`
+ A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
- An OAuth access token that can be used with a remote MCP server, either
- with a custom MCP server URL or a service connector. Your application
- must handle the OAuth authorization flow and provide the token here.
+ - `source: object { content, type } or object { id, type }`
- - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
+ Determines what populates the `item` namespace in the data source.
- Identifier for service connectors, like those available in ChatGPT. One of
- `server_url` or `connector_id` must be provided. Learn more about service
- connectors [here](/docs/guides/tools-remote-mcp#connectors).
+ - `EvalJSONLFileContentSource object { content, type }`
- Currently supported `connector_id` values are:
+ - `content: array of object { item, sample }`
- - Dropbox: `connector_dropbox`
- - Gmail: `connector_gmail`
- - Google Calendar: `connector_googlecalendar`
- - Google Drive: `connector_googledrive`
- - Microsoft Teams: `connector_microsoftteams`
- - Outlook Calendar: `connector_outlookcalendar`
- - Outlook Email: `connector_outlookemail`
- - SharePoint: `connector_sharepoint`
+ The content of the jsonl file.
- - `"connector_dropbox"`
+ - `item: map[unknown]`
- - `"connector_gmail"`
+ - `sample: optional map[unknown]`
- - `"connector_googlecalendar"`
+ - `type: "file_content"`
- - `"connector_googledrive"`
+ The type of jsonl source. Always `file_content`.
- - `"connector_microsoftteams"`
+ - `"file_content"`
- - `"connector_outlookcalendar"`
+ - `EvalJSONLFileIDSource object { id, type }`
- - `"connector_outlookemail"`
+ - `id: string`
- - `"connector_sharepoint"`
+ The identifier of the file.
- - `defer_loading: optional boolean`
+ - `type: "file_id"`
- Whether this MCP tool is deferred and discovered via tool search.
+ The type of jsonl source. Always `file_id`.
- - `headers: optional map[string]`
+ - `"file_id"`
- Optional HTTP headers to send to the MCP server. Use for authentication
- or other purposes.
+ - `type: "jsonl"`
- - `require_approval: optional object { always, never } or "always" or "never"`
+ The type of data source. Always `jsonl`.
- Specify which of the MCP server's tools require approval.
+ - `"jsonl"`
- - `McpToolApprovalFilter = object { always, never }`
+ - `CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }`
- Specify which of the MCP server's tools require approval. Can be
- `always`, `never`, or a filter object associated with tools
- that require approval.
+ A CompletionsRunDataSource object describing a model sampling configuration.
- - `always: optional object { read_only, tool_names }`
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
- A filter object to specify which tools are allowed.
+ Determines what populates the `item` namespace in this run's data source.
- - `read_only: optional boolean`
+ - `EvalJSONLFileContentSource object { content, type }`
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ - `content: array of object { item, sample }`
- - `tool_names: optional array of string`
+ The content of the jsonl file.
- List of allowed tool names.
+ - `item: map[unknown]`
- - `never: optional object { read_only, tool_names }`
+ - `sample: optional map[unknown]`
- A filter object to specify which tools are allowed.
+ - `type: "file_content"`
- - `read_only: optional boolean`
+ The type of jsonl source. Always `file_content`.
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ - `"file_content"`
- - `tool_names: optional array of string`
+ - `EvalJSONLFileIDSource object { id, type }`
- List of allowed tool names.
+ - `id: string`
- - `McpToolApprovalSetting = "always" or "never"`
+ The identifier of the file.
- Specify a single approval policy for all tools. One of `always` or
- `never`. When set to `always`, all tools will require approval. When
- set to `never`, all tools will not require approval.
+ - `type: "file_id"`
- - `"always"`
+ The type of jsonl source. Always `file_id`.
- - `"never"`
+ - `"file_id"`
- - `server_description: optional string`
+ - `StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more }`
- Optional description of the MCP server, used to provide more context.
+ A StoredCompletionsRunDataSource configuration describing a set of filters
- - `server_url: optional string`
+ - `type: "stored_completions"`
- The URL for the MCP server. One of `server_url` or `connector_id` must be
- provided.
+ The type of source. Always `stored_completions`.
- - `CodeInterpreter = object { container, type }`
+ - `"stored_completions"`
- A tool that runs Python code to help generate a response to a prompt.
+ - `created_after: optional number`
- - `container: string or object { type, file_ids, memory_limit, network_policy }`
+ An optional Unix timestamp to filter items created after this time.
- The code interpreter container. Can be a container ID or an object that
- specifies uploaded file IDs to make available to your code, along with an
- optional `memory_limit` setting.
+ - `created_before: optional number`
- - `string`
+ An optional Unix timestamp to filter items created before this time.
- The container ID.
+ - `limit: optional number`
- - `CodeInterpreterToolAuto = object { type, file_ids, memory_limit, network_policy }`
+ An optional maximum number of items to return.
- Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
+ - `metadata: optional Metadata`
- - `type: "auto"`
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- Always `auto`.
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `"auto"`
+ - `model: optional string`
- - `file_ids: optional array of string`
+ An optional model to filter by (e.g., 'gpt-4o').
- An optional list of uploaded files to make available to your code.
+ - `type: "completions"`
- - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+ The type of run data source. Always `completions`.
- The memory limit for the code interpreter container.
+ - `"completions"`
- - `"1g"`
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
- - `"4g"`
-
- - `"16g"`
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
- - `"64g"`
+ - `TemplateInputMessages object { template, type }`
- - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+ - `template: array of EasyInputMessage or object { content, role, type }`
- Network access policy for the container.
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
- - `ContainerNetworkPolicyDisabled = object { type }`
+ - `EasyInputMessage object { content, role, phase, type }`
- - `type: "disabled"`
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- Disable outbound network access. Always `disabled`.
+ - `content: string or ResponseInputMessageContentList`
- - `"disabled"`
+ Text, image, or audio input to the model, used to generate a response.
+ Can also contain previous assistant responses.
- - `ContainerNetworkPolicyAllowlist = object { allowed_domains, type, domain_secrets }`
+ - `TextInput = string`
- - `allowed_domains: array of string`
+ A text input to the model.
- A list of allowed domains when type is `allowlist`.
+ - `ResponseInputMessageContentList = array of ResponseInputContent`
- - `type: "allowlist"`
+ A list of one or many input items to the model, containing different content
+ types.
- Allow outbound network access only to specified domains. Always `allowlist`.
+ - `ResponseInputText object { text, type }`
- - `"allowlist"`
+ A text input to the model.
- - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+ - `text: string`
- Optional domain-scoped secrets for allowlisted domains.
+ The text input to the model.
- - `domain: string`
+ - `type: "input_text"`
- The domain associated with the secret.
+ The type of the input item. Always `input_text`.
- - `name: string`
+ - `"input_text"`
- The name of the secret to inject for the domain.
+ - `ResponseInputImage object { detail, type, file_id, image_url }`
- - `value: string`
+ An image input to the model. Learn about [image inputs](/docs/guides/vision).
- The secret value to inject for the domain.
+ - `detail: "low" or "high" or "auto" or "original"`
- - `type: "code_interpreter"`
+ The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
- The type of the code interpreter tool. Always `code_interpreter`.
+ - `"low"`
- - `"code_interpreter"`
+ - `"high"`
- - `ImageGeneration = object { type, action, background, 9 more }`
+ - `"auto"`
- A tool that generates images using the GPT image models.
+ - `"original"`
- - `type: "image_generation"`
+ - `type: "input_image"`
- The type of the image generation tool. Always `image_generation`.
+ The type of the input item. Always `input_image`.
- - `"image_generation"`
+ - `"input_image"`
- - `action: optional "generate" or "edit" or "auto"`
+ - `file_id: optional string`
- Whether to generate a new image or edit an existing image. Default: `auto`.
+ The ID of the file to be sent to the model.
- - `"generate"`
+ - `image_url: optional string`
- - `"edit"`
+ The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
- - `"auto"`
+ - `ResponseInputFile object { type, detail, file_data, 3 more }`
- - `background: optional "transparent" or "opaque" or "auto"`
+ A file input to the model.
- Background type for the generated image. One of `transparent`,
- `opaque`, or `auto`. Default: `auto`.
+ - `type: "input_file"`
- - `"transparent"`
+ The type of the input item. Always `input_file`.
- - `"opaque"`
+ - `"input_file"`
- - `"auto"`
+ - `detail: optional "low" or "high"`
- - `input_fidelity: optional "high" or "low"`
+ The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
- Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
+ - `"low"`
- `"high"`
- - `"low"`
-
- - `input_image_mask: optional object { file_id, image_url }`
+ - `file_data: optional string`
- Optional mask for inpainting. Contains `image_url`
- (string, optional) and `file_id` (string, optional).
+ The content of the file to be sent to the model.
- `file_id: optional string`
- File ID for the mask image.
+ The ID of the file to be sent to the model.
- - `image_url: optional string`
+ - `file_url: optional string`
- Base64-encoded mask image.
+ The URL of the file to be sent to the model.
- - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+ - `filename: optional string`
- The image generation model to use. Default: `gpt-image-1`.
+ The name of the file to be sent to the model.
- - `string`
+ - `role: "user" or "assistant" or "system" or "developer"`
- - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- The image generation model to use. Default: `gpt-image-1`.
+ - `"user"`
- - `"gpt-image-1"`
+ - `"assistant"`
- - `"gpt-image-1-mini"`
+ - `"system"`
- - `"gpt-image-1.5"`
+ - `"developer"`
- - `moderation: optional "auto" or "low"`
+ - `phase: optional "commentary" or "final_answer"`
- Moderation level for the generated image. Default: `auto`.
+ Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
+ For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
+ phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
- - `"auto"`
+ - `"commentary"`
- - `"low"`
+ - `"final_answer"`
- - `output_compression: optional number`
+ - `type: optional "message"`
- Compression level for the output image. Default: 100.
+ The type of the message input. Always `message`.
- - `output_format: optional "png" or "webp" or "jpeg"`
+ - `"message"`
- The output format of the generated image. One of `png`, `webp`, or
- `jpeg`. Default: `png`.
+ - `EvalMessageObject object { content, role, type }`
- - `"png"`
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- - `"webp"`
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- - `"jpeg"`
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- - `partial_images: optional number`
+ - `TextInput = string`
- Number of partial images to generate in streaming mode, from 0 (default value) to 3.
+ A text input to the model.
- - `quality: optional "low" or "medium" or "high" or "auto"`
+ - `ResponseInputText object { text, type }`
- The quality of the generated image. One of `low`, `medium`, `high`,
- or `auto`. Default: `auto`.
+ A text input to the model.
- - `"low"`
+ - `OutputText object { text, type }`
- - `"medium"`
+ A text output from the model.
- - `"high"`
+ - `text: string`
- - `"auto"`
+ The text output from the model.
- - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
+ - `type: "output_text"`
- The size of the generated image. One of `1024x1024`, `1024x1536`,
- `1536x1024`, or `auto`. Default: `auto`.
+ The type of the output text. Always `output_text`.
- - `"1024x1024"`
+ - `"output_text"`
- - `"1024x1536"`
+ - `InputImage object { image_url, type, detail }`
- - `"1536x1024"`
+ An image input block used within EvalItem content arrays.
- - `"auto"`
+ - `image_url: string`
- - `LocalShell = object { type }`
+ The URL of the image input.
- A tool that allows the model to execute shell commands in a local environment.
+ - `type: "input_image"`
- - `type: "local_shell"`
+ The type of the image input. Always `input_image`.
- The type of the local shell tool. Always `local_shell`.
+ - `"input_image"`
- - `"local_shell"`
+ - `detail: optional string`
- - `Shell = object { type, environment }`
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- A tool that allows the model to execute shell commands.
+ - `ResponseInputAudio object { input_audio, type }`
- - `type: "shell"`
+ An audio input to the model.
- The type of the shell tool. Always `shell`.
+ - `input_audio: object { data, format }`
- - `"shell"`
+ - `data: string`
- - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
+ Base64-encoded audio data.
- - `ContainerAuto = object { type, file_ids, memory_limit, 2 more }`
+ - `format: "mp3" or "wav"`
- - `type: "container_auto"`
+ The format of the audio data. Currently supported formats are `mp3` and
+ `wav`.
- Automatically creates a container for this request
+ - `"mp3"`
- - `"container_auto"`
+ - `"wav"`
- - `file_ids: optional array of string`
+ - `type: "input_audio"`
- An optional list of uploaded files to make available to your code.
+ The type of the input item. Always `input_audio`.
- - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+ - `"input_audio"`
- The memory limit for the container.
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
- - `"1g"`
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
- - `"4g"`
+ - `TextInput = string`
- - `"16g"`
+ A text input to the model.
- - `"64g"`
+ - `ResponseInputText object { text, type }`
- - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+ A text input to the model.
- Network access policy for the container.
+ - `OutputText object { text, type }`
- - `ContainerNetworkPolicyDisabled = object { type }`
+ A text output from the model.
- - `type: "disabled"`
+ - `text: string`
- Disable outbound network access. Always `disabled`.
+ The text output from the model.
- - `"disabled"`
+ - `type: "output_text"`
- - `ContainerNetworkPolicyAllowlist = object { allowed_domains, type, domain_secrets }`
+ The type of the output text. Always `output_text`.
- - `allowed_domains: array of string`
+ - `"output_text"`
- A list of allowed domains when type is `allowlist`.
+ - `InputImage object { image_url, type, detail }`
- - `type: "allowlist"`
+ An image input block used within EvalItem content arrays.
- Allow outbound network access only to specified domains. Always `allowlist`.
+ - `image_url: string`
- - `"allowlist"`
+ The URL of the image input.
- - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+ - `type: "input_image"`
- Optional domain-scoped secrets for allowlisted domains.
+ The type of the image input. Always `input_image`.
- - `domain: string`
+ - `"input_image"`
- The domain associated with the secret.
+ - `detail: optional string`
- - `name: string`
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- The name of the secret to inject for the domain.
+ - `ResponseInputAudio object { input_audio, type }`
- - `value: string`
+ An audio input to the model.
- The secret value to inject for the domain.
+ - `role: "user" or "assistant" or "system" or "developer"`
- - `skills: optional array of SkillReference or InlineSkill`
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- An optional list of skills referenced by id or inline data.
+ - `"user"`
- - `SkillReference = object { skill_id, type, version }`
+ - `"assistant"`
- - `skill_id: string`
+ - `"system"`
- The ID of the referenced skill.
+ - `"developer"`
- - `type: "skill_reference"`
+ - `type: optional "message"`
- References a skill created with the /v1/skills endpoint.
+ The type of the message input. Always `message`.
- - `"skill_reference"`
+ - `"message"`
- - `version: optional string`
+ - `type: "template"`
- Optional skill version. Use a positive integer or 'latest'. Omit for default.
+ The type of input messages. Always `template`.
- - `InlineSkill = object { description, name, source, type }`
+ - `"template"`
- - `description: string`
+ - `ItemReferenceInputMessages object { item_reference, type }`
- The description of the skill.
+ - `item_reference: string`
- - `name: string`
+ A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
- The name of the skill.
+ - `type: "item_reference"`
- - `source: InlineSkillSource`
+ The type of input messages. Always `item_reference`.
- Inline skill payload
+ - `"item_reference"`
- - `data: string`
+ - `model: optional string`
- Base64-encoded skill zip bundle.
+ The name of the model to use for generating completions (e.g. "o3-mini").
- - `media_type: "application/zip"`
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
- The media type of the inline skill payload. Must be `application/zip`.
+ - `max_completion_tokens: optional number`
- - `"application/zip"`
+ The maximum number of tokens in the generated output.
- - `type: "base64"`
+ - `reasoning_effort: optional ReasoningEffort`
- The type of the inline skill source. Must be `base64`.
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- - `"base64"`
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- - `type: "inline"`
+ - `"none"`
- Defines an inline skill for this request.
+ - `"minimal"`
- - `"inline"`
+ - `"low"`
- - `LocalEnvironment = object { type, skills }`
+ - `"medium"`
- - `type: "local"`
+ - `"high"`
- Use a local computer environment.
+ - `"xhigh"`
- - `"local"`
+ - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
- - `skills: optional array of LocalSkill`
+ An object specifying the format that the model must output.
- An optional list of skills.
+ Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
+ Structured Outputs which ensures the model will match your supplied JSON
+ schema. Learn more in the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- - `description: string`
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
- The description of the skill.
+ - `ResponseFormatText object { type }`
- - `name: string`
+ Default response format. Used to generate text responses.
- The name of the skill.
+ - `type: "text"`
- - `path: string`
+ The type of response format being defined. Always `text`.
- The path to the directory containing the skill.
+ - `"text"`
- - `ContainerReference = object { container_id, type }`
+ - `ResponseFormatJSONSchema object { json_schema, type }`
- - `container_id: string`
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
- The ID of the referenced container.
+ - `json_schema: object { name, description, schema, strict }`
- - `type: "container_reference"`
+ Structured Outputs configuration options, including a JSON Schema.
- References a container created with the /v1/containers endpoint
+ - `name: string`
- - `"container_reference"`
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
- - `Custom = object { name, type, defer_loading, 2 more }`
+ - `description: optional string`
- A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
- - `name: string`
+ - `schema: optional map[unknown]`
- The name of the custom tool, used to identify it in tool calls.
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- - `type: "custom"`
+ - `strict: optional boolean`
- The type of the custom tool. Always `custom`.
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- - `"custom"`
+ - `type: "json_schema"`
- - `defer_loading: optional boolean`
+ The type of response format being defined. Always `json_schema`.
- Whether this tool should be deferred and discovered via tool search.
+ - `"json_schema"`
- - `description: optional string`
+ - `ResponseFormatJSONObject object { type }`
- Optional description of the custom tool, used to provide more context.
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
- - `format: optional CustomToolInputFormat`
+ - `type: "json_object"`
- The input format for the custom tool. Default is unconstrained text.
+ The type of response format being defined. Always `json_object`.
- - `Text = object { type }`
+ - `"json_object"`
- Unconstrained free-form text.
+ - `seed: optional number`
- - `type: "text"`
+ A seed value to initialize the randomness, during sampling.
- Unconstrained text format. Always `text`.
+ - `temperature: optional number`
- - `"text"`
+ A higher temperature increases randomness in the outputs.
- - `Grammar = object { definition, syntax, type }`
+ - `tools: optional array of ChatCompletionFunctionTool`
- A grammar defined by the user.
+ A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
- - `definition: string`
+ - `function: FunctionDefinition`
- The grammar definition.
+ - `name: string`
- - `syntax: "lark" or "regex"`
+ The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
- The syntax of the grammar definition. One of `lark` or `regex`.
+ - `description: optional string`
- - `"lark"`
+ A description of what the function does, used by the model to choose when and how to call the function.
- - `"regex"`
+ - `parameters: optional FunctionParameters`
- - `type: "grammar"`
+ The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
- Grammar format. Always `grammar`.
+ Omitting `parameters` defines a function with an empty parameter list.
- - `"grammar"`
+ - `strict: optional boolean`
- - `Namespace = object { description, name, tools, type }`
+ Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
- Groups function/custom tools under a shared namespace.
+ - `type: "function"`
- - `description: string`
+ The type of the tool. Currently, only `function` is supported.
- A description of the namespace shown to the model.
+ - `"function"`
- - `name: string`
+ - `top_p: optional number`
- The namespace name used in tool calls (for example, `crm`).
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
+ - `ResponsesRunDataSource object { source, type, input_messages, 2 more }`
- The function/custom tools available inside this namespace.
+ A ResponsesRunDataSource object describing a model sampling configuration.
- - `Function = object { name, type, defer_loading, 3 more }`
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
- - `name: string`
+ Determines what populates the `item` namespace in this run's data source.
- - `type: "function"`
+ - `EvalJSONLFileContentSource object { content, type }`
- - `"function"`
+ - `content: array of object { item, sample }`
- - `defer_loading: optional boolean`
+ The content of the jsonl file.
- Whether this function should be deferred and discovered via tool search.
+ - `item: map[unknown]`
- - `description: optional string`
+ - `sample: optional map[unknown]`
- - `parameters: optional unknown`
+ - `type: "file_content"`
- - `strict: optional boolean`
+ The type of jsonl source. Always `file_content`.
- - `Custom = object { name, type, defer_loading, 2 more }`
+ - `"file_content"`
- A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+ - `EvalJSONLFileIDSource object { id, type }`
- - `name: string`
+ - `id: string`
- The name of the custom tool, used to identify it in tool calls.
+ The identifier of the file.
- - `type: "custom"`
+ - `type: "file_id"`
- The type of the custom tool. Always `custom`.
+ The type of jsonl source. Always `file_id`.
- - `"custom"`
+ - `"file_id"`
- - `defer_loading: optional boolean`
+ - `EvalResponsesSource object { type, created_after, created_before, 8 more }`
- Whether this tool should be deferred and discovered via tool search.
+ A EvalResponsesSource object describing a run data source configuration.
- - `description: optional string`
+ - `type: "responses"`
- Optional description of the custom tool, used to provide more context.
+ The type of run data source. Always `responses`.
- - `format: optional CustomToolInputFormat`
+ - `"responses"`
- The input format for the custom tool. Default is unconstrained text.
+ - `created_after: optional number`
- - `Text = object { type }`
+ Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
- Unconstrained free-form text.
+ - `created_before: optional number`
- - `type: "text"`
+ Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
- Unconstrained text format. Always `text`.
+ - `instructions_search: optional string`
- - `"text"`
+ Optional string to search the 'instructions' field. This is a query parameter used to select responses.
- - `Grammar = object { definition, syntax, type }`
+ - `metadata: optional unknown`
- A grammar defined by the user.
+ Metadata filter for the responses. This is a query parameter used to select responses.
- - `definition: string`
+ - `model: optional string`
- The grammar definition.
+ The name of the model to find responses for. This is a query parameter used to select responses.
- - `syntax: "lark" or "regex"`
+ - `reasoning_effort: optional ReasoningEffort`
- The syntax of the grammar definition. One of `lark` or `regex`.
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- - `"lark"`
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- - `"regex"`
+ - `temperature: optional number`
- - `type: "grammar"`
+ Sampling temperature. This is a query parameter used to select responses.
- Grammar format. Always `grammar`.
+ - `tools: optional array of string`
- - `"grammar"`
+ List of tool names. This is a query parameter used to select responses.
- - `type: "namespace"`
+ - `top_p: optional number`
- The type of the tool. Always `namespace`.
+ Nucleus sampling parameter. This is a query parameter used to select responses.
- - `"namespace"`
+ - `users: optional array of string`
- - `ToolSearch = object { type, description, execution, parameters }`
+ List of user identifiers. This is a query parameter used to select responses.
- Hosted or BYOT tool search configuration for deferred tools.
+ - `type: "responses"`
- - `type: "tool_search"`
+ The type of run data source. Always `responses`.
- The type of the tool. Always `tool_search`.
+ - `"responses"`
- - `"tool_search"`
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
- - `description: optional string`
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
- Description shown to the model for a client-executed tool search tool.
+ - `InputMessagesTemplate object { template, type }`
- - `execution: optional "server" or "client"`
+ - `template: array of object { content, role } or object { content, role, type }`
- Whether tool search is executed by the server or by the client.
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
- - `"server"`
+ - `ChatMessage object { content, role }`
- - `"client"`
+ - `content: string`
- - `parameters: optional unknown`
+ The content of the message.
- Parameter schema for a client-executed tool search tool.
+ - `role: string`
- - `WebSearchPreview = object { type, search_content_types, search_context_size, user_location }`
+ The role of the message (e.g. "system", "assistant", "user").
- This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
+ - `EvalMessageObject object { content, role, type }`
- - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- - `"web_search_preview"`
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- - `"web_search_preview_2025_03_11"`
+ - `TextInput = string`
- - `search_content_types: optional array of "text" or "image"`
+ A text input to the model.
- - `"text"`
+ - `ResponseInputText object { text, type }`
- - `"image"`
+ A text input to the model.
- - `search_context_size: optional "low" or "medium" or "high"`
+ - `OutputText object { text, type }`
- High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+ A text output from the model.
- - `"low"`
+ - `text: string`
- - `"medium"`
+ The text output from the model.
- - `"high"`
+ - `type: "output_text"`
- - `user_location: optional object { type, city, country, 2 more }`
+ The type of the output text. Always `output_text`.
- The user's location.
+ - `"output_text"`
- - `type: "approximate"`
+ - `InputImage object { image_url, type, detail }`
- The type of location approximation. Always `approximate`.
+ An image input block used within EvalItem content arrays.
- - `"approximate"`
+ - `image_url: string`
- - `city: optional string`
+ The URL of the image input.
- Free text input for the city of the user, e.g. `San Francisco`.
+ - `type: "input_image"`
- - `country: optional string`
+ The type of the image input. Always `input_image`.
- The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+ - `"input_image"`
- - `region: optional string`
+ - `detail: optional string`
- Free text input for the region of the user, e.g. `California`.
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `timezone: optional string`
+ - `ResponseInputAudio object { input_audio, type }`
- The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+ An audio input to the model.
- - `ApplyPatch = object { type }`
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
- Allows the assistant to create, delete, or update files using unified diffs.
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
- - `type: "apply_patch"`
+ - `role: "user" or "assistant" or "system" or "developer"`
- The type of the tool. Always `apply_patch`.
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- - `"apply_patch"`
+ - `"user"`
- - `top_p: optional number`
+ - `"assistant"`
- An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+ - `"system"`
-- `error: EvalAPIError`
+ - `"developer"`
- An object representing an error response from the Eval API.
+ - `type: optional "message"`
- - `code: string`
+ The type of the message input. Always `message`.
- The error code.
+ - `"message"`
- - `message: string`
+ - `type: "template"`
- The error message.
+ The type of input messages. Always `template`.
-- `eval_id: string`
+ - `"template"`
- The identifier of the associated evaluation.
+ - `InputMessagesItemReference object { item_reference, type }`
-- `metadata: Metadata`
+ - `item_reference: string`
- Set of 16 key-value pairs that can be attached to an object. This can be
- useful for storing additional information about the object in a structured
- format, and querying for objects via API or the dashboard.
+ A reference to a variable in the `item` namespace. Ie, "item.name"
- Keys are strings with a maximum length of 64 characters. Values are strings
- with a maximum length of 512 characters.
+ - `type: "item_reference"`
-- `model: string`
+ The type of input messages. Always `item_reference`.
- The model that is evaluated, if applicable.
+ - `"item_reference"`
-- `name: string`
+ - `model: optional string`
- The name of the evaluation run.
+ The name of the model to use for generating completions (e.g. "o3-mini").
-- `object: "eval.run"`
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
- The type of the object. Always "eval.run".
+ - `max_completion_tokens: optional number`
- - `"eval.run"`
+ The maximum number of tokens in the generated output.
-- `per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }`
+ - `reasoning_effort: optional ReasoningEffort`
- Usage statistics for each model during the evaluation run.
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- - `cached_tokens: number`
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- The number of tokens retrieved from cache.
+ - `seed: optional number`
- - `completion_tokens: number`
+ A seed value to initialize the randomness, during sampling.
- The number of completion tokens generated.
+ - `temperature: optional number`
- - `invocation_count: number`
+ A higher temperature increases randomness in the outputs.
- The number of invocations.
+ - `text: optional object { format }`
- - `model_name: string`
+ Configuration options for a text response from the model. Can be plain
+ text or structured JSON data. Learn more:
- The name of the model.
+ - [Text inputs and outputs](/docs/guides/text)
+ - [Structured Outputs](/docs/guides/structured-outputs)
- - `prompt_tokens: number`
+ - `format: optional ResponseFormatTextConfig`
- The number of prompt tokens used.
+ An object specifying the format that the model must output.
- - `total_tokens: number`
+ Configuring `{ "type": "json_schema" }` enables Structured Outputs,
+ which ensures the model will match your supplied JSON schema. Learn more in the
+ [Structured Outputs guide](/docs/guides/structured-outputs).
- The total number of tokens used.
+ The default format is `{ "type": "text" }` with no additional options.
-- `per_testing_criteria_results: array of object { failed, passed, testing_criteria }`
+ **Not recommended for gpt-4o and newer models:**
- Results per testing criteria applied during the evaluation run.
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
- - `failed: number`
+ - `ResponseFormatText object { type }`
- Number of tests failed for this criteria.
+ Default response format. Used to generate text responses.
- - `passed: number`
+ - `ResponseFormatTextJSONSchemaConfig object { name, schema, type, 2 more }`
- Number of tests passed for this criteria.
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
- - `testing_criteria: string`
+ - `name: string`
- A description of the testing criteria.
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
-- `report_url: string`
+ - `schema: map[unknown]`
- The URL to the rendered evaluation run report on the UI dashboard.
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
-- `result_counts: object { errored, failed, passed, total }`
+ - `type: "json_schema"`
- Counters summarizing the outcomes of the evaluation run.
+ The type of response format being defined. Always `json_schema`.
- - `errored: number`
+ - `"json_schema"`
- Number of output items that resulted in an error.
+ - `description: optional string`
- - `failed: number`
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
- Number of output items that failed to pass the evaluation.
+ - `strict: optional boolean`
- - `passed: number`
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- Number of output items that passed the evaluation.
+ - `ResponseFormatJSONObject object { type }`
- - `total: number`
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
- Total number of executed output items.
+ - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
-- `status: string`
+ An array of tools the model may call while generating a response. You
+ can specify which tool to use by setting the `tool_choice` parameter.
- The status of the evaluation run.
+ The two categories of tools you can provide the model are:
-### Example
+ - **Built-in tools**: Tools that are provided by OpenAI that extend the
+ model's capabilities, like [web search](/docs/guides/tools-web-search)
+ or [file search](/docs/guides/tools-file-search). Learn more about
+ [built-in tools](/docs/guides/tools).
+ - **Function calls (custom tools)**: Functions that are defined by you,
+ enabling the model to call your own code. Learn more about
+ [function calling](/docs/guides/function-calling).
-```http
-curl https://api.openai.com/v1/evals/$EVAL_ID/runs \
- -H 'Content-Type: application/json' \
- -H "Authorization: Bearer $OPENAI_API_KEY" \
- -d '{
- "data_source": {
- "source": {
- "content": [
- {
- "item": {
- "foo": "bar"
- }
- }
- ],
- "type": "file_content"
- },
- "type": "jsonl"
- }
- }'
-```
+ - `Function object { name, parameters, strict, 3 more }`
-#### Response
+ Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
-```json
-{
- "id": "id",
- "created_at": 0,
- "data_source": {
- "source": {
- "content": [
- {
- "item": {
- "foo": "bar"
- },
- "sample": {
- "foo": "bar"
- }
- }
- ],
- "type": "file_content"
- },
- "type": "jsonl"
- },
- "error": {
- "code": "code",
- "message": "message"
- },
- "eval_id": "eval_id",
- "metadata": {
- "foo": "string"
- },
- "model": "model",
- "name": "name",
- "object": "eval.run",
- "per_model_usage": [
- {
- "cached_tokens": 0,
- "completion_tokens": 0,
- "invocation_count": 0,
- "model_name": "model_name",
- "prompt_tokens": 0,
- "total_tokens": 0
- }
- ],
- "per_testing_criteria_results": [
- {
- "failed": 0,
- "passed": 0,
- "testing_criteria": "testing_criteria"
- }
- ],
- "report_url": "report_url",
- "result_counts": {
- "errored": 0,
- "failed": 0,
- "passed": 0,
- "total": 0
- },
- "status": "status"
-}
-```
+ - `name: string`
-### Example
+ The name of the function to call.
-```http
-curl https://api.openai.com/v1/evals/eval_67e579652b548190aaa83ada4b125f47/runs \
- -X POST \
- -H "Authorization: Bearer $OPENAI_API_KEY" \
- -H "Content-Type: application/json" \
- -d '{"name":"gpt-4o-mini","data_source":{"type":"completions","input_messages":{"type":"template","template":[{"role":"developer","content":"Categorize a given news headline into one of the following topics: Technology, Markets, World, Business, or Sports.\n\n# Steps\n\n1. Analyze the content of the news headline to understand its primary focus.\n2. Extract the subject matter, identifying any key indicators or keywords.\n3. Use the identified indicators to determine the most suitable category out of the five options: Technology, Markets, World, Business, or Sports.\n4. Ensure only one category is selected per headline.\n\n# Output Format\n\nRespond with the chosen category as a single word. For instance: \"Technology\", \"Markets\", \"World\", \"Business\", or \"Sports\".\n\n# Examples\n\n**Input**: \"Apple Unveils New iPhone Model, Featuring Advanced AI Features\" \n**Output**: \"Technology\"\n\n**Input**: \"Global Stocks Mixed as Investors Await Central Bank Decisions\" \n**Output**: \"Markets\"\n\n**Input**: \"War in Ukraine: Latest Updates on Negotiation Status\" \n**Output**: \"World\"\n\n**Input**: \"Microsoft in Talks to Acquire Gaming Company for $2 Billion\" \n**Output**: \"Business\"\n\n**Input**: \"Manchester United Secures Win in Premier League Football Match\" \n**Output**: \"Sports\" \n\n# Notes\n\n- If the headline appears to fit into more than one category, choose the most dominant theme.\n- Keywords or phrases such as \"stocks\", \"company acquisition\", \"match\", or technological brands can be good indicators for classification.\n"} , {"role":"user","content":"{{item.input}}"}]} ,"sampling_params":{"temperature":1,"max_completions_tokens":2048,"top_p":1,"seed":42},"model":"gpt-4o-mini","source":{"type":"file_content","content":[{"item":{"input":"Tech Company Launches Advanced Artificial Intelligence Platform","ground_truth":"Technology"}}]}}'
-```
+ - `parameters: map[unknown]`
-#### Response
+ A JSON schema object describing the parameters of the function.
-```json
-{
- "object": "eval.run",
- "id": "evalrun_67e57965b480819094274e3a32235e4c",
- "eval_id": "eval_67e579652b548190aaa83ada4b125f47",
- "report_url": "https://platform.openai.com/evaluations/eval_67e579652b548190aaa83ada4b125f47&run_id=evalrun_67e57965b480819094274e3a32235e4c",
- "status": "queued",
- "model": "gpt-4o-mini",
- "name": "gpt-4o-mini",
- "created_at": 1743092069,
- "result_counts": {
- "total": 0,
- "errored": 0,
- "failed": 0,
- "passed": 0
- },
- "per_model_usage": null,
- "per_testing_criteria_results": null,
- "data_source": {
- "type": "completions",
- "source": {
- "type": "file_content",
- "content": [
- {
- "item": {
- "input": "Tech Company Launches Advanced Artificial Intelligence Platform",
- "ground_truth": "Technology"
- }
- }
- ]
- },
- "input_messages": {
- "type": "template",
- "template": [
- {
- "type": "message",
- "role": "developer",
- "content": {
- "type": "input_text",
- "text": "Categorize a given news headline into one of the following topics: Technology, Markets, World, Business, or Sports.\n\n# Steps\n\n1. Analyze the content of the news headline to understand its primary focus.\n2. Extract the subject matter, identifying any key indicators or keywords.\n3. Use the identified indicators to determine the most suitable category out of the five options: Technology, Markets, World, Business, or Sports.\n4. Ensure only one category is selected per headline.\n\n# Output Format\n\nRespond with the chosen category as a single word. For instance: \"Technology\", \"Markets\", \"World\", \"Business\", or \"Sports\".\n\n# Examples\n\n**Input**: \"Apple Unveils New iPhone Model, Featuring Advanced AI Features\" \n**Output**: \"Technology\"\n\n**Input**: \"Global Stocks Mixed as Investors Await Central Bank Decisions\" \n**Output**: \"Markets\"\n\n**Input**: \"War in Ukraine: Latest Updates on Negotiation Status\" \n**Output**: \"World\"\n\n**Input**: \"Microsoft in Talks to Acquire Gaming Company for $2 Billion\" \n**Output**: \"Business\"\n\n**Input**: \"Manchester United Secures Win in Premier League Football Match\" \n**Output**: \"Sports\" \n\n# Notes\n\n- If the headline appears to fit into more than one category, choose the most dominant theme.\n- Keywords or phrases such as \"stocks\", \"company acquisition\", \"match\", or technological brands can be good indicators for classification.\n"
- }
- },
- {
- "type": "message",
- "role": "user",
- "content": {
- "type": "input_text",
- "text": "{{item.input}}"
- }
- }
- ]
- },
- "model": "gpt-4o-mini",
- "sampling_params": {
- "seed": 42,
- "temperature": 1.0,
- "top_p": 1.0,
- "max_completions_tokens": 2048
- }
- },
- "error": null,
- "metadata": {}
-}
-```
+ - `strict: boolean`
-## Get an eval run
+ Whether to enforce strict parameter validation. Default `true`.
-**get** `/evals/{eval_id}/runs/{run_id}`
+ - `type: "function"`
-Get an evaluation run by ID.
+ The type of the function tool. Always `function`.
-### Path Parameters
+ - `"function"`
-- `eval_id: string`
+ - `defer_loading: optional boolean`
-- `run_id: string`
+ Whether this function is deferred and loaded via tool search.
-### Returns
+ - `description: optional string`
-- `id: string`
+ A description of the function. Used by the model to determine whether or not to call the function.
- Unique identifier for the evaluation run.
+ - `FileSearch object { type, vector_store_ids, filters, 2 more }`
-- `created_at: number`
+ A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
- Unix timestamp (in seconds) when the evaluation run was created.
+ - `type: "file_search"`
-- `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
+ The type of the file search tool. Always `file_search`.
- Information about the run's data source.
+ - `"file_search"`
- - `CreateEvalJSONLRunDataSource = object { source, type }`
+ - `vector_store_ids: array of string`
- A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
+ The IDs of the vector stores to search.
- - `source: object { content, type } or object { id, type }`
+ - `filters: optional ComparisonFilter or CompoundFilter`
- Determines what populates the `item` namespace in the data source.
+ A filter to apply.
- - `EvalJSONLFileContentSource = object { content, type }`
+ - `ComparisonFilter object { key, type, value }`
- - `content: array of object { item, sample }`
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
- The content of the jsonl file.
+ - `key: string`
- - `item: map[unknown]`
+ The key to compare against the value.
- - `sample: optional map[unknown]`
+ - `type: "eq" or "ne" or "gt" or 5 more`
- - `type: "file_content"`
+ Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
- The type of jsonl source. Always `file_content`.
+ - `eq`: equals
+ - `ne`: not equal
+ - `gt`: greater than
+ - `gte`: greater than or equal
+ - `lt`: less than
+ - `lte`: less than or equal
+ - `in`: in
+ - `nin`: not in
- - `"file_content"`
+ - `"eq"`
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `"ne"`
- - `id: string`
+ - `"gt"`
- The identifier of the file.
+ - `"gte"`
- - `type: "file_id"`
+ - `"lt"`
- The type of jsonl source. Always `file_id`.
+ - `"lte"`
- - `"file_id"`
+ - `"in"`
- - `type: "jsonl"`
+ - `"nin"`
- The type of data source. Always `jsonl`.
+ - `value: string or number or boolean or array of string or number`
- - `"jsonl"`
+ The value to compare against the attribute key; supports string, number, or boolean types.
- - `CreateEvalCompletionsRunDataSource = object { source, type, input_messages, 2 more }`
+ - `string`
- A CompletionsRunDataSource object describing a model sampling configuration.
+ - `number`
- - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
+ - `boolean`
- Determines what populates the `item` namespace in this run's data source.
+ - `array of string or number`
- - `EvalJSONLFileContentSource = object { content, type }`
+ - `string`
- - `content: array of object { item, sample }`
+ - `number`
- The content of the jsonl file.
+ - `CompoundFilter object { filters, type }`
- - `item: map[unknown]`
+ Combine multiple filters using `and` or `or`.
- - `sample: optional map[unknown]`
+ - `filters: array of ComparisonFilter or unknown`
- - `type: "file_content"`
+ Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
- The type of jsonl source. Always `file_content`.
+ - `ComparisonFilter object { key, type, value }`
- - `"file_content"`
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `unknown`
- - `id: string`
+ - `type: "and" or "or"`
- The identifier of the file.
+ Type of operation: `and` or `or`.
- - `type: "file_id"`
+ - `"and"`
- The type of jsonl source. Always `file_id`.
+ - `"or"`
- - `"file_id"`
+ - `max_num_results: optional number`
- - `StoredCompletionsRunDataSource = object { type, created_after, created_before, 3 more }`
+ The maximum number of results to return. This number should be between 1 and 50 inclusive.
- A StoredCompletionsRunDataSource configuration describing a set of filters
+ - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
- - `type: "stored_completions"`
+ Ranking options for search.
- The type of source. Always `stored_completions`.
+ - `hybrid_search: optional object { embedding_weight, text_weight }`
- - `"stored_completions"`
+ Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
- - `created_after: optional number`
+ - `embedding_weight: number`
- An optional Unix timestamp to filter items created after this time.
+ The weight of the embedding in the reciprocal ranking fusion.
- - `created_before: optional number`
+ - `text_weight: number`
- An optional Unix timestamp to filter items created before this time.
+ The weight of the text in the reciprocal ranking fusion.
- - `limit: optional number`
+ - `ranker: optional "auto" or "default-2024-11-15"`
- An optional maximum number of items to return.
+ The ranker to use for the file search.
- - `metadata: optional Metadata`
+ - `"auto"`
- Set of 16 key-value pairs that can be attached to an object. This can be
- useful for storing additional information about the object in a structured
- format, and querying for objects via API or the dashboard.
+ - `"default-2024-11-15"`
- Keys are strings with a maximum length of 64 characters. Values are strings
- with a maximum length of 512 characters.
+ - `score_threshold: optional number`
- - `model: optional string`
+ The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
- An optional model to filter by (e.g., 'gpt-4o').
+ - `Computer object { type }`
- - `type: "completions"`
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
- The type of run data source. Always `completions`.
+ - `type: "computer"`
- - `"completions"`
+ The type of the computer tool. Always `computer`.
- - `input_messages: optional object { template, type } or object { item_reference, type }`
+ - `"computer"`
- Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+ - `ComputerUsePreview object { display_height, display_width, environment, type }`
- - `TemplateInputMessages = object { template, type }`
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
- - `template: array of EasyInputMessage or object { content, role, type }`
+ - `display_height: number`
- A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+ The height of the computer display.
- - `EasyInputMessage = object { content, role, phase, type }`
+ - `display_width: number`
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ The width of the computer display.
- - `content: string or ResponseInputMessageContentList`
+ - `environment: "windows" or "mac" or "linux" or 2 more`
- Text, image, or audio input to the model, used to generate a response.
- Can also contain previous assistant responses.
+ The type of computer environment to control.
- - `TextInput = string`
+ - `"windows"`
- A text input to the model.
+ - `"mac"`
- - `ResponseInputMessageContentList = array of ResponseInputContent`
+ - `"linux"`
- A list of one or many input items to the model, containing different content
- types.
+ - `"ubuntu"`
- - `ResponseInputText = object { text, type }`
+ - `"browser"`
- A text input to the model.
+ - `type: "computer_use_preview"`
- - `text: string`
+ The type of the computer use tool. Always `computer_use_preview`.
- The text input to the model.
+ - `"computer_use_preview"`
- - `type: "input_text"`
+ - `WebSearch object { type, filters, search_context_size, user_location }`
- The type of the input item. Always `input_text`.
+ Search the Internet for sources related to the prompt. Learn more about the
+ [web search tool](/docs/guides/tools-web-search).
- - `"input_text"`
+ - `type: "web_search" or "web_search_2025_08_26"`
- - `ResponseInputImage = object { detail, type, file_id, image_url }`
+ The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
- An image input to the model. Learn about [image inputs](/docs/guides/vision).
+ - `"web_search"`
- - `detail: "low" or "high" or "auto" or "original"`
+ - `"web_search_2025_08_26"`
- The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
+ - `filters: optional object { allowed_domains }`
- - `"low"`
+ Filters for the search.
- - `"high"`
+ - `allowed_domains: optional array of string`
- - `"auto"`
+ Allowed domains for the search. If not provided, all domains are allowed.
+ Subdomains of the provided domains are allowed as well.
- - `"original"`
+ Example: `["pubmed.ncbi.nlm.nih.gov"]`
- - `type: "input_image"`
+ - `search_context_size: optional "low" or "medium" or "high"`
- The type of the input item. Always `input_image`.
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
- - `"input_image"`
+ - `"low"`
- - `file_id: optional string`
+ - `"medium"`
- The ID of the file to be sent to the model.
+ - `"high"`
- - `image_url: optional string`
+ - `user_location: optional object { city, country, region, 2 more }`
- The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
+ The approximate location of the user.
- - `ResponseInputFile = object { type, detail, file_data, 3 more }`
+ - `city: optional string`
- A file input to the model.
+ Free text input for the city of the user, e.g. `San Francisco`.
- - `type: "input_file"`
+ - `country: optional string`
- The type of the input item. Always `input_file`.
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
- - `"input_file"`
+ - `region: optional string`
- - `detail: optional "low" or "high"`
+ Free text input for the region of the user, e.g. `California`.
- The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
+ - `timezone: optional string`
- - `"low"`
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
- - `"high"`
+ - `type: optional "approximate"`
- - `file_data: optional string`
+ The type of location approximation. Always `approximate`.
- The content of the file to be sent to the model.
+ - `"approximate"`
- - `file_id: optional string`
+ - `Mcp object { server_label, type, allowed_tools, 7 more }`
- The ID of the file to be sent to the model.
+ Give the model access to additional tools via remote Model Context Protocol
+ (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
- - `file_url: optional string`
+ - `server_label: string`
- The URL of the file to be sent to the model.
+ A label for this MCP server, used to identify it in tool calls.
- - `filename: optional string`
+ - `type: "mcp"`
- The name of the file to be sent to the model.
+ The type of the MCP tool. Always `mcp`.
- - `role: "user" or "assistant" or "system" or "developer"`
+ - `"mcp"`
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
+ - `allowed_tools: optional array of string or object { read_only, tool_names }`
- - `"user"`
+ List of allowed tool names or a filter object.
- - `"assistant"`
+ - `McpAllowedTools = array of string`
- - `"system"`
+ A string array of allowed tool names
- - `"developer"`
+ - `McpToolFilter object { read_only, tool_names }`
- - `phase: optional "commentary" or "final_answer"`
+ A filter object to specify which tools are allowed.
- Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
- For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
- phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
+ - `read_only: optional boolean`
- - `"commentary"`
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- - `"final_answer"`
+ - `tool_names: optional array of string`
- - `type: optional "message"`
+ List of allowed tool names.
- The type of the message input. Always `message`.
+ - `authorization: optional string`
- - `"message"`
+ An OAuth access token that can be used with a remote MCP server, either
+ with a custom MCP server URL or a service connector. Your application
+ must handle the OAuth authorization flow and provide the token here.
- - `EvalMessageObject = object { content, role, type }`
+ - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ Identifier for service connectors, like those available in ChatGPT. One of
+ `server_url` or `connector_id` must be provided. Learn more about service
+ connectors [here](/docs/guides/tools-remote-mcp#connectors).
- - `content: string or ResponseInputText or object { text, type } or 3 more`
+ Currently supported `connector_id` values are:
- Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+ - Dropbox: `connector_dropbox`
+ - Gmail: `connector_gmail`
+ - Google Calendar: `connector_googlecalendar`
+ - Google Drive: `connector_googledrive`
+ - Microsoft Teams: `connector_microsoftteams`
+ - Outlook Calendar: `connector_outlookcalendar`
+ - Outlook Email: `connector_outlookemail`
+ - SharePoint: `connector_sharepoint`
- - `TextInput = string`
+ - `"connector_dropbox"`
- A text input to the model.
+ - `"connector_gmail"`
- - `ResponseInputText = object { text, type }`
+ - `"connector_googlecalendar"`
- A text input to the model.
+ - `"connector_googledrive"`
- - `text: string`
+ - `"connector_microsoftteams"`
- The text input to the model.
+ - `"connector_outlookcalendar"`
- - `type: "input_text"`
+ - `"connector_outlookemail"`
- The type of the input item. Always `input_text`.
+ - `"connector_sharepoint"`
- - `"input_text"`
+ - `defer_loading: optional boolean`
- - `OutputText = object { text, type }`
+ Whether this MCP tool is deferred and discovered via tool search.
- A text output from the model.
+ - `headers: optional map[string]`
- - `text: string`
+ Optional HTTP headers to send to the MCP server. Use for authentication
+ or other purposes.
- The text output from the model.
+ - `require_approval: optional object { always, never } or "always" or "never"`
- - `type: "output_text"`
+ Specify which of the MCP server's tools require approval.
- The type of the output text. Always `output_text`.
+ - `McpToolApprovalFilter object { always, never }`
- - `"output_text"`
+ Specify which of the MCP server's tools require approval. Can be
+ `always`, `never`, or a filter object associated with tools
+ that require approval.
- - `InputImage = object { image_url, type, detail }`
+ - `always: optional object { read_only, tool_names }`
- An image input block used within EvalItem content arrays.
+ A filter object to specify which tools are allowed.
- - `image_url: string`
+ - `read_only: optional boolean`
- The URL of the image input.
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- - `type: "input_image"`
+ - `tool_names: optional array of string`
- The type of the image input. Always `input_image`.
+ List of allowed tool names.
- - `"input_image"`
+ - `never: optional object { read_only, tool_names }`
- - `detail: optional string`
+ A filter object to specify which tools are allowed.
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ - `read_only: optional boolean`
- - `ResponseInputAudio = object { input_audio, type }`
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- An audio input to the model.
+ - `tool_names: optional array of string`
- - `input_audio: object { data, format }`
+ List of allowed tool names.
- - `data: string`
+ - `McpToolApprovalSetting = "always" or "never"`
- Base64-encoded audio data.
+ Specify a single approval policy for all tools. One of `always` or
+ `never`. When set to `always`, all tools will require approval. When
+ set to `never`, all tools will not require approval.
- - `format: "mp3" or "wav"`
+ - `"always"`
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ - `"never"`
- - `"mp3"`
+ - `server_description: optional string`
- - `"wav"`
+ Optional description of the MCP server, used to provide more context.
- - `type: "input_audio"`
+ - `server_url: optional string`
- The type of the input item. Always `input_audio`.
+ The URL for the MCP server. One of `server_url` or `connector_id` must be
+ provided.
- - `"input_audio"`
+ - `CodeInterpreter object { container, type }`
- - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+ A tool that runs Python code to help generate a response to a prompt.
- A list of inputs, each of which may be either an input text, output text, input
- image, or input audio object.
+ - `container: string or object { type, file_ids, memory_limit, network_policy }`
- - `TextInput = string`
+ The code interpreter container. Can be a container ID or an object that
+ specifies uploaded file IDs to make available to your code, along with an
+ optional `memory_limit` setting.
- A text input to the model.
+ - `string`
- - `ResponseInputText = object { text, type }`
+ The container ID.
- A text input to the model.
+ - `CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy }`
- - `text: string`
+ Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
- The text input to the model.
+ - `type: "auto"`
- - `type: "input_text"`
+ Always `auto`.
- The type of the input item. Always `input_text`.
+ - `"auto"`
- - `"input_text"`
+ - `file_ids: optional array of string`
- - `OutputText = object { text, type }`
+ An optional list of uploaded files to make available to your code.
- A text output from the model.
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
- - `text: string`
+ The memory limit for the code interpreter container.
- The text output from the model.
+ - `"1g"`
- - `type: "output_text"`
+ - `"4g"`
- The type of the output text. Always `output_text`.
+ - `"16g"`
- - `"output_text"`
+ - `"64g"`
- - `InputImage = object { image_url, type, detail }`
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
- An image input block used within EvalItem content arrays.
+ Network access policy for the container.
- - `image_url: string`
+ - `ContainerNetworkPolicyDisabled object { type }`
- The URL of the image input.
+ - `type: "disabled"`
- - `type: "input_image"`
+ Disable outbound network access. Always `disabled`.
- The type of the image input. Always `input_image`.
+ - `"disabled"`
- - `"input_image"`
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
- - `detail: optional string`
+ - `allowed_domains: array of string`
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ A list of allowed domains when type is `allowlist`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `type: "allowlist"`
- An audio input to the model.
+ Allow outbound network access only to specified domains. Always `allowlist`.
- - `input_audio: object { data, format }`
+ - `"allowlist"`
- - `data: string`
+ - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
- Base64-encoded audio data.
+ Optional domain-scoped secrets for allowlisted domains.
- - `format: "mp3" or "wav"`
+ - `domain: string`
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ The domain associated with the secret.
- - `"mp3"`
+ - `name: string`
- - `"wav"`
+ The name of the secret to inject for the domain.
- - `type: "input_audio"`
+ - `value: string`
- The type of the input item. Always `input_audio`.
+ The secret value to inject for the domain.
- - `"input_audio"`
+ - `type: "code_interpreter"`
- - `role: "user" or "assistant" or "system" or "developer"`
+ The type of the code interpreter tool. Always `code_interpreter`.
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
+ - `"code_interpreter"`
- - `"user"`
+ - `ImageGeneration object { type, action, background, 9 more }`
- - `"assistant"`
+ A tool that generates images using the GPT image models.
- - `"system"`
+ - `type: "image_generation"`
- - `"developer"`
+ The type of the image generation tool. Always `image_generation`.
- - `type: optional "message"`
+ - `"image_generation"`
- The type of the message input. Always `message`.
+ - `action: optional "generate" or "edit" or "auto"`
- - `"message"`
+ Whether to generate a new image or edit an existing image. Default: `auto`.
- - `type: "template"`
+ - `"generate"`
- The type of input messages. Always `template`.
+ - `"edit"`
- - `"template"`
+ - `"auto"`
- - `ItemReferenceInputMessages = object { item_reference, type }`
+ - `background: optional "transparent" or "opaque" or "auto"`
- - `item_reference: string`
+ Background type for the generated image. One of `transparent`,
+ `opaque`, or `auto`. Default: `auto`.
- A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
+ - `"transparent"`
- - `type: "item_reference"`
+ - `"opaque"`
- The type of input messages. Always `item_reference`.
+ - `"auto"`
- - `"item_reference"`
+ - `input_fidelity: optional "high" or "low"`
- - `model: optional string`
+ Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
- The name of the model to use for generating completions (e.g. "o3-mini").
+ - `"high"`
- - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
+ - `"low"`
- - `max_completion_tokens: optional number`
+ - `input_image_mask: optional object { file_id, image_url }`
- The maximum number of tokens in the generated output.
+ Optional mask for inpainting. Contains `image_url`
+ (string, optional) and `file_id` (string, optional).
- - `reasoning_effort: optional ReasoningEffort`
+ - `file_id: optional string`
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ File ID for the mask image.
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ - `image_url: optional string`
- - `"none"`
+ Base64-encoded mask image.
- - `"minimal"`
+ - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
- - `"low"`
+ The image generation model to use. Default: `gpt-image-1`.
- - `"medium"`
+ - `string`
- - `"high"`
+ - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
- - `"xhigh"`
+ The image generation model to use. Default: `gpt-image-1`.
- - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
+ - `"gpt-image-1"`
- An object specifying the format that the model must output.
+ - `"gpt-image-1-mini"`
- Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
- Structured Outputs which ensures the model will match your supplied JSON
- schema. Learn more in the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ - `"gpt-image-1.5"`
- Setting to `{ "type": "json_object" }` enables the older JSON mode, which
- ensures the message the model generates is valid JSON. Using `json_schema`
- is preferred for models that support it.
+ - `moderation: optional "auto" or "low"`
- - `ResponseFormatText = object { type }`
+ Moderation level for the generated image. Default: `auto`.
- Default response format. Used to generate text responses.
+ - `"auto"`
- - `type: "text"`
+ - `"low"`
- The type of response format being defined. Always `text`.
+ - `output_compression: optional number`
- - `"text"`
+ Compression level for the output image. Default: 100.
- - `ResponseFormatJSONSchema = object { json_schema, type }`
+ - `output_format: optional "png" or "webp" or "jpeg"`
- JSON Schema response format. Used to generate structured JSON responses.
- Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+ The output format of the generated image. One of `png`, `webp`, or
+ `jpeg`. Default: `png`.
- - `json_schema: object { name, description, schema, strict }`
+ - `"png"`
- Structured Outputs configuration options, including a JSON Schema.
+ - `"webp"`
- - `name: string`
+ - `"jpeg"`
- The name of the response format. Must be a-z, A-Z, 0-9, or contain
- underscores and dashes, with a maximum length of 64.
+ - `partial_images: optional number`
- - `description: optional string`
+ Number of partial images to generate in streaming mode, from 0 (default value) to 3.
- A description of what the response format is for, used by the model to
- determine how to respond in the format.
+ - `quality: optional "low" or "medium" or "high" or "auto"`
- - `schema: optional map[unknown]`
+ The quality of the generated image. One of `low`, `medium`, `high`,
+ or `auto`. Default: `auto`.
- The schema for the response format, described as a JSON Schema object.
- Learn how to build JSON schemas [here](https://json-schema.org/).
+ - `"low"`
- - `strict: optional boolean`
+ - `"medium"`
- Whether to enable strict schema adherence when generating the output.
- If set to true, the model will always follow the exact schema defined
- in the `schema` field. Only a subset of JSON Schema is supported when
- `strict` is `true`. To learn more, read the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ - `"high"`
- - `type: "json_schema"`
+ - `"auto"`
- The type of response format being defined. Always `json_schema`.
+ - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
- - `"json_schema"`
+ The size of the generated image. One of `1024x1024`, `1024x1536`,
+ `1536x1024`, or `auto`. Default: `auto`.
- - `ResponseFormatJSONObject = object { type }`
+ - `"1024x1024"`
- JSON object response format. An older method of generating JSON responses.
- Using `json_schema` is recommended for models that support it. Note that the
- model will not generate JSON without a system or user message instructing it
- to do so.
+ - `"1024x1536"`
- - `type: "json_object"`
+ - `"1536x1024"`
- The type of response format being defined. Always `json_object`.
+ - `"auto"`
- - `"json_object"`
+ - `LocalShell object { type }`
- - `seed: optional number`
+ A tool that allows the model to execute shell commands in a local environment.
- A seed value to initialize the randomness, during sampling.
+ - `type: "local_shell"`
- - `temperature: optional number`
+ The type of the local shell tool. Always `local_shell`.
- A higher temperature increases randomness in the outputs.
+ - `"local_shell"`
- - `tools: optional array of ChatCompletionFunctionTool`
+ - `Shell object { type, environment }`
- A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
+ A tool that allows the model to execute shell commands.
- - `function: FunctionDefinition`
+ - `type: "shell"`
- - `name: string`
+ The type of the shell tool. Always `shell`.
- The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
+ - `"shell"`
- - `description: optional string`
+ - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
- A description of what the function does, used by the model to choose when and how to call the function.
+ - `ContainerAuto object { type, file_ids, memory_limit, 2 more }`
- - `parameters: optional FunctionParameters`
+ - `type: "container_auto"`
- The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
+ Automatically creates a container for this request
- Omitting `parameters` defines a function with an empty parameter list.
+ - `"container_auto"`
- - `strict: optional boolean`
+ - `file_ids: optional array of string`
- Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
+ An optional list of uploaded files to make available to your code.
- - `type: "function"`
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
- The type of the tool. Currently, only `function` is supported.
+ The memory limit for the container.
- - `"function"`
+ - `"1g"`
- - `top_p: optional number`
+ - `"4g"`
- An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+ - `"16g"`
- - `ResponsesRunDataSource = object { source, type, input_messages, 2 more }`
+ - `"64g"`
- A ResponsesRunDataSource object describing a model sampling configuration.
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
- - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
+ Network access policy for the container.
- Determines what populates the `item` namespace in this run's data source.
+ - `ContainerNetworkPolicyDisabled object { type }`
- - `EvalJSONLFileContentSource = object { content, type }`
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
- - `content: array of object { item, sample }`
+ - `skills: optional array of SkillReference or InlineSkill`
- The content of the jsonl file.
+ An optional list of skills referenced by id or inline data.
- - `item: map[unknown]`
+ - `SkillReference object { skill_id, type, version }`
- - `sample: optional map[unknown]`
+ - `skill_id: string`
- - `type: "file_content"`
+ The ID of the referenced skill.
- The type of jsonl source. Always `file_content`.
+ - `type: "skill_reference"`
- - `"file_content"`
+ References a skill created with the /v1/skills endpoint.
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `"skill_reference"`
- - `id: string`
+ - `version: optional string`
- The identifier of the file.
+ Optional skill version. Use a positive integer or 'latest'. Omit for default.
- - `type: "file_id"`
+ - `InlineSkill object { description, name, source, type }`
- The type of jsonl source. Always `file_id`.
+ - `description: string`
- - `"file_id"`
+ The description of the skill.
- - `EvalResponsesSource = object { type, created_after, created_before, 8 more }`
+ - `name: string`
- A EvalResponsesSource object describing a run data source configuration.
+ The name of the skill.
- - `type: "responses"`
+ - `source: InlineSkillSource`
- The type of run data source. Always `responses`.
+ Inline skill payload
- - `"responses"`
+ - `data: string`
- - `created_after: optional number`
+ Base64-encoded skill zip bundle.
- Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
+ - `media_type: "application/zip"`
- - `created_before: optional number`
+ The media type of the inline skill payload. Must be `application/zip`.
- Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
+ - `"application/zip"`
- - `instructions_search: optional string`
+ - `type: "base64"`
- Optional string to search the 'instructions' field. This is a query parameter used to select responses.
+ The type of the inline skill source. Must be `base64`.
- - `metadata: optional unknown`
+ - `"base64"`
- Metadata filter for the responses. This is a query parameter used to select responses.
+ - `type: "inline"`
- - `model: optional string`
+ Defines an inline skill for this request.
- The name of the model to find responses for. This is a query parameter used to select responses.
+ - `"inline"`
- - `reasoning_effort: optional ReasoningEffort`
+ - `LocalEnvironment object { type, skills }`
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ - `type: "local"`
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ Use a local computer environment.
- - `"none"`
+ - `"local"`
- - `"minimal"`
+ - `skills: optional array of LocalSkill`
- - `"low"`
+ An optional list of skills.
- - `"medium"`
+ - `description: string`
- - `"high"`
+ The description of the skill.
- - `"xhigh"`
+ - `name: string`
- - `temperature: optional number`
+ The name of the skill.
- Sampling temperature. This is a query parameter used to select responses.
+ - `path: string`
- - `tools: optional array of string`
+ The path to the directory containing the skill.
- List of tool names. This is a query parameter used to select responses.
+ - `ContainerReference object { container_id, type }`
- - `top_p: optional number`
+ - `container_id: string`
- Nucleus sampling parameter. This is a query parameter used to select responses.
+ The ID of the referenced container.
- - `users: optional array of string`
+ - `type: "container_reference"`
- List of user identifiers. This is a query parameter used to select responses.
+ References a container created with the /v1/containers endpoint
- - `type: "responses"`
+ - `"container_reference"`
- The type of run data source. Always `responses`.
+ - `Custom object { name, type, defer_loading, 2 more }`
- - `"responses"`
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
- - `input_messages: optional object { template, type } or object { item_reference, type }`
+ - `name: string`
- Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+ The name of the custom tool, used to identify it in tool calls.
- - `InputMessagesTemplate = object { template, type }`
+ - `type: "custom"`
- - `template: array of object { content, role } or object { content, role, type }`
+ The type of the custom tool. Always `custom`.
- A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+ - `"custom"`
- - `ChatMessage = object { content, role }`
+ - `defer_loading: optional boolean`
- - `content: string`
+ Whether this tool should be deferred and discovered via tool search.
- The content of the message.
+ - `description: optional string`
- - `role: string`
+ Optional description of the custom tool, used to provide more context.
- The role of the message (e.g. "system", "assistant", "user").
+ - `format: optional CustomToolInputFormat`
- - `EvalMessageObject = object { content, role, type }`
+ The input format for the custom tool. Default is unconstrained text.
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ - `Text object { type }`
- - `content: string or ResponseInputText or object { text, type } or 3 more`
+ Unconstrained free-form text.
- Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+ - `type: "text"`
- - `TextInput = string`
+ Unconstrained text format. Always `text`.
- A text input to the model.
+ - `"text"`
- - `ResponseInputText = object { text, type }`
+ - `Grammar object { definition, syntax, type }`
- A text input to the model.
+ A grammar defined by the user.
- - `text: string`
+ - `definition: string`
- The text input to the model.
+ The grammar definition.
- - `type: "input_text"`
+ - `syntax: "lark" or "regex"`
- The type of the input item. Always `input_text`.
+ The syntax of the grammar definition. One of `lark` or `regex`.
- - `"input_text"`
+ - `"lark"`
- - `OutputText = object { text, type }`
+ - `"regex"`
- A text output from the model.
+ - `type: "grammar"`
- - `text: string`
+ Grammar format. Always `grammar`.
- The text output from the model.
+ - `"grammar"`
- - `type: "output_text"`
+ - `Namespace object { description, name, tools, type }`
- The type of the output text. Always `output_text`.
+ Groups function/custom tools under a shared namespace.
- - `"output_text"`
+ - `description: string`
- - `InputImage = object { image_url, type, detail }`
+ A description of the namespace shown to the model.
- An image input block used within EvalItem content arrays.
+ - `name: string`
- - `image_url: string`
+ The namespace name used in tool calls (for example, `crm`).
- The URL of the image input.
+ - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
- - `type: "input_image"`
+ The function/custom tools available inside this namespace.
- The type of the image input. Always `input_image`.
+ - `Function object { name, type, defer_loading, 3 more }`
- - `"input_image"`
+ - `name: string`
- - `detail: optional string`
+ - `type: "function"`
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ - `"function"`
- - `ResponseInputAudio = object { input_audio, type }`
+ - `defer_loading: optional boolean`
- An audio input to the model.
+ Whether this function should be deferred and discovered via tool search.
- - `input_audio: object { data, format }`
+ - `description: optional string`
- - `data: string`
+ - `parameters: optional unknown`
- Base64-encoded audio data.
+ - `strict: optional boolean`
- - `format: "mp3" or "wav"`
+ - `Custom object { name, type, defer_loading, 2 more }`
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
- - `"mp3"`
+ - `name: string`
- - `"wav"`
+ The name of the custom tool, used to identify it in tool calls.
- - `type: "input_audio"`
+ - `type: "custom"`
- The type of the input item. Always `input_audio`.
+ The type of the custom tool. Always `custom`.
- - `"input_audio"`
+ - `"custom"`
- - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+ - `defer_loading: optional boolean`
- A list of inputs, each of which may be either an input text, output text, input
- image, or input audio object.
+ Whether this tool should be deferred and discovered via tool search.
- - `TextInput = string`
+ - `description: optional string`
- A text input to the model.
+ Optional description of the custom tool, used to provide more context.
- - `ResponseInputText = object { text, type }`
+ - `format: optional CustomToolInputFormat`
- A text input to the model.
+ The input format for the custom tool. Default is unconstrained text.
- - `text: string`
+ - `type: "namespace"`
- The text input to the model.
+ The type of the tool. Always `namespace`.
- - `type: "input_text"`
+ - `"namespace"`
- The type of the input item. Always `input_text`.
+ - `ToolSearch object { type, description, execution, parameters }`
- - `"input_text"`
+ Hosted or BYOT tool search configuration for deferred tools.
- - `OutputText = object { text, type }`
+ - `type: "tool_search"`
- A text output from the model.
+ The type of the tool. Always `tool_search`.
- - `text: string`
+ - `"tool_search"`
- The text output from the model.
+ - `description: optional string`
- - `type: "output_text"`
+ Description shown to the model for a client-executed tool search tool.
- The type of the output text. Always `output_text`.
+ - `execution: optional "server" or "client"`
- - `"output_text"`
+ Whether tool search is executed by the server or by the client.
- - `InputImage = object { image_url, type, detail }`
+ - `"server"`
- An image input block used within EvalItem content arrays.
+ - `"client"`
- - `image_url: string`
+ - `parameters: optional unknown`
- The URL of the image input.
+ Parameter schema for a client-executed tool search tool.
- - `type: "input_image"`
+ - `WebSearchPreview object { type, search_content_types, search_context_size, user_location }`
- The type of the image input. Always `input_image`.
+ This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
- - `"input_image"`
+ - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
- - `detail: optional string`
+ The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ - `"web_search_preview"`
- - `ResponseInputAudio = object { input_audio, type }`
+ - `"web_search_preview_2025_03_11"`
- An audio input to the model.
+ - `search_content_types: optional array of "text" or "image"`
- - `input_audio: object { data, format }`
+ - `"text"`
- - `data: string`
+ - `"image"`
- Base64-encoded audio data.
+ - `search_context_size: optional "low" or "medium" or "high"`
- - `format: "mp3" or "wav"`
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
-
- - `"mp3"`
-
- - `"wav"`
-
- - `type: "input_audio"`
-
- The type of the input item. Always `input_audio`.
-
- - `"input_audio"`
-
- - `role: "user" or "assistant" or "system" or "developer"`
-
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
-
- - `"user"`
-
- - `"assistant"`
-
- - `"system"`
-
- - `"developer"`
-
- - `type: optional "message"`
-
- The type of the message input. Always `message`.
-
- - `"message"`
-
- - `type: "template"`
-
- The type of input messages. Always `template`.
-
- - `"template"`
-
- - `InputMessagesItemReference = object { item_reference, type }`
-
- - `item_reference: string`
-
- A reference to a variable in the `item` namespace. Ie, "item.name"
-
- - `type: "item_reference"`
-
- The type of input messages. Always `item_reference`.
-
- - `"item_reference"`
-
- - `model: optional string`
-
- The name of the model to use for generating completions (e.g. "o3-mini").
-
- - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
-
- - `max_completion_tokens: optional number`
-
- The maximum number of tokens in the generated output.
-
- - `reasoning_effort: optional ReasoningEffort`
-
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
-
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
-
- - `"none"`
-
- - `"minimal"`
-
- - `"low"`
+ - `"low"`
- `"medium"`
- `"high"`
- - `"xhigh"`
-
- - `seed: optional number`
-
- A seed value to initialize the randomness, during sampling.
+ - `user_location: optional object { type, city, country, 2 more }`
- - `temperature: optional number`
+ The user's location.
- A higher temperature increases randomness in the outputs.
+ - `type: "approximate"`
- - `text: optional object { format }`
+ The type of location approximation. Always `approximate`.
- Configuration options for a text response from the model. Can be plain
- text or structured JSON data. Learn more:
+ - `"approximate"`
- - [Text inputs and outputs](/docs/guides/text)
- - [Structured Outputs](/docs/guides/structured-outputs)
+ - `city: optional string`
- - `format: optional ResponseFormatTextConfig`
+ Free text input for the city of the user, e.g. `San Francisco`.
- An object specifying the format that the model must output.
+ - `country: optional string`
- Configuring `{ "type": "json_schema" }` enables Structured Outputs,
- which ensures the model will match your supplied JSON schema. Learn more in the
- [Structured Outputs guide](/docs/guides/structured-outputs).
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
- The default format is `{ "type": "text" }` with no additional options.
+ - `region: optional string`
- **Not recommended for gpt-4o and newer models:**
+ Free text input for the region of the user, e.g. `California`.
- Setting to `{ "type": "json_object" }` enables the older JSON mode, which
- ensures the message the model generates is valid JSON. Using `json_schema`
- is preferred for models that support it.
+ - `timezone: optional string`
- - `ResponseFormatText = object { type }`
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
- Default response format. Used to generate text responses.
+ - `ApplyPatch object { type }`
- - `type: "text"`
+ Allows the assistant to create, delete, or update files using unified diffs.
- The type of response format being defined. Always `text`.
+ - `type: "apply_patch"`
- - `"text"`
+ The type of the tool. Always `apply_patch`.
- - `ResponseFormatTextJSONSchemaConfig = object { name, schema, type, 2 more }`
+ - `"apply_patch"`
- JSON Schema response format. Used to generate structured JSON responses.
- Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+ - `top_p: optional number`
- - `name: string`
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- The name of the response format. Must be a-z, A-Z, 0-9, or contain
- underscores and dashes, with a maximum length of 64.
+- `error: EvalAPIError`
- - `schema: map[unknown]`
+ An object representing an error response from the Eval API.
- The schema for the response format, described as a JSON Schema object.
- Learn how to build JSON schemas [here](https://json-schema.org/).
+ - `code: string`
- - `type: "json_schema"`
+ The error code.
- The type of response format being defined. Always `json_schema`.
+ - `message: string`
- - `"json_schema"`
+ The error message.
- - `description: optional string`
+- `eval_id: string`
- A description of what the response format is for, used by the model to
- determine how to respond in the format.
+ The identifier of the associated evaluation.
- - `strict: optional boolean`
+- `metadata: Metadata`
- Whether to enable strict schema adherence when generating the output.
- If set to true, the model will always follow the exact schema defined
- in the `schema` field. Only a subset of JSON Schema is supported when
- `strict` is `true`. To learn more, read the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - `ResponseFormatJSONObject = object { type }`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- JSON object response format. An older method of generating JSON responses.
- Using `json_schema` is recommended for models that support it. Note that the
- model will not generate JSON without a system or user message instructing it
- to do so.
+- `model: string`
- - `type: "json_object"`
+ The model that is evaluated, if applicable.
- The type of response format being defined. Always `json_object`.
+- `name: string`
- - `"json_object"`
+ The name of the evaluation run.
- - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
+- `object: "eval.run"`
- An array of tools the model may call while generating a response. You
- can specify which tool to use by setting the `tool_choice` parameter.
+ The type of the object. Always "eval.run".
- The two categories of tools you can provide the model are:
+ - `"eval.run"`
- - **Built-in tools**: Tools that are provided by OpenAI that extend the
- model's capabilities, like [web search](/docs/guides/tools-web-search)
- or [file search](/docs/guides/tools-file-search). Learn more about
- [built-in tools](/docs/guides/tools).
- - **Function calls (custom tools)**: Functions that are defined by you,
- enabling the model to call your own code. Learn more about
- [function calling](/docs/guides/function-calling).
+- `per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }`
- - `Function = object { name, parameters, strict, 3 more }`
+ Usage statistics for each model during the evaluation run.
- Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
+ - `cached_tokens: number`
- - `name: string`
+ The number of tokens retrieved from cache.
- The name of the function to call.
+ - `completion_tokens: number`
- - `parameters: map[unknown]`
+ The number of completion tokens generated.
- A JSON schema object describing the parameters of the function.
+ - `invocation_count: number`
- - `strict: boolean`
+ The number of invocations.
- Whether to enforce strict parameter validation. Default `true`.
+ - `model_name: string`
- - `type: "function"`
+ The name of the model.
- The type of the function tool. Always `function`.
+ - `prompt_tokens: number`
- - `"function"`
+ The number of prompt tokens used.
- - `defer_loading: optional boolean`
+ - `total_tokens: number`
- Whether this function is deferred and loaded via tool search.
+ The total number of tokens used.
- - `description: optional string`
+- `per_testing_criteria_results: array of object { failed, passed, testing_criteria }`
- A description of the function. Used by the model to determine whether or not to call the function.
+ Results per testing criteria applied during the evaluation run.
- - `FileSearch = object { type, vector_store_ids, filters, 2 more }`
+ - `failed: number`
- A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
+ Number of tests failed for this criteria.
- - `type: "file_search"`
+ - `passed: number`
- The type of the file search tool. Always `file_search`.
+ Number of tests passed for this criteria.
- - `"file_search"`
+ - `testing_criteria: string`
- - `vector_store_ids: array of string`
+ A description of the testing criteria.
- The IDs of the vector stores to search.
+- `report_url: string`
- - `filters: optional ComparisonFilter or CompoundFilter`
+ The URL to the rendered evaluation run report on the UI dashboard.
- A filter to apply.
+- `result_counts: object { errored, failed, passed, total }`
- - `ComparisonFilter = object { key, type, value }`
+ Counters summarizing the outcomes of the evaluation run.
- A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+ - `errored: number`
- - `key: string`
+ Number of output items that resulted in an error.
- The key to compare against the value.
+ - `failed: number`
- - `type: "eq" or "ne" or "gt" or 5 more`
+ Number of output items that failed to pass the evaluation.
- Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+ - `passed: number`
- - `eq`: equals
- - `ne`: not equal
- - `gt`: greater than
- - `gte`: greater than or equal
- - `lt`: less than
- - `lte`: less than or equal
- - `in`: in
- - `nin`: not in
+ Number of output items that passed the evaluation.
- - `"eq"`
+ - `total: number`
- - `"ne"`
+ Total number of executed output items.
- - `"gt"`
+- `status: string`
- - `"gte"`
+ The status of the evaluation run.
- - `"lt"`
+### Example
- - `"lte"`
+```http
+curl https://api.openai.com/v1/evals/$EVAL_ID/runs \
+ -H 'Content-Type: application/json' \
+ -H "Authorization: Bearer $OPENAI_API_KEY" \
+ -d '{
+ "data_source": {
+ "source": {
+ "content": [
+ {
+ "item": {
+ "foo": "bar"
+ }
+ }
+ ],
+ "type": "file_content"
+ },
+ "type": "jsonl"
+ }
+ }'
+```
- - `"in"`
+#### Response
- - `"nin"`
+```json
+{
+ "id": "id",
+ "created_at": 0,
+ "data_source": {
+ "source": {
+ "content": [
+ {
+ "item": {
+ "foo": "bar"
+ },
+ "sample": {
+ "foo": "bar"
+ }
+ }
+ ],
+ "type": "file_content"
+ },
+ "type": "jsonl"
+ },
+ "error": {
+ "code": "code",
+ "message": "message"
+ },
+ "eval_id": "eval_id",
+ "metadata": {
+ "foo": "string"
+ },
+ "model": "model",
+ "name": "name",
+ "object": "eval.run",
+ "per_model_usage": [
+ {
+ "cached_tokens": 0,
+ "completion_tokens": 0,
+ "invocation_count": 0,
+ "model_name": "model_name",
+ "prompt_tokens": 0,
+ "total_tokens": 0
+ }
+ ],
+ "per_testing_criteria_results": [
+ {
+ "failed": 0,
+ "passed": 0,
+ "testing_criteria": "testing_criteria"
+ }
+ ],
+ "report_url": "report_url",
+ "result_counts": {
+ "errored": 0,
+ "failed": 0,
+ "passed": 0,
+ "total": 0
+ },
+ "status": "status"
+}
+```
- - `value: string or number or boolean or array of string or number`
+### Example
- The value to compare against the attribute key; supports string, number, or boolean types.
+```http
+curl https://api.openai.com/v1/evals/eval_67e579652b548190aaa83ada4b125f47/runs \
+ -X POST \
+ -H "Authorization: Bearer $OPENAI_API_KEY" \
+ -H "Content-Type: application/json" \
+ -d '{"name":"gpt-4o-mini","data_source":{"type":"completions","input_messages":{"type":"template","template":[{"role":"developer","content":"Categorize a given news headline into one of the following topics: Technology, Markets, World, Business, or Sports.\n\n# Steps\n\n1. Analyze the content of the news headline to understand its primary focus.\n2. Extract the subject matter, identifying any key indicators or keywords.\n3. Use the identified indicators to determine the most suitable category out of the five options: Technology, Markets, World, Business, or Sports.\n4. Ensure only one category is selected per headline.\n\n# Output Format\n\nRespond with the chosen category as a single word. For instance: \"Technology\", \"Markets\", \"World\", \"Business\", or \"Sports\".\n\n# Examples\n\n**Input**: \"Apple Unveils New iPhone Model, Featuring Advanced AI Features\" \n**Output**: \"Technology\"\n\n**Input**: \"Global Stocks Mixed as Investors Await Central Bank Decisions\" \n**Output**: \"Markets\"\n\n**Input**: \"War in Ukraine: Latest Updates on Negotiation Status\" \n**Output**: \"World\"\n\n**Input**: \"Microsoft in Talks to Acquire Gaming Company for $2 Billion\" \n**Output**: \"Business\"\n\n**Input**: \"Manchester United Secures Win in Premier League Football Match\" \n**Output**: \"Sports\" \n\n# Notes\n\n- If the headline appears to fit into more than one category, choose the most dominant theme.\n- Keywords or phrases such as \"stocks\", \"company acquisition\", \"match\", or technological brands can be good indicators for classification.\n"} , {"role":"user","content":"{{item.input}}"}]} ,"sampling_params":{"temperature":1,"max_completions_tokens":2048,"top_p":1,"seed":42},"model":"gpt-4o-mini","source":{"type":"file_content","content":[{"item":{"input":"Tech Company Launches Advanced Artificial Intelligence Platform","ground_truth":"Technology"}}]}}'
+```
- - `string`
+#### Response
- - `number`
+```json
+{
+ "object": "eval.run",
+ "id": "evalrun_67e57965b480819094274e3a32235e4c",
+ "eval_id": "eval_67e579652b548190aaa83ada4b125f47",
+ "report_url": "https://platform.openai.com/evaluations/eval_67e579652b548190aaa83ada4b125f47&run_id=evalrun_67e57965b480819094274e3a32235e4c",
+ "status": "queued",
+ "model": "gpt-4o-mini",
+ "name": "gpt-4o-mini",
+ "created_at": 1743092069,
+ "result_counts": {
+ "total": 0,
+ "errored": 0,
+ "failed": 0,
+ "passed": 0
+ },
+ "per_model_usage": null,
+ "per_testing_criteria_results": null,
+ "data_source": {
+ "type": "completions",
+ "source": {
+ "type": "file_content",
+ "content": [
+ {
+ "item": {
+ "input": "Tech Company Launches Advanced Artificial Intelligence Platform",
+ "ground_truth": "Technology"
+ }
+ }
+ ]
+ },
+ "input_messages": {
+ "type": "template",
+ "template": [
+ {
+ "type": "message",
+ "role": "developer",
+ "content": {
+ "type": "input_text",
+ "text": "Categorize a given news headline into one of the following topics: Technology, Markets, World, Business, or Sports.\n\n# Steps\n\n1. Analyze the content of the news headline to understand its primary focus.\n2. Extract the subject matter, identifying any key indicators or keywords.\n3. Use the identified indicators to determine the most suitable category out of the five options: Technology, Markets, World, Business, or Sports.\n4. Ensure only one category is selected per headline.\n\n# Output Format\n\nRespond with the chosen category as a single word. For instance: \"Technology\", \"Markets\", \"World\", \"Business\", or \"Sports\".\n\n# Examples\n\n**Input**: \"Apple Unveils New iPhone Model, Featuring Advanced AI Features\" \n**Output**: \"Technology\"\n\n**Input**: \"Global Stocks Mixed as Investors Await Central Bank Decisions\" \n**Output**: \"Markets\"\n\n**Input**: \"War in Ukraine: Latest Updates on Negotiation Status\" \n**Output**: \"World\"\n\n**Input**: \"Microsoft in Talks to Acquire Gaming Company for $2 Billion\" \n**Output**: \"Business\"\n\n**Input**: \"Manchester United Secures Win in Premier League Football Match\" \n**Output**: \"Sports\" \n\n# Notes\n\n- If the headline appears to fit into more than one category, choose the most dominant theme.\n- Keywords or phrases such as \"stocks\", \"company acquisition\", \"match\", or technological brands can be good indicators for classification.\n"
+ }
+ },
+ {
+ "type": "message",
+ "role": "user",
+ "content": {
+ "type": "input_text",
+ "text": "{{item.input}}"
+ }
+ }
+ ]
+ },
+ "model": "gpt-4o-mini",
+ "sampling_params": {
+ "seed": 42,
+ "temperature": 1.0,
+ "top_p": 1.0,
+ "max_completions_tokens": 2048
+ }
+ },
+ "error": null,
+ "metadata": {}
+}
+```
- - `boolean`
+## Get an eval run
- - `array of string or number`
+**get** `/evals/{eval_id}/runs/{run_id}`
- - `string`
+Get an evaluation run by ID.
- - `number`
+### Path Parameters
- - `CompoundFilter = object { filters, type }`
+- `eval_id: string`
- Combine multiple filters using `and` or `or`.
+- `run_id: string`
- - `filters: array of ComparisonFilter or unknown`
+### Returns
- Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
+- `id: string`
- - `ComparisonFilter = object { key, type, value }`
+ Unique identifier for the evaluation run.
- A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+- `created_at: number`
- - `key: string`
+ Unix timestamp (in seconds) when the evaluation run was created.
- The key to compare against the value.
+- `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
- - `type: "eq" or "ne" or "gt" or 5 more`
+ Information about the run's data source.
- Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+ - `CreateEvalJSONLRunDataSource object { source, type }`
- - `eq`: equals
- - `ne`: not equal
- - `gt`: greater than
- - `gte`: greater than or equal
- - `lt`: less than
- - `lte`: less than or equal
- - `in`: in
- - `nin`: not in
+ A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
- - `"eq"`
+ - `source: object { content, type } or object { id, type }`
- - `"ne"`
+ Determines what populates the `item` namespace in the data source.
- - `"gt"`
+ - `EvalJSONLFileContentSource object { content, type }`
- - `"gte"`
+ - `content: array of object { item, sample }`
- - `"lt"`
+ The content of the jsonl file.
- - `"lte"`
+ - `item: map[unknown]`
- - `"in"`
+ - `sample: optional map[unknown]`
- - `"nin"`
+ - `type: "file_content"`
- - `value: string or number or boolean or array of string or number`
+ The type of jsonl source. Always `file_content`.
- The value to compare against the attribute key; supports string, number, or boolean types.
+ - `"file_content"`
- - `string`
+ - `EvalJSONLFileIDSource object { id, type }`
- - `number`
+ - `id: string`
- - `boolean`
+ The identifier of the file.
- - `array of string or number`
+ - `type: "file_id"`
- - `string`
+ The type of jsonl source. Always `file_id`.
- - `number`
+ - `"file_id"`
- - `unknown`
+ - `type: "jsonl"`
- - `type: "and" or "or"`
+ The type of data source. Always `jsonl`.
- Type of operation: `and` or `or`.
+ - `"jsonl"`
- - `"and"`
+ - `CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }`
- - `"or"`
+ A CompletionsRunDataSource object describing a model sampling configuration.
- - `max_num_results: optional number`
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
- The maximum number of results to return. This number should be between 1 and 50 inclusive.
+ Determines what populates the `item` namespace in this run's data source.
- - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
+ - `EvalJSONLFileContentSource object { content, type }`
- Ranking options for search.
+ - `content: array of object { item, sample }`
- - `hybrid_search: optional object { embedding_weight, text_weight }`
+ The content of the jsonl file.
- Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
+ - `item: map[unknown]`
- - `embedding_weight: number`
+ - `sample: optional map[unknown]`
- The weight of the embedding in the reciprocal ranking fusion.
+ - `type: "file_content"`
- - `text_weight: number`
+ The type of jsonl source. Always `file_content`.
- The weight of the text in the reciprocal ranking fusion.
+ - `"file_content"`
- - `ranker: optional "auto" or "default-2024-11-15"`
+ - `EvalJSONLFileIDSource object { id, type }`
- The ranker to use for the file search.
+ - `id: string`
- - `"auto"`
+ The identifier of the file.
- - `"default-2024-11-15"`
+ - `type: "file_id"`
- - `score_threshold: optional number`
+ The type of jsonl source. Always `file_id`.
- The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
+ - `"file_id"`
- - `Computer = object { type }`
+ - `StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more }`
- A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+ A StoredCompletionsRunDataSource configuration describing a set of filters
- - `type: "computer"`
+ - `type: "stored_completions"`
- The type of the computer tool. Always `computer`.
+ The type of source. Always `stored_completions`.
- - `"computer"`
+ - `"stored_completions"`
- - `ComputerUsePreview = object { display_height, display_width, environment, type }`
+ - `created_after: optional number`
- A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+ An optional Unix timestamp to filter items created after this time.
- - `display_height: number`
+ - `created_before: optional number`
- The height of the computer display.
+ An optional Unix timestamp to filter items created before this time.
- - `display_width: number`
+ - `limit: optional number`
- The width of the computer display.
+ An optional maximum number of items to return.
- - `environment: "windows" or "mac" or "linux" or 2 more`
+ - `metadata: optional Metadata`
- The type of computer environment to control.
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - `"windows"`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `"mac"`
+ - `model: optional string`
- - `"linux"`
+ An optional model to filter by (e.g., 'gpt-4o').
- - `"ubuntu"`
+ - `type: "completions"`
- - `"browser"`
+ The type of run data source. Always `completions`.
- - `type: "computer_use_preview"`
+ - `"completions"`
- The type of the computer use tool. Always `computer_use_preview`.
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
- - `"computer_use_preview"`
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
- - `WebSearch = object { type, filters, search_context_size, user_location }`
+ - `TemplateInputMessages object { template, type }`
- Search the Internet for sources related to the prompt. Learn more about the
- [web search tool](/docs/guides/tools-web-search).
+ - `template: array of EasyInputMessage or object { content, role, type }`
- - `type: "web_search" or "web_search_2025_08_26"`
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
- The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
+ - `EasyInputMessage object { content, role, phase, type }`
- - `"web_search"`
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- - `"web_search_2025_08_26"`
+ - `content: string or ResponseInputMessageContentList`
- - `filters: optional object { allowed_domains }`
+ Text, image, or audio input to the model, used to generate a response.
+ Can also contain previous assistant responses.
- Filters for the search.
+ - `TextInput = string`
- - `allowed_domains: optional array of string`
+ A text input to the model.
- Allowed domains for the search. If not provided, all domains are allowed.
- Subdomains of the provided domains are allowed as well.
+ - `ResponseInputMessageContentList = array of ResponseInputContent`
- Example: `["pubmed.ncbi.nlm.nih.gov"]`
+ A list of one or many input items to the model, containing different content
+ types.
- - `search_context_size: optional "low" or "medium" or "high"`
+ - `ResponseInputText object { text, type }`
- High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+ A text input to the model.
- - `"low"`
+ - `text: string`
- - `"medium"`
+ The text input to the model.
- - `"high"`
+ - `type: "input_text"`
- - `user_location: optional object { city, country, region, 2 more }`
+ The type of the input item. Always `input_text`.
- The approximate location of the user.
+ - `"input_text"`
- - `city: optional string`
+ - `ResponseInputImage object { detail, type, file_id, image_url }`
- Free text input for the city of the user, e.g. `San Francisco`.
+ An image input to the model. Learn about [image inputs](/docs/guides/vision).
- - `country: optional string`
+ - `detail: "low" or "high" or "auto" or "original"`
- The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+ The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
- - `region: optional string`
+ - `"low"`
- Free text input for the region of the user, e.g. `California`.
+ - `"high"`
- - `timezone: optional string`
+ - `"auto"`
- The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+ - `"original"`
- - `type: optional "approximate"`
+ - `type: "input_image"`
- The type of location approximation. Always `approximate`.
+ The type of the input item. Always `input_image`.
- - `"approximate"`
+ - `"input_image"`
- - `Mcp = object { server_label, type, allowed_tools, 7 more }`
+ - `file_id: optional string`
- Give the model access to additional tools via remote Model Context Protocol
- (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
+ The ID of the file to be sent to the model.
- - `server_label: string`
+ - `image_url: optional string`
- A label for this MCP server, used to identify it in tool calls.
+ The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
- - `type: "mcp"`
+ - `ResponseInputFile object { type, detail, file_data, 3 more }`
- The type of the MCP tool. Always `mcp`.
+ A file input to the model.
- - `"mcp"`
+ - `type: "input_file"`
- - `allowed_tools: optional array of string or object { read_only, tool_names }`
+ The type of the input item. Always `input_file`.
- List of allowed tool names or a filter object.
+ - `"input_file"`
- - `McpAllowedTools = array of string`
+ - `detail: optional "low" or "high"`
- A string array of allowed tool names
+ The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
- - `McpToolFilter = object { read_only, tool_names }`
+ - `"low"`
- A filter object to specify which tools are allowed.
+ - `"high"`
- - `read_only: optional boolean`
+ - `file_data: optional string`
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ The content of the file to be sent to the model.
- - `tool_names: optional array of string`
+ - `file_id: optional string`
- List of allowed tool names.
+ The ID of the file to be sent to the model.
- - `authorization: optional string`
+ - `file_url: optional string`
- An OAuth access token that can be used with a remote MCP server, either
- with a custom MCP server URL or a service connector. Your application
- must handle the OAuth authorization flow and provide the token here.
+ The URL of the file to be sent to the model.
- - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
+ - `filename: optional string`
- Identifier for service connectors, like those available in ChatGPT. One of
- `server_url` or `connector_id` must be provided. Learn more about service
- connectors [here](/docs/guides/tools-remote-mcp#connectors).
+ The name of the file to be sent to the model.
- Currently supported `connector_id` values are:
+ - `role: "user" or "assistant" or "system" or "developer"`
- - Dropbox: `connector_dropbox`
- - Gmail: `connector_gmail`
- - Google Calendar: `connector_googlecalendar`
- - Google Drive: `connector_googledrive`
- - Microsoft Teams: `connector_microsoftteams`
- - Outlook Calendar: `connector_outlookcalendar`
- - Outlook Email: `connector_outlookemail`
- - SharePoint: `connector_sharepoint`
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- - `"connector_dropbox"`
+ - `"user"`
- - `"connector_gmail"`
+ - `"assistant"`
- - `"connector_googlecalendar"`
+ - `"system"`
- - `"connector_googledrive"`
+ - `"developer"`
- - `"connector_microsoftteams"`
+ - `phase: optional "commentary" or "final_answer"`
- - `"connector_outlookcalendar"`
+ Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
+ For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
+ phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
- - `"connector_outlookemail"`
+ - `"commentary"`
- - `"connector_sharepoint"`
+ - `"final_answer"`
- - `defer_loading: optional boolean`
+ - `type: optional "message"`
- Whether this MCP tool is deferred and discovered via tool search.
+ The type of the message input. Always `message`.
- - `headers: optional map[string]`
+ - `"message"`
- Optional HTTP headers to send to the MCP server. Use for authentication
- or other purposes.
+ - `EvalMessageObject object { content, role, type }`
- - `require_approval: optional object { always, never } or "always" or "never"`
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- Specify which of the MCP server's tools require approval.
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- - `McpToolApprovalFilter = object { always, never }`
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- Specify which of the MCP server's tools require approval. Can be
- `always`, `never`, or a filter object associated with tools
- that require approval.
+ - `TextInput = string`
- - `always: optional object { read_only, tool_names }`
+ A text input to the model.
- A filter object to specify which tools are allowed.
+ - `ResponseInputText object { text, type }`
- - `read_only: optional boolean`
+ A text input to the model.
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ - `OutputText object { text, type }`
- - `tool_names: optional array of string`
+ A text output from the model.
- List of allowed tool names.
+ - `text: string`
- - `never: optional object { read_only, tool_names }`
+ The text output from the model.
- A filter object to specify which tools are allowed.
+ - `type: "output_text"`
- - `read_only: optional boolean`
+ The type of the output text. Always `output_text`.
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ - `"output_text"`
- - `tool_names: optional array of string`
+ - `InputImage object { image_url, type, detail }`
- List of allowed tool names.
+ An image input block used within EvalItem content arrays.
- - `McpToolApprovalSetting = "always" or "never"`
+ - `image_url: string`
- Specify a single approval policy for all tools. One of `always` or
- `never`. When set to `always`, all tools will require approval. When
- set to `never`, all tools will not require approval.
+ The URL of the image input.
- - `"always"`
+ - `type: "input_image"`
- - `"never"`
+ The type of the image input. Always `input_image`.
- - `server_description: optional string`
+ - `"input_image"`
- Optional description of the MCP server, used to provide more context.
+ - `detail: optional string`
- - `server_url: optional string`
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- The URL for the MCP server. One of `server_url` or `connector_id` must be
- provided.
+ - `ResponseInputAudio object { input_audio, type }`
- - `CodeInterpreter = object { container, type }`
+ An audio input to the model.
- A tool that runs Python code to help generate a response to a prompt.
+ - `input_audio: object { data, format }`
- - `container: string or object { type, file_ids, memory_limit, network_policy }`
+ - `data: string`
- The code interpreter container. Can be a container ID or an object that
- specifies uploaded file IDs to make available to your code, along with an
- optional `memory_limit` setting.
+ Base64-encoded audio data.
- - `string`
+ - `format: "mp3" or "wav"`
- The container ID.
+ The format of the audio data. Currently supported formats are `mp3` and
+ `wav`.
- - `CodeInterpreterToolAuto = object { type, file_ids, memory_limit, network_policy }`
+ - `"mp3"`
- Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
+ - `"wav"`
- - `type: "auto"`
+ - `type: "input_audio"`
- Always `auto`.
+ The type of the input item. Always `input_audio`.
- - `"auto"`
+ - `"input_audio"`
- - `file_ids: optional array of string`
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
- An optional list of uploaded files to make available to your code.
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
- - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+ - `TextInput = string`
- The memory limit for the code interpreter container.
+ A text input to the model.
- - `"1g"`
+ - `ResponseInputText object { text, type }`
- - `"4g"`
+ A text input to the model.
- - `"16g"`
+ - `OutputText object { text, type }`
- - `"64g"`
+ A text output from the model.
- - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+ - `text: string`
- Network access policy for the container.
+ The text output from the model.
- - `ContainerNetworkPolicyDisabled = object { type }`
+ - `type: "output_text"`
- - `type: "disabled"`
+ The type of the output text. Always `output_text`.
- Disable outbound network access. Always `disabled`.
+ - `"output_text"`
- - `"disabled"`
+ - `InputImage object { image_url, type, detail }`
- - `ContainerNetworkPolicyAllowlist = object { allowed_domains, type, domain_secrets }`
+ An image input block used within EvalItem content arrays.
- - `allowed_domains: array of string`
+ - `image_url: string`
- A list of allowed domains when type is `allowlist`.
+ The URL of the image input.
- - `type: "allowlist"`
+ - `type: "input_image"`
- Allow outbound network access only to specified domains. Always `allowlist`.
+ The type of the image input. Always `input_image`.
- - `"allowlist"`
+ - `"input_image"`
- - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+ - `detail: optional string`
- Optional domain-scoped secrets for allowlisted domains.
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `domain: string`
+ - `ResponseInputAudio object { input_audio, type }`
- The domain associated with the secret.
+ An audio input to the model.
- - `name: string`
+ - `role: "user" or "assistant" or "system" or "developer"`
- The name of the secret to inject for the domain.
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- - `value: string`
+ - `"user"`
- The secret value to inject for the domain.
+ - `"assistant"`
- - `type: "code_interpreter"`
+ - `"system"`
- The type of the code interpreter tool. Always `code_interpreter`.
+ - `"developer"`
- - `"code_interpreter"`
+ - `type: optional "message"`
- - `ImageGeneration = object { type, action, background, 9 more }`
+ The type of the message input. Always `message`.
- A tool that generates images using the GPT image models.
+ - `"message"`
- - `type: "image_generation"`
+ - `type: "template"`
- The type of the image generation tool. Always `image_generation`.
+ The type of input messages. Always `template`.
- - `"image_generation"`
+ - `"template"`
- - `action: optional "generate" or "edit" or "auto"`
+ - `ItemReferenceInputMessages object { item_reference, type }`
- Whether to generate a new image or edit an existing image. Default: `auto`.
+ - `item_reference: string`
- - `"generate"`
+ A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
- - `"edit"`
+ - `type: "item_reference"`
- - `"auto"`
+ The type of input messages. Always `item_reference`.
- - `background: optional "transparent" or "opaque" or "auto"`
+ - `"item_reference"`
- Background type for the generated image. One of `transparent`,
- `opaque`, or `auto`. Default: `auto`.
+ - `model: optional string`
- - `"transparent"`
+ The name of the model to use for generating completions (e.g. "o3-mini").
- - `"opaque"`
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
- - `"auto"`
+ - `max_completion_tokens: optional number`
- - `input_fidelity: optional "high" or "low"`
+ The maximum number of tokens in the generated output.
- Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
+ - `reasoning_effort: optional ReasoningEffort`
- - `"high"`
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- - `"low"`
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- - `input_image_mask: optional object { file_id, image_url }`
+ - `"none"`
- Optional mask for inpainting. Contains `image_url`
- (string, optional) and `file_id` (string, optional).
+ - `"minimal"`
- - `file_id: optional string`
+ - `"low"`
- File ID for the mask image.
+ - `"medium"`
- - `image_url: optional string`
+ - `"high"`
- Base64-encoded mask image.
+ - `"xhigh"`
- - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+ - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
- The image generation model to use. Default: `gpt-image-1`.
+ An object specifying the format that the model must output.
- - `string`
+ Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
+ Structured Outputs which ensures the model will match your supplied JSON
+ schema. Learn more in the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
- The image generation model to use. Default: `gpt-image-1`.
+ - `ResponseFormatText object { type }`
- - `"gpt-image-1"`
+ Default response format. Used to generate text responses.
- - `"gpt-image-1-mini"`
+ - `type: "text"`
- - `"gpt-image-1.5"`
+ The type of response format being defined. Always `text`.
- - `moderation: optional "auto" or "low"`
+ - `"text"`
- Moderation level for the generated image. Default: `auto`.
+ - `ResponseFormatJSONSchema object { json_schema, type }`
- - `"auto"`
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
- - `"low"`
+ - `json_schema: object { name, description, schema, strict }`
- - `output_compression: optional number`
+ Structured Outputs configuration options, including a JSON Schema.
- Compression level for the output image. Default: 100.
+ - `name: string`
- - `output_format: optional "png" or "webp" or "jpeg"`
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
- The output format of the generated image. One of `png`, `webp`, or
- `jpeg`. Default: `png`.
+ - `description: optional string`
- - `"png"`
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
- - `"webp"`
+ - `schema: optional map[unknown]`
- - `"jpeg"`
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- - `partial_images: optional number`
+ - `strict: optional boolean`
- Number of partial images to generate in streaming mode, from 0 (default value) to 3.
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- - `quality: optional "low" or "medium" or "high" or "auto"`
+ - `type: "json_schema"`
- The quality of the generated image. One of `low`, `medium`, `high`,
- or `auto`. Default: `auto`.
+ The type of response format being defined. Always `json_schema`.
- - `"low"`
+ - `"json_schema"`
- - `"medium"`
+ - `ResponseFormatJSONObject object { type }`
- - `"high"`
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
- - `"auto"`
+ - `type: "json_object"`
- - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
+ The type of response format being defined. Always `json_object`.
- The size of the generated image. One of `1024x1024`, `1024x1536`,
- `1536x1024`, or `auto`. Default: `auto`.
+ - `"json_object"`
- - `"1024x1024"`
+ - `seed: optional number`
- - `"1024x1536"`
+ A seed value to initialize the randomness, during sampling.
- - `"1536x1024"`
+ - `temperature: optional number`
- - `"auto"`
+ A higher temperature increases randomness in the outputs.
- - `LocalShell = object { type }`
+ - `tools: optional array of ChatCompletionFunctionTool`
- A tool that allows the model to execute shell commands in a local environment.
+ A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
- - `type: "local_shell"`
+ - `function: FunctionDefinition`
- The type of the local shell tool. Always `local_shell`.
+ - `name: string`
- - `"local_shell"`
+ The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
- - `Shell = object { type, environment }`
+ - `description: optional string`
- A tool that allows the model to execute shell commands.
+ A description of what the function does, used by the model to choose when and how to call the function.
- - `type: "shell"`
+ - `parameters: optional FunctionParameters`
- The type of the shell tool. Always `shell`.
+ The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
- - `"shell"`
+ Omitting `parameters` defines a function with an empty parameter list.
- - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
+ - `strict: optional boolean`
- - `ContainerAuto = object { type, file_ids, memory_limit, 2 more }`
+ Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
- - `type: "container_auto"`
+ - `type: "function"`
- Automatically creates a container for this request
+ The type of the tool. Currently, only `function` is supported.
- - `"container_auto"`
+ - `"function"`
- - `file_ids: optional array of string`
+ - `top_p: optional number`
- An optional list of uploaded files to make available to your code.
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+ - `ResponsesRunDataSource object { source, type, input_messages, 2 more }`
- The memory limit for the container.
+ A ResponsesRunDataSource object describing a model sampling configuration.
- - `"1g"`
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
- - `"4g"`
+ Determines what populates the `item` namespace in this run's data source.
- - `"16g"`
+ - `EvalJSONLFileContentSource object { content, type }`
- - `"64g"`
+ - `content: array of object { item, sample }`
- - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+ The content of the jsonl file.
- Network access policy for the container.
+ - `item: map[unknown]`
- - `ContainerNetworkPolicyDisabled = object { type }`
+ - `sample: optional map[unknown]`
- - `type: "disabled"`
+ - `type: "file_content"`
- Disable outbound network access. Always `disabled`.
+ The type of jsonl source. Always `file_content`.
- - `"disabled"`
+ - `"file_content"`
- - `ContainerNetworkPolicyAllowlist = object { allowed_domains, type, domain_secrets }`
+ - `EvalJSONLFileIDSource object { id, type }`
- - `allowed_domains: array of string`
+ - `id: string`
- A list of allowed domains when type is `allowlist`.
+ The identifier of the file.
- - `type: "allowlist"`
+ - `type: "file_id"`
- Allow outbound network access only to specified domains. Always `allowlist`.
+ The type of jsonl source. Always `file_id`.
- - `"allowlist"`
+ - `"file_id"`
- - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+ - `EvalResponsesSource object { type, created_after, created_before, 8 more }`
- Optional domain-scoped secrets for allowlisted domains.
+ A EvalResponsesSource object describing a run data source configuration.
- - `domain: string`
+ - `type: "responses"`
- The domain associated with the secret.
+ The type of run data source. Always `responses`.
- - `name: string`
+ - `"responses"`
- The name of the secret to inject for the domain.
+ - `created_after: optional number`
- - `value: string`
+ Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
- The secret value to inject for the domain.
+ - `created_before: optional number`
- - `skills: optional array of SkillReference or InlineSkill`
+ Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
- An optional list of skills referenced by id or inline data.
+ - `instructions_search: optional string`
- - `SkillReference = object { skill_id, type, version }`
+ Optional string to search the 'instructions' field. This is a query parameter used to select responses.
- - `skill_id: string`
+ - `metadata: optional unknown`
- The ID of the referenced skill.
+ Metadata filter for the responses. This is a query parameter used to select responses.
- - `type: "skill_reference"`
+ - `model: optional string`
- References a skill created with the /v1/skills endpoint.
+ The name of the model to find responses for. This is a query parameter used to select responses.
- - `"skill_reference"`
+ - `reasoning_effort: optional ReasoningEffort`
- - `version: optional string`
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- Optional skill version. Use a positive integer or 'latest'. Omit for default.
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- - `InlineSkill = object { description, name, source, type }`
+ - `temperature: optional number`
- - `description: string`
+ Sampling temperature. This is a query parameter used to select responses.
- The description of the skill.
+ - `tools: optional array of string`
- - `name: string`
+ List of tool names. This is a query parameter used to select responses.
- The name of the skill.
+ - `top_p: optional number`
- - `source: InlineSkillSource`
+ Nucleus sampling parameter. This is a query parameter used to select responses.
- Inline skill payload
+ - `users: optional array of string`
- - `data: string`
+ List of user identifiers. This is a query parameter used to select responses.
- Base64-encoded skill zip bundle.
+ - `type: "responses"`
- - `media_type: "application/zip"`
+ The type of run data source. Always `responses`.
- The media type of the inline skill payload. Must be `application/zip`.
+ - `"responses"`
- - `"application/zip"`
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
- - `type: "base64"`
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
- The type of the inline skill source. Must be `base64`.
+ - `InputMessagesTemplate object { template, type }`
- - `"base64"`
+ - `template: array of object { content, role } or object { content, role, type }`
- - `type: "inline"`
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
- Defines an inline skill for this request.
+ - `ChatMessage object { content, role }`
- - `"inline"`
+ - `content: string`
- - `LocalEnvironment = object { type, skills }`
+ The content of the message.
- - `type: "local"`
+ - `role: string`
- Use a local computer environment.
+ The role of the message (e.g. "system", "assistant", "user").
- - `"local"`
+ - `EvalMessageObject object { content, role, type }`
- - `skills: optional array of LocalSkill`
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- An optional list of skills.
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- - `description: string`
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- The description of the skill.
+ - `TextInput = string`
- - `name: string`
+ A text input to the model.
- The name of the skill.
+ - `ResponseInputText object { text, type }`
- - `path: string`
+ A text input to the model.
- The path to the directory containing the skill.
+ - `OutputText object { text, type }`
- - `ContainerReference = object { container_id, type }`
+ A text output from the model.
- - `container_id: string`
+ - `text: string`
- The ID of the referenced container.
+ The text output from the model.
- - `type: "container_reference"`
+ - `type: "output_text"`
- References a container created with the /v1/containers endpoint
+ The type of the output text. Always `output_text`.
- - `"container_reference"`
+ - `"output_text"`
- - `Custom = object { name, type, defer_loading, 2 more }`
+ - `InputImage object { image_url, type, detail }`
- A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+ An image input block used within EvalItem content arrays.
- - `name: string`
+ - `image_url: string`
- The name of the custom tool, used to identify it in tool calls.
+ The URL of the image input.
- - `type: "custom"`
+ - `type: "input_image"`
- The type of the custom tool. Always `custom`.
+ The type of the image input. Always `input_image`.
- - `"custom"`
+ - `"input_image"`
- - `defer_loading: optional boolean`
+ - `detail: optional string`
- Whether this tool should be deferred and discovered via tool search.
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `description: optional string`
+ - `ResponseInputAudio object { input_audio, type }`
- Optional description of the custom tool, used to provide more context.
+ An audio input to the model.
- - `format: optional CustomToolInputFormat`
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
- The input format for the custom tool. Default is unconstrained text.
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
- - `Text = object { type }`
+ - `role: "user" or "assistant" or "system" or "developer"`
- Unconstrained free-form text.
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- - `type: "text"`
+ - `"user"`
- Unconstrained text format. Always `text`.
+ - `"assistant"`
- - `"text"`
+ - `"system"`
- - `Grammar = object { definition, syntax, type }`
+ - `"developer"`
- A grammar defined by the user.
+ - `type: optional "message"`
- - `definition: string`
+ The type of the message input. Always `message`.
- The grammar definition.
+ - `"message"`
- - `syntax: "lark" or "regex"`
+ - `type: "template"`
- The syntax of the grammar definition. One of `lark` or `regex`.
+ The type of input messages. Always `template`.
- - `"lark"`
+ - `"template"`
- - `"regex"`
+ - `InputMessagesItemReference object { item_reference, type }`
- - `type: "grammar"`
+ - `item_reference: string`
- Grammar format. Always `grammar`.
+ A reference to a variable in the `item` namespace. Ie, "item.name"
- - `"grammar"`
+ - `type: "item_reference"`
- - `Namespace = object { description, name, tools, type }`
+ The type of input messages. Always `item_reference`.
- Groups function/custom tools under a shared namespace.
+ - `"item_reference"`
- - `description: string`
+ - `model: optional string`
- A description of the namespace shown to the model.
+ The name of the model to use for generating completions (e.g. "o3-mini").
- - `name: string`
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
- The namespace name used in tool calls (for example, `crm`).
+ - `max_completion_tokens: optional number`
- - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
+ The maximum number of tokens in the generated output.
- The function/custom tools available inside this namespace.
+ - `reasoning_effort: optional ReasoningEffort`
- - `Function = object { name, type, defer_loading, 3 more }`
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- - `name: string`
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- - `type: "function"`
+ - `seed: optional number`
- - `"function"`
+ A seed value to initialize the randomness, during sampling.
- - `defer_loading: optional boolean`
+ - `temperature: optional number`
- Whether this function should be deferred and discovered via tool search.
+ A higher temperature increases randomness in the outputs.
- - `description: optional string`
+ - `text: optional object { format }`
- - `parameters: optional unknown`
+ Configuration options for a text response from the model. Can be plain
+ text or structured JSON data. Learn more:
- - `strict: optional boolean`
+ - [Text inputs and outputs](/docs/guides/text)
+ - [Structured Outputs](/docs/guides/structured-outputs)
- - `Custom = object { name, type, defer_loading, 2 more }`
+ - `format: optional ResponseFormatTextConfig`
- A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+ An object specifying the format that the model must output.
- - `name: string`
+ Configuring `{ "type": "json_schema" }` enables Structured Outputs,
+ which ensures the model will match your supplied JSON schema. Learn more in the
+ [Structured Outputs guide](/docs/guides/structured-outputs).
- The name of the custom tool, used to identify it in tool calls.
+ The default format is `{ "type": "text" }` with no additional options.
- - `type: "custom"`
+ **Not recommended for gpt-4o and newer models:**
- The type of the custom tool. Always `custom`.
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
- - `"custom"`
+ - `ResponseFormatText object { type }`
- - `defer_loading: optional boolean`
+ Default response format. Used to generate text responses.
- Whether this tool should be deferred and discovered via tool search.
+ - `ResponseFormatTextJSONSchemaConfig object { name, schema, type, 2 more }`
- - `description: optional string`
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
- Optional description of the custom tool, used to provide more context.
+ - `name: string`
- - `format: optional CustomToolInputFormat`
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
- The input format for the custom tool. Default is unconstrained text.
+ - `schema: map[unknown]`
- - `Text = object { type }`
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- Unconstrained free-form text.
+ - `type: "json_schema"`
- - `type: "text"`
+ The type of response format being defined. Always `json_schema`.
- Unconstrained text format. Always `text`.
+ - `"json_schema"`
- - `"text"`
+ - `description: optional string`
- - `Grammar = object { definition, syntax, type }`
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
- A grammar defined by the user.
+ - `strict: optional boolean`
- - `definition: string`
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- The grammar definition.
+ - `ResponseFormatJSONObject object { type }`
- - `syntax: "lark" or "regex"`
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
- The syntax of the grammar definition. One of `lark` or `regex`.
+ - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
- - `"lark"`
+ An array of tools the model may call while generating a response. You
+ can specify which tool to use by setting the `tool_choice` parameter.
- - `"regex"`
+ The two categories of tools you can provide the model are:
- - `type: "grammar"`
+ - **Built-in tools**: Tools that are provided by OpenAI that extend the
+ model's capabilities, like [web search](/docs/guides/tools-web-search)
+ or [file search](/docs/guides/tools-file-search). Learn more about
+ [built-in tools](/docs/guides/tools).
+ - **Function calls (custom tools)**: Functions that are defined by you,
+ enabling the model to call your own code. Learn more about
+ [function calling](/docs/guides/function-calling).
- Grammar format. Always `grammar`.
+ - `Function object { name, parameters, strict, 3 more }`
- - `"grammar"`
+ Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
- - `type: "namespace"`
+ - `name: string`
- The type of the tool. Always `namespace`.
+ The name of the function to call.
- - `"namespace"`
+ - `parameters: map[unknown]`
- - `ToolSearch = object { type, description, execution, parameters }`
+ A JSON schema object describing the parameters of the function.
- Hosted or BYOT tool search configuration for deferred tools.
+ - `strict: boolean`
- - `type: "tool_search"`
+ Whether to enforce strict parameter validation. Default `true`.
- The type of the tool. Always `tool_search`.
+ - `type: "function"`
- - `"tool_search"`
+ The type of the function tool. Always `function`.
- - `description: optional string`
+ - `"function"`
- Description shown to the model for a client-executed tool search tool.
+ - `defer_loading: optional boolean`
- - `execution: optional "server" or "client"`
+ Whether this function is deferred and loaded via tool search.
- Whether tool search is executed by the server or by the client.
+ - `description: optional string`
- - `"server"`
+ A description of the function. Used by the model to determine whether or not to call the function.
- - `"client"`
+ - `FileSearch object { type, vector_store_ids, filters, 2 more }`
- - `parameters: optional unknown`
+ A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
- Parameter schema for a client-executed tool search tool.
+ - `type: "file_search"`
- - `WebSearchPreview = object { type, search_content_types, search_context_size, user_location }`
+ The type of the file search tool. Always `file_search`.
- This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
+ - `"file_search"`
- - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
+ - `vector_store_ids: array of string`
- The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
+ The IDs of the vector stores to search.
- - `"web_search_preview"`
+ - `filters: optional ComparisonFilter or CompoundFilter`
- - `"web_search_preview_2025_03_11"`
+ A filter to apply.
- - `search_content_types: optional array of "text" or "image"`
+ - `ComparisonFilter object { key, type, value }`
- - `"text"`
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
- - `"image"`
+ - `key: string`
- - `search_context_size: optional "low" or "medium" or "high"`
+ The key to compare against the value.
- High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+ - `type: "eq" or "ne" or "gt" or 5 more`
- - `"low"`
+ Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
- - `"medium"`
+ - `eq`: equals
+ - `ne`: not equal
+ - `gt`: greater than
+ - `gte`: greater than or equal
+ - `lt`: less than
+ - `lte`: less than or equal
+ - `in`: in
+ - `nin`: not in
- - `"high"`
+ - `"eq"`
- - `user_location: optional object { type, city, country, 2 more }`
+ - `"ne"`
- The user's location.
+ - `"gt"`
- - `type: "approximate"`
+ - `"gte"`
- The type of location approximation. Always `approximate`.
+ - `"lt"`
- - `"approximate"`
+ - `"lte"`
- - `city: optional string`
+ - `"in"`
- Free text input for the city of the user, e.g. `San Francisco`.
+ - `"nin"`
- - `country: optional string`
+ - `value: string or number or boolean or array of string or number`
- The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+ The value to compare against the attribute key; supports string, number, or boolean types.
- - `region: optional string`
+ - `string`
- Free text input for the region of the user, e.g. `California`.
+ - `number`
- - `timezone: optional string`
+ - `boolean`
- The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+ - `array of string or number`
- - `ApplyPatch = object { type }`
+ - `string`
- Allows the assistant to create, delete, or update files using unified diffs.
+ - `number`
- - `type: "apply_patch"`
+ - `CompoundFilter object { filters, type }`
- The type of the tool. Always `apply_patch`.
+ Combine multiple filters using `and` or `or`.
- - `"apply_patch"`
+ - `filters: array of ComparisonFilter or unknown`
- - `top_p: optional number`
+ Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
- An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+ - `ComparisonFilter object { key, type, value }`
-- `error: EvalAPIError`
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
- An object representing an error response from the Eval API.
+ - `unknown`
- - `code: string`
+ - `type: "and" or "or"`
- The error code.
+ Type of operation: `and` or `or`.
- - `message: string`
+ - `"and"`
- The error message.
+ - `"or"`
-- `eval_id: string`
+ - `max_num_results: optional number`
- The identifier of the associated evaluation.
+ The maximum number of results to return. This number should be between 1 and 50 inclusive.
-- `metadata: Metadata`
+ - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
- Set of 16 key-value pairs that can be attached to an object. This can be
- useful for storing additional information about the object in a structured
- format, and querying for objects via API or the dashboard.
+ Ranking options for search.
- Keys are strings with a maximum length of 64 characters. Values are strings
- with a maximum length of 512 characters.
+ - `hybrid_search: optional object { embedding_weight, text_weight }`
-- `model: string`
+ Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
- The model that is evaluated, if applicable.
+ - `embedding_weight: number`
-- `name: string`
+ The weight of the embedding in the reciprocal ranking fusion.
- The name of the evaluation run.
+ - `text_weight: number`
-- `object: "eval.run"`
+ The weight of the text in the reciprocal ranking fusion.
- The type of the object. Always "eval.run".
+ - `ranker: optional "auto" or "default-2024-11-15"`
- - `"eval.run"`
+ The ranker to use for the file search.
-- `per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }`
+ - `"auto"`
- Usage statistics for each model during the evaluation run.
+ - `"default-2024-11-15"`
- - `cached_tokens: number`
+ - `score_threshold: optional number`
- The number of tokens retrieved from cache.
+ The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
- - `completion_tokens: number`
+ - `Computer object { type }`
- The number of completion tokens generated.
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
- - `invocation_count: number`
+ - `type: "computer"`
- The number of invocations.
+ The type of the computer tool. Always `computer`.
- - `model_name: string`
+ - `"computer"`
- The name of the model.
+ - `ComputerUsePreview object { display_height, display_width, environment, type }`
- - `prompt_tokens: number`
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
- The number of prompt tokens used.
+ - `display_height: number`
- - `total_tokens: number`
+ The height of the computer display.
- The total number of tokens used.
+ - `display_width: number`
-- `per_testing_criteria_results: array of object { failed, passed, testing_criteria }`
+ The width of the computer display.
- Results per testing criteria applied during the evaluation run.
+ - `environment: "windows" or "mac" or "linux" or 2 more`
- - `failed: number`
+ The type of computer environment to control.
- Number of tests failed for this criteria.
+ - `"windows"`
- - `passed: number`
+ - `"mac"`
- Number of tests passed for this criteria.
+ - `"linux"`
- - `testing_criteria: string`
+ - `"ubuntu"`
- A description of the testing criteria.
+ - `"browser"`
-- `report_url: string`
+ - `type: "computer_use_preview"`
- The URL to the rendered evaluation run report on the UI dashboard.
+ The type of the computer use tool. Always `computer_use_preview`.
-- `result_counts: object { errored, failed, passed, total }`
+ - `"computer_use_preview"`
- Counters summarizing the outcomes of the evaluation run.
+ - `WebSearch object { type, filters, search_context_size, user_location }`
- - `errored: number`
+ Search the Internet for sources related to the prompt. Learn more about the
+ [web search tool](/docs/guides/tools-web-search).
- Number of output items that resulted in an error.
+ - `type: "web_search" or "web_search_2025_08_26"`
- - `failed: number`
+ The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
- Number of output items that failed to pass the evaluation.
+ - `"web_search"`
- - `passed: number`
+ - `"web_search_2025_08_26"`
- Number of output items that passed the evaluation.
+ - `filters: optional object { allowed_domains }`
- - `total: number`
+ Filters for the search.
- Total number of executed output items.
+ - `allowed_domains: optional array of string`
-- `status: string`
+ Allowed domains for the search. If not provided, all domains are allowed.
+ Subdomains of the provided domains are allowed as well.
- The status of the evaluation run.
+ Example: `["pubmed.ncbi.nlm.nih.gov"]`
-### Example
+ - `search_context_size: optional "low" or "medium" or "high"`
-```http
-curl https://api.openai.com/v1/evals/$EVAL_ID/runs/$RUN_ID \
- -H "Authorization: Bearer $OPENAI_API_KEY"
-```
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
-#### Response
+ - `"low"`
-```json
-{
- "id": "id",
- "created_at": 0,
- "data_source": {
- "source": {
- "content": [
- {
- "item": {
- "foo": "bar"
- },
- "sample": {
- "foo": "bar"
- }
- }
- ],
- "type": "file_content"
- },
- "type": "jsonl"
- },
- "error": {
- "code": "code",
- "message": "message"
- },
- "eval_id": "eval_id",
- "metadata": {
- "foo": "string"
- },
- "model": "model",
- "name": "name",
- "object": "eval.run",
- "per_model_usage": [
- {
- "cached_tokens": 0,
- "completion_tokens": 0,
- "invocation_count": 0,
- "model_name": "model_name",
- "prompt_tokens": 0,
- "total_tokens": 0
- }
- ],
- "per_testing_criteria_results": [
- {
- "failed": 0,
- "passed": 0,
- "testing_criteria": "testing_criteria"
- }
- ],
- "report_url": "report_url",
- "result_counts": {
- "errored": 0,
- "failed": 0,
- "passed": 0,
- "total": 0
- },
- "status": "status"
-}
-```
+ - `"medium"`
-### Example
+ - `"high"`
-```http
-curl https://api.openai.com/v1/evals/eval_67abd54d9b0081909a86353f6fb9317a/runs/evalrun_67abd54d60ec8190832b46859da808f7 \
- -H "Authorization: Bearer $OPENAI_API_KEY" \
- -H "Content-Type: application/json"
-```
+ - `user_location: optional object { city, country, region, 2 more }`
-#### Response
+ The approximate location of the user.
-```json
-{
- "object": "eval.run",
- "id": "evalrun_67abd54d60ec8190832b46859da808f7",
- "eval_id": "eval_67abd54d9b0081909a86353f6fb9317a",
- "report_url": "https://platform.openai.com/evaluations/eval_67abd54d9b0081909a86353f6fb9317a?run_id=evalrun_67abd54d60ec8190832b46859da808f7",
- "status": "queued",
- "model": "gpt-4o-mini",
- "name": "gpt-4o-mini",
- "created_at": 1743092069,
- "result_counts": {
- "total": 0,
- "errored": 0,
- "failed": 0,
- "passed": 0
- },
- "per_model_usage": null,
- "per_testing_criteria_results": null,
- "data_source": {
- "type": "completions",
- "source": {
- "type": "file_content",
- "content": [
- {
- "item": {
- "input": "Tech Company Launches Advanced Artificial Intelligence Platform",
- "ground_truth": "Technology"
- }
- },
- {
- "item": {
- "input": "Central Bank Increases Interest Rates Amid Inflation Concerns",
- "ground_truth": "Markets"
- }
- },
- {
- "item": {
- "input": "International Summit Addresses Climate Change Strategies",
- "ground_truth": "World"
- }
- },
- {
- "item": {
- "input": "Major Retailer Reports Record-Breaking Holiday Sales",
- "ground_truth": "Business"
- }
- },
- {
- "item": {
- "input": "National Team Qualifies for World Championship Finals",
- "ground_truth": "Sports"
- }
- },
- {
- "item": {
- "input": "Stock Markets Rally After Positive Economic Data Released",
- "ground_truth": "Markets"
- }
- },
- {
- "item": {
- "input": "Global Manufacturer Announces Merger with Competitor",
- "ground_truth": "Business"
- }
- },
- {
- "item": {
- "input": "Breakthrough in Renewable Energy Technology Unveiled",
- "ground_truth": "Technology"
- }
- },
- {
- "item": {
- "input": "World Leaders Sign Historic Climate Agreement",
- "ground_truth": "World"
- }
- },
- {
- "item": {
- "input": "Professional Athlete Sets New Record in Championship Event",
- "ground_truth": "Sports"
- }
- },
- {
- "item": {
- "input": "Financial Institutions Adapt to New Regulatory Requirements",
- "ground_truth": "Business"
- }
- },
- {
- "item": {
- "input": "Tech Conference Showcases Advances in Artificial Intelligence",
- "ground_truth": "Technology"
- }
- },
- {
- "item": {
- "input": "Global Markets Respond to Oil Price Fluctuations",
- "ground_truth": "Markets"
- }
- },
- {
- "item": {
- "input": "International Cooperation Strengthened Through New Treaty",
- "ground_truth": "World"
- }
- },
- {
- "item": {
- "input": "Sports League Announces Revised Schedule for Upcoming Season",
- "ground_truth": "Sports"
- }
- }
- ]
- },
- "input_messages": {
- "type": "template",
- "template": [
- {
- "type": "message",
- "role": "developer",
- "content": {
- "type": "input_text",
- "text": "Categorize a given news headline into one of the following topics: Technology, Markets, World, Business, or Sports.\n\n# Steps\n\n1. Analyze the content of the news headline to understand its primary focus.\n2. Extract the subject matter, identifying any key indicators or keywords.\n3. Use the identified indicators to determine the most suitable category out of the five options: Technology, Markets, World, Business, or Sports.\n4. Ensure only one category is selected per headline.\n\n# Output Format\n\nRespond with the chosen category as a single word. For instance: \"Technology\", \"Markets\", \"World\", \"Business\", or \"Sports\".\n\n# Examples\n\n**Input**: \"Apple Unveils New iPhone Model, Featuring Advanced AI Features\" \n**Output**: \"Technology\"\n\n**Input**: \"Global Stocks Mixed as Investors Await Central Bank Decisions\" \n**Output**: \"Markets\"\n\n**Input**: \"War in Ukraine: Latest Updates on Negotiation Status\" \n**Output**: \"World\"\n\n**Input**: \"Microsoft in Talks to Acquire Gaming Company for $2 Billion\" \n**Output**: \"Business\"\n\n**Input**: \"Manchester United Secures Win in Premier League Football Match\" \n**Output**: \"Sports\" \n\n# Notes\n\n- If the headline appears to fit into more than one category, choose the most dominant theme.\n- Keywords or phrases such as \"stocks\", \"company acquisition\", \"match\", or technological brands can be good indicators for classification.\n"
- }
- },
- {
- "type": "message",
- "role": "user",
- "content": {
- "type": "input_text",
- "text": "{{item.input}}"
- }
- }
- ]
- },
- "model": "gpt-4o-mini",
- "sampling_params": {
- "seed": 42,
- "temperature": 1.0,
- "top_p": 1.0,
- "max_completions_tokens": 2048
- }
- },
- "error": null,
- "metadata": {}
-}
-```
+ - `city: optional string`
-## Cancel eval run
+ Free text input for the city of the user, e.g. `San Francisco`.
-**post** `/evals/{eval_id}/runs/{run_id}`
+ - `country: optional string`
-Cancel an ongoing evaluation run.
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
-### Path Parameters
+ - `region: optional string`
-- `eval_id: string`
+ Free text input for the region of the user, e.g. `California`.
-- `run_id: string`
+ - `timezone: optional string`
-### Returns
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
-- `id: string`
+ - `type: optional "approximate"`
- Unique identifier for the evaluation run.
+ The type of location approximation. Always `approximate`.
-- `created_at: number`
+ - `"approximate"`
- Unix timestamp (in seconds) when the evaluation run was created.
+ - `Mcp object { server_label, type, allowed_tools, 7 more }`
-- `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
+ Give the model access to additional tools via remote Model Context Protocol
+ (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
- Information about the run's data source.
+ - `server_label: string`
- - `CreateEvalJSONLRunDataSource = object { source, type }`
+ A label for this MCP server, used to identify it in tool calls.
- A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
+ - `type: "mcp"`
- - `source: object { content, type } or object { id, type }`
+ The type of the MCP tool. Always `mcp`.
- Determines what populates the `item` namespace in the data source.
+ - `"mcp"`
- - `EvalJSONLFileContentSource = object { content, type }`
+ - `allowed_tools: optional array of string or object { read_only, tool_names }`
- - `content: array of object { item, sample }`
+ List of allowed tool names or a filter object.
- The content of the jsonl file.
+ - `McpAllowedTools = array of string`
- - `item: map[unknown]`
+ A string array of allowed tool names
- - `sample: optional map[unknown]`
+ - `McpToolFilter object { read_only, tool_names }`
- - `type: "file_content"`
+ A filter object to specify which tools are allowed.
- The type of jsonl source. Always `file_content`.
+ - `read_only: optional boolean`
- - `"file_content"`
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `tool_names: optional array of string`
- - `id: string`
+ List of allowed tool names.
- The identifier of the file.
+ - `authorization: optional string`
- - `type: "file_id"`
+ An OAuth access token that can be used with a remote MCP server, either
+ with a custom MCP server URL or a service connector. Your application
+ must handle the OAuth authorization flow and provide the token here.
- The type of jsonl source. Always `file_id`.
+ - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
- - `"file_id"`
+ Identifier for service connectors, like those available in ChatGPT. One of
+ `server_url` or `connector_id` must be provided. Learn more about service
+ connectors [here](/docs/guides/tools-remote-mcp#connectors).
- - `type: "jsonl"`
+ Currently supported `connector_id` values are:
- The type of data source. Always `jsonl`.
+ - Dropbox: `connector_dropbox`
+ - Gmail: `connector_gmail`
+ - Google Calendar: `connector_googlecalendar`
+ - Google Drive: `connector_googledrive`
+ - Microsoft Teams: `connector_microsoftteams`
+ - Outlook Calendar: `connector_outlookcalendar`
+ - Outlook Email: `connector_outlookemail`
+ - SharePoint: `connector_sharepoint`
- - `"jsonl"`
+ - `"connector_dropbox"`
- - `CreateEvalCompletionsRunDataSource = object { source, type, input_messages, 2 more }`
+ - `"connector_gmail"`
- A CompletionsRunDataSource object describing a model sampling configuration.
+ - `"connector_googlecalendar"`
- - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
+ - `"connector_googledrive"`
- Determines what populates the `item` namespace in this run's data source.
+ - `"connector_microsoftteams"`
- - `EvalJSONLFileContentSource = object { content, type }`
+ - `"connector_outlookcalendar"`
- - `content: array of object { item, sample }`
+ - `"connector_outlookemail"`
- The content of the jsonl file.
+ - `"connector_sharepoint"`
- - `item: map[unknown]`
+ - `defer_loading: optional boolean`
- - `sample: optional map[unknown]`
+ Whether this MCP tool is deferred and discovered via tool search.
- - `type: "file_content"`
+ - `headers: optional map[string]`
- The type of jsonl source. Always `file_content`.
+ Optional HTTP headers to send to the MCP server. Use for authentication
+ or other purposes.
- - `"file_content"`
+ - `require_approval: optional object { always, never } or "always" or "never"`
- - `EvalJSONLFileIDSource = object { id, type }`
+ Specify which of the MCP server's tools require approval.
- - `id: string`
+ - `McpToolApprovalFilter object { always, never }`
- The identifier of the file.
+ Specify which of the MCP server's tools require approval. Can be
+ `always`, `never`, or a filter object associated with tools
+ that require approval.
- - `type: "file_id"`
+ - `always: optional object { read_only, tool_names }`
- The type of jsonl source. Always `file_id`.
+ A filter object to specify which tools are allowed.
- - `"file_id"`
+ - `read_only: optional boolean`
- - `StoredCompletionsRunDataSource = object { type, created_after, created_before, 3 more }`
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- A StoredCompletionsRunDataSource configuration describing a set of filters
+ - `tool_names: optional array of string`
- - `type: "stored_completions"`
+ List of allowed tool names.
- The type of source. Always `stored_completions`.
+ - `never: optional object { read_only, tool_names }`
- - `"stored_completions"`
+ A filter object to specify which tools are allowed.
- - `created_after: optional number`
+ - `read_only: optional boolean`
- An optional Unix timestamp to filter items created after this time.
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- - `created_before: optional number`
+ - `tool_names: optional array of string`
- An optional Unix timestamp to filter items created before this time.
+ List of allowed tool names.
- - `limit: optional number`
+ - `McpToolApprovalSetting = "always" or "never"`
- An optional maximum number of items to return.
+ Specify a single approval policy for all tools. One of `always` or
+ `never`. When set to `always`, all tools will require approval. When
+ set to `never`, all tools will not require approval.
- - `metadata: optional Metadata`
+ - `"always"`
- Set of 16 key-value pairs that can be attached to an object. This can be
- useful for storing additional information about the object in a structured
- format, and querying for objects via API or the dashboard.
+ - `"never"`
- Keys are strings with a maximum length of 64 characters. Values are strings
- with a maximum length of 512 characters.
+ - `server_description: optional string`
- - `model: optional string`
+ Optional description of the MCP server, used to provide more context.
- An optional model to filter by (e.g., 'gpt-4o').
+ - `server_url: optional string`
- - `type: "completions"`
+ The URL for the MCP server. One of `server_url` or `connector_id` must be
+ provided.
- The type of run data source. Always `completions`.
+ - `CodeInterpreter object { container, type }`
- - `"completions"`
+ A tool that runs Python code to help generate a response to a prompt.
- - `input_messages: optional object { template, type } or object { item_reference, type }`
+ - `container: string or object { type, file_ids, memory_limit, network_policy }`
- Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+ The code interpreter container. Can be a container ID or an object that
+ specifies uploaded file IDs to make available to your code, along with an
+ optional `memory_limit` setting.
- - `TemplateInputMessages = object { template, type }`
+ - `string`
- - `template: array of EasyInputMessage or object { content, role, type }`
+ The container ID.
- A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+ - `CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy }`
- - `EasyInputMessage = object { content, role, phase, type }`
+ Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ - `type: "auto"`
- - `content: string or ResponseInputMessageContentList`
+ Always `auto`.
- Text, image, or audio input to the model, used to generate a response.
- Can also contain previous assistant responses.
+ - `"auto"`
- - `TextInput = string`
+ - `file_ids: optional array of string`
- A text input to the model.
+ An optional list of uploaded files to make available to your code.
- - `ResponseInputMessageContentList = array of ResponseInputContent`
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
- A list of one or many input items to the model, containing different content
- types.
+ The memory limit for the code interpreter container.
- - `ResponseInputText = object { text, type }`
+ - `"1g"`
- A text input to the model.
+ - `"4g"`
- - `text: string`
+ - `"16g"`
- The text input to the model.
+ - `"64g"`
- - `type: "input_text"`
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
- The type of the input item. Always `input_text`.
+ Network access policy for the container.
- - `"input_text"`
+ - `ContainerNetworkPolicyDisabled object { type }`
- - `ResponseInputImage = object { detail, type, file_id, image_url }`
+ - `type: "disabled"`
- An image input to the model. Learn about [image inputs](/docs/guides/vision).
+ Disable outbound network access. Always `disabled`.
- - `detail: "low" or "high" or "auto" or "original"`
+ - `"disabled"`
- The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
- - `"low"`
+ - `allowed_domains: array of string`
- - `"high"`
+ A list of allowed domains when type is `allowlist`.
- - `"auto"`
+ - `type: "allowlist"`
- - `"original"`
+ Allow outbound network access only to specified domains. Always `allowlist`.
- - `type: "input_image"`
+ - `"allowlist"`
- The type of the input item. Always `input_image`.
+ - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
- - `"input_image"`
+ Optional domain-scoped secrets for allowlisted domains.
- - `file_id: optional string`
+ - `domain: string`
- The ID of the file to be sent to the model.
+ The domain associated with the secret.
- - `image_url: optional string`
+ - `name: string`
- The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
+ The name of the secret to inject for the domain.
- - `ResponseInputFile = object { type, detail, file_data, 3 more }`
+ - `value: string`
- A file input to the model.
+ The secret value to inject for the domain.
- - `type: "input_file"`
+ - `type: "code_interpreter"`
- The type of the input item. Always `input_file`.
+ The type of the code interpreter tool. Always `code_interpreter`.
- - `"input_file"`
+ - `"code_interpreter"`
- - `detail: optional "low" or "high"`
+ - `ImageGeneration object { type, action, background, 9 more }`
- The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
+ A tool that generates images using the GPT image models.
- - `"low"`
+ - `type: "image_generation"`
- - `"high"`
+ The type of the image generation tool. Always `image_generation`.
- - `file_data: optional string`
+ - `"image_generation"`
- The content of the file to be sent to the model.
+ - `action: optional "generate" or "edit" or "auto"`
- - `file_id: optional string`
+ Whether to generate a new image or edit an existing image. Default: `auto`.
- The ID of the file to be sent to the model.
+ - `"generate"`
- - `file_url: optional string`
+ - `"edit"`
- The URL of the file to be sent to the model.
+ - `"auto"`
- - `filename: optional string`
+ - `background: optional "transparent" or "opaque" or "auto"`
- The name of the file to be sent to the model.
+ Background type for the generated image. One of `transparent`,
+ `opaque`, or `auto`. Default: `auto`.
- - `role: "user" or "assistant" or "system" or "developer"`
+ - `"transparent"`
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
+ - `"opaque"`
- - `"user"`
+ - `"auto"`
- - `"assistant"`
+ - `input_fidelity: optional "high" or "low"`
- - `"system"`
+ Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
- - `"developer"`
+ - `"high"`
- - `phase: optional "commentary" or "final_answer"`
+ - `"low"`
- Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
- For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
- phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
+ - `input_image_mask: optional object { file_id, image_url }`
- - `"commentary"`
+ Optional mask for inpainting. Contains `image_url`
+ (string, optional) and `file_id` (string, optional).
- - `"final_answer"`
+ - `file_id: optional string`
- - `type: optional "message"`
+ File ID for the mask image.
- The type of the message input. Always `message`.
+ - `image_url: optional string`
- - `"message"`
+ Base64-encoded mask image.
- - `EvalMessageObject = object { content, role, type }`
+ - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ The image generation model to use. Default: `gpt-image-1`.
- - `content: string or ResponseInputText or object { text, type } or 3 more`
+ - `string`
- Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+ - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
- - `TextInput = string`
+ The image generation model to use. Default: `gpt-image-1`.
- A text input to the model.
+ - `"gpt-image-1"`
- - `ResponseInputText = object { text, type }`
+ - `"gpt-image-1-mini"`
- A text input to the model.
+ - `"gpt-image-1.5"`
- - `text: string`
+ - `moderation: optional "auto" or "low"`
- The text input to the model.
+ Moderation level for the generated image. Default: `auto`.
- - `type: "input_text"`
+ - `"auto"`
- The type of the input item. Always `input_text`.
+ - `"low"`
- - `"input_text"`
+ - `output_compression: optional number`
- - `OutputText = object { text, type }`
+ Compression level for the output image. Default: 100.
- A text output from the model.
+ - `output_format: optional "png" or "webp" or "jpeg"`
- - `text: string`
+ The output format of the generated image. One of `png`, `webp`, or
+ `jpeg`. Default: `png`.
- The text output from the model.
+ - `"png"`
- - `type: "output_text"`
+ - `"webp"`
- The type of the output text. Always `output_text`.
+ - `"jpeg"`
- - `"output_text"`
+ - `partial_images: optional number`
- - `InputImage = object { image_url, type, detail }`
+ Number of partial images to generate in streaming mode, from 0 (default value) to 3.
- An image input block used within EvalItem content arrays.
+ - `quality: optional "low" or "medium" or "high" or "auto"`
- - `image_url: string`
+ The quality of the generated image. One of `low`, `medium`, `high`,
+ or `auto`. Default: `auto`.
- The URL of the image input.
+ - `"low"`
- - `type: "input_image"`
+ - `"medium"`
- The type of the image input. Always `input_image`.
+ - `"high"`
- - `"input_image"`
+ - `"auto"`
- - `detail: optional string`
+ - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ The size of the generated image. One of `1024x1024`, `1024x1536`,
+ `1536x1024`, or `auto`. Default: `auto`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `"1024x1024"`
- An audio input to the model.
+ - `"1024x1536"`
- - `input_audio: object { data, format }`
+ - `"1536x1024"`
- - `data: string`
+ - `"auto"`
- Base64-encoded audio data.
+ - `LocalShell object { type }`
- - `format: "mp3" or "wav"`
+ A tool that allows the model to execute shell commands in a local environment.
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ - `type: "local_shell"`
- - `"mp3"`
+ The type of the local shell tool. Always `local_shell`.
- - `"wav"`
+ - `"local_shell"`
- - `type: "input_audio"`
+ - `Shell object { type, environment }`
- The type of the input item. Always `input_audio`.
+ A tool that allows the model to execute shell commands.
- - `"input_audio"`
+ - `type: "shell"`
- - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+ The type of the shell tool. Always `shell`.
- A list of inputs, each of which may be either an input text, output text, input
- image, or input audio object.
+ - `"shell"`
- - `TextInput = string`
+ - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
- A text input to the model.
+ - `ContainerAuto object { type, file_ids, memory_limit, 2 more }`
- - `ResponseInputText = object { text, type }`
+ - `type: "container_auto"`
- A text input to the model.
+ Automatically creates a container for this request
- - `text: string`
+ - `"container_auto"`
- The text input to the model.
+ - `file_ids: optional array of string`
- - `type: "input_text"`
+ An optional list of uploaded files to make available to your code.
- The type of the input item. Always `input_text`.
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
- - `"input_text"`
+ The memory limit for the container.
- - `OutputText = object { text, type }`
+ - `"1g"`
- A text output from the model.
+ - `"4g"`
- - `text: string`
+ - `"16g"`
- The text output from the model.
+ - `"64g"`
- - `type: "output_text"`
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
- The type of the output text. Always `output_text`.
+ Network access policy for the container.
- - `"output_text"`
+ - `ContainerNetworkPolicyDisabled object { type }`
- - `InputImage = object { image_url, type, detail }`
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
- An image input block used within EvalItem content arrays.
+ - `skills: optional array of SkillReference or InlineSkill`
- - `image_url: string`
+ An optional list of skills referenced by id or inline data.
- The URL of the image input.
+ - `SkillReference object { skill_id, type, version }`
- - `type: "input_image"`
+ - `skill_id: string`
- The type of the image input. Always `input_image`.
+ The ID of the referenced skill.
- - `"input_image"`
+ - `type: "skill_reference"`
- - `detail: optional string`
+ References a skill created with the /v1/skills endpoint.
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ - `"skill_reference"`
- - `ResponseInputAudio = object { input_audio, type }`
+ - `version: optional string`
- An audio input to the model.
+ Optional skill version. Use a positive integer or 'latest'. Omit for default.
- - `input_audio: object { data, format }`
+ - `InlineSkill object { description, name, source, type }`
- - `data: string`
+ - `description: string`
- Base64-encoded audio data.
+ The description of the skill.
- - `format: "mp3" or "wav"`
+ - `name: string`
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ The name of the skill.
- - `"mp3"`
+ - `source: InlineSkillSource`
- - `"wav"`
+ Inline skill payload
- - `type: "input_audio"`
+ - `data: string`
- The type of the input item. Always `input_audio`.
+ Base64-encoded skill zip bundle.
- - `"input_audio"`
+ - `media_type: "application/zip"`
- - `role: "user" or "assistant" or "system" or "developer"`
+ The media type of the inline skill payload. Must be `application/zip`.
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
+ - `"application/zip"`
- - `"user"`
+ - `type: "base64"`
- - `"assistant"`
+ The type of the inline skill source. Must be `base64`.
- - `"system"`
+ - `"base64"`
- - `"developer"`
+ - `type: "inline"`
- - `type: optional "message"`
+ Defines an inline skill for this request.
- The type of the message input. Always `message`.
+ - `"inline"`
- - `"message"`
+ - `LocalEnvironment object { type, skills }`
- - `type: "template"`
+ - `type: "local"`
- The type of input messages. Always `template`.
+ Use a local computer environment.
- - `"template"`
+ - `"local"`
- - `ItemReferenceInputMessages = object { item_reference, type }`
+ - `skills: optional array of LocalSkill`
- - `item_reference: string`
+ An optional list of skills.
- A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
+ - `description: string`
- - `type: "item_reference"`
+ The description of the skill.
- The type of input messages. Always `item_reference`.
+ - `name: string`
- - `"item_reference"`
+ The name of the skill.
- - `model: optional string`
+ - `path: string`
- The name of the model to use for generating completions (e.g. "o3-mini").
+ The path to the directory containing the skill.
- - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
+ - `ContainerReference object { container_id, type }`
- - `max_completion_tokens: optional number`
+ - `container_id: string`
- The maximum number of tokens in the generated output.
+ The ID of the referenced container.
- - `reasoning_effort: optional ReasoningEffort`
+ - `type: "container_reference"`
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ References a container created with the /v1/containers endpoint
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ - `"container_reference"`
- - `"none"`
+ - `Custom object { name, type, defer_loading, 2 more }`
- - `"minimal"`
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
- - `"low"`
+ - `name: string`
- - `"medium"`
+ The name of the custom tool, used to identify it in tool calls.
- - `"high"`
+ - `type: "custom"`
- - `"xhigh"`
+ The type of the custom tool. Always `custom`.
- - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
+ - `"custom"`
- An object specifying the format that the model must output.
+ - `defer_loading: optional boolean`
- Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
- Structured Outputs which ensures the model will match your supplied JSON
- schema. Learn more in the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ Whether this tool should be deferred and discovered via tool search.
- Setting to `{ "type": "json_object" }` enables the older JSON mode, which
- ensures the message the model generates is valid JSON. Using `json_schema`
- is preferred for models that support it.
+ - `description: optional string`
+
+ Optional description of the custom tool, used to provide more context.
- - `ResponseFormatText = object { type }`
+ - `format: optional CustomToolInputFormat`
- Default response format. Used to generate text responses.
+ The input format for the custom tool. Default is unconstrained text.
+
+ - `Text object { type }`
+
+ Unconstrained free-form text.
- `type: "text"`
- The type of response format being defined. Always `text`.
+ Unconstrained text format. Always `text`.
- `"text"`
- - `ResponseFormatJSONSchema = object { json_schema, type }`
+ - `Grammar object { definition, syntax, type }`
- JSON Schema response format. Used to generate structured JSON responses.
- Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+ A grammar defined by the user.
- - `json_schema: object { name, description, schema, strict }`
+ - `definition: string`
- Structured Outputs configuration options, including a JSON Schema.
+ The grammar definition.
- - `name: string`
+ - `syntax: "lark" or "regex"`
- The name of the response format. Must be a-z, A-Z, 0-9, or contain
- underscores and dashes, with a maximum length of 64.
-
- - `description: optional string`
-
- A description of what the response format is for, used by the model to
- determine how to respond in the format.
-
- - `schema: optional map[unknown]`
-
- The schema for the response format, described as a JSON Schema object.
- Learn how to build JSON schemas [here](https://json-schema.org/).
-
- - `strict: optional boolean`
-
- Whether to enable strict schema adherence when generating the output.
- If set to true, the model will always follow the exact schema defined
- in the `schema` field. Only a subset of JSON Schema is supported when
- `strict` is `true`. To learn more, read the [Structured Outputs
- guide](/docs/guides/structured-outputs).
-
- - `type: "json_schema"`
+ The syntax of the grammar definition. One of `lark` or `regex`.
- The type of response format being defined. Always `json_schema`.
+ - `"lark"`
- - `"json_schema"`
+ - `"regex"`
- - `ResponseFormatJSONObject = object { type }`
+ - `type: "grammar"`
- JSON object response format. An older method of generating JSON responses.
- Using `json_schema` is recommended for models that support it. Note that the
- model will not generate JSON without a system or user message instructing it
- to do so.
+ Grammar format. Always `grammar`.
- - `type: "json_object"`
+ - `"grammar"`
- The type of response format being defined. Always `json_object`.
+ - `Namespace object { description, name, tools, type }`
- - `"json_object"`
+ Groups function/custom tools under a shared namespace.
- - `seed: optional number`
+ - `description: string`
- A seed value to initialize the randomness, during sampling.
+ A description of the namespace shown to the model.
- - `temperature: optional number`
+ - `name: string`
- A higher temperature increases randomness in the outputs.
+ The namespace name used in tool calls (for example, `crm`).
- - `tools: optional array of ChatCompletionFunctionTool`
+ - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
- A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
+ The function/custom tools available inside this namespace.
- - `function: FunctionDefinition`
+ - `Function object { name, type, defer_loading, 3 more }`
- `name: string`
- The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
+ - `type: "function"`
- - `description: optional string`
+ - `"function"`
- A description of what the function does, used by the model to choose when and how to call the function.
+ - `defer_loading: optional boolean`
- - `parameters: optional FunctionParameters`
+ Whether this function should be deferred and discovered via tool search.
- The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
+ - `description: optional string`
- Omitting `parameters` defines a function with an empty parameter list.
+ - `parameters: optional unknown`
- `strict: optional boolean`
- Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
-
- - `type: "function"`
-
- The type of the tool. Currently, only `function` is supported.
-
- - `"function"`
-
- - `top_p: optional number`
+ - `Custom object { name, type, defer_loading, 2 more }`
- An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
- - `ResponsesRunDataSource = object { source, type, input_messages, 2 more }`
+ - `name: string`
- A ResponsesRunDataSource object describing a model sampling configuration.
+ The name of the custom tool, used to identify it in tool calls.
- - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
+ - `type: "custom"`
- Determines what populates the `item` namespace in this run's data source.
+ The type of the custom tool. Always `custom`.
- - `EvalJSONLFileContentSource = object { content, type }`
+ - `"custom"`
- - `content: array of object { item, sample }`
+ - `defer_loading: optional boolean`
- The content of the jsonl file.
+ Whether this tool should be deferred and discovered via tool search.
- - `item: map[unknown]`
+ - `description: optional string`
- - `sample: optional map[unknown]`
+ Optional description of the custom tool, used to provide more context.
- - `type: "file_content"`
+ - `format: optional CustomToolInputFormat`
- The type of jsonl source. Always `file_content`.
+ The input format for the custom tool. Default is unconstrained text.
- - `"file_content"`
+ - `type: "namespace"`
- - `EvalJSONLFileIDSource = object { id, type }`
+ The type of the tool. Always `namespace`.
- - `id: string`
+ - `"namespace"`
- The identifier of the file.
+ - `ToolSearch object { type, description, execution, parameters }`
- - `type: "file_id"`
+ Hosted or BYOT tool search configuration for deferred tools.
- The type of jsonl source. Always `file_id`.
+ - `type: "tool_search"`
- - `"file_id"`
+ The type of the tool. Always `tool_search`.
- - `EvalResponsesSource = object { type, created_after, created_before, 8 more }`
+ - `"tool_search"`
- A EvalResponsesSource object describing a run data source configuration.
+ - `description: optional string`
- - `type: "responses"`
+ Description shown to the model for a client-executed tool search tool.
- The type of run data source. Always `responses`.
+ - `execution: optional "server" or "client"`
- - `"responses"`
+ Whether tool search is executed by the server or by the client.
- - `created_after: optional number`
+ - `"server"`
- Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
+ - `"client"`
- - `created_before: optional number`
+ - `parameters: optional unknown`
- Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
+ Parameter schema for a client-executed tool search tool.
- - `instructions_search: optional string`
+ - `WebSearchPreview object { type, search_content_types, search_context_size, user_location }`
- Optional string to search the 'instructions' field. This is a query parameter used to select responses.
+ This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
- - `metadata: optional unknown`
+ - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
- Metadata filter for the responses. This is a query parameter used to select responses.
+ The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
- - `model: optional string`
+ - `"web_search_preview"`
- The name of the model to find responses for. This is a query parameter used to select responses.
+ - `"web_search_preview_2025_03_11"`
- - `reasoning_effort: optional ReasoningEffort`
+ - `search_content_types: optional array of "text" or "image"`
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ - `"text"`
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ - `"image"`
- - `"none"`
+ - `search_context_size: optional "low" or "medium" or "high"`
- - `"minimal"`
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
- `"low"`
@@ -13035,1425 +12004,1536 @@ Cancel an ongoing evaluation run.
- `"high"`
- - `"xhigh"`
+ - `user_location: optional object { type, city, country, 2 more }`
- - `temperature: optional number`
+ The user's location.
- Sampling temperature. This is a query parameter used to select responses.
+ - `type: "approximate"`
- - `tools: optional array of string`
+ The type of location approximation. Always `approximate`.
- List of tool names. This is a query parameter used to select responses.
+ - `"approximate"`
- - `top_p: optional number`
+ - `city: optional string`
- Nucleus sampling parameter. This is a query parameter used to select responses.
+ Free text input for the city of the user, e.g. `San Francisco`.
- - `users: optional array of string`
+ - `country: optional string`
- List of user identifiers. This is a query parameter used to select responses.
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
- - `type: "responses"`
+ - `region: optional string`
- The type of run data source. Always `responses`.
+ Free text input for the region of the user, e.g. `California`.
- - `"responses"`
+ - `timezone: optional string`
- - `input_messages: optional object { template, type } or object { item_reference, type }`
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
- Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+ - `ApplyPatch object { type }`
- - `InputMessagesTemplate = object { template, type }`
+ Allows the assistant to create, delete, or update files using unified diffs.
- - `template: array of object { content, role } or object { content, role, type }`
+ - `type: "apply_patch"`
- A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+ The type of the tool. Always `apply_patch`.
- - `ChatMessage = object { content, role }`
+ - `"apply_patch"`
- - `content: string`
+ - `top_p: optional number`
- The content of the message.
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- - `role: string`
+- `error: EvalAPIError`
- The role of the message (e.g. "system", "assistant", "user").
+ An object representing an error response from the Eval API.
- - `EvalMessageObject = object { content, role, type }`
+ - `code: string`
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ The error code.
- - `content: string or ResponseInputText or object { text, type } or 3 more`
+ - `message: string`
- Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+ The error message.
- - `TextInput = string`
+- `eval_id: string`
- A text input to the model.
+ The identifier of the associated evaluation.
- - `ResponseInputText = object { text, type }`
+- `metadata: Metadata`
- A text input to the model.
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - `text: string`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- The text input to the model.
+- `model: string`
- - `type: "input_text"`
+ The model that is evaluated, if applicable.
- The type of the input item. Always `input_text`.
+- `name: string`
- - `"input_text"`
+ The name of the evaluation run.
- - `OutputText = object { text, type }`
+- `object: "eval.run"`
- A text output from the model.
+ The type of the object. Always "eval.run".
- - `text: string`
+ - `"eval.run"`
- The text output from the model.
+- `per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }`
- - `type: "output_text"`
+ Usage statistics for each model during the evaluation run.
- The type of the output text. Always `output_text`.
+ - `cached_tokens: number`
- - `"output_text"`
+ The number of tokens retrieved from cache.
- - `InputImage = object { image_url, type, detail }`
+ - `completion_tokens: number`
- An image input block used within EvalItem content arrays.
+ The number of completion tokens generated.
- - `image_url: string`
+ - `invocation_count: number`
- The URL of the image input.
+ The number of invocations.
- - `type: "input_image"`
+ - `model_name: string`
- The type of the image input. Always `input_image`.
+ The name of the model.
- - `"input_image"`
+ - `prompt_tokens: number`
- - `detail: optional string`
+ The number of prompt tokens used.
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ - `total_tokens: number`
- - `ResponseInputAudio = object { input_audio, type }`
+ The total number of tokens used.
- An audio input to the model.
+- `per_testing_criteria_results: array of object { failed, passed, testing_criteria }`
- - `input_audio: object { data, format }`
+ Results per testing criteria applied during the evaluation run.
- - `data: string`
-
- Base64-encoded audio data.
-
- - `format: "mp3" or "wav"`
-
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
-
- - `"mp3"`
-
- - `"wav"`
-
- - `type: "input_audio"`
-
- The type of the input item. Always `input_audio`.
-
- - `"input_audio"`
-
- - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
-
- A list of inputs, each of which may be either an input text, output text, input
- image, or input audio object.
-
- - `TextInput = string`
-
- A text input to the model.
-
- - `ResponseInputText = object { text, type }`
-
- A text input to the model.
-
- - `text: string`
-
- The text input to the model.
-
- - `type: "input_text"`
-
- The type of the input item. Always `input_text`.
-
- - `"input_text"`
-
- - `OutputText = object { text, type }`
-
- A text output from the model.
-
- - `text: string`
-
- The text output from the model.
-
- - `type: "output_text"`
-
- The type of the output text. Always `output_text`.
-
- - `"output_text"`
-
- - `InputImage = object { image_url, type, detail }`
-
- An image input block used within EvalItem content arrays.
-
- - `image_url: string`
-
- The URL of the image input.
-
- - `type: "input_image"`
-
- The type of the image input. Always `input_image`.
-
- - `"input_image"`
-
- - `detail: optional string`
-
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
-
- - `ResponseInputAudio = object { input_audio, type }`
-
- An audio input to the model.
-
- - `input_audio: object { data, format }`
-
- - `data: string`
-
- Base64-encoded audio data.
-
- - `format: "mp3" or "wav"`
-
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
-
- - `"mp3"`
-
- - `"wav"`
-
- - `type: "input_audio"`
-
- The type of the input item. Always `input_audio`.
-
- - `"input_audio"`
-
- - `role: "user" or "assistant" or "system" or "developer"`
-
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
-
- - `"user"`
-
- - `"assistant"`
-
- - `"system"`
-
- - `"developer"`
-
- - `type: optional "message"`
-
- The type of the message input. Always `message`.
-
- - `"message"`
-
- - `type: "template"`
-
- The type of input messages. Always `template`.
-
- - `"template"`
-
- - `InputMessagesItemReference = object { item_reference, type }`
-
- - `item_reference: string`
-
- A reference to a variable in the `item` namespace. Ie, "item.name"
-
- - `type: "item_reference"`
-
- The type of input messages. Always `item_reference`.
+ - `failed: number`
- - `"item_reference"`
+ Number of tests failed for this criteria.
- - `model: optional string`
+ - `passed: number`
- The name of the model to use for generating completions (e.g. "o3-mini").
+ Number of tests passed for this criteria.
- - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
+ - `testing_criteria: string`
- - `max_completion_tokens: optional number`
+ A description of the testing criteria.
- The maximum number of tokens in the generated output.
+- `report_url: string`
- - `reasoning_effort: optional ReasoningEffort`
+ The URL to the rendered evaluation run report on the UI dashboard.
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+- `result_counts: object { errored, failed, passed, total }`
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+ Counters summarizing the outcomes of the evaluation run.
- - `"none"`
+ - `errored: number`
- - `"minimal"`
+ Number of output items that resulted in an error.
- - `"low"`
+ - `failed: number`
- - `"medium"`
+ Number of output items that failed to pass the evaluation.
- - `"high"`
+ - `passed: number`
- - `"xhigh"`
+ Number of output items that passed the evaluation.
- - `seed: optional number`
+ - `total: number`
- A seed value to initialize the randomness, during sampling.
+ Total number of executed output items.
- - `temperature: optional number`
+- `status: string`
- A higher temperature increases randomness in the outputs.
+ The status of the evaluation run.
- - `text: optional object { format }`
+### Example
- Configuration options for a text response from the model. Can be plain
- text or structured JSON data. Learn more:
+```http
+curl https://api.openai.com/v1/evals/$EVAL_ID/runs/$RUN_ID \
+ -H "Authorization: Bearer $OPENAI_API_KEY"
+```
- - [Text inputs and outputs](/docs/guides/text)
- - [Structured Outputs](/docs/guides/structured-outputs)
+#### Response
- - `format: optional ResponseFormatTextConfig`
+```json
+{
+ "id": "id",
+ "created_at": 0,
+ "data_source": {
+ "source": {
+ "content": [
+ {
+ "item": {
+ "foo": "bar"
+ },
+ "sample": {
+ "foo": "bar"
+ }
+ }
+ ],
+ "type": "file_content"
+ },
+ "type": "jsonl"
+ },
+ "error": {
+ "code": "code",
+ "message": "message"
+ },
+ "eval_id": "eval_id",
+ "metadata": {
+ "foo": "string"
+ },
+ "model": "model",
+ "name": "name",
+ "object": "eval.run",
+ "per_model_usage": [
+ {
+ "cached_tokens": 0,
+ "completion_tokens": 0,
+ "invocation_count": 0,
+ "model_name": "model_name",
+ "prompt_tokens": 0,
+ "total_tokens": 0
+ }
+ ],
+ "per_testing_criteria_results": [
+ {
+ "failed": 0,
+ "passed": 0,
+ "testing_criteria": "testing_criteria"
+ }
+ ],
+ "report_url": "report_url",
+ "result_counts": {
+ "errored": 0,
+ "failed": 0,
+ "passed": 0,
+ "total": 0
+ },
+ "status": "status"
+}
+```
- An object specifying the format that the model must output.
+### Example
- Configuring `{ "type": "json_schema" }` enables Structured Outputs,
- which ensures the model will match your supplied JSON schema. Learn more in the
- [Structured Outputs guide](/docs/guides/structured-outputs).
+```http
+curl https://api.openai.com/v1/evals/eval_67abd54d9b0081909a86353f6fb9317a/runs/evalrun_67abd54d60ec8190832b46859da808f7 \
+ -H "Authorization: Bearer $OPENAI_API_KEY" \
+ -H "Content-Type: application/json"
+```
- The default format is `{ "type": "text" }` with no additional options.
+#### Response
- **Not recommended for gpt-4o and newer models:**
+```json
+{
+ "object": "eval.run",
+ "id": "evalrun_67abd54d60ec8190832b46859da808f7",
+ "eval_id": "eval_67abd54d9b0081909a86353f6fb9317a",
+ "report_url": "https://platform.openai.com/evaluations/eval_67abd54d9b0081909a86353f6fb9317a?run_id=evalrun_67abd54d60ec8190832b46859da808f7",
+ "status": "queued",
+ "model": "gpt-4o-mini",
+ "name": "gpt-4o-mini",
+ "created_at": 1743092069,
+ "result_counts": {
+ "total": 0,
+ "errored": 0,
+ "failed": 0,
+ "passed": 0
+ },
+ "per_model_usage": null,
+ "per_testing_criteria_results": null,
+ "data_source": {
+ "type": "completions",
+ "source": {
+ "type": "file_content",
+ "content": [
+ {
+ "item": {
+ "input": "Tech Company Launches Advanced Artificial Intelligence Platform",
+ "ground_truth": "Technology"
+ }
+ },
+ {
+ "item": {
+ "input": "Central Bank Increases Interest Rates Amid Inflation Concerns",
+ "ground_truth": "Markets"
+ }
+ },
+ {
+ "item": {
+ "input": "International Summit Addresses Climate Change Strategies",
+ "ground_truth": "World"
+ }
+ },
+ {
+ "item": {
+ "input": "Major Retailer Reports Record-Breaking Holiday Sales",
+ "ground_truth": "Business"
+ }
+ },
+ {
+ "item": {
+ "input": "National Team Qualifies for World Championship Finals",
+ "ground_truth": "Sports"
+ }
+ },
+ {
+ "item": {
+ "input": "Stock Markets Rally After Positive Economic Data Released",
+ "ground_truth": "Markets"
+ }
+ },
+ {
+ "item": {
+ "input": "Global Manufacturer Announces Merger with Competitor",
+ "ground_truth": "Business"
+ }
+ },
+ {
+ "item": {
+ "input": "Breakthrough in Renewable Energy Technology Unveiled",
+ "ground_truth": "Technology"
+ }
+ },
+ {
+ "item": {
+ "input": "World Leaders Sign Historic Climate Agreement",
+ "ground_truth": "World"
+ }
+ },
+ {
+ "item": {
+ "input": "Professional Athlete Sets New Record in Championship Event",
+ "ground_truth": "Sports"
+ }
+ },
+ {
+ "item": {
+ "input": "Financial Institutions Adapt to New Regulatory Requirements",
+ "ground_truth": "Business"
+ }
+ },
+ {
+ "item": {
+ "input": "Tech Conference Showcases Advances in Artificial Intelligence",
+ "ground_truth": "Technology"
+ }
+ },
+ {
+ "item": {
+ "input": "Global Markets Respond to Oil Price Fluctuations",
+ "ground_truth": "Markets"
+ }
+ },
+ {
+ "item": {
+ "input": "International Cooperation Strengthened Through New Treaty",
+ "ground_truth": "World"
+ }
+ },
+ {
+ "item": {
+ "input": "Sports League Announces Revised Schedule for Upcoming Season",
+ "ground_truth": "Sports"
+ }
+ }
+ ]
+ },
+ "input_messages": {
+ "type": "template",
+ "template": [
+ {
+ "type": "message",
+ "role": "developer",
+ "content": {
+ "type": "input_text",
+ "text": "Categorize a given news headline into one of the following topics: Technology, Markets, World, Business, or Sports.\n\n# Steps\n\n1. Analyze the content of the news headline to understand its primary focus.\n2. Extract the subject matter, identifying any key indicators or keywords.\n3. Use the identified indicators to determine the most suitable category out of the five options: Technology, Markets, World, Business, or Sports.\n4. Ensure only one category is selected per headline.\n\n# Output Format\n\nRespond with the chosen category as a single word. For instance: \"Technology\", \"Markets\", \"World\", \"Business\", or \"Sports\".\n\n# Examples\n\n**Input**: \"Apple Unveils New iPhone Model, Featuring Advanced AI Features\" \n**Output**: \"Technology\"\n\n**Input**: \"Global Stocks Mixed as Investors Await Central Bank Decisions\" \n**Output**: \"Markets\"\n\n**Input**: \"War in Ukraine: Latest Updates on Negotiation Status\" \n**Output**: \"World\"\n\n**Input**: \"Microsoft in Talks to Acquire Gaming Company for $2 Billion\" \n**Output**: \"Business\"\n\n**Input**: \"Manchester United Secures Win in Premier League Football Match\" \n**Output**: \"Sports\" \n\n# Notes\n\n- If the headline appears to fit into more than one category, choose the most dominant theme.\n- Keywords or phrases such as \"stocks\", \"company acquisition\", \"match\", or technological brands can be good indicators for classification.\n"
+ }
+ },
+ {
+ "type": "message",
+ "role": "user",
+ "content": {
+ "type": "input_text",
+ "text": "{{item.input}}"
+ }
+ }
+ ]
+ },
+ "model": "gpt-4o-mini",
+ "sampling_params": {
+ "seed": 42,
+ "temperature": 1.0,
+ "top_p": 1.0,
+ "max_completions_tokens": 2048
+ }
+ },
+ "error": null,
+ "metadata": {}
+}
+```
- Setting to `{ "type": "json_object" }` enables the older JSON mode, which
- ensures the message the model generates is valid JSON. Using `json_schema`
- is preferred for models that support it.
+## Cancel eval run
- - `ResponseFormatText = object { type }`
+**post** `/evals/{eval_id}/runs/{run_id}`
- Default response format. Used to generate text responses.
+Cancel an ongoing evaluation run.
- - `type: "text"`
+### Path Parameters
- The type of response format being defined. Always `text`.
+- `eval_id: string`
- - `"text"`
+- `run_id: string`
- - `ResponseFormatTextJSONSchemaConfig = object { name, schema, type, 2 more }`
+### Returns
- JSON Schema response format. Used to generate structured JSON responses.
- Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+- `id: string`
- - `name: string`
+ Unique identifier for the evaluation run.
- The name of the response format. Must be a-z, A-Z, 0-9, or contain
- underscores and dashes, with a maximum length of 64.
+- `created_at: number`
- - `schema: map[unknown]`
+ Unix timestamp (in seconds) when the evaluation run was created.
- The schema for the response format, described as a JSON Schema object.
- Learn how to build JSON schemas [here](https://json-schema.org/).
+- `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
- - `type: "json_schema"`
+ Information about the run's data source.
- The type of response format being defined. Always `json_schema`.
+ - `CreateEvalJSONLRunDataSource object { source, type }`
- - `"json_schema"`
+ A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
- - `description: optional string`
+ - `source: object { content, type } or object { id, type }`
- A description of what the response format is for, used by the model to
- determine how to respond in the format.
+ Determines what populates the `item` namespace in the data source.
- - `strict: optional boolean`
+ - `EvalJSONLFileContentSource object { content, type }`
- Whether to enable strict schema adherence when generating the output.
- If set to true, the model will always follow the exact schema defined
- in the `schema` field. Only a subset of JSON Schema is supported when
- `strict` is `true`. To learn more, read the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ - `content: array of object { item, sample }`
- - `ResponseFormatJSONObject = object { type }`
+ The content of the jsonl file.
- JSON object response format. An older method of generating JSON responses.
- Using `json_schema` is recommended for models that support it. Note that the
- model will not generate JSON without a system or user message instructing it
- to do so.
+ - `item: map[unknown]`
- - `type: "json_object"`
+ - `sample: optional map[unknown]`
- The type of response format being defined. Always `json_object`.
+ - `type: "file_content"`
- - `"json_object"`
+ The type of jsonl source. Always `file_content`.
- - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
+ - `"file_content"`
- An array of tools the model may call while generating a response. You
- can specify which tool to use by setting the `tool_choice` parameter.
+ - `EvalJSONLFileIDSource object { id, type }`
- The two categories of tools you can provide the model are:
+ - `id: string`
- - **Built-in tools**: Tools that are provided by OpenAI that extend the
- model's capabilities, like [web search](/docs/guides/tools-web-search)
- or [file search](/docs/guides/tools-file-search). Learn more about
- [built-in tools](/docs/guides/tools).
- - **Function calls (custom tools)**: Functions that are defined by you,
- enabling the model to call your own code. Learn more about
- [function calling](/docs/guides/function-calling).
+ The identifier of the file.
- - `Function = object { name, parameters, strict, 3 more }`
+ - `type: "file_id"`
- Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
+ The type of jsonl source. Always `file_id`.
- - `name: string`
+ - `"file_id"`
- The name of the function to call.
+ - `type: "jsonl"`
- - `parameters: map[unknown]`
+ The type of data source. Always `jsonl`.
- A JSON schema object describing the parameters of the function.
+ - `"jsonl"`
- - `strict: boolean`
+ - `CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }`
- Whether to enforce strict parameter validation. Default `true`.
+ A CompletionsRunDataSource object describing a model sampling configuration.
- - `type: "function"`
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
- The type of the function tool. Always `function`.
+ Determines what populates the `item` namespace in this run's data source.
- - `"function"`
+ - `EvalJSONLFileContentSource object { content, type }`
- - `defer_loading: optional boolean`
+ - `content: array of object { item, sample }`
- Whether this function is deferred and loaded via tool search.
+ The content of the jsonl file.
- - `description: optional string`
+ - `item: map[unknown]`
- A description of the function. Used by the model to determine whether or not to call the function.
+ - `sample: optional map[unknown]`
- - `FileSearch = object { type, vector_store_ids, filters, 2 more }`
+ - `type: "file_content"`
- A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
+ The type of jsonl source. Always `file_content`.
- - `type: "file_search"`
+ - `"file_content"`
- The type of the file search tool. Always `file_search`.
+ - `EvalJSONLFileIDSource object { id, type }`
- - `"file_search"`
+ - `id: string`
- - `vector_store_ids: array of string`
+ The identifier of the file.
- The IDs of the vector stores to search.
+ - `type: "file_id"`
- - `filters: optional ComparisonFilter or CompoundFilter`
+ The type of jsonl source. Always `file_id`.
- A filter to apply.
+ - `"file_id"`
- - `ComparisonFilter = object { key, type, value }`
+ - `StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more }`
- A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+ A StoredCompletionsRunDataSource configuration describing a set of filters
- - `key: string`
+ - `type: "stored_completions"`
- The key to compare against the value.
+ The type of source. Always `stored_completions`.
- - `type: "eq" or "ne" or "gt" or 5 more`
+ - `"stored_completions"`
- Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+ - `created_after: optional number`
- - `eq`: equals
- - `ne`: not equal
- - `gt`: greater than
- - `gte`: greater than or equal
- - `lt`: less than
- - `lte`: less than or equal
- - `in`: in
- - `nin`: not in
+ An optional Unix timestamp to filter items created after this time.
- - `"eq"`
+ - `created_before: optional number`
- - `"ne"`
+ An optional Unix timestamp to filter items created before this time.
- - `"gt"`
+ - `limit: optional number`
- - `"gte"`
+ An optional maximum number of items to return.
- - `"lt"`
+ - `metadata: optional Metadata`
- - `"lte"`
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- - `"in"`
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `"nin"`
+ - `model: optional string`
- - `value: string or number or boolean or array of string or number`
+ An optional model to filter by (e.g., 'gpt-4o').
- The value to compare against the attribute key; supports string, number, or boolean types.
+ - `type: "completions"`
- - `string`
+ The type of run data source. Always `completions`.
- - `number`
+ - `"completions"`
- - `boolean`
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
- - `array of string or number`
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
- - `string`
+ - `TemplateInputMessages object { template, type }`
- - `number`
+ - `template: array of EasyInputMessage or object { content, role, type }`
- - `CompoundFilter = object { filters, type }`
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
- Combine multiple filters using `and` or `or`.
+ - `EasyInputMessage object { content, role, phase, type }`
- - `filters: array of ComparisonFilter or unknown`
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
+ - `content: string or ResponseInputMessageContentList`
- - `ComparisonFilter = object { key, type, value }`
+ Text, image, or audio input to the model, used to generate a response.
+ Can also contain previous assistant responses.
- A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+ - `TextInput = string`
- - `key: string`
+ A text input to the model.
- The key to compare against the value.
+ - `ResponseInputMessageContentList = array of ResponseInputContent`
- - `type: "eq" or "ne" or "gt" or 5 more`
+ A list of one or many input items to the model, containing different content
+ types.
- Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+ - `ResponseInputText object { text, type }`
- - `eq`: equals
- - `ne`: not equal
- - `gt`: greater than
- - `gte`: greater than or equal
- - `lt`: less than
- - `lte`: less than or equal
- - `in`: in
- - `nin`: not in
+ A text input to the model.
- - `"eq"`
+ - `text: string`
- - `"ne"`
+ The text input to the model.
- - `"gt"`
+ - `type: "input_text"`
- - `"gte"`
+ The type of the input item. Always `input_text`.
- - `"lt"`
+ - `"input_text"`
- - `"lte"`
+ - `ResponseInputImage object { detail, type, file_id, image_url }`
- - `"in"`
+ An image input to the model. Learn about [image inputs](/docs/guides/vision).
- - `"nin"`
+ - `detail: "low" or "high" or "auto" or "original"`
- - `value: string or number or boolean or array of string or number`
+ The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
- The value to compare against the attribute key; supports string, number, or boolean types.
+ - `"low"`
- - `string`
+ - `"high"`
- - `number`
+ - `"auto"`
- - `boolean`
+ - `"original"`
- - `array of string or number`
+ - `type: "input_image"`
- - `string`
+ The type of the input item. Always `input_image`.
- - `number`
+ - `"input_image"`
- - `unknown`
+ - `file_id: optional string`
- - `type: "and" or "or"`
+ The ID of the file to be sent to the model.
- Type of operation: `and` or `or`.
+ - `image_url: optional string`
- - `"and"`
+ The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
- - `"or"`
+ - `ResponseInputFile object { type, detail, file_data, 3 more }`
- - `max_num_results: optional number`
+ A file input to the model.
- The maximum number of results to return. This number should be between 1 and 50 inclusive.
+ - `type: "input_file"`
- - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
+ The type of the input item. Always `input_file`.
- Ranking options for search.
+ - `"input_file"`
- - `hybrid_search: optional object { embedding_weight, text_weight }`
+ - `detail: optional "low" or "high"`
- Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
+ The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
- - `embedding_weight: number`
+ - `"low"`
- The weight of the embedding in the reciprocal ranking fusion.
+ - `"high"`
- - `text_weight: number`
+ - `file_data: optional string`
- The weight of the text in the reciprocal ranking fusion.
+ The content of the file to be sent to the model.
- - `ranker: optional "auto" or "default-2024-11-15"`
+ - `file_id: optional string`
- The ranker to use for the file search.
+ The ID of the file to be sent to the model.
- - `"auto"`
+ - `file_url: optional string`
- - `"default-2024-11-15"`
+ The URL of the file to be sent to the model.
- - `score_threshold: optional number`
+ - `filename: optional string`
- The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
+ The name of the file to be sent to the model.
- - `Computer = object { type }`
+ - `role: "user" or "assistant" or "system" or "developer"`
- A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- - `type: "computer"`
+ - `"user"`
- The type of the computer tool. Always `computer`.
+ - `"assistant"`
- - `"computer"`
+ - `"system"`
- - `ComputerUsePreview = object { display_height, display_width, environment, type }`
+ - `"developer"`
- A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+ - `phase: optional "commentary" or "final_answer"`
- - `display_height: number`
+ Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
+ For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
+ phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
- The height of the computer display.
+ - `"commentary"`
- - `display_width: number`
+ - `"final_answer"`
- The width of the computer display.
+ - `type: optional "message"`
- - `environment: "windows" or "mac" or "linux" or 2 more`
+ The type of the message input. Always `message`.
- The type of computer environment to control.
+ - `"message"`
- - `"windows"`
+ - `EvalMessageObject object { content, role, type }`
- - `"mac"`
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- - `"linux"`
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- - `"ubuntu"`
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- - `"browser"`
+ - `TextInput = string`
- - `type: "computer_use_preview"`
+ A text input to the model.
- The type of the computer use tool. Always `computer_use_preview`.
+ - `ResponseInputText object { text, type }`
- - `"computer_use_preview"`
+ A text input to the model.
- - `WebSearch = object { type, filters, search_context_size, user_location }`
+ - `OutputText object { text, type }`
- Search the Internet for sources related to the prompt. Learn more about the
- [web search tool](/docs/guides/tools-web-search).
+ A text output from the model.
- - `type: "web_search" or "web_search_2025_08_26"`
+ - `text: string`
- The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
+ The text output from the model.
- - `"web_search"`
+ - `type: "output_text"`
- - `"web_search_2025_08_26"`
+ The type of the output text. Always `output_text`.
- - `filters: optional object { allowed_domains }`
+ - `"output_text"`
- Filters for the search.
+ - `InputImage object { image_url, type, detail }`
- - `allowed_domains: optional array of string`
+ An image input block used within EvalItem content arrays.
- Allowed domains for the search. If not provided, all domains are allowed.
- Subdomains of the provided domains are allowed as well.
+ - `image_url: string`
- Example: `["pubmed.ncbi.nlm.nih.gov"]`
+ The URL of the image input.
- - `search_context_size: optional "low" or "medium" or "high"`
+ - `type: "input_image"`
- High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+ The type of the image input. Always `input_image`.
- - `"low"`
+ - `"input_image"`
- - `"medium"`
+ - `detail: optional string`
- - `"high"`
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `user_location: optional object { city, country, region, 2 more }`
+ - `ResponseInputAudio object { input_audio, type }`
- The approximate location of the user.
+ An audio input to the model.
- - `city: optional string`
+ - `input_audio: object { data, format }`
- Free text input for the city of the user, e.g. `San Francisco`.
+ - `data: string`
- - `country: optional string`
+ Base64-encoded audio data.
- The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+ - `format: "mp3" or "wav"`
- - `region: optional string`
+ The format of the audio data. Currently supported formats are `mp3` and
+ `wav`.
- Free text input for the region of the user, e.g. `California`.
+ - `"mp3"`
- - `timezone: optional string`
+ - `"wav"`
- The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+ - `type: "input_audio"`
- - `type: optional "approximate"`
+ The type of the input item. Always `input_audio`.
- The type of location approximation. Always `approximate`.
+ - `"input_audio"`
- - `"approximate"`
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
- - `Mcp = object { server_label, type, allowed_tools, 7 more }`
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
- Give the model access to additional tools via remote Model Context Protocol
- (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
+ - `TextInput = string`
- - `server_label: string`
+ A text input to the model.
- A label for this MCP server, used to identify it in tool calls.
+ - `ResponseInputText object { text, type }`
- - `type: "mcp"`
+ A text input to the model.
- The type of the MCP tool. Always `mcp`.
+ - `OutputText object { text, type }`
- - `"mcp"`
+ A text output from the model.
- - `allowed_tools: optional array of string or object { read_only, tool_names }`
+ - `text: string`
- List of allowed tool names or a filter object.
+ The text output from the model.
- - `McpAllowedTools = array of string`
+ - `type: "output_text"`
- A string array of allowed tool names
+ The type of the output text. Always `output_text`.
- - `McpToolFilter = object { read_only, tool_names }`
+ - `"output_text"`
- A filter object to specify which tools are allowed.
+ - `InputImage object { image_url, type, detail }`
- - `read_only: optional boolean`
+ An image input block used within EvalItem content arrays.
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ - `image_url: string`
- - `tool_names: optional array of string`
+ The URL of the image input.
- List of allowed tool names.
+ - `type: "input_image"`
- - `authorization: optional string`
+ The type of the image input. Always `input_image`.
- An OAuth access token that can be used with a remote MCP server, either
- with a custom MCP server URL or a service connector. Your application
- must handle the OAuth authorization flow and provide the token here.
+ - `"input_image"`
- - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
+ - `detail: optional string`
- Identifier for service connectors, like those available in ChatGPT. One of
- `server_url` or `connector_id` must be provided. Learn more about service
- connectors [here](/docs/guides/tools-remote-mcp#connectors).
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- Currently supported `connector_id` values are:
+ - `ResponseInputAudio object { input_audio, type }`
- - Dropbox: `connector_dropbox`
- - Gmail: `connector_gmail`
- - Google Calendar: `connector_googlecalendar`
- - Google Drive: `connector_googledrive`
- - Microsoft Teams: `connector_microsoftteams`
- - Outlook Calendar: `connector_outlookcalendar`
- - Outlook Email: `connector_outlookemail`
- - SharePoint: `connector_sharepoint`
+ An audio input to the model.
- - `"connector_dropbox"`
+ - `role: "user" or "assistant" or "system" or "developer"`
- - `"connector_gmail"`
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- - `"connector_googlecalendar"`
+ - `"user"`
- - `"connector_googledrive"`
+ - `"assistant"`
- - `"connector_microsoftteams"`
+ - `"system"`
- - `"connector_outlookcalendar"`
+ - `"developer"`
- - `"connector_outlookemail"`
+ - `type: optional "message"`
- - `"connector_sharepoint"`
+ The type of the message input. Always `message`.
- - `defer_loading: optional boolean`
+ - `"message"`
- Whether this MCP tool is deferred and discovered via tool search.
+ - `type: "template"`
- - `headers: optional map[string]`
+ The type of input messages. Always `template`.
- Optional HTTP headers to send to the MCP server. Use for authentication
- or other purposes.
+ - `"template"`
- - `require_approval: optional object { always, never } or "always" or "never"`
+ - `ItemReferenceInputMessages object { item_reference, type }`
- Specify which of the MCP server's tools require approval.
+ - `item_reference: string`
- - `McpToolApprovalFilter = object { always, never }`
+ A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
- Specify which of the MCP server's tools require approval. Can be
- `always`, `never`, or a filter object associated with tools
- that require approval.
+ - `type: "item_reference"`
- - `always: optional object { read_only, tool_names }`
+ The type of input messages. Always `item_reference`.
- A filter object to specify which tools are allowed.
+ - `"item_reference"`
- - `read_only: optional boolean`
+ - `model: optional string`
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ The name of the model to use for generating completions (e.g. "o3-mini").
- - `tool_names: optional array of string`
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
- List of allowed tool names.
+ - `max_completion_tokens: optional number`
- - `never: optional object { read_only, tool_names }`
+ The maximum number of tokens in the generated output.
- A filter object to specify which tools are allowed.
+ - `reasoning_effort: optional ReasoningEffort`
- - `read_only: optional boolean`
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- Indicates whether or not a tool modifies data or is read-only. If an
- MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
- it will match this filter.
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- - `tool_names: optional array of string`
+ - `"none"`
- List of allowed tool names.
+ - `"minimal"`
- - `McpToolApprovalSetting = "always" or "never"`
+ - `"low"`
- Specify a single approval policy for all tools. One of `always` or
- `never`. When set to `always`, all tools will require approval. When
- set to `never`, all tools will not require approval.
+ - `"medium"`
- - `"always"`
+ - `"high"`
- - `"never"`
+ - `"xhigh"`
- - `server_description: optional string`
+ - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
- Optional description of the MCP server, used to provide more context.
+ An object specifying the format that the model must output.
- - `server_url: optional string`
+ Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
+ Structured Outputs which ensures the model will match your supplied JSON
+ schema. Learn more in the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- The URL for the MCP server. One of `server_url` or `connector_id` must be
- provided.
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
- - `CodeInterpreter = object { container, type }`
+ - `ResponseFormatText object { type }`
- A tool that runs Python code to help generate a response to a prompt.
+ Default response format. Used to generate text responses.
- - `container: string or object { type, file_ids, memory_limit, network_policy }`
+ - `type: "text"`
- The code interpreter container. Can be a container ID or an object that
- specifies uploaded file IDs to make available to your code, along with an
- optional `memory_limit` setting.
+ The type of response format being defined. Always `text`.
- - `string`
+ - `"text"`
- The container ID.
+ - `ResponseFormatJSONSchema object { json_schema, type }`
- - `CodeInterpreterToolAuto = object { type, file_ids, memory_limit, network_policy }`
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
- Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
+ - `json_schema: object { name, description, schema, strict }`
- - `type: "auto"`
+ Structured Outputs configuration options, including a JSON Schema.
- Always `auto`.
+ - `name: string`
- - `"auto"`
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
- - `file_ids: optional array of string`
+ - `description: optional string`
- An optional list of uploaded files to make available to your code.
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
- - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+ - `schema: optional map[unknown]`
- The memory limit for the code interpreter container.
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- - `"1g"`
+ - `strict: optional boolean`
- - `"4g"`
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- - `"16g"`
+ - `type: "json_schema"`
- - `"64g"`
+ The type of response format being defined. Always `json_schema`.
- - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+ - `"json_schema"`
- Network access policy for the container.
+ - `ResponseFormatJSONObject object { type }`
- - `ContainerNetworkPolicyDisabled = object { type }`
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
- - `type: "disabled"`
+ - `type: "json_object"`
- Disable outbound network access. Always `disabled`.
+ The type of response format being defined. Always `json_object`.
- - `"disabled"`
+ - `"json_object"`
- - `ContainerNetworkPolicyAllowlist = object { allowed_domains, type, domain_secrets }`
+ - `seed: optional number`
- - `allowed_domains: array of string`
+ A seed value to initialize the randomness, during sampling.
- A list of allowed domains when type is `allowlist`.
+ - `temperature: optional number`
- - `type: "allowlist"`
+ A higher temperature increases randomness in the outputs.
- Allow outbound network access only to specified domains. Always `allowlist`.
+ - `tools: optional array of ChatCompletionFunctionTool`
- - `"allowlist"`
+ A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
- - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+ - `function: FunctionDefinition`
- Optional domain-scoped secrets for allowlisted domains.
+ - `name: string`
- - `domain: string`
+ The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
- The domain associated with the secret.
+ - `description: optional string`
- - `name: string`
+ A description of what the function does, used by the model to choose when and how to call the function.
- The name of the secret to inject for the domain.
+ - `parameters: optional FunctionParameters`
- - `value: string`
+ The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
- The secret value to inject for the domain.
+ Omitting `parameters` defines a function with an empty parameter list.
- - `type: "code_interpreter"`
+ - `strict: optional boolean`
- The type of the code interpreter tool. Always `code_interpreter`.
+ Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
- - `"code_interpreter"`
+ - `type: "function"`
- - `ImageGeneration = object { type, action, background, 9 more }`
+ The type of the tool. Currently, only `function` is supported.
- A tool that generates images using the GPT image models.
+ - `"function"`
- - `type: "image_generation"`
+ - `top_p: optional number`
- The type of the image generation tool. Always `image_generation`.
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- - `"image_generation"`
+ - `ResponsesRunDataSource object { source, type, input_messages, 2 more }`
- - `action: optional "generate" or "edit" or "auto"`
+ A ResponsesRunDataSource object describing a model sampling configuration.
- Whether to generate a new image or edit an existing image. Default: `auto`.
-
- - `"generate"`
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
- - `"edit"`
+ Determines what populates the `item` namespace in this run's data source.
- - `"auto"`
+ - `EvalJSONLFileContentSource object { content, type }`
- - `background: optional "transparent" or "opaque" or "auto"`
+ - `content: array of object { item, sample }`
- Background type for the generated image. One of `transparent`,
- `opaque`, or `auto`. Default: `auto`.
+ The content of the jsonl file.
- - `"transparent"`
+ - `item: map[unknown]`
- - `"opaque"`
+ - `sample: optional map[unknown]`
- - `"auto"`
+ - `type: "file_content"`
- - `input_fidelity: optional "high" or "low"`
+ The type of jsonl source. Always `file_content`.
- Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
+ - `"file_content"`
- - `"high"`
+ - `EvalJSONLFileIDSource object { id, type }`
- - `"low"`
+ - `id: string`
- - `input_image_mask: optional object { file_id, image_url }`
+ The identifier of the file.
- Optional mask for inpainting. Contains `image_url`
- (string, optional) and `file_id` (string, optional).
+ - `type: "file_id"`
- - `file_id: optional string`
+ The type of jsonl source. Always `file_id`.
- File ID for the mask image.
+ - `"file_id"`
- - `image_url: optional string`
+ - `EvalResponsesSource object { type, created_after, created_before, 8 more }`
- Base64-encoded mask image.
+ A EvalResponsesSource object describing a run data source configuration.
- - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+ - `type: "responses"`
- The image generation model to use. Default: `gpt-image-1`.
+ The type of run data source. Always `responses`.
- - `string`
+ - `"responses"`
- - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+ - `created_after: optional number`
- The image generation model to use. Default: `gpt-image-1`.
+ Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
- - `"gpt-image-1"`
+ - `created_before: optional number`
- - `"gpt-image-1-mini"`
+ Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
- - `"gpt-image-1.5"`
+ - `instructions_search: optional string`
- - `moderation: optional "auto" or "low"`
+ Optional string to search the 'instructions' field. This is a query parameter used to select responses.
- Moderation level for the generated image. Default: `auto`.
+ - `metadata: optional unknown`
- - `"auto"`
+ Metadata filter for the responses. This is a query parameter used to select responses.
- - `"low"`
+ - `model: optional string`
- - `output_compression: optional number`
+ The name of the model to find responses for. This is a query parameter used to select responses.
- Compression level for the output image. Default: 100.
+ - `reasoning_effort: optional ReasoningEffort`
- - `output_format: optional "png" or "webp" or "jpeg"`
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- The output format of the generated image. One of `png`, `webp`, or
- `jpeg`. Default: `png`.
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- - `"png"`
+ - `temperature: optional number`
- - `"webp"`
+ Sampling temperature. This is a query parameter used to select responses.
- - `"jpeg"`
+ - `tools: optional array of string`
- - `partial_images: optional number`
+ List of tool names. This is a query parameter used to select responses.
- Number of partial images to generate in streaming mode, from 0 (default value) to 3.
+ - `top_p: optional number`
- - `quality: optional "low" or "medium" or "high" or "auto"`
+ Nucleus sampling parameter. This is a query parameter used to select responses.
- The quality of the generated image. One of `low`, `medium`, `high`,
- or `auto`. Default: `auto`.
+ - `users: optional array of string`
- - `"low"`
+ List of user identifiers. This is a query parameter used to select responses.
- - `"medium"`
+ - `type: "responses"`
- - `"high"`
+ The type of run data source. Always `responses`.
- - `"auto"`
+ - `"responses"`
- - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
- The size of the generated image. One of `1024x1024`, `1024x1536`,
- `1536x1024`, or `auto`. Default: `auto`.
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
- - `"1024x1024"`
+ - `InputMessagesTemplate object { template, type }`
- - `"1024x1536"`
+ - `template: array of object { content, role } or object { content, role, type }`
- - `"1536x1024"`
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
- - `"auto"`
+ - `ChatMessage object { content, role }`
- - `LocalShell = object { type }`
+ - `content: string`
- A tool that allows the model to execute shell commands in a local environment.
+ The content of the message.
- - `type: "local_shell"`
+ - `role: string`
- The type of the local shell tool. Always `local_shell`.
+ The role of the message (e.g. "system", "assistant", "user").
- - `"local_shell"`
+ - `EvalMessageObject object { content, role, type }`
- - `Shell = object { type, environment }`
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
- A tool that allows the model to execute shell commands.
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
- - `type: "shell"`
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
- The type of the shell tool. Always `shell`.
+ - `TextInput = string`
- - `"shell"`
+ A text input to the model.
- - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
+ - `ResponseInputText object { text, type }`
- - `ContainerAuto = object { type, file_ids, memory_limit, 2 more }`
+ A text input to the model.
- - `type: "container_auto"`
+ - `OutputText object { text, type }`
- Automatically creates a container for this request
+ A text output from the model.
- - `"container_auto"`
+ - `text: string`
- - `file_ids: optional array of string`
+ The text output from the model.
- An optional list of uploaded files to make available to your code.
+ - `type: "output_text"`
- - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+ The type of the output text. Always `output_text`.
- The memory limit for the container.
+ - `"output_text"`
- - `"1g"`
+ - `InputImage object { image_url, type, detail }`
- - `"4g"`
+ An image input block used within EvalItem content arrays.
- - `"16g"`
+ - `image_url: string`
- - `"64g"`
+ The URL of the image input.
- - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+ - `type: "input_image"`
- Network access policy for the container.
+ The type of the image input. Always `input_image`.
- - `ContainerNetworkPolicyDisabled = object { type }`
+ - `"input_image"`
- - `type: "disabled"`
+ - `detail: optional string`
- Disable outbound network access. Always `disabled`.
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
- - `"disabled"`
+ - `ResponseInputAudio object { input_audio, type }`
- - `ContainerNetworkPolicyAllowlist = object { allowed_domains, type, domain_secrets }`
+ An audio input to the model.
- - `allowed_domains: array of string`
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
- A list of allowed domains when type is `allowlist`.
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
- - `type: "allowlist"`
+ - `role: "user" or "assistant" or "system" or "developer"`
- Allow outbound network access only to specified domains. Always `allowlist`.
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
- - `"allowlist"`
+ - `"user"`
- - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+ - `"assistant"`
- Optional domain-scoped secrets for allowlisted domains.
+ - `"system"`
- - `domain: string`
+ - `"developer"`
- The domain associated with the secret.
+ - `type: optional "message"`
- - `name: string`
+ The type of the message input. Always `message`.
- The name of the secret to inject for the domain.
+ - `"message"`
- - `value: string`
+ - `type: "template"`
- The secret value to inject for the domain.
+ The type of input messages. Always `template`.
- - `skills: optional array of SkillReference or InlineSkill`
+ - `"template"`
- An optional list of skills referenced by id or inline data.
+ - `InputMessagesItemReference object { item_reference, type }`
- - `SkillReference = object { skill_id, type, version }`
+ - `item_reference: string`
- - `skill_id: string`
+ A reference to a variable in the `item` namespace. Ie, "item.name"
- The ID of the referenced skill.
+ - `type: "item_reference"`
- - `type: "skill_reference"`
+ The type of input messages. Always `item_reference`.
- References a skill created with the /v1/skills endpoint.
+ - `"item_reference"`
- - `"skill_reference"`
+ - `model: optional string`
- - `version: optional string`
+ The name of the model to use for generating completions (e.g. "o3-mini").
- Optional skill version. Use a positive integer or 'latest'. Omit for default.
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
- - `InlineSkill = object { description, name, source, type }`
+ - `max_completion_tokens: optional number`
- - `description: string`
+ The maximum number of tokens in the generated output.
- The description of the skill.
+ - `reasoning_effort: optional ReasoningEffort`
- - `name: string`
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
- The name of the skill.
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
- - `source: InlineSkillSource`
+ - `seed: optional number`
- Inline skill payload
+ A seed value to initialize the randomness, during sampling.
- - `data: string`
+ - `temperature: optional number`
- Base64-encoded skill zip bundle.
+ A higher temperature increases randomness in the outputs.
- - `media_type: "application/zip"`
+ - `text: optional object { format }`
- The media type of the inline skill payload. Must be `application/zip`.
+ Configuration options for a text response from the model. Can be plain
+ text or structured JSON data. Learn more:
- - `"application/zip"`
+ - [Text inputs and outputs](/docs/guides/text)
+ - [Structured Outputs](/docs/guides/structured-outputs)
- - `type: "base64"`
+ - `format: optional ResponseFormatTextConfig`
- The type of the inline skill source. Must be `base64`.
+ An object specifying the format that the model must output.
- - `"base64"`
+ Configuring `{ "type": "json_schema" }` enables Structured Outputs,
+ which ensures the model will match your supplied JSON schema. Learn more in the
+ [Structured Outputs guide](/docs/guides/structured-outputs).
- - `type: "inline"`
+ The default format is `{ "type": "text" }` with no additional options.
- Defines an inline skill for this request.
+ **Not recommended for gpt-4o and newer models:**
- - `"inline"`
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
- - `LocalEnvironment = object { type, skills }`
+ - `ResponseFormatText object { type }`
- - `type: "local"`
+ Default response format. Used to generate text responses.
- Use a local computer environment.
+ - `ResponseFormatTextJSONSchemaConfig object { name, schema, type, 2 more }`
- - `"local"`
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
- - `skills: optional array of LocalSkill`
+ - `name: string`
- An optional list of skills.
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
- - `description: string`
+ - `schema: map[unknown]`
- The description of the skill.
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
- - `name: string`
+ - `type: "json_schema"`
- The name of the skill.
+ The type of response format being defined. Always `json_schema`.
- - `path: string`
+ - `"json_schema"`
- The path to the directory containing the skill.
+ - `description: optional string`
- - `ContainerReference = object { container_id, type }`
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
- - `container_id: string`
+ - `strict: optional boolean`
- The ID of the referenced container.
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
- - `type: "container_reference"`
+ - `ResponseFormatJSONObject object { type }`
- References a container created with the /v1/containers endpoint
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
- - `"container_reference"`
+ - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
- - `Custom = object { name, type, defer_loading, 2 more }`
+ An array of tools the model may call while generating a response. You
+ can specify which tool to use by setting the `tool_choice` parameter.
- A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+ The two categories of tools you can provide the model are:
+
+ - **Built-in tools**: Tools that are provided by OpenAI that extend the
+ model's capabilities, like [web search](/docs/guides/tools-web-search)
+ or [file search](/docs/guides/tools-file-search). Learn more about
+ [built-in tools](/docs/guides/tools).
+ - **Function calls (custom tools)**: Functions that are defined by you,
+ enabling the model to call your own code. Learn more about
+ [function calling](/docs/guides/function-calling).
+
+ - `Function object { name, parameters, strict, 3 more }`
+
+ Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
- `name: string`
- The name of the custom tool, used to identify it in tool calls.
+ The name of the function to call.
- - `type: "custom"`
+ - `parameters: map[unknown]`
- The type of the custom tool. Always `custom`.
+ A JSON schema object describing the parameters of the function.
- - `"custom"`
+ - `strict: boolean`
+
+ Whether to enforce strict parameter validation. Default `true`.
+
+ - `type: "function"`
+
+ The type of the function tool. Always `function`.
+
+ - `"function"`
- `defer_loading: optional boolean`
- Whether this tool should be deferred and discovered via tool search.
+ Whether this function is deferred and loaded via tool search.
- `description: optional string`
- Optional description of the custom tool, used to provide more context.
+ A description of the function. Used by the model to determine whether or not to call the function.
- - `format: optional CustomToolInputFormat`
+ - `FileSearch object { type, vector_store_ids, filters, 2 more }`
- The input format for the custom tool. Default is unconstrained text.
+ A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
- - `Text = object { type }`
+ - `type: "file_search"`
- Unconstrained free-form text.
+ The type of the file search tool. Always `file_search`.
- - `type: "text"`
+ - `"file_search"`
- Unconstrained text format. Always `text`.
+ - `vector_store_ids: array of string`
- - `"text"`
+ The IDs of the vector stores to search.
- - `Grammar = object { definition, syntax, type }`
+ - `filters: optional ComparisonFilter or CompoundFilter`
- A grammar defined by the user.
+ A filter to apply.
- - `definition: string`
+ - `ComparisonFilter object { key, type, value }`
- The grammar definition.
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
- - `syntax: "lark" or "regex"`
+ - `key: string`
- The syntax of the grammar definition. One of `lark` or `regex`.
+ The key to compare against the value.
- - `"lark"`
+ - `type: "eq" or "ne" or "gt" or 5 more`
- - `"regex"`
+ Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
- - `type: "grammar"`
+ - `eq`: equals
+ - `ne`: not equal
+ - `gt`: greater than
+ - `gte`: greater than or equal
+ - `lt`: less than
+ - `lte`: less than or equal
+ - `in`: in
+ - `nin`: not in
- Grammar format. Always `grammar`.
+ - `"eq"`
- - `"grammar"`
+ - `"ne"`
- - `Namespace = object { description, name, tools, type }`
+ - `"gt"`
- Groups function/custom tools under a shared namespace.
+ - `"gte"`
- - `description: string`
+ - `"lt"`
- A description of the namespace shown to the model.
+ - `"lte"`
- - `name: string`
+ - `"in"`
- The namespace name used in tool calls (for example, `crm`).
+ - `"nin"`
- - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
+ - `value: string or number or boolean or array of string or number`
- The function/custom tools available inside this namespace.
+ The value to compare against the attribute key; supports string, number, or boolean types.
- - `Function = object { name, type, defer_loading, 3 more }`
+ - `string`
- - `name: string`
+ - `number`
- - `type: "function"`
+ - `boolean`
- - `"function"`
+ - `array of string or number`
- - `defer_loading: optional boolean`
+ - `string`
- Whether this function should be deferred and discovered via tool search.
+ - `number`
- - `description: optional string`
+ - `CompoundFilter object { filters, type }`
- - `parameters: optional unknown`
+ Combine multiple filters using `and` or `or`.
- - `strict: optional boolean`
+ - `filters: array of ComparisonFilter or unknown`
- - `Custom = object { name, type, defer_loading, 2 more }`
+ Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
- A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+ - `ComparisonFilter object { key, type, value }`
- - `name: string`
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
- The name of the custom tool, used to identify it in tool calls.
+ - `unknown`
- - `type: "custom"`
+ - `type: "and" or "or"`
- The type of the custom tool. Always `custom`.
+ Type of operation: `and` or `or`.
- - `"custom"`
+ - `"and"`
- - `defer_loading: optional boolean`
+ - `"or"`
- Whether this tool should be deferred and discovered via tool search.
+ - `max_num_results: optional number`
- - `description: optional string`
+ The maximum number of results to return. This number should be between 1 and 50 inclusive.
- Optional description of the custom tool, used to provide more context.
+ - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
- - `format: optional CustomToolInputFormat`
+ Ranking options for search.
- The input format for the custom tool. Default is unconstrained text.
+ - `hybrid_search: optional object { embedding_weight, text_weight }`
- - `Text = object { type }`
+ Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
- Unconstrained free-form text.
+ - `embedding_weight: number`
- - `type: "text"`
+ The weight of the embedding in the reciprocal ranking fusion.
- Unconstrained text format. Always `text`.
+ - `text_weight: number`
- - `"text"`
+ The weight of the text in the reciprocal ranking fusion.
- - `Grammar = object { definition, syntax, type }`
+ - `ranker: optional "auto" or "default-2024-11-15"`
- A grammar defined by the user.
+ The ranker to use for the file search.
- - `definition: string`
+ - `"auto"`
- The grammar definition.
+ - `"default-2024-11-15"`
- - `syntax: "lark" or "regex"`
+ - `score_threshold: optional number`
- The syntax of the grammar definition. One of `lark` or `regex`.
+ The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
- - `"lark"`
+ - `Computer object { type }`
- - `"regex"`
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
- - `type: "grammar"`
+ - `type: "computer"`
- Grammar format. Always `grammar`.
+ The type of the computer tool. Always `computer`.
- - `"grammar"`
+ - `"computer"`
- - `type: "namespace"`
+ - `ComputerUsePreview object { display_height, display_width, environment, type }`
- The type of the tool. Always `namespace`.
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
- - `"namespace"`
+ - `display_height: number`
- - `ToolSearch = object { type, description, execution, parameters }`
+ The height of the computer display.
- Hosted or BYOT tool search configuration for deferred tools.
+ - `display_width: number`
- - `type: "tool_search"`
+ The width of the computer display.
- The type of the tool. Always `tool_search`.
+ - `environment: "windows" or "mac" or "linux" or 2 more`
- - `"tool_search"`
+ The type of computer environment to control.
- - `description: optional string`
+ - `"windows"`
- Description shown to the model for a client-executed tool search tool.
+ - `"mac"`
- - `execution: optional "server" or "client"`
+ - `"linux"`
- Whether tool search is executed by the server or by the client.
+ - `"ubuntu"`
- - `"server"`
+ - `"browser"`
- - `"client"`
+ - `type: "computer_use_preview"`
- - `parameters: optional unknown`
+ The type of the computer use tool. Always `computer_use_preview`.
- Parameter schema for a client-executed tool search tool.
+ - `"computer_use_preview"`
- - `WebSearchPreview = object { type, search_content_types, search_context_size, user_location }`
+ - `WebSearch object { type, filters, search_context_size, user_location }`
- This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
+ Search the Internet for sources related to the prompt. Learn more about the
+ [web search tool](/docs/guides/tools-web-search).
- - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
+ - `type: "web_search" or "web_search_2025_08_26"`
- The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
+ The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
- - `"web_search_preview"`
+ - `"web_search"`
- - `"web_search_preview_2025_03_11"`
+ - `"web_search_2025_08_26"`
- - `search_content_types: optional array of "text" or "image"`
+ - `filters: optional object { allowed_domains }`
- - `"text"`
+ Filters for the search.
- - `"image"`
+ - `allowed_domains: optional array of string`
+
+ Allowed domains for the search. If not provided, all domains are allowed.
+ Subdomains of the provided domains are allowed as well.
+
+ Example: `["pubmed.ncbi.nlm.nih.gov"]`
- `search_context_size: optional "low" or "medium" or "high"`
@@ -14465,15 +13545,9 @@ Cancel an ongoing evaluation run.
- `"high"`
- - `user_location: optional object { type, city, country, 2 more }`
-
- The user's location.
-
- - `type: "approximate"`
-
- The type of location approximation. Always `approximate`.
+ - `user_location: optional object { city, country, region, 2 more }`
- - `"approximate"`
+ The approximate location of the user.
- `city: optional string`
@@ -14491,232 +13565,937 @@ Cancel an ongoing evaluation run.
The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
- - `ApplyPatch = object { type }`
+ - `type: optional "approximate"`
- Allows the assistant to create, delete, or update files using unified diffs.
+ The type of location approximation. Always `approximate`.
- - `type: "apply_patch"`
+ - `"approximate"`
- The type of the tool. Always `apply_patch`.
+ - `Mcp object { server_label, type, allowed_tools, 7 more }`
- - `"apply_patch"`
+ Give the model access to additional tools via remote Model Context Protocol
+ (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
- - `top_p: optional number`
+ - `server_label: string`
- An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+ A label for this MCP server, used to identify it in tool calls.
-- `error: EvalAPIError`
+ - `type: "mcp"`
- An object representing an error response from the Eval API.
+ The type of the MCP tool. Always `mcp`.
- - `code: string`
+ - `"mcp"`
- The error code.
+ - `allowed_tools: optional array of string or object { read_only, tool_names }`
- - `message: string`
+ List of allowed tool names or a filter object.
- The error message.
+ - `McpAllowedTools = array of string`
-- `eval_id: string`
+ A string array of allowed tool names
- The identifier of the associated evaluation.
+ - `McpToolFilter object { read_only, tool_names }`
-- `metadata: Metadata`
+ A filter object to specify which tools are allowed.
- Set of 16 key-value pairs that can be attached to an object. This can be
- useful for storing additional information about the object in a structured
- format, and querying for objects via API or the dashboard.
+ - `read_only: optional boolean`
- Keys are strings with a maximum length of 64 characters. Values are strings
- with a maximum length of 512 characters.
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
-- `model: string`
+ - `tool_names: optional array of string`
- The model that is evaluated, if applicable.
+ List of allowed tool names.
-- `name: string`
+ - `authorization: optional string`
- The name of the evaluation run.
+ An OAuth access token that can be used with a remote MCP server, either
+ with a custom MCP server URL or a service connector. Your application
+ must handle the OAuth authorization flow and provide the token here.
-- `object: "eval.run"`
+ - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
- The type of the object. Always "eval.run".
+ Identifier for service connectors, like those available in ChatGPT. One of
+ `server_url` or `connector_id` must be provided. Learn more about service
+ connectors [here](/docs/guides/tools-remote-mcp#connectors).
- - `"eval.run"`
+ Currently supported `connector_id` values are:
-- `per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }`
+ - Dropbox: `connector_dropbox`
+ - Gmail: `connector_gmail`
+ - Google Calendar: `connector_googlecalendar`
+ - Google Drive: `connector_googledrive`
+ - Microsoft Teams: `connector_microsoftteams`
+ - Outlook Calendar: `connector_outlookcalendar`
+ - Outlook Email: `connector_outlookemail`
+ - SharePoint: `connector_sharepoint`
- Usage statistics for each model during the evaluation run.
+ - `"connector_dropbox"`
- - `cached_tokens: number`
+ - `"connector_gmail"`
- The number of tokens retrieved from cache.
+ - `"connector_googlecalendar"`
- - `completion_tokens: number`
+ - `"connector_googledrive"`
- The number of completion tokens generated.
+ - `"connector_microsoftteams"`
- - `invocation_count: number`
+ - `"connector_outlookcalendar"`
- The number of invocations.
+ - `"connector_outlookemail"`
- - `model_name: string`
+ - `"connector_sharepoint"`
- The name of the model.
+ - `defer_loading: optional boolean`
- - `prompt_tokens: number`
+ Whether this MCP tool is deferred and discovered via tool search.
- The number of prompt tokens used.
+ - `headers: optional map[string]`
- - `total_tokens: number`
+ Optional HTTP headers to send to the MCP server. Use for authentication
+ or other purposes.
- The total number of tokens used.
+ - `require_approval: optional object { always, never } or "always" or "never"`
-- `per_testing_criteria_results: array of object { failed, passed, testing_criteria }`
+ Specify which of the MCP server's tools require approval.
- Results per testing criteria applied during the evaluation run.
+ - `McpToolApprovalFilter object { always, never }`
- - `failed: number`
+ Specify which of the MCP server's tools require approval. Can be
+ `always`, `never`, or a filter object associated with tools
+ that require approval.
- Number of tests failed for this criteria.
+ - `always: optional object { read_only, tool_names }`
- - `passed: number`
+ A filter object to specify which tools are allowed.
- Number of tests passed for this criteria.
+ - `read_only: optional boolean`
- - `testing_criteria: string`
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- A description of the testing criteria.
+ - `tool_names: optional array of string`
-- `report_url: string`
+ List of allowed tool names.
- The URL to the rendered evaluation run report on the UI dashboard.
+ - `never: optional object { read_only, tool_names }`
-- `result_counts: object { errored, failed, passed, total }`
+ A filter object to specify which tools are allowed.
- Counters summarizing the outcomes of the evaluation run.
+ - `read_only: optional boolean`
- - `errored: number`
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
- Number of output items that resulted in an error.
+ - `tool_names: optional array of string`
- - `failed: number`
+ List of allowed tool names.
- Number of output items that failed to pass the evaluation.
+ - `McpToolApprovalSetting = "always" or "never"`
- - `passed: number`
+ Specify a single approval policy for all tools. One of `always` or
+ `never`. When set to `always`, all tools will require approval. When
+ set to `never`, all tools will not require approval.
- Number of output items that passed the evaluation.
+ - `"always"`
- - `total: number`
+ - `"never"`
- Total number of executed output items.
+ - `server_description: optional string`
-- `status: string`
+ Optional description of the MCP server, used to provide more context.
- The status of the evaluation run.
+ - `server_url: optional string`
-### Example
+ The URL for the MCP server. One of `server_url` or `connector_id` must be
+ provided.
-```http
-curl https://api.openai.com/v1/evals/$EVAL_ID/runs/$RUN_ID \
- -X POST \
- -H "Authorization: Bearer $OPENAI_API_KEY"
-```
+ - `CodeInterpreter object { container, type }`
-#### Response
+ A tool that runs Python code to help generate a response to a prompt.
-```json
-{
- "id": "id",
- "created_at": 0,
- "data_source": {
- "source": {
- "content": [
- {
- "item": {
- "foo": "bar"
- },
- "sample": {
- "foo": "bar"
- }
- }
- ],
- "type": "file_content"
- },
- "type": "jsonl"
- },
- "error": {
- "code": "code",
- "message": "message"
- },
- "eval_id": "eval_id",
- "metadata": {
- "foo": "string"
- },
- "model": "model",
- "name": "name",
- "object": "eval.run",
- "per_model_usage": [
- {
- "cached_tokens": 0,
- "completion_tokens": 0,
- "invocation_count": 0,
- "model_name": "model_name",
- "prompt_tokens": 0,
- "total_tokens": 0
- }
- ],
- "per_testing_criteria_results": [
- {
- "failed": 0,
- "passed": 0,
- "testing_criteria": "testing_criteria"
- }
- ],
- "report_url": "report_url",
- "result_counts": {
- "errored": 0,
- "failed": 0,
- "passed": 0,
- "total": 0
- },
- "status": "status"
-}
-```
+ - `container: string or object { type, file_ids, memory_limit, network_policy }`
-### Example
+ The code interpreter container. Can be a container ID or an object that
+ specifies uploaded file IDs to make available to your code, along with an
+ optional `memory_limit` setting.
-```http
-curl https://api.openai.com/v1/evals/eval_67abd54d9b0081909a86353f6fb9317a/runs/evalrun_67abd54d60ec8190832b46859da808f7/cancel \
- -X POST \
- -H "Authorization: Bearer $OPENAI_API_KEY" \
- -H "Content-Type: application/json"
-```
+ - `string`
-#### Response
+ The container ID.
-```json
-{
- "object": "eval.run",
- "id": "evalrun_67abd54d60ec8190832b46859da808f7",
- "eval_id": "eval_67abd54d9b0081909a86353f6fb9317a",
- "report_url": "https://platform.openai.com/evaluations/eval_67abd54d9b0081909a86353f6fb9317a?run_id=evalrun_67abd54d60ec8190832b46859da808f7",
- "status": "canceled",
- "model": "gpt-4o-mini",
- "name": "gpt-4o-mini",
- "created_at": 1743092069,
- "result_counts": {
- "total": 0,
- "errored": 0,
- "failed": 0,
- "passed": 0
- },
- "per_model_usage": null,
- "per_testing_criteria_results": null,
- "data_source": {
- "type": "completions",
- "source": {
+ - `CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy }`
+
+ Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
+
+ - `type: "auto"`
+
+ Always `auto`.
+
+ - `"auto"`
+
+ - `file_ids: optional array of string`
+
+ An optional list of uploaded files to make available to your code.
+
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+
+ The memory limit for the code interpreter container.
+
+ - `"1g"`
+
+ - `"4g"`
+
+ - `"16g"`
+
+ - `"64g"`
+
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+
+ Network access policy for the container.
+
+ - `ContainerNetworkPolicyDisabled object { type }`
+
+ - `type: "disabled"`
+
+ Disable outbound network access. Always `disabled`.
+
+ - `"disabled"`
+
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
+
+ - `allowed_domains: array of string`
+
+ A list of allowed domains when type is `allowlist`.
+
+ - `type: "allowlist"`
+
+ Allow outbound network access only to specified domains. Always `allowlist`.
+
+ - `"allowlist"`
+
+ - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+
+ Optional domain-scoped secrets for allowlisted domains.
+
+ - `domain: string`
+
+ The domain associated with the secret.
+
+ - `name: string`
+
+ The name of the secret to inject for the domain.
+
+ - `value: string`
+
+ The secret value to inject for the domain.
+
+ - `type: "code_interpreter"`
+
+ The type of the code interpreter tool. Always `code_interpreter`.
+
+ - `"code_interpreter"`
+
+ - `ImageGeneration object { type, action, background, 9 more }`
+
+ A tool that generates images using the GPT image models.
+
+ - `type: "image_generation"`
+
+ The type of the image generation tool. Always `image_generation`.
+
+ - `"image_generation"`
+
+ - `action: optional "generate" or "edit" or "auto"`
+
+ Whether to generate a new image or edit an existing image. Default: `auto`.
+
+ - `"generate"`
+
+ - `"edit"`
+
+ - `"auto"`
+
+ - `background: optional "transparent" or "opaque" or "auto"`
+
+ Background type for the generated image. One of `transparent`,
+ `opaque`, or `auto`. Default: `auto`.
+
+ - `"transparent"`
+
+ - `"opaque"`
+
+ - `"auto"`
+
+ - `input_fidelity: optional "high" or "low"`
+
+ Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
+
+ - `"high"`
+
+ - `"low"`
+
+ - `input_image_mask: optional object { file_id, image_url }`
+
+ Optional mask for inpainting. Contains `image_url`
+ (string, optional) and `file_id` (string, optional).
+
+ - `file_id: optional string`
+
+ File ID for the mask image.
+
+ - `image_url: optional string`
+
+ Base64-encoded mask image.
+
+ - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+
+ The image generation model to use. Default: `gpt-image-1`.
+
+ - `string`
+
+ - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+
+ The image generation model to use. Default: `gpt-image-1`.
+
+ - `"gpt-image-1"`
+
+ - `"gpt-image-1-mini"`
+
+ - `"gpt-image-1.5"`
+
+ - `moderation: optional "auto" or "low"`
+
+ Moderation level for the generated image. Default: `auto`.
+
+ - `"auto"`
+
+ - `"low"`
+
+ - `output_compression: optional number`
+
+ Compression level for the output image. Default: 100.
+
+ - `output_format: optional "png" or "webp" or "jpeg"`
+
+ The output format of the generated image. One of `png`, `webp`, or
+ `jpeg`. Default: `png`.
+
+ - `"png"`
+
+ - `"webp"`
+
+ - `"jpeg"`
+
+ - `partial_images: optional number`
+
+ Number of partial images to generate in streaming mode, from 0 (default value) to 3.
+
+ - `quality: optional "low" or "medium" or "high" or "auto"`
+
+ The quality of the generated image. One of `low`, `medium`, `high`,
+ or `auto`. Default: `auto`.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `"auto"`
+
+ - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
+
+ The size of the generated image. One of `1024x1024`, `1024x1536`,
+ `1536x1024`, or `auto`. Default: `auto`.
+
+ - `"1024x1024"`
+
+ - `"1024x1536"`
+
+ - `"1536x1024"`
+
+ - `"auto"`
+
+ - `LocalShell object { type }`
+
+ A tool that allows the model to execute shell commands in a local environment.
+
+ - `type: "local_shell"`
+
+ The type of the local shell tool. Always `local_shell`.
+
+ - `"local_shell"`
+
+ - `Shell object { type, environment }`
+
+ A tool that allows the model to execute shell commands.
+
+ - `type: "shell"`
+
+ The type of the shell tool. Always `shell`.
+
+ - `"shell"`
+
+ - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
+
+ - `ContainerAuto object { type, file_ids, memory_limit, 2 more }`
+
+ - `type: "container_auto"`
+
+ Automatically creates a container for this request
+
+ - `"container_auto"`
+
+ - `file_ids: optional array of string`
+
+ An optional list of uploaded files to make available to your code.
+
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+
+ The memory limit for the container.
+
+ - `"1g"`
+
+ - `"4g"`
+
+ - `"16g"`
+
+ - `"64g"`
+
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+
+ Network access policy for the container.
+
+ - `ContainerNetworkPolicyDisabled object { type }`
+
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
+
+ - `skills: optional array of SkillReference or InlineSkill`
+
+ An optional list of skills referenced by id or inline data.
+
+ - `SkillReference object { skill_id, type, version }`
+
+ - `skill_id: string`
+
+ The ID of the referenced skill.
+
+ - `type: "skill_reference"`
+
+ References a skill created with the /v1/skills endpoint.
+
+ - `"skill_reference"`
+
+ - `version: optional string`
+
+ Optional skill version. Use a positive integer or 'latest'. Omit for default.
+
+ - `InlineSkill object { description, name, source, type }`
+
+ - `description: string`
+
+ The description of the skill.
+
+ - `name: string`
+
+ The name of the skill.
+
+ - `source: InlineSkillSource`
+
+ Inline skill payload
+
+ - `data: string`
+
+ Base64-encoded skill zip bundle.
+
+ - `media_type: "application/zip"`
+
+ The media type of the inline skill payload. Must be `application/zip`.
+
+ - `"application/zip"`
+
+ - `type: "base64"`
+
+ The type of the inline skill source. Must be `base64`.
+
+ - `"base64"`
+
+ - `type: "inline"`
+
+ Defines an inline skill for this request.
+
+ - `"inline"`
+
+ - `LocalEnvironment object { type, skills }`
+
+ - `type: "local"`
+
+ Use a local computer environment.
+
+ - `"local"`
+
+ - `skills: optional array of LocalSkill`
+
+ An optional list of skills.
+
+ - `description: string`
+
+ The description of the skill.
+
+ - `name: string`
+
+ The name of the skill.
+
+ - `path: string`
+
+ The path to the directory containing the skill.
+
+ - `ContainerReference object { container_id, type }`
+
+ - `container_id: string`
+
+ The ID of the referenced container.
+
+ - `type: "container_reference"`
+
+ References a container created with the /v1/containers endpoint
+
+ - `"container_reference"`
+
+ - `Custom object { name, type, defer_loading, 2 more }`
+
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+
+ - `name: string`
+
+ The name of the custom tool, used to identify it in tool calls.
+
+ - `type: "custom"`
+
+ The type of the custom tool. Always `custom`.
+
+ - `"custom"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this tool should be deferred and discovered via tool search.
+
+ - `description: optional string`
+
+ Optional description of the custom tool, used to provide more context.
+
+ - `format: optional CustomToolInputFormat`
+
+ The input format for the custom tool. Default is unconstrained text.
+
+ - `Text object { type }`
+
+ Unconstrained free-form text.
+
+ - `type: "text"`
+
+ Unconstrained text format. Always `text`.
+
+ - `"text"`
+
+ - `Grammar object { definition, syntax, type }`
+
+ A grammar defined by the user.
+
+ - `definition: string`
+
+ The grammar definition.
+
+ - `syntax: "lark" or "regex"`
+
+ The syntax of the grammar definition. One of `lark` or `regex`.
+
+ - `"lark"`
+
+ - `"regex"`
+
+ - `type: "grammar"`
+
+ Grammar format. Always `grammar`.
+
+ - `"grammar"`
+
+ - `Namespace object { description, name, tools, type }`
+
+ Groups function/custom tools under a shared namespace.
+
+ - `description: string`
+
+ A description of the namespace shown to the model.
+
+ - `name: string`
+
+ The namespace name used in tool calls (for example, `crm`).
+
+ - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
+
+ The function/custom tools available inside this namespace.
+
+ - `Function object { name, type, defer_loading, 3 more }`
+
+ - `name: string`
+
+ - `type: "function"`
+
+ - `"function"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this function should be deferred and discovered via tool search.
+
+ - `description: optional string`
+
+ - `parameters: optional unknown`
+
+ - `strict: optional boolean`
+
+ - `Custom object { name, type, defer_loading, 2 more }`
+
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+
+ - `name: string`
+
+ The name of the custom tool, used to identify it in tool calls.
+
+ - `type: "custom"`
+
+ The type of the custom tool. Always `custom`.
+
+ - `"custom"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this tool should be deferred and discovered via tool search.
+
+ - `description: optional string`
+
+ Optional description of the custom tool, used to provide more context.
+
+ - `format: optional CustomToolInputFormat`
+
+ The input format for the custom tool. Default is unconstrained text.
+
+ - `type: "namespace"`
+
+ The type of the tool. Always `namespace`.
+
+ - `"namespace"`
+
+ - `ToolSearch object { type, description, execution, parameters }`
+
+ Hosted or BYOT tool search configuration for deferred tools.
+
+ - `type: "tool_search"`
+
+ The type of the tool. Always `tool_search`.
+
+ - `"tool_search"`
+
+ - `description: optional string`
+
+ Description shown to the model for a client-executed tool search tool.
+
+ - `execution: optional "server" or "client"`
+
+ Whether tool search is executed by the server or by the client.
+
+ - `"server"`
+
+ - `"client"`
+
+ - `parameters: optional unknown`
+
+ Parameter schema for a client-executed tool search tool.
+
+ - `WebSearchPreview object { type, search_content_types, search_context_size, user_location }`
+
+ This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
+
+ - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
+
+ The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
+
+ - `"web_search_preview"`
+
+ - `"web_search_preview_2025_03_11"`
+
+ - `search_content_types: optional array of "text" or "image"`
+
+ - `"text"`
+
+ - `"image"`
+
+ - `search_context_size: optional "low" or "medium" or "high"`
+
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `user_location: optional object { type, city, country, 2 more }`
+
+ The user's location.
+
+ - `type: "approximate"`
+
+ The type of location approximation. Always `approximate`.
+
+ - `"approximate"`
+
+ - `city: optional string`
+
+ Free text input for the city of the user, e.g. `San Francisco`.
+
+ - `country: optional string`
+
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+
+ - `region: optional string`
+
+ Free text input for the region of the user, e.g. `California`.
+
+ - `timezone: optional string`
+
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+
+ - `ApplyPatch object { type }`
+
+ Allows the assistant to create, delete, or update files using unified diffs.
+
+ - `type: "apply_patch"`
+
+ The type of the tool. Always `apply_patch`.
+
+ - `"apply_patch"`
+
+ - `top_p: optional number`
+
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+
+- `error: EvalAPIError`
+
+ An object representing an error response from the Eval API.
+
+ - `code: string`
+
+ The error code.
+
+ - `message: string`
+
+ The error message.
+
+- `eval_id: string`
+
+ The identifier of the associated evaluation.
+
+- `metadata: Metadata`
+
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
+
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
+
+- `model: string`
+
+ The model that is evaluated, if applicable.
+
+- `name: string`
+
+ The name of the evaluation run.
+
+- `object: "eval.run"`
+
+ The type of the object. Always "eval.run".
+
+ - `"eval.run"`
+
+- `per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }`
+
+ Usage statistics for each model during the evaluation run.
+
+ - `cached_tokens: number`
+
+ The number of tokens retrieved from cache.
+
+ - `completion_tokens: number`
+
+ The number of completion tokens generated.
+
+ - `invocation_count: number`
+
+ The number of invocations.
+
+ - `model_name: string`
+
+ The name of the model.
+
+ - `prompt_tokens: number`
+
+ The number of prompt tokens used.
+
+ - `total_tokens: number`
+
+ The total number of tokens used.
+
+- `per_testing_criteria_results: array of object { failed, passed, testing_criteria }`
+
+ Results per testing criteria applied during the evaluation run.
+
+ - `failed: number`
+
+ Number of tests failed for this criteria.
+
+ - `passed: number`
+
+ Number of tests passed for this criteria.
+
+ - `testing_criteria: string`
+
+ A description of the testing criteria.
+
+- `report_url: string`
+
+ The URL to the rendered evaluation run report on the UI dashboard.
+
+- `result_counts: object { errored, failed, passed, total }`
+
+ Counters summarizing the outcomes of the evaluation run.
+
+ - `errored: number`
+
+ Number of output items that resulted in an error.
+
+ - `failed: number`
+
+ Number of output items that failed to pass the evaluation.
+
+ - `passed: number`
+
+ Number of output items that passed the evaluation.
+
+ - `total: number`
+
+ Total number of executed output items.
+
+- `status: string`
+
+ The status of the evaluation run.
+
+### Example
+
+```http
+curl https://api.openai.com/v1/evals/$EVAL_ID/runs/$RUN_ID \
+ -X POST \
+ -H "Authorization: Bearer $OPENAI_API_KEY"
+```
+
+#### Response
+
+```json
+{
+ "id": "id",
+ "created_at": 0,
+ "data_source": {
+ "source": {
+ "content": [
+ {
+ "item": {
+ "foo": "bar"
+ },
+ "sample": {
+ "foo": "bar"
+ }
+ }
+ ],
+ "type": "file_content"
+ },
+ "type": "jsonl"
+ },
+ "error": {
+ "code": "code",
+ "message": "message"
+ },
+ "eval_id": "eval_id",
+ "metadata": {
+ "foo": "string"
+ },
+ "model": "model",
+ "name": "name",
+ "object": "eval.run",
+ "per_model_usage": [
+ {
+ "cached_tokens": 0,
+ "completion_tokens": 0,
+ "invocation_count": 0,
+ "model_name": "model_name",
+ "prompt_tokens": 0,
+ "total_tokens": 0
+ }
+ ],
+ "per_testing_criteria_results": [
+ {
+ "failed": 0,
+ "passed": 0,
+ "testing_criteria": "testing_criteria"
+ }
+ ],
+ "report_url": "report_url",
+ "result_counts": {
+ "errored": 0,
+ "failed": 0,
+ "passed": 0,
+ "total": 0
+ },
+ "status": "status"
+}
+```
+
+### Example
+
+```http
+curl https://api.openai.com/v1/evals/eval_67abd54d9b0081909a86353f6fb9317a/runs/evalrun_67abd54d60ec8190832b46859da808f7/cancel \
+ -X POST \
+ -H "Authorization: Bearer $OPENAI_API_KEY" \
+ -H "Content-Type: application/json"
+```
+
+#### Response
+
+```json
+{
+ "object": "eval.run",
+ "id": "evalrun_67abd54d60ec8190832b46859da808f7",
+ "eval_id": "eval_67abd54d9b0081909a86353f6fb9317a",
+ "report_url": "https://platform.openai.com/evaluations/eval_67abd54d9b0081909a86353f6fb9317a?run_id=evalrun_67abd54d60ec8190832b46859da808f7",
+ "status": "canceled",
+ "model": "gpt-4o-mini",
+ "name": "gpt-4o-mini",
+ "created_at": 1743092069,
+ "result_counts": {
+ "total": 0,
+ "errored": 0,
+ "failed": 0,
+ "passed": 0
+ },
+ "per_model_usage": null,
+ "per_testing_criteria_results": null,
+ "data_source": {
+ "type": "completions",
+ "source": {
"type": "file_content",
"content": [
{
@@ -14845,770 +14624,9090 @@ curl https://api.openai.com/v1/evals/eval_67abd54d9b0081909a86353f6fb9317a/runs/
}
```
-## Delete eval run
+## Delete eval run
+
+**delete** `/evals/{eval_id}/runs/{run_id}`
+
+Delete an eval run.
+
+### Path Parameters
+
+- `eval_id: string`
+
+- `run_id: string`
+
+### Returns
+
+- `deleted: optional boolean`
+
+- `object: optional string`
+
+- `run_id: optional string`
+
+### Example
+
+```http
+curl https://api.openai.com/v1/evals/$EVAL_ID/runs/$RUN_ID \
+ -X DELETE \
+ -H "Authorization: Bearer $OPENAI_API_KEY"
+```
+
+#### Response
+
+```json
+{
+ "deleted": true,
+ "object": "eval.run.deleted",
+ "run_id": "evalrun_677469f564d48190807532a852da3afb"
+}
+```
+
+### Example
+
+```http
+curl https://api.openai.com/v1/evals/eval_123abc/runs/evalrun_abc456 \
+ -X DELETE \
+ -H "Authorization: Bearer $OPENAI_API_KEY" \
+ -H "Content-Type: application/json"
+```
+
+#### Response
+
+```json
+{
+ "object": "eval.run.deleted",
+ "deleted": true,
+ "run_id": "evalrun_abc456"
+}
+```
+
+## Domain Types
+
+### Create Eval Completions Run Data Source
+
+- `CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }`
+
+ A CompletionsRunDataSource object describing a model sampling configuration.
+
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
+
+ Determines what populates the `item` namespace in this run's data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more }`
+
+ A StoredCompletionsRunDataSource configuration describing a set of filters
+
+ - `type: "stored_completions"`
+
+ The type of source. Always `stored_completions`.
+
+ - `"stored_completions"`
+
+ - `created_after: optional number`
+
+ An optional Unix timestamp to filter items created after this time.
+
+ - `created_before: optional number`
+
+ An optional Unix timestamp to filter items created before this time.
+
+ - `limit: optional number`
+
+ An optional maximum number of items to return.
+
+ - `metadata: optional Metadata`
+
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
+
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
+
+ - `model: optional string`
+
+ An optional model to filter by (e.g., 'gpt-4o').
+
+ - `type: "completions"`
+
+ The type of run data source. Always `completions`.
+
+ - `"completions"`
+
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
+
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+
+ - `TemplateInputMessages object { template, type }`
+
+ - `template: array of EasyInputMessage or object { content, role, type }`
+
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+
+ - `EasyInputMessage object { content, role, phase, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputMessageContentList`
+
+ Text, image, or audio input to the model, used to generate a response.
+ Can also contain previous assistant responses.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputMessageContentList = array of ResponseInputContent`
+
+ A list of one or many input items to the model, containing different content
+ types.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `text: string`
+
+ The text input to the model.
+
+ - `type: "input_text"`
+
+ The type of the input item. Always `input_text`.
+
+ - `"input_text"`
+
+ - `ResponseInputImage object { detail, type, file_id, image_url }`
+
+ An image input to the model. Learn about [image inputs](/docs/guides/vision).
+
+ - `detail: "low" or "high" or "auto" or "original"`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
+
+ - `"low"`
+
+ - `"high"`
+
+ - `"auto"`
+
+ - `"original"`
+
+ - `type: "input_image"`
+
+ The type of the input item. Always `input_image`.
+
+ - `"input_image"`
+
+ - `file_id: optional string`
+
+ The ID of the file to be sent to the model.
+
+ - `image_url: optional string`
+
+ The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
+
+ - `ResponseInputFile object { type, detail, file_data, 3 more }`
+
+ A file input to the model.
+
+ - `type: "input_file"`
+
+ The type of the input item. Always `input_file`.
+
+ - `"input_file"`
+
+ - `detail: optional "low" or "high"`
+
+ The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
+
+ - `"low"`
+
+ - `"high"`
+
+ - `file_data: optional string`
+
+ The content of the file to be sent to the model.
+
+ - `file_id: optional string`
+
+ The ID of the file to be sent to the model.
+
+ - `file_url: optional string`
+
+ The URL of the file to be sent to the model.
+
+ - `filename: optional string`
+
+ The name of the file to be sent to the model.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `phase: optional "commentary" or "final_answer"`
+
+ Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
+ For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
+ phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
+
+ - `"commentary"`
+
+ - `"final_answer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `EvalMessageObject object { content, role, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
+
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `input_audio: object { data, format }`
+
+ - `data: string`
+
+ Base64-encoded audio data.
+
+ - `format: "mp3" or "wav"`
+
+ The format of the audio data. Currently supported formats are `mp3` and
+ `wav`.
+
+ - `"mp3"`
+
+ - `"wav"`
+
+ - `type: "input_audio"`
+
+ The type of the input item. Always `input_audio`.
+
+ - `"input_audio"`
+
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `type: "template"`
+
+ The type of input messages. Always `template`.
+
+ - `"template"`
+
+ - `ItemReferenceInputMessages object { item_reference, type }`
+
+ - `item_reference: string`
+
+ A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
+
+ - `type: "item_reference"`
+
+ The type of input messages. Always `item_reference`.
+
+ - `"item_reference"`
+
+ - `model: optional string`
+
+ The name of the model to use for generating completions (e.g. "o3-mini").
+
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
+
+ - `max_completion_tokens: optional number`
+
+ The maximum number of tokens in the generated output.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `"none"`
+
+ - `"minimal"`
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `"xhigh"`
+
+ - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
+
+ An object specifying the format that the model must output.
+
+ Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
+ Structured Outputs which ensures the model will match your supplied JSON
+ schema. Learn more in the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
+
+ - `ResponseFormatText object { type }`
+
+ Default response format. Used to generate text responses.
+
+ - `type: "text"`
+
+ The type of response format being defined. Always `text`.
+
+ - `"text"`
+
+ - `ResponseFormatJSONSchema object { json_schema, type }`
+
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+
+ - `json_schema: object { name, description, schema, strict }`
+
+ Structured Outputs configuration options, including a JSON Schema.
+
+ - `name: string`
+
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
+
+ - `description: optional string`
+
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
+
+ - `schema: optional map[unknown]`
+
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ - `type: "json_schema"`
+
+ The type of response format being defined. Always `json_schema`.
+
+ - `"json_schema"`
+
+ - `ResponseFormatJSONObject object { type }`
+
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
+
+ - `type: "json_object"`
+
+ The type of response format being defined. Always `json_object`.
+
+ - `"json_object"`
+
+ - `seed: optional number`
+
+ A seed value to initialize the randomness, during sampling.
+
+ - `temperature: optional number`
+
+ A higher temperature increases randomness in the outputs.
+
+ - `tools: optional array of ChatCompletionFunctionTool`
+
+ A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
+
+ - `function: FunctionDefinition`
+
+ - `name: string`
+
+ The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
+
+ - `description: optional string`
+
+ A description of what the function does, used by the model to choose when and how to call the function.
+
+ - `parameters: optional FunctionParameters`
+
+ The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
+
+ Omitting `parameters` defines a function with an empty parameter list.
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
+
+ - `type: "function"`
+
+ The type of the tool. Currently, only `function` is supported.
+
+ - `"function"`
+
+ - `top_p: optional number`
+
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+
+### Create Eval JSONL Run Data Source
+
+- `CreateEvalJSONLRunDataSource object { source, type }`
+
+ A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
+
+ - `source: object { content, type } or object { id, type }`
+
+ Determines what populates the `item` namespace in the data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `type: "jsonl"`
+
+ The type of data source. Always `jsonl`.
+
+ - `"jsonl"`
+
+### Eval API Error
+
+- `EvalAPIError object { code, message }`
+
+ An object representing an error response from the Eval API.
+
+ - `code: string`
+
+ The error code.
+
+ - `message: string`
+
+ The error message.
+
+### Run List Response
+
+- `RunListResponse object { id, created_at, data_source, 11 more }`
+
+ A schema representing an evaluation run.
+
+ - `id: string`
+
+ Unique identifier for the evaluation run.
+
+ - `created_at: number`
+
+ Unix timestamp (in seconds) when the evaluation run was created.
+
+ - `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
+
+ Information about the run's data source.
+
+ - `CreateEvalJSONLRunDataSource object { source, type }`
+
+ A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
+
+ - `source: object { content, type } or object { id, type }`
+
+ Determines what populates the `item` namespace in the data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `type: "jsonl"`
+
+ The type of data source. Always `jsonl`.
+
+ - `"jsonl"`
+
+ - `CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }`
+
+ A CompletionsRunDataSource object describing a model sampling configuration.
+
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
+
+ Determines what populates the `item` namespace in this run's data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more }`
+
+ A StoredCompletionsRunDataSource configuration describing a set of filters
+
+ - `type: "stored_completions"`
+
+ The type of source. Always `stored_completions`.
+
+ - `"stored_completions"`
+
+ - `created_after: optional number`
+
+ An optional Unix timestamp to filter items created after this time.
+
+ - `created_before: optional number`
+
+ An optional Unix timestamp to filter items created before this time.
+
+ - `limit: optional number`
+
+ An optional maximum number of items to return.
+
+ - `metadata: optional Metadata`
+
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
+
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
+
+ - `model: optional string`
+
+ An optional model to filter by (e.g., 'gpt-4o').
+
+ - `type: "completions"`
+
+ The type of run data source. Always `completions`.
+
+ - `"completions"`
+
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
+
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+
+ - `TemplateInputMessages object { template, type }`
+
+ - `template: array of EasyInputMessage or object { content, role, type }`
+
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+
+ - `EasyInputMessage object { content, role, phase, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputMessageContentList`
+
+ Text, image, or audio input to the model, used to generate a response.
+ Can also contain previous assistant responses.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputMessageContentList = array of ResponseInputContent`
+
+ A list of one or many input items to the model, containing different content
+ types.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `text: string`
+
+ The text input to the model.
+
+ - `type: "input_text"`
+
+ The type of the input item. Always `input_text`.
+
+ - `"input_text"`
+
+ - `ResponseInputImage object { detail, type, file_id, image_url }`
+
+ An image input to the model. Learn about [image inputs](/docs/guides/vision).
+
+ - `detail: "low" or "high" or "auto" or "original"`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
+
+ - `"low"`
+
+ - `"high"`
+
+ - `"auto"`
+
+ - `"original"`
+
+ - `type: "input_image"`
+
+ The type of the input item. Always `input_image`.
+
+ - `"input_image"`
+
+ - `file_id: optional string`
+
+ The ID of the file to be sent to the model.
+
+ - `image_url: optional string`
+
+ The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
+
+ - `ResponseInputFile object { type, detail, file_data, 3 more }`
+
+ A file input to the model.
+
+ - `type: "input_file"`
+
+ The type of the input item. Always `input_file`.
+
+ - `"input_file"`
+
+ - `detail: optional "low" or "high"`
+
+ The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
+
+ - `"low"`
+
+ - `"high"`
+
+ - `file_data: optional string`
+
+ The content of the file to be sent to the model.
+
+ - `file_id: optional string`
+
+ The ID of the file to be sent to the model.
+
+ - `file_url: optional string`
+
+ The URL of the file to be sent to the model.
+
+ - `filename: optional string`
+
+ The name of the file to be sent to the model.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `phase: optional "commentary" or "final_answer"`
+
+ Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
+ For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
+ phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
+
+ - `"commentary"`
+
+ - `"final_answer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `EvalMessageObject object { content, role, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
+
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `input_audio: object { data, format }`
+
+ - `data: string`
+
+ Base64-encoded audio data.
+
+ - `format: "mp3" or "wav"`
+
+ The format of the audio data. Currently supported formats are `mp3` and
+ `wav`.
+
+ - `"mp3"`
+
+ - `"wav"`
+
+ - `type: "input_audio"`
+
+ The type of the input item. Always `input_audio`.
+
+ - `"input_audio"`
+
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `type: "template"`
+
+ The type of input messages. Always `template`.
+
+ - `"template"`
+
+ - `ItemReferenceInputMessages object { item_reference, type }`
+
+ - `item_reference: string`
+
+ A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
+
+ - `type: "item_reference"`
+
+ The type of input messages. Always `item_reference`.
+
+ - `"item_reference"`
+
+ - `model: optional string`
+
+ The name of the model to use for generating completions (e.g. "o3-mini").
+
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
+
+ - `max_completion_tokens: optional number`
+
+ The maximum number of tokens in the generated output.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `"none"`
+
+ - `"minimal"`
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `"xhigh"`
+
+ - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
+
+ An object specifying the format that the model must output.
+
+ Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
+ Structured Outputs which ensures the model will match your supplied JSON
+ schema. Learn more in the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
+
+ - `ResponseFormatText object { type }`
+
+ Default response format. Used to generate text responses.
+
+ - `type: "text"`
+
+ The type of response format being defined. Always `text`.
+
+ - `"text"`
+
+ - `ResponseFormatJSONSchema object { json_schema, type }`
+
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+
+ - `json_schema: object { name, description, schema, strict }`
+
+ Structured Outputs configuration options, including a JSON Schema.
+
+ - `name: string`
+
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
+
+ - `description: optional string`
+
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
+
+ - `schema: optional map[unknown]`
+
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ - `type: "json_schema"`
+
+ The type of response format being defined. Always `json_schema`.
+
+ - `"json_schema"`
+
+ - `ResponseFormatJSONObject object { type }`
+
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
+
+ - `type: "json_object"`
+
+ The type of response format being defined. Always `json_object`.
+
+ - `"json_object"`
+
+ - `seed: optional number`
+
+ A seed value to initialize the randomness, during sampling.
+
+ - `temperature: optional number`
+
+ A higher temperature increases randomness in the outputs.
+
+ - `tools: optional array of ChatCompletionFunctionTool`
+
+ A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
+
+ - `function: FunctionDefinition`
+
+ - `name: string`
+
+ The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
+
+ - `description: optional string`
+
+ A description of what the function does, used by the model to choose when and how to call the function.
+
+ - `parameters: optional FunctionParameters`
+
+ The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
+
+ Omitting `parameters` defines a function with an empty parameter list.
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
+
+ - `type: "function"`
+
+ The type of the tool. Currently, only `function` is supported.
+
+ - `"function"`
+
+ - `top_p: optional number`
+
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+
+ - `ResponsesRunDataSource object { source, type, input_messages, 2 more }`
+
+ A ResponsesRunDataSource object describing a model sampling configuration.
+
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
+
+ Determines what populates the `item` namespace in this run's data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `EvalResponsesSource object { type, created_after, created_before, 8 more }`
+
+ A EvalResponsesSource object describing a run data source configuration.
+
+ - `type: "responses"`
+
+ The type of run data source. Always `responses`.
+
+ - `"responses"`
+
+ - `created_after: optional number`
+
+ Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
+
+ - `created_before: optional number`
+
+ Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
+
+ - `instructions_search: optional string`
+
+ Optional string to search the 'instructions' field. This is a query parameter used to select responses.
+
+ - `metadata: optional unknown`
+
+ Metadata filter for the responses. This is a query parameter used to select responses.
+
+ - `model: optional string`
+
+ The name of the model to find responses for. This is a query parameter used to select responses.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `temperature: optional number`
+
+ Sampling temperature. This is a query parameter used to select responses.
+
+ - `tools: optional array of string`
+
+ List of tool names. This is a query parameter used to select responses.
+
+ - `top_p: optional number`
+
+ Nucleus sampling parameter. This is a query parameter used to select responses.
+
+ - `users: optional array of string`
+
+ List of user identifiers. This is a query parameter used to select responses.
+
+ - `type: "responses"`
+
+ The type of run data source. Always `responses`.
+
+ - `"responses"`
+
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
+
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+
+ - `InputMessagesTemplate object { template, type }`
+
+ - `template: array of object { content, role } or object { content, role, type }`
+
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+
+ - `ChatMessage object { content, role }`
+
+ - `content: string`
+
+ The content of the message.
+
+ - `role: string`
+
+ The role of the message (e.g. "system", "assistant", "user").
+
+ - `EvalMessageObject object { content, role, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
+
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `type: "template"`
+
+ The type of input messages. Always `template`.
+
+ - `"template"`
+
+ - `InputMessagesItemReference object { item_reference, type }`
+
+ - `item_reference: string`
+
+ A reference to a variable in the `item` namespace. Ie, "item.name"
+
+ - `type: "item_reference"`
+
+ The type of input messages. Always `item_reference`.
+
+ - `"item_reference"`
+
+ - `model: optional string`
+
+ The name of the model to use for generating completions (e.g. "o3-mini").
+
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
+
+ - `max_completion_tokens: optional number`
+
+ The maximum number of tokens in the generated output.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `seed: optional number`
+
+ A seed value to initialize the randomness, during sampling.
+
+ - `temperature: optional number`
+
+ A higher temperature increases randomness in the outputs.
+
+ - `text: optional object { format }`
+
+ Configuration options for a text response from the model. Can be plain
+ text or structured JSON data. Learn more:
+
+ - [Text inputs and outputs](/docs/guides/text)
+ - [Structured Outputs](/docs/guides/structured-outputs)
+
+ - `format: optional ResponseFormatTextConfig`
+
+ An object specifying the format that the model must output.
+
+ Configuring `{ "type": "json_schema" }` enables Structured Outputs,
+ which ensures the model will match your supplied JSON schema. Learn more in the
+ [Structured Outputs guide](/docs/guides/structured-outputs).
+
+ The default format is `{ "type": "text" }` with no additional options.
+
+ **Not recommended for gpt-4o and newer models:**
+
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
+
+ - `ResponseFormatText object { type }`
+
+ Default response format. Used to generate text responses.
+
+ - `ResponseFormatTextJSONSchemaConfig object { name, schema, type, 2 more }`
+
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+
+ - `name: string`
+
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
+
+ - `schema: map[unknown]`
+
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
+
+ - `type: "json_schema"`
+
+ The type of response format being defined. Always `json_schema`.
+
+ - `"json_schema"`
+
+ - `description: optional string`
+
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ - `ResponseFormatJSONObject object { type }`
+
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
+
+ - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
+
+ An array of tools the model may call while generating a response. You
+ can specify which tool to use by setting the `tool_choice` parameter.
+
+ The two categories of tools you can provide the model are:
+
+ - **Built-in tools**: Tools that are provided by OpenAI that extend the
+ model's capabilities, like [web search](/docs/guides/tools-web-search)
+ or [file search](/docs/guides/tools-file-search). Learn more about
+ [built-in tools](/docs/guides/tools).
+ - **Function calls (custom tools)**: Functions that are defined by you,
+ enabling the model to call your own code. Learn more about
+ [function calling](/docs/guides/function-calling).
+
+ - `Function object { name, parameters, strict, 3 more }`
+
+ Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
+
+ - `name: string`
+
+ The name of the function to call.
+
+ - `parameters: map[unknown]`
+
+ A JSON schema object describing the parameters of the function.
+
+ - `strict: boolean`
+
+ Whether to enforce strict parameter validation. Default `true`.
+
+ - `type: "function"`
+
+ The type of the function tool. Always `function`.
+
+ - `"function"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this function is deferred and loaded via tool search.
+
+ - `description: optional string`
+
+ A description of the function. Used by the model to determine whether or not to call the function.
+
+ - `FileSearch object { type, vector_store_ids, filters, 2 more }`
+
+ A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
+
+ - `type: "file_search"`
+
+ The type of the file search tool. Always `file_search`.
+
+ - `"file_search"`
+
+ - `vector_store_ids: array of string`
+
+ The IDs of the vector stores to search.
+
+ - `filters: optional ComparisonFilter or CompoundFilter`
+
+ A filter to apply.
+
+ - `ComparisonFilter object { key, type, value }`
+
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+
+ - `key: string`
+
+ The key to compare against the value.
+
+ - `type: "eq" or "ne" or "gt" or 5 more`
+
+ Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+
+ - `eq`: equals
+ - `ne`: not equal
+ - `gt`: greater than
+ - `gte`: greater than or equal
+ - `lt`: less than
+ - `lte`: less than or equal
+ - `in`: in
+ - `nin`: not in
+
+ - `"eq"`
+
+ - `"ne"`
+
+ - `"gt"`
+
+ - `"gte"`
+
+ - `"lt"`
+
+ - `"lte"`
+
+ - `"in"`
+
+ - `"nin"`
+
+ - `value: string or number or boolean or array of string or number`
+
+ The value to compare against the attribute key; supports string, number, or boolean types.
+
+ - `string`
+
+ - `number`
+
+ - `boolean`
+
+ - `array of string or number`
+
+ - `string`
+
+ - `number`
+
+ - `CompoundFilter object { filters, type }`
+
+ Combine multiple filters using `and` or `or`.
+
+ - `filters: array of ComparisonFilter or unknown`
+
+ Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
+
+ - `ComparisonFilter object { key, type, value }`
+
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+
+ - `unknown`
+
+ - `type: "and" or "or"`
+
+ Type of operation: `and` or `or`.
+
+ - `"and"`
+
+ - `"or"`
+
+ - `max_num_results: optional number`
+
+ The maximum number of results to return. This number should be between 1 and 50 inclusive.
+
+ - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
+
+ Ranking options for search.
+
+ - `hybrid_search: optional object { embedding_weight, text_weight }`
+
+ Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
+
+ - `embedding_weight: number`
+
+ The weight of the embedding in the reciprocal ranking fusion.
+
+ - `text_weight: number`
+
+ The weight of the text in the reciprocal ranking fusion.
+
+ - `ranker: optional "auto" or "default-2024-11-15"`
+
+ The ranker to use for the file search.
+
+ - `"auto"`
+
+ - `"default-2024-11-15"`
+
+ - `score_threshold: optional number`
+
+ The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
+
+ - `Computer object { type }`
+
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+
+ - `type: "computer"`
+
+ The type of the computer tool. Always `computer`.
+
+ - `"computer"`
+
+ - `ComputerUsePreview object { display_height, display_width, environment, type }`
+
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+
+ - `display_height: number`
+
+ The height of the computer display.
+
+ - `display_width: number`
+
+ The width of the computer display.
+
+ - `environment: "windows" or "mac" or "linux" or 2 more`
+
+ The type of computer environment to control.
+
+ - `"windows"`
+
+ - `"mac"`
+
+ - `"linux"`
+
+ - `"ubuntu"`
+
+ - `"browser"`
+
+ - `type: "computer_use_preview"`
+
+ The type of the computer use tool. Always `computer_use_preview`.
+
+ - `"computer_use_preview"`
+
+ - `WebSearch object { type, filters, search_context_size, user_location }`
+
+ Search the Internet for sources related to the prompt. Learn more about the
+ [web search tool](/docs/guides/tools-web-search).
+
+ - `type: "web_search" or "web_search_2025_08_26"`
+
+ The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
+
+ - `"web_search"`
+
+ - `"web_search_2025_08_26"`
+
+ - `filters: optional object { allowed_domains }`
+
+ Filters for the search.
+
+ - `allowed_domains: optional array of string`
+
+ Allowed domains for the search. If not provided, all domains are allowed.
+ Subdomains of the provided domains are allowed as well.
+
+ Example: `["pubmed.ncbi.nlm.nih.gov"]`
+
+ - `search_context_size: optional "low" or "medium" or "high"`
+
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `user_location: optional object { city, country, region, 2 more }`
+
+ The approximate location of the user.
+
+ - `city: optional string`
+
+ Free text input for the city of the user, e.g. `San Francisco`.
+
+ - `country: optional string`
+
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+
+ - `region: optional string`
+
+ Free text input for the region of the user, e.g. `California`.
+
+ - `timezone: optional string`
+
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+
+ - `type: optional "approximate"`
+
+ The type of location approximation. Always `approximate`.
+
+ - `"approximate"`
+
+ - `Mcp object { server_label, type, allowed_tools, 7 more }`
+
+ Give the model access to additional tools via remote Model Context Protocol
+ (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
+
+ - `server_label: string`
+
+ A label for this MCP server, used to identify it in tool calls.
+
+ - `type: "mcp"`
+
+ The type of the MCP tool. Always `mcp`.
+
+ - `"mcp"`
+
+ - `allowed_tools: optional array of string or object { read_only, tool_names }`
+
+ List of allowed tool names or a filter object.
+
+ - `McpAllowedTools = array of string`
+
+ A string array of allowed tool names
+
+ - `McpToolFilter object { read_only, tool_names }`
+
+ A filter object to specify which tools are allowed.
+
+ - `read_only: optional boolean`
+
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
+
+ - `tool_names: optional array of string`
+
+ List of allowed tool names.
+
+ - `authorization: optional string`
+
+ An OAuth access token that can be used with a remote MCP server, either
+ with a custom MCP server URL or a service connector. Your application
+ must handle the OAuth authorization flow and provide the token here.
+
+ - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
+
+ Identifier for service connectors, like those available in ChatGPT. One of
+ `server_url` or `connector_id` must be provided. Learn more about service
+ connectors [here](/docs/guides/tools-remote-mcp#connectors).
+
+ Currently supported `connector_id` values are:
+
+ - Dropbox: `connector_dropbox`
+ - Gmail: `connector_gmail`
+ - Google Calendar: `connector_googlecalendar`
+ - Google Drive: `connector_googledrive`
+ - Microsoft Teams: `connector_microsoftteams`
+ - Outlook Calendar: `connector_outlookcalendar`
+ - Outlook Email: `connector_outlookemail`
+ - SharePoint: `connector_sharepoint`
+
+ - `"connector_dropbox"`
+
+ - `"connector_gmail"`
+
+ - `"connector_googlecalendar"`
+
+ - `"connector_googledrive"`
+
+ - `"connector_microsoftteams"`
+
+ - `"connector_outlookcalendar"`
+
+ - `"connector_outlookemail"`
+
+ - `"connector_sharepoint"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this MCP tool is deferred and discovered via tool search.
+
+ - `headers: optional map[string]`
+
+ Optional HTTP headers to send to the MCP server. Use for authentication
+ or other purposes.
+
+ - `require_approval: optional object { always, never } or "always" or "never"`
+
+ Specify which of the MCP server's tools require approval.
+
+ - `McpToolApprovalFilter object { always, never }`
+
+ Specify which of the MCP server's tools require approval. Can be
+ `always`, `never`, or a filter object associated with tools
+ that require approval.
+
+ - `always: optional object { read_only, tool_names }`
+
+ A filter object to specify which tools are allowed.
+
+ - `read_only: optional boolean`
+
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
+
+ - `tool_names: optional array of string`
+
+ List of allowed tool names.
+
+ - `never: optional object { read_only, tool_names }`
+
+ A filter object to specify which tools are allowed.
+
+ - `read_only: optional boolean`
+
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
+
+ - `tool_names: optional array of string`
+
+ List of allowed tool names.
+
+ - `McpToolApprovalSetting = "always" or "never"`
+
+ Specify a single approval policy for all tools. One of `always` or
+ `never`. When set to `always`, all tools will require approval. When
+ set to `never`, all tools will not require approval.
+
+ - `"always"`
+
+ - `"never"`
+
+ - `server_description: optional string`
+
+ Optional description of the MCP server, used to provide more context.
+
+ - `server_url: optional string`
+
+ The URL for the MCP server. One of `server_url` or `connector_id` must be
+ provided.
+
+ - `CodeInterpreter object { container, type }`
+
+ A tool that runs Python code to help generate a response to a prompt.
+
+ - `container: string or object { type, file_ids, memory_limit, network_policy }`
+
+ The code interpreter container. Can be a container ID or an object that
+ specifies uploaded file IDs to make available to your code, along with an
+ optional `memory_limit` setting.
+
+ - `string`
+
+ The container ID.
+
+ - `CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy }`
+
+ Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
+
+ - `type: "auto"`
+
+ Always `auto`.
+
+ - `"auto"`
+
+ - `file_ids: optional array of string`
+
+ An optional list of uploaded files to make available to your code.
+
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+
+ The memory limit for the code interpreter container.
+
+ - `"1g"`
+
+ - `"4g"`
+
+ - `"16g"`
+
+ - `"64g"`
+
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+
+ Network access policy for the container.
+
+ - `ContainerNetworkPolicyDisabled object { type }`
+
+ - `type: "disabled"`
+
+ Disable outbound network access. Always `disabled`.
+
+ - `"disabled"`
+
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
+
+ - `allowed_domains: array of string`
+
+ A list of allowed domains when type is `allowlist`.
+
+ - `type: "allowlist"`
+
+ Allow outbound network access only to specified domains. Always `allowlist`.
+
+ - `"allowlist"`
+
+ - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+
+ Optional domain-scoped secrets for allowlisted domains.
+
+ - `domain: string`
+
+ The domain associated with the secret.
+
+ - `name: string`
+
+ The name of the secret to inject for the domain.
+
+ - `value: string`
+
+ The secret value to inject for the domain.
+
+ - `type: "code_interpreter"`
+
+ The type of the code interpreter tool. Always `code_interpreter`.
+
+ - `"code_interpreter"`
+
+ - `ImageGeneration object { type, action, background, 9 more }`
+
+ A tool that generates images using the GPT image models.
+
+ - `type: "image_generation"`
+
+ The type of the image generation tool. Always `image_generation`.
+
+ - `"image_generation"`
+
+ - `action: optional "generate" or "edit" or "auto"`
+
+ Whether to generate a new image or edit an existing image. Default: `auto`.
+
+ - `"generate"`
+
+ - `"edit"`
+
+ - `"auto"`
+
+ - `background: optional "transparent" or "opaque" or "auto"`
+
+ Background type for the generated image. One of `transparent`,
+ `opaque`, or `auto`. Default: `auto`.
+
+ - `"transparent"`
+
+ - `"opaque"`
+
+ - `"auto"`
+
+ - `input_fidelity: optional "high" or "low"`
+
+ Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
+
+ - `"high"`
+
+ - `"low"`
+
+ - `input_image_mask: optional object { file_id, image_url }`
+
+ Optional mask for inpainting. Contains `image_url`
+ (string, optional) and `file_id` (string, optional).
+
+ - `file_id: optional string`
+
+ File ID for the mask image.
+
+ - `image_url: optional string`
+
+ Base64-encoded mask image.
+
+ - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+
+ The image generation model to use. Default: `gpt-image-1`.
+
+ - `string`
+
+ - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+
+ The image generation model to use. Default: `gpt-image-1`.
+
+ - `"gpt-image-1"`
+
+ - `"gpt-image-1-mini"`
+
+ - `"gpt-image-1.5"`
+
+ - `moderation: optional "auto" or "low"`
+
+ Moderation level for the generated image. Default: `auto`.
+
+ - `"auto"`
+
+ - `"low"`
+
+ - `output_compression: optional number`
+
+ Compression level for the output image. Default: 100.
+
+ - `output_format: optional "png" or "webp" or "jpeg"`
+
+ The output format of the generated image. One of `png`, `webp`, or
+ `jpeg`. Default: `png`.
+
+ - `"png"`
+
+ - `"webp"`
+
+ - `"jpeg"`
+
+ - `partial_images: optional number`
+
+ Number of partial images to generate in streaming mode, from 0 (default value) to 3.
+
+ - `quality: optional "low" or "medium" or "high" or "auto"`
+
+ The quality of the generated image. One of `low`, `medium`, `high`,
+ or `auto`. Default: `auto`.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `"auto"`
+
+ - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
+
+ The size of the generated image. One of `1024x1024`, `1024x1536`,
+ `1536x1024`, or `auto`. Default: `auto`.
+
+ - `"1024x1024"`
+
+ - `"1024x1536"`
+
+ - `"1536x1024"`
+
+ - `"auto"`
+
+ - `LocalShell object { type }`
+
+ A tool that allows the model to execute shell commands in a local environment.
+
+ - `type: "local_shell"`
+
+ The type of the local shell tool. Always `local_shell`.
+
+ - `"local_shell"`
+
+ - `Shell object { type, environment }`
+
+ A tool that allows the model to execute shell commands.
+
+ - `type: "shell"`
+
+ The type of the shell tool. Always `shell`.
+
+ - `"shell"`
+
+ - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
+
+ - `ContainerAuto object { type, file_ids, memory_limit, 2 more }`
+
+ - `type: "container_auto"`
+
+ Automatically creates a container for this request
+
+ - `"container_auto"`
+
+ - `file_ids: optional array of string`
+
+ An optional list of uploaded files to make available to your code.
+
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+
+ The memory limit for the container.
+
+ - `"1g"`
+
+ - `"4g"`
+
+ - `"16g"`
+
+ - `"64g"`
+
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+
+ Network access policy for the container.
+
+ - `ContainerNetworkPolicyDisabled object { type }`
+
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
+
+ - `skills: optional array of SkillReference or InlineSkill`
+
+ An optional list of skills referenced by id or inline data.
+
+ - `SkillReference object { skill_id, type, version }`
+
+ - `skill_id: string`
+
+ The ID of the referenced skill.
+
+ - `type: "skill_reference"`
+
+ References a skill created with the /v1/skills endpoint.
+
+ - `"skill_reference"`
+
+ - `version: optional string`
+
+ Optional skill version. Use a positive integer or 'latest'. Omit for default.
+
+ - `InlineSkill object { description, name, source, type }`
+
+ - `description: string`
+
+ The description of the skill.
+
+ - `name: string`
+
+ The name of the skill.
+
+ - `source: InlineSkillSource`
+
+ Inline skill payload
+
+ - `data: string`
+
+ Base64-encoded skill zip bundle.
+
+ - `media_type: "application/zip"`
+
+ The media type of the inline skill payload. Must be `application/zip`.
+
+ - `"application/zip"`
+
+ - `type: "base64"`
+
+ The type of the inline skill source. Must be `base64`.
+
+ - `"base64"`
+
+ - `type: "inline"`
+
+ Defines an inline skill for this request.
+
+ - `"inline"`
+
+ - `LocalEnvironment object { type, skills }`
+
+ - `type: "local"`
+
+ Use a local computer environment.
+
+ - `"local"`
+
+ - `skills: optional array of LocalSkill`
+
+ An optional list of skills.
+
+ - `description: string`
+
+ The description of the skill.
+
+ - `name: string`
+
+ The name of the skill.
+
+ - `path: string`
+
+ The path to the directory containing the skill.
+
+ - `ContainerReference object { container_id, type }`
+
+ - `container_id: string`
+
+ The ID of the referenced container.
+
+ - `type: "container_reference"`
+
+ References a container created with the /v1/containers endpoint
+
+ - `"container_reference"`
+
+ - `Custom object { name, type, defer_loading, 2 more }`
+
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+
+ - `name: string`
+
+ The name of the custom tool, used to identify it in tool calls.
+
+ - `type: "custom"`
+
+ The type of the custom tool. Always `custom`.
+
+ - `"custom"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this tool should be deferred and discovered via tool search.
+
+ - `description: optional string`
+
+ Optional description of the custom tool, used to provide more context.
+
+ - `format: optional CustomToolInputFormat`
+
+ The input format for the custom tool. Default is unconstrained text.
+
+ - `Text object { type }`
+
+ Unconstrained free-form text.
+
+ - `type: "text"`
+
+ Unconstrained text format. Always `text`.
+
+ - `"text"`
+
+ - `Grammar object { definition, syntax, type }`
+
+ A grammar defined by the user.
+
+ - `definition: string`
+
+ The grammar definition.
+
+ - `syntax: "lark" or "regex"`
+
+ The syntax of the grammar definition. One of `lark` or `regex`.
+
+ - `"lark"`
+
+ - `"regex"`
+
+ - `type: "grammar"`
+
+ Grammar format. Always `grammar`.
+
+ - `"grammar"`
+
+ - `Namespace object { description, name, tools, type }`
+
+ Groups function/custom tools under a shared namespace.
+
+ - `description: string`
+
+ A description of the namespace shown to the model.
+
+ - `name: string`
+
+ The namespace name used in tool calls (for example, `crm`).
+
+ - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
+
+ The function/custom tools available inside this namespace.
+
+ - `Function object { name, type, defer_loading, 3 more }`
+
+ - `name: string`
+
+ - `type: "function"`
+
+ - `"function"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this function should be deferred and discovered via tool search.
+
+ - `description: optional string`
+
+ - `parameters: optional unknown`
+
+ - `strict: optional boolean`
+
+ - `Custom object { name, type, defer_loading, 2 more }`
+
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+
+ - `name: string`
+
+ The name of the custom tool, used to identify it in tool calls.
+
+ - `type: "custom"`
+
+ The type of the custom tool. Always `custom`.
+
+ - `"custom"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this tool should be deferred and discovered via tool search.
+
+ - `description: optional string`
+
+ Optional description of the custom tool, used to provide more context.
+
+ - `format: optional CustomToolInputFormat`
+
+ The input format for the custom tool. Default is unconstrained text.
+
+ - `type: "namespace"`
+
+ The type of the tool. Always `namespace`.
+
+ - `"namespace"`
+
+ - `ToolSearch object { type, description, execution, parameters }`
+
+ Hosted or BYOT tool search configuration for deferred tools.
+
+ - `type: "tool_search"`
+
+ The type of the tool. Always `tool_search`.
+
+ - `"tool_search"`
+
+ - `description: optional string`
+
+ Description shown to the model for a client-executed tool search tool.
+
+ - `execution: optional "server" or "client"`
+
+ Whether tool search is executed by the server or by the client.
+
+ - `"server"`
+
+ - `"client"`
+
+ - `parameters: optional unknown`
+
+ Parameter schema for a client-executed tool search tool.
+
+ - `WebSearchPreview object { type, search_content_types, search_context_size, user_location }`
+
+ This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
+
+ - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
+
+ The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
+
+ - `"web_search_preview"`
+
+ - `"web_search_preview_2025_03_11"`
+
+ - `search_content_types: optional array of "text" or "image"`
+
+ - `"text"`
+
+ - `"image"`
+
+ - `search_context_size: optional "low" or "medium" or "high"`
+
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `user_location: optional object { type, city, country, 2 more }`
+
+ The user's location.
+
+ - `type: "approximate"`
+
+ The type of location approximation. Always `approximate`.
+
+ - `"approximate"`
+
+ - `city: optional string`
+
+ Free text input for the city of the user, e.g. `San Francisco`.
+
+ - `country: optional string`
+
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+
+ - `region: optional string`
+
+ Free text input for the region of the user, e.g. `California`.
+
+ - `timezone: optional string`
+
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+
+ - `ApplyPatch object { type }`
+
+ Allows the assistant to create, delete, or update files using unified diffs.
+
+ - `type: "apply_patch"`
+
+ The type of the tool. Always `apply_patch`.
+
+ - `"apply_patch"`
+
+ - `top_p: optional number`
+
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+
+ - `error: EvalAPIError`
+
+ An object representing an error response from the Eval API.
+
+ - `code: string`
+
+ The error code.
+
+ - `message: string`
+
+ The error message.
+
+ - `eval_id: string`
+
+ The identifier of the associated evaluation.
+
+ - `metadata: Metadata`
+
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
+
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
+
+ - `model: string`
+
+ The model that is evaluated, if applicable.
+
+ - `name: string`
+
+ The name of the evaluation run.
+
+ - `object: "eval.run"`
+
+ The type of the object. Always "eval.run".
+
+ - `"eval.run"`
+
+ - `per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }`
+
+ Usage statistics for each model during the evaluation run.
+
+ - `cached_tokens: number`
+
+ The number of tokens retrieved from cache.
+
+ - `completion_tokens: number`
+
+ The number of completion tokens generated.
+
+ - `invocation_count: number`
+
+ The number of invocations.
+
+ - `model_name: string`
+
+ The name of the model.
+
+ - `prompt_tokens: number`
+
+ The number of prompt tokens used.
+
+ - `total_tokens: number`
+
+ The total number of tokens used.
+
+ - `per_testing_criteria_results: array of object { failed, passed, testing_criteria }`
+
+ Results per testing criteria applied during the evaluation run.
+
+ - `failed: number`
+
+ Number of tests failed for this criteria.
+
+ - `passed: number`
+
+ Number of tests passed for this criteria.
+
+ - `testing_criteria: string`
+
+ A description of the testing criteria.
+
+ - `report_url: string`
+
+ The URL to the rendered evaluation run report on the UI dashboard.
+
+ - `result_counts: object { errored, failed, passed, total }`
+
+ Counters summarizing the outcomes of the evaluation run.
+
+ - `errored: number`
+
+ Number of output items that resulted in an error.
+
+ - `failed: number`
+
+ Number of output items that failed to pass the evaluation.
+
+ - `passed: number`
+
+ Number of output items that passed the evaluation.
+
+ - `total: number`
+
+ Total number of executed output items.
+
+ - `status: string`
+
+ The status of the evaluation run.
+
+### Run Create Response
+
+- `RunCreateResponse object { id, created_at, data_source, 11 more }`
+
+ A schema representing an evaluation run.
+
+ - `id: string`
+
+ Unique identifier for the evaluation run.
+
+ - `created_at: number`
+
+ Unix timestamp (in seconds) when the evaluation run was created.
+
+ - `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
+
+ Information about the run's data source.
+
+ - `CreateEvalJSONLRunDataSource object { source, type }`
+
+ A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
+
+ - `source: object { content, type } or object { id, type }`
+
+ Determines what populates the `item` namespace in the data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `type: "jsonl"`
+
+ The type of data source. Always `jsonl`.
+
+ - `"jsonl"`
+
+ - `CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }`
+
+ A CompletionsRunDataSource object describing a model sampling configuration.
+
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
+
+ Determines what populates the `item` namespace in this run's data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more }`
+
+ A StoredCompletionsRunDataSource configuration describing a set of filters
+
+ - `type: "stored_completions"`
+
+ The type of source. Always `stored_completions`.
+
+ - `"stored_completions"`
+
+ - `created_after: optional number`
+
+ An optional Unix timestamp to filter items created after this time.
+
+ - `created_before: optional number`
+
+ An optional Unix timestamp to filter items created before this time.
+
+ - `limit: optional number`
+
+ An optional maximum number of items to return.
+
+ - `metadata: optional Metadata`
+
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
+
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
+
+ - `model: optional string`
+
+ An optional model to filter by (e.g., 'gpt-4o').
+
+ - `type: "completions"`
+
+ The type of run data source. Always `completions`.
+
+ - `"completions"`
+
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
+
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+
+ - `TemplateInputMessages object { template, type }`
+
+ - `template: array of EasyInputMessage or object { content, role, type }`
+
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+
+ - `EasyInputMessage object { content, role, phase, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputMessageContentList`
+
+ Text, image, or audio input to the model, used to generate a response.
+ Can also contain previous assistant responses.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputMessageContentList = array of ResponseInputContent`
+
+ A list of one or many input items to the model, containing different content
+ types.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `text: string`
+
+ The text input to the model.
+
+ - `type: "input_text"`
+
+ The type of the input item. Always `input_text`.
+
+ - `"input_text"`
+
+ - `ResponseInputImage object { detail, type, file_id, image_url }`
+
+ An image input to the model. Learn about [image inputs](/docs/guides/vision).
+
+ - `detail: "low" or "high" or "auto" or "original"`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
+
+ - `"low"`
+
+ - `"high"`
+
+ - `"auto"`
+
+ - `"original"`
+
+ - `type: "input_image"`
+
+ The type of the input item. Always `input_image`.
+
+ - `"input_image"`
+
+ - `file_id: optional string`
+
+ The ID of the file to be sent to the model.
+
+ - `image_url: optional string`
+
+ The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
+
+ - `ResponseInputFile object { type, detail, file_data, 3 more }`
+
+ A file input to the model.
+
+ - `type: "input_file"`
+
+ The type of the input item. Always `input_file`.
+
+ - `"input_file"`
+
+ - `detail: optional "low" or "high"`
+
+ The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
+
+ - `"low"`
+
+ - `"high"`
+
+ - `file_data: optional string`
+
+ The content of the file to be sent to the model.
+
+ - `file_id: optional string`
+
+ The ID of the file to be sent to the model.
+
+ - `file_url: optional string`
+
+ The URL of the file to be sent to the model.
+
+ - `filename: optional string`
+
+ The name of the file to be sent to the model.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `phase: optional "commentary" or "final_answer"`
+
+ Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
+ For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
+ phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
+
+ - `"commentary"`
+
+ - `"final_answer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `EvalMessageObject object { content, role, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
+
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `input_audio: object { data, format }`
+
+ - `data: string`
+
+ Base64-encoded audio data.
+
+ - `format: "mp3" or "wav"`
+
+ The format of the audio data. Currently supported formats are `mp3` and
+ `wav`.
+
+ - `"mp3"`
+
+ - `"wav"`
+
+ - `type: "input_audio"`
+
+ The type of the input item. Always `input_audio`.
+
+ - `"input_audio"`
+
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `type: "template"`
+
+ The type of input messages. Always `template`.
+
+ - `"template"`
+
+ - `ItemReferenceInputMessages object { item_reference, type }`
+
+ - `item_reference: string`
+
+ A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
+
+ - `type: "item_reference"`
+
+ The type of input messages. Always `item_reference`.
+
+ - `"item_reference"`
+
+ - `model: optional string`
+
+ The name of the model to use for generating completions (e.g. "o3-mini").
+
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
+
+ - `max_completion_tokens: optional number`
+
+ The maximum number of tokens in the generated output.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `"none"`
+
+ - `"minimal"`
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `"xhigh"`
+
+ - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
+
+ An object specifying the format that the model must output.
+
+ Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
+ Structured Outputs which ensures the model will match your supplied JSON
+ schema. Learn more in the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
+
+ - `ResponseFormatText object { type }`
+
+ Default response format. Used to generate text responses.
+
+ - `type: "text"`
+
+ The type of response format being defined. Always `text`.
+
+ - `"text"`
+
+ - `ResponseFormatJSONSchema object { json_schema, type }`
+
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+
+ - `json_schema: object { name, description, schema, strict }`
+
+ Structured Outputs configuration options, including a JSON Schema.
+
+ - `name: string`
+
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
+
+ - `description: optional string`
+
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
+
+ - `schema: optional map[unknown]`
+
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ - `type: "json_schema"`
+
+ The type of response format being defined. Always `json_schema`.
+
+ - `"json_schema"`
+
+ - `ResponseFormatJSONObject object { type }`
+
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
+
+ - `type: "json_object"`
+
+ The type of response format being defined. Always `json_object`.
+
+ - `"json_object"`
+
+ - `seed: optional number`
+
+ A seed value to initialize the randomness, during sampling.
+
+ - `temperature: optional number`
+
+ A higher temperature increases randomness in the outputs.
+
+ - `tools: optional array of ChatCompletionFunctionTool`
+
+ A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
+
+ - `function: FunctionDefinition`
+
+ - `name: string`
+
+ The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
+
+ - `description: optional string`
+
+ A description of what the function does, used by the model to choose when and how to call the function.
+
+ - `parameters: optional FunctionParameters`
+
+ The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
+
+ Omitting `parameters` defines a function with an empty parameter list.
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
+
+ - `type: "function"`
+
+ The type of the tool. Currently, only `function` is supported.
+
+ - `"function"`
+
+ - `top_p: optional number`
+
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+
+ - `ResponsesRunDataSource object { source, type, input_messages, 2 more }`
+
+ A ResponsesRunDataSource object describing a model sampling configuration.
+
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
+
+ Determines what populates the `item` namespace in this run's data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `EvalResponsesSource object { type, created_after, created_before, 8 more }`
+
+ A EvalResponsesSource object describing a run data source configuration.
+
+ - `type: "responses"`
+
+ The type of run data source. Always `responses`.
+
+ - `"responses"`
+
+ - `created_after: optional number`
+
+ Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
+
+ - `created_before: optional number`
+
+ Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
+
+ - `instructions_search: optional string`
+
+ Optional string to search the 'instructions' field. This is a query parameter used to select responses.
+
+ - `metadata: optional unknown`
+
+ Metadata filter for the responses. This is a query parameter used to select responses.
+
+ - `model: optional string`
+
+ The name of the model to find responses for. This is a query parameter used to select responses.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `temperature: optional number`
+
+ Sampling temperature. This is a query parameter used to select responses.
+
+ - `tools: optional array of string`
+
+ List of tool names. This is a query parameter used to select responses.
+
+ - `top_p: optional number`
+
+ Nucleus sampling parameter. This is a query parameter used to select responses.
+
+ - `users: optional array of string`
+
+ List of user identifiers. This is a query parameter used to select responses.
+
+ - `type: "responses"`
+
+ The type of run data source. Always `responses`.
+
+ - `"responses"`
+
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
+
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+
+ - `InputMessagesTemplate object { template, type }`
+
+ - `template: array of object { content, role } or object { content, role, type }`
+
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+
+ - `ChatMessage object { content, role }`
+
+ - `content: string`
+
+ The content of the message.
+
+ - `role: string`
+
+ The role of the message (e.g. "system", "assistant", "user").
+
+ - `EvalMessageObject object { content, role, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
+
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `type: "template"`
+
+ The type of input messages. Always `template`.
+
+ - `"template"`
+
+ - `InputMessagesItemReference object { item_reference, type }`
+
+ - `item_reference: string`
+
+ A reference to a variable in the `item` namespace. Ie, "item.name"
+
+ - `type: "item_reference"`
+
+ The type of input messages. Always `item_reference`.
+
+ - `"item_reference"`
+
+ - `model: optional string`
+
+ The name of the model to use for generating completions (e.g. "o3-mini").
+
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
+
+ - `max_completion_tokens: optional number`
+
+ The maximum number of tokens in the generated output.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `seed: optional number`
+
+ A seed value to initialize the randomness, during sampling.
+
+ - `temperature: optional number`
+
+ A higher temperature increases randomness in the outputs.
+
+ - `text: optional object { format }`
+
+ Configuration options for a text response from the model. Can be plain
+ text or structured JSON data. Learn more:
+
+ - [Text inputs and outputs](/docs/guides/text)
+ - [Structured Outputs](/docs/guides/structured-outputs)
+
+ - `format: optional ResponseFormatTextConfig`
+
+ An object specifying the format that the model must output.
+
+ Configuring `{ "type": "json_schema" }` enables Structured Outputs,
+ which ensures the model will match your supplied JSON schema. Learn more in the
+ [Structured Outputs guide](/docs/guides/structured-outputs).
+
+ The default format is `{ "type": "text" }` with no additional options.
+
+ **Not recommended for gpt-4o and newer models:**
+
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
+
+ - `ResponseFormatText object { type }`
+
+ Default response format. Used to generate text responses.
+
+ - `ResponseFormatTextJSONSchemaConfig object { name, schema, type, 2 more }`
+
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+
+ - `name: string`
+
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
+
+ - `schema: map[unknown]`
+
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
+
+ - `type: "json_schema"`
+
+ The type of response format being defined. Always `json_schema`.
+
+ - `"json_schema"`
+
+ - `description: optional string`
+
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ - `ResponseFormatJSONObject object { type }`
+
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
+
+ - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
+
+ An array of tools the model may call while generating a response. You
+ can specify which tool to use by setting the `tool_choice` parameter.
+
+ The two categories of tools you can provide the model are:
+
+ - **Built-in tools**: Tools that are provided by OpenAI that extend the
+ model's capabilities, like [web search](/docs/guides/tools-web-search)
+ or [file search](/docs/guides/tools-file-search). Learn more about
+ [built-in tools](/docs/guides/tools).
+ - **Function calls (custom tools)**: Functions that are defined by you,
+ enabling the model to call your own code. Learn more about
+ [function calling](/docs/guides/function-calling).
+
+ - `Function object { name, parameters, strict, 3 more }`
+
+ Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
+
+ - `name: string`
+
+ The name of the function to call.
+
+ - `parameters: map[unknown]`
+
+ A JSON schema object describing the parameters of the function.
+
+ - `strict: boolean`
+
+ Whether to enforce strict parameter validation. Default `true`.
+
+ - `type: "function"`
+
+ The type of the function tool. Always `function`.
+
+ - `"function"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this function is deferred and loaded via tool search.
+
+ - `description: optional string`
+
+ A description of the function. Used by the model to determine whether or not to call the function.
+
+ - `FileSearch object { type, vector_store_ids, filters, 2 more }`
+
+ A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
+
+ - `type: "file_search"`
+
+ The type of the file search tool. Always `file_search`.
+
+ - `"file_search"`
+
+ - `vector_store_ids: array of string`
+
+ The IDs of the vector stores to search.
+
+ - `filters: optional ComparisonFilter or CompoundFilter`
+
+ A filter to apply.
+
+ - `ComparisonFilter object { key, type, value }`
+
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+
+ - `key: string`
+
+ The key to compare against the value.
+
+ - `type: "eq" or "ne" or "gt" or 5 more`
+
+ Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+
+ - `eq`: equals
+ - `ne`: not equal
+ - `gt`: greater than
+ - `gte`: greater than or equal
+ - `lt`: less than
+ - `lte`: less than or equal
+ - `in`: in
+ - `nin`: not in
+
+ - `"eq"`
+
+ - `"ne"`
+
+ - `"gt"`
+
+ - `"gte"`
+
+ - `"lt"`
+
+ - `"lte"`
+
+ - `"in"`
+
+ - `"nin"`
+
+ - `value: string or number or boolean or array of string or number`
+
+ The value to compare against the attribute key; supports string, number, or boolean types.
+
+ - `string`
+
+ - `number`
+
+ - `boolean`
+
+ - `array of string or number`
+
+ - `string`
+
+ - `number`
+
+ - `CompoundFilter object { filters, type }`
+
+ Combine multiple filters using `and` or `or`.
+
+ - `filters: array of ComparisonFilter or unknown`
+
+ Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
+
+ - `ComparisonFilter object { key, type, value }`
+
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+
+ - `unknown`
+
+ - `type: "and" or "or"`
+
+ Type of operation: `and` or `or`.
+
+ - `"and"`
+
+ - `"or"`
+
+ - `max_num_results: optional number`
+
+ The maximum number of results to return. This number should be between 1 and 50 inclusive.
+
+ - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
+
+ Ranking options for search.
+
+ - `hybrid_search: optional object { embedding_weight, text_weight }`
+
+ Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
+
+ - `embedding_weight: number`
+
+ The weight of the embedding in the reciprocal ranking fusion.
+
+ - `text_weight: number`
+
+ The weight of the text in the reciprocal ranking fusion.
+
+ - `ranker: optional "auto" or "default-2024-11-15"`
+
+ The ranker to use for the file search.
+
+ - `"auto"`
+
+ - `"default-2024-11-15"`
+
+ - `score_threshold: optional number`
+
+ The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
+
+ - `Computer object { type }`
+
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+
+ - `type: "computer"`
+
+ The type of the computer tool. Always `computer`.
+
+ - `"computer"`
+
+ - `ComputerUsePreview object { display_height, display_width, environment, type }`
+
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+
+ - `display_height: number`
+
+ The height of the computer display.
+
+ - `display_width: number`
+
+ The width of the computer display.
+
+ - `environment: "windows" or "mac" or "linux" or 2 more`
+
+ The type of computer environment to control.
+
+ - `"windows"`
+
+ - `"mac"`
+
+ - `"linux"`
+
+ - `"ubuntu"`
+
+ - `"browser"`
+
+ - `type: "computer_use_preview"`
+
+ The type of the computer use tool. Always `computer_use_preview`.
+
+ - `"computer_use_preview"`
+
+ - `WebSearch object { type, filters, search_context_size, user_location }`
+
+ Search the Internet for sources related to the prompt. Learn more about the
+ [web search tool](/docs/guides/tools-web-search).
+
+ - `type: "web_search" or "web_search_2025_08_26"`
+
+ The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
+
+ - `"web_search"`
+
+ - `"web_search_2025_08_26"`
+
+ - `filters: optional object { allowed_domains }`
+
+ Filters for the search.
+
+ - `allowed_domains: optional array of string`
+
+ Allowed domains for the search. If not provided, all domains are allowed.
+ Subdomains of the provided domains are allowed as well.
+
+ Example: `["pubmed.ncbi.nlm.nih.gov"]`
+
+ - `search_context_size: optional "low" or "medium" or "high"`
+
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `user_location: optional object { city, country, region, 2 more }`
+
+ The approximate location of the user.
+
+ - `city: optional string`
+
+ Free text input for the city of the user, e.g. `San Francisco`.
+
+ - `country: optional string`
+
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+
+ - `region: optional string`
+
+ Free text input for the region of the user, e.g. `California`.
+
+ - `timezone: optional string`
+
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+
+ - `type: optional "approximate"`
+
+ The type of location approximation. Always `approximate`.
+
+ - `"approximate"`
+
+ - `Mcp object { server_label, type, allowed_tools, 7 more }`
+
+ Give the model access to additional tools via remote Model Context Protocol
+ (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
+
+ - `server_label: string`
+
+ A label for this MCP server, used to identify it in tool calls.
+
+ - `type: "mcp"`
+
+ The type of the MCP tool. Always `mcp`.
+
+ - `"mcp"`
+
+ - `allowed_tools: optional array of string or object { read_only, tool_names }`
+
+ List of allowed tool names or a filter object.
+
+ - `McpAllowedTools = array of string`
+
+ A string array of allowed tool names
+
+ - `McpToolFilter object { read_only, tool_names }`
+
+ A filter object to specify which tools are allowed.
+
+ - `read_only: optional boolean`
+
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
+
+ - `tool_names: optional array of string`
+
+ List of allowed tool names.
+
+ - `authorization: optional string`
+
+ An OAuth access token that can be used with a remote MCP server, either
+ with a custom MCP server URL or a service connector. Your application
+ must handle the OAuth authorization flow and provide the token here.
+
+ - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
+
+ Identifier for service connectors, like those available in ChatGPT. One of
+ `server_url` or `connector_id` must be provided. Learn more about service
+ connectors [here](/docs/guides/tools-remote-mcp#connectors).
+
+ Currently supported `connector_id` values are:
+
+ - Dropbox: `connector_dropbox`
+ - Gmail: `connector_gmail`
+ - Google Calendar: `connector_googlecalendar`
+ - Google Drive: `connector_googledrive`
+ - Microsoft Teams: `connector_microsoftteams`
+ - Outlook Calendar: `connector_outlookcalendar`
+ - Outlook Email: `connector_outlookemail`
+ - SharePoint: `connector_sharepoint`
+
+ - `"connector_dropbox"`
+
+ - `"connector_gmail"`
+
+ - `"connector_googlecalendar"`
+
+ - `"connector_googledrive"`
+
+ - `"connector_microsoftteams"`
+
+ - `"connector_outlookcalendar"`
+
+ - `"connector_outlookemail"`
+
+ - `"connector_sharepoint"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this MCP tool is deferred and discovered via tool search.
+
+ - `headers: optional map[string]`
+
+ Optional HTTP headers to send to the MCP server. Use for authentication
+ or other purposes.
+
+ - `require_approval: optional object { always, never } or "always" or "never"`
+
+ Specify which of the MCP server's tools require approval.
+
+ - `McpToolApprovalFilter object { always, never }`
+
+ Specify which of the MCP server's tools require approval. Can be
+ `always`, `never`, or a filter object associated with tools
+ that require approval.
+
+ - `always: optional object { read_only, tool_names }`
+
+ A filter object to specify which tools are allowed.
+
+ - `read_only: optional boolean`
+
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
+
+ - `tool_names: optional array of string`
+
+ List of allowed tool names.
+
+ - `never: optional object { read_only, tool_names }`
+
+ A filter object to specify which tools are allowed.
+
+ - `read_only: optional boolean`
+
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
+
+ - `tool_names: optional array of string`
+
+ List of allowed tool names.
+
+ - `McpToolApprovalSetting = "always" or "never"`
+
+ Specify a single approval policy for all tools. One of `always` or
+ `never`. When set to `always`, all tools will require approval. When
+ set to `never`, all tools will not require approval.
+
+ - `"always"`
+
+ - `"never"`
+
+ - `server_description: optional string`
+
+ Optional description of the MCP server, used to provide more context.
+
+ - `server_url: optional string`
+
+ The URL for the MCP server. One of `server_url` or `connector_id` must be
+ provided.
+
+ - `CodeInterpreter object { container, type }`
+
+ A tool that runs Python code to help generate a response to a prompt.
+
+ - `container: string or object { type, file_ids, memory_limit, network_policy }`
+
+ The code interpreter container. Can be a container ID or an object that
+ specifies uploaded file IDs to make available to your code, along with an
+ optional `memory_limit` setting.
+
+ - `string`
+
+ The container ID.
+
+ - `CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy }`
+
+ Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
+
+ - `type: "auto"`
+
+ Always `auto`.
+
+ - `"auto"`
+
+ - `file_ids: optional array of string`
+
+ An optional list of uploaded files to make available to your code.
+
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+
+ The memory limit for the code interpreter container.
+
+ - `"1g"`
+
+ - `"4g"`
+
+ - `"16g"`
+
+ - `"64g"`
+
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+
+ Network access policy for the container.
+
+ - `ContainerNetworkPolicyDisabled object { type }`
+
+ - `type: "disabled"`
+
+ Disable outbound network access. Always `disabled`.
+
+ - `"disabled"`
+
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
+
+ - `allowed_domains: array of string`
+
+ A list of allowed domains when type is `allowlist`.
+
+ - `type: "allowlist"`
+
+ Allow outbound network access only to specified domains. Always `allowlist`.
+
+ - `"allowlist"`
+
+ - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+
+ Optional domain-scoped secrets for allowlisted domains.
+
+ - `domain: string`
+
+ The domain associated with the secret.
+
+ - `name: string`
+
+ The name of the secret to inject for the domain.
+
+ - `value: string`
+
+ The secret value to inject for the domain.
+
+ - `type: "code_interpreter"`
+
+ The type of the code interpreter tool. Always `code_interpreter`.
+
+ - `"code_interpreter"`
+
+ - `ImageGeneration object { type, action, background, 9 more }`
+
+ A tool that generates images using the GPT image models.
+
+ - `type: "image_generation"`
+
+ The type of the image generation tool. Always `image_generation`.
+
+ - `"image_generation"`
+
+ - `action: optional "generate" or "edit" or "auto"`
+
+ Whether to generate a new image or edit an existing image. Default: `auto`.
+
+ - `"generate"`
+
+ - `"edit"`
+
+ - `"auto"`
+
+ - `background: optional "transparent" or "opaque" or "auto"`
+
+ Background type for the generated image. One of `transparent`,
+ `opaque`, or `auto`. Default: `auto`.
+
+ - `"transparent"`
+
+ - `"opaque"`
+
+ - `"auto"`
+
+ - `input_fidelity: optional "high" or "low"`
+
+ Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
+
+ - `"high"`
+
+ - `"low"`
+
+ - `input_image_mask: optional object { file_id, image_url }`
+
+ Optional mask for inpainting. Contains `image_url`
+ (string, optional) and `file_id` (string, optional).
+
+ - `file_id: optional string`
+
+ File ID for the mask image.
+
+ - `image_url: optional string`
+
+ Base64-encoded mask image.
+
+ - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+
+ The image generation model to use. Default: `gpt-image-1`.
+
+ - `string`
+
+ - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+
+ The image generation model to use. Default: `gpt-image-1`.
+
+ - `"gpt-image-1"`
+
+ - `"gpt-image-1-mini"`
+
+ - `"gpt-image-1.5"`
+
+ - `moderation: optional "auto" or "low"`
+
+ Moderation level for the generated image. Default: `auto`.
+
+ - `"auto"`
+
+ - `"low"`
+
+ - `output_compression: optional number`
+
+ Compression level for the output image. Default: 100.
+
+ - `output_format: optional "png" or "webp" or "jpeg"`
+
+ The output format of the generated image. One of `png`, `webp`, or
+ `jpeg`. Default: `png`.
+
+ - `"png"`
+
+ - `"webp"`
+
+ - `"jpeg"`
+
+ - `partial_images: optional number`
+
+ Number of partial images to generate in streaming mode, from 0 (default value) to 3.
+
+ - `quality: optional "low" or "medium" or "high" or "auto"`
+
+ The quality of the generated image. One of `low`, `medium`, `high`,
+ or `auto`. Default: `auto`.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `"auto"`
+
+ - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
+
+ The size of the generated image. One of `1024x1024`, `1024x1536`,
+ `1536x1024`, or `auto`. Default: `auto`.
+
+ - `"1024x1024"`
+
+ - `"1024x1536"`
+
+ - `"1536x1024"`
+
+ - `"auto"`
+
+ - `LocalShell object { type }`
+
+ A tool that allows the model to execute shell commands in a local environment.
+
+ - `type: "local_shell"`
+
+ The type of the local shell tool. Always `local_shell`.
+
+ - `"local_shell"`
+
+ - `Shell object { type, environment }`
+
+ A tool that allows the model to execute shell commands.
+
+ - `type: "shell"`
+
+ The type of the shell tool. Always `shell`.
+
+ - `"shell"`
+
+ - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
+
+ - `ContainerAuto object { type, file_ids, memory_limit, 2 more }`
+
+ - `type: "container_auto"`
+
+ Automatically creates a container for this request
+
+ - `"container_auto"`
+
+ - `file_ids: optional array of string`
+
+ An optional list of uploaded files to make available to your code.
+
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+
+ The memory limit for the container.
+
+ - `"1g"`
+
+ - `"4g"`
+
+ - `"16g"`
+
+ - `"64g"`
+
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+
+ Network access policy for the container.
+
+ - `ContainerNetworkPolicyDisabled object { type }`
+
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
+
+ - `skills: optional array of SkillReference or InlineSkill`
+
+ An optional list of skills referenced by id or inline data.
+
+ - `SkillReference object { skill_id, type, version }`
+
+ - `skill_id: string`
+
+ The ID of the referenced skill.
+
+ - `type: "skill_reference"`
+
+ References a skill created with the /v1/skills endpoint.
+
+ - `"skill_reference"`
+
+ - `version: optional string`
+
+ Optional skill version. Use a positive integer or 'latest'. Omit for default.
+
+ - `InlineSkill object { description, name, source, type }`
+
+ - `description: string`
+
+ The description of the skill.
+
+ - `name: string`
+
+ The name of the skill.
+
+ - `source: InlineSkillSource`
+
+ Inline skill payload
+
+ - `data: string`
+
+ Base64-encoded skill zip bundle.
+
+ - `media_type: "application/zip"`
+
+ The media type of the inline skill payload. Must be `application/zip`.
+
+ - `"application/zip"`
+
+ - `type: "base64"`
+
+ The type of the inline skill source. Must be `base64`.
+
+ - `"base64"`
+
+ - `type: "inline"`
+
+ Defines an inline skill for this request.
+
+ - `"inline"`
+
+ - `LocalEnvironment object { type, skills }`
+
+ - `type: "local"`
+
+ Use a local computer environment.
+
+ - `"local"`
+
+ - `skills: optional array of LocalSkill`
+
+ An optional list of skills.
+
+ - `description: string`
+
+ The description of the skill.
+
+ - `name: string`
+
+ The name of the skill.
+
+ - `path: string`
+
+ The path to the directory containing the skill.
+
+ - `ContainerReference object { container_id, type }`
+
+ - `container_id: string`
+
+ The ID of the referenced container.
+
+ - `type: "container_reference"`
+
+ References a container created with the /v1/containers endpoint
+
+ - `"container_reference"`
+
+ - `Custom object { name, type, defer_loading, 2 more }`
+
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+
+ - `name: string`
+
+ The name of the custom tool, used to identify it in tool calls.
+
+ - `type: "custom"`
+
+ The type of the custom tool. Always `custom`.
+
+ - `"custom"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this tool should be deferred and discovered via tool search.
+
+ - `description: optional string`
+
+ Optional description of the custom tool, used to provide more context.
+
+ - `format: optional CustomToolInputFormat`
+
+ The input format for the custom tool. Default is unconstrained text.
+
+ - `Text object { type }`
+
+ Unconstrained free-form text.
+
+ - `type: "text"`
+
+ Unconstrained text format. Always `text`.
+
+ - `"text"`
+
+ - `Grammar object { definition, syntax, type }`
+
+ A grammar defined by the user.
+
+ - `definition: string`
+
+ The grammar definition.
+
+ - `syntax: "lark" or "regex"`
+
+ The syntax of the grammar definition. One of `lark` or `regex`.
+
+ - `"lark"`
+
+ - `"regex"`
+
+ - `type: "grammar"`
+
+ Grammar format. Always `grammar`.
+
+ - `"grammar"`
+
+ - `Namespace object { description, name, tools, type }`
+
+ Groups function/custom tools under a shared namespace.
+
+ - `description: string`
+
+ A description of the namespace shown to the model.
+
+ - `name: string`
+
+ The namespace name used in tool calls (for example, `crm`).
+
+ - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
+
+ The function/custom tools available inside this namespace.
+
+ - `Function object { name, type, defer_loading, 3 more }`
+
+ - `name: string`
+
+ - `type: "function"`
+
+ - `"function"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this function should be deferred and discovered via tool search.
+
+ - `description: optional string`
+
+ - `parameters: optional unknown`
+
+ - `strict: optional boolean`
+
+ - `Custom object { name, type, defer_loading, 2 more }`
+
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+
+ - `name: string`
+
+ The name of the custom tool, used to identify it in tool calls.
+
+ - `type: "custom"`
+
+ The type of the custom tool. Always `custom`.
+
+ - `"custom"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this tool should be deferred and discovered via tool search.
+
+ - `description: optional string`
+
+ Optional description of the custom tool, used to provide more context.
+
+ - `format: optional CustomToolInputFormat`
+
+ The input format for the custom tool. Default is unconstrained text.
+
+ - `type: "namespace"`
+
+ The type of the tool. Always `namespace`.
+
+ - `"namespace"`
+
+ - `ToolSearch object { type, description, execution, parameters }`
+
+ Hosted or BYOT tool search configuration for deferred tools.
+
+ - `type: "tool_search"`
+
+ The type of the tool. Always `tool_search`.
+
+ - `"tool_search"`
+
+ - `description: optional string`
+
+ Description shown to the model for a client-executed tool search tool.
+
+ - `execution: optional "server" or "client"`
+
+ Whether tool search is executed by the server or by the client.
+
+ - `"server"`
+
+ - `"client"`
+
+ - `parameters: optional unknown`
+
+ Parameter schema for a client-executed tool search tool.
+
+ - `WebSearchPreview object { type, search_content_types, search_context_size, user_location }`
+
+ This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
+
+ - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
+
+ The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
+
+ - `"web_search_preview"`
+
+ - `"web_search_preview_2025_03_11"`
+
+ - `search_content_types: optional array of "text" or "image"`
+
+ - `"text"`
+
+ - `"image"`
+
+ - `search_context_size: optional "low" or "medium" or "high"`
+
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `user_location: optional object { type, city, country, 2 more }`
+
+ The user's location.
+
+ - `type: "approximate"`
+
+ The type of location approximation. Always `approximate`.
+
+ - `"approximate"`
+
+ - `city: optional string`
+
+ Free text input for the city of the user, e.g. `San Francisco`.
+
+ - `country: optional string`
+
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+
+ - `region: optional string`
+
+ Free text input for the region of the user, e.g. `California`.
+
+ - `timezone: optional string`
+
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+
+ - `ApplyPatch object { type }`
+
+ Allows the assistant to create, delete, or update files using unified diffs.
+
+ - `type: "apply_patch"`
+
+ The type of the tool. Always `apply_patch`.
+
+ - `"apply_patch"`
+
+ - `top_p: optional number`
+
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+
+ - `error: EvalAPIError`
+
+ An object representing an error response from the Eval API.
+
+ - `code: string`
+
+ The error code.
+
+ - `message: string`
+
+ The error message.
+
+ - `eval_id: string`
+
+ The identifier of the associated evaluation.
+
+ - `metadata: Metadata`
+
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
+
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
+
+ - `model: string`
+
+ The model that is evaluated, if applicable.
+
+ - `name: string`
+
+ The name of the evaluation run.
+
+ - `object: "eval.run"`
+
+ The type of the object. Always "eval.run".
+
+ - `"eval.run"`
+
+ - `per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }`
+
+ Usage statistics for each model during the evaluation run.
+
+ - `cached_tokens: number`
+
+ The number of tokens retrieved from cache.
+
+ - `completion_tokens: number`
+
+ The number of completion tokens generated.
+
+ - `invocation_count: number`
+
+ The number of invocations.
+
+ - `model_name: string`
+
+ The name of the model.
+
+ - `prompt_tokens: number`
+
+ The number of prompt tokens used.
+
+ - `total_tokens: number`
+
+ The total number of tokens used.
+
+ - `per_testing_criteria_results: array of object { failed, passed, testing_criteria }`
+
+ Results per testing criteria applied during the evaluation run.
+
+ - `failed: number`
+
+ Number of tests failed for this criteria.
+
+ - `passed: number`
+
+ Number of tests passed for this criteria.
+
+ - `testing_criteria: string`
+
+ A description of the testing criteria.
+
+ - `report_url: string`
+
+ The URL to the rendered evaluation run report on the UI dashboard.
+
+ - `result_counts: object { errored, failed, passed, total }`
+
+ Counters summarizing the outcomes of the evaluation run.
+
+ - `errored: number`
+
+ Number of output items that resulted in an error.
+
+ - `failed: number`
+
+ Number of output items that failed to pass the evaluation.
+
+ - `passed: number`
+
+ Number of output items that passed the evaluation.
+
+ - `total: number`
+
+ Total number of executed output items.
+
+ - `status: string`
+
+ The status of the evaluation run.
+
+### Run Retrieve Response
+
+- `RunRetrieveResponse object { id, created_at, data_source, 11 more }`
+
+ A schema representing an evaluation run.
+
+ - `id: string`
+
+ Unique identifier for the evaluation run.
+
+ - `created_at: number`
+
+ Unix timestamp (in seconds) when the evaluation run was created.
+
+ - `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
+
+ Information about the run's data source.
+
+ - `CreateEvalJSONLRunDataSource object { source, type }`
+
+ A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
+
+ - `source: object { content, type } or object { id, type }`
+
+ Determines what populates the `item` namespace in the data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `type: "jsonl"`
+
+ The type of data source. Always `jsonl`.
+
+ - `"jsonl"`
+
+ - `CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }`
+
+ A CompletionsRunDataSource object describing a model sampling configuration.
+
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
+
+ Determines what populates the `item` namespace in this run's data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more }`
+
+ A StoredCompletionsRunDataSource configuration describing a set of filters
+
+ - `type: "stored_completions"`
+
+ The type of source. Always `stored_completions`.
+
+ - `"stored_completions"`
+
+ - `created_after: optional number`
+
+ An optional Unix timestamp to filter items created after this time.
+
+ - `created_before: optional number`
+
+ An optional Unix timestamp to filter items created before this time.
+
+ - `limit: optional number`
+
+ An optional maximum number of items to return.
+
+ - `metadata: optional Metadata`
+
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
+
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
+
+ - `model: optional string`
+
+ An optional model to filter by (e.g., 'gpt-4o').
+
+ - `type: "completions"`
+
+ The type of run data source. Always `completions`.
+
+ - `"completions"`
+
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
+
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+
+ - `TemplateInputMessages object { template, type }`
+
+ - `template: array of EasyInputMessage or object { content, role, type }`
+
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+
+ - `EasyInputMessage object { content, role, phase, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputMessageContentList`
+
+ Text, image, or audio input to the model, used to generate a response.
+ Can also contain previous assistant responses.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputMessageContentList = array of ResponseInputContent`
+
+ A list of one or many input items to the model, containing different content
+ types.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `text: string`
+
+ The text input to the model.
+
+ - `type: "input_text"`
+
+ The type of the input item. Always `input_text`.
+
+ - `"input_text"`
+
+ - `ResponseInputImage object { detail, type, file_id, image_url }`
+
+ An image input to the model. Learn about [image inputs](/docs/guides/vision).
+
+ - `detail: "low" or "high" or "auto" or "original"`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
+
+ - `"low"`
+
+ - `"high"`
+
+ - `"auto"`
+
+ - `"original"`
+
+ - `type: "input_image"`
+
+ The type of the input item. Always `input_image`.
+
+ - `"input_image"`
+
+ - `file_id: optional string`
+
+ The ID of the file to be sent to the model.
+
+ - `image_url: optional string`
+
+ The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
+
+ - `ResponseInputFile object { type, detail, file_data, 3 more }`
+
+ A file input to the model.
+
+ - `type: "input_file"`
+
+ The type of the input item. Always `input_file`.
+
+ - `"input_file"`
+
+ - `detail: optional "low" or "high"`
+
+ The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
+
+ - `"low"`
+
+ - `"high"`
+
+ - `file_data: optional string`
+
+ The content of the file to be sent to the model.
+
+ - `file_id: optional string`
+
+ The ID of the file to be sent to the model.
+
+ - `file_url: optional string`
+
+ The URL of the file to be sent to the model.
+
+ - `filename: optional string`
+
+ The name of the file to be sent to the model.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `phase: optional "commentary" or "final_answer"`
+
+ Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
+ For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
+ phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
+
+ - `"commentary"`
+
+ - `"final_answer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `EvalMessageObject object { content, role, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
+
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `input_audio: object { data, format }`
+
+ - `data: string`
+
+ Base64-encoded audio data.
+
+ - `format: "mp3" or "wav"`
+
+ The format of the audio data. Currently supported formats are `mp3` and
+ `wav`.
+
+ - `"mp3"`
+
+ - `"wav"`
+
+ - `type: "input_audio"`
+
+ The type of the input item. Always `input_audio`.
+
+ - `"input_audio"`
+
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `type: "template"`
+
+ The type of input messages. Always `template`.
+
+ - `"template"`
+
+ - `ItemReferenceInputMessages object { item_reference, type }`
+
+ - `item_reference: string`
+
+ A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
+
+ - `type: "item_reference"`
+
+ The type of input messages. Always `item_reference`.
+
+ - `"item_reference"`
+
+ - `model: optional string`
+
+ The name of the model to use for generating completions (e.g. "o3-mini").
+
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
+
+ - `max_completion_tokens: optional number`
+
+ The maximum number of tokens in the generated output.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `"none"`
+
+ - `"minimal"`
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `"xhigh"`
+
+ - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
+
+ An object specifying the format that the model must output.
+
+ Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
+ Structured Outputs which ensures the model will match your supplied JSON
+ schema. Learn more in the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
+
+ - `ResponseFormatText object { type }`
+
+ Default response format. Used to generate text responses.
+
+ - `type: "text"`
+
+ The type of response format being defined. Always `text`.
+
+ - `"text"`
+
+ - `ResponseFormatJSONSchema object { json_schema, type }`
+
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+
+ - `json_schema: object { name, description, schema, strict }`
+
+ Structured Outputs configuration options, including a JSON Schema.
+
+ - `name: string`
+
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
+
+ - `description: optional string`
+
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
+
+ - `schema: optional map[unknown]`
+
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ - `type: "json_schema"`
+
+ The type of response format being defined. Always `json_schema`.
+
+ - `"json_schema"`
+
+ - `ResponseFormatJSONObject object { type }`
+
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
+
+ - `type: "json_object"`
+
+ The type of response format being defined. Always `json_object`.
+
+ - `"json_object"`
+
+ - `seed: optional number`
+
+ A seed value to initialize the randomness, during sampling.
+
+ - `temperature: optional number`
+
+ A higher temperature increases randomness in the outputs.
+
+ - `tools: optional array of ChatCompletionFunctionTool`
+
+ A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
+
+ - `function: FunctionDefinition`
+
+ - `name: string`
+
+ The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
+
+ - `description: optional string`
+
+ A description of what the function does, used by the model to choose when and how to call the function.
+
+ - `parameters: optional FunctionParameters`
+
+ The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
+
+ Omitting `parameters` defines a function with an empty parameter list.
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
+
+ - `type: "function"`
+
+ The type of the tool. Currently, only `function` is supported.
+
+ - `"function"`
+
+ - `top_p: optional number`
+
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+
+ - `ResponsesRunDataSource object { source, type, input_messages, 2 more }`
+
+ A ResponsesRunDataSource object describing a model sampling configuration.
+
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
+
+ Determines what populates the `item` namespace in this run's data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `EvalResponsesSource object { type, created_after, created_before, 8 more }`
+
+ A EvalResponsesSource object describing a run data source configuration.
+
+ - `type: "responses"`
+
+ The type of run data source. Always `responses`.
+
+ - `"responses"`
+
+ - `created_after: optional number`
+
+ Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
+
+ - `created_before: optional number`
+
+ Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
+
+ - `instructions_search: optional string`
+
+ Optional string to search the 'instructions' field. This is a query parameter used to select responses.
+
+ - `metadata: optional unknown`
+
+ Metadata filter for the responses. This is a query parameter used to select responses.
+
+ - `model: optional string`
+
+ The name of the model to find responses for. This is a query parameter used to select responses.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `temperature: optional number`
+
+ Sampling temperature. This is a query parameter used to select responses.
+
+ - `tools: optional array of string`
+
+ List of tool names. This is a query parameter used to select responses.
+
+ - `top_p: optional number`
+
+ Nucleus sampling parameter. This is a query parameter used to select responses.
+
+ - `users: optional array of string`
+
+ List of user identifiers. This is a query parameter used to select responses.
+
+ - `type: "responses"`
+
+ The type of run data source. Always `responses`.
+
+ - `"responses"`
+
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
+
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+
+ - `InputMessagesTemplate object { template, type }`
+
+ - `template: array of object { content, role } or object { content, role, type }`
+
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+
+ - `ChatMessage object { content, role }`
+
+ - `content: string`
+
+ The content of the message.
+
+ - `role: string`
+
+ The role of the message (e.g. "system", "assistant", "user").
+
+ - `EvalMessageObject object { content, role, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
+
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `type: "template"`
+
+ The type of input messages. Always `template`.
+
+ - `"template"`
+
+ - `InputMessagesItemReference object { item_reference, type }`
+
+ - `item_reference: string`
+
+ A reference to a variable in the `item` namespace. Ie, "item.name"
+
+ - `type: "item_reference"`
+
+ The type of input messages. Always `item_reference`.
+
+ - `"item_reference"`
+
+ - `model: optional string`
+
+ The name of the model to use for generating completions (e.g. "o3-mini").
+
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
+
+ - `max_completion_tokens: optional number`
+
+ The maximum number of tokens in the generated output.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `seed: optional number`
+
+ A seed value to initialize the randomness, during sampling.
+
+ - `temperature: optional number`
+
+ A higher temperature increases randomness in the outputs.
+
+ - `text: optional object { format }`
+
+ Configuration options for a text response from the model. Can be plain
+ text or structured JSON data. Learn more:
+
+ - [Text inputs and outputs](/docs/guides/text)
+ - [Structured Outputs](/docs/guides/structured-outputs)
+
+ - `format: optional ResponseFormatTextConfig`
+
+ An object specifying the format that the model must output.
+
+ Configuring `{ "type": "json_schema" }` enables Structured Outputs,
+ which ensures the model will match your supplied JSON schema. Learn more in the
+ [Structured Outputs guide](/docs/guides/structured-outputs).
+
+ The default format is `{ "type": "text" }` with no additional options.
+
+ **Not recommended for gpt-4o and newer models:**
+
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
+
+ - `ResponseFormatText object { type }`
+
+ Default response format. Used to generate text responses.
+
+ - `ResponseFormatTextJSONSchemaConfig object { name, schema, type, 2 more }`
+
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+
+ - `name: string`
+
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
+
+ - `schema: map[unknown]`
+
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
+
+ - `type: "json_schema"`
+
+ The type of response format being defined. Always `json_schema`.
+
+ - `"json_schema"`
+
+ - `description: optional string`
+
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ - `ResponseFormatJSONObject object { type }`
+
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
+
+ - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
+
+ An array of tools the model may call while generating a response. You
+ can specify which tool to use by setting the `tool_choice` parameter.
+
+ The two categories of tools you can provide the model are:
+
+ - **Built-in tools**: Tools that are provided by OpenAI that extend the
+ model's capabilities, like [web search](/docs/guides/tools-web-search)
+ or [file search](/docs/guides/tools-file-search). Learn more about
+ [built-in tools](/docs/guides/tools).
+ - **Function calls (custom tools)**: Functions that are defined by you,
+ enabling the model to call your own code. Learn more about
+ [function calling](/docs/guides/function-calling).
+
+ - `Function object { name, parameters, strict, 3 more }`
+
+ Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
+
+ - `name: string`
+
+ The name of the function to call.
+
+ - `parameters: map[unknown]`
+
+ A JSON schema object describing the parameters of the function.
+
+ - `strict: boolean`
+
+ Whether to enforce strict parameter validation. Default `true`.
+
+ - `type: "function"`
+
+ The type of the function tool. Always `function`.
+
+ - `"function"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this function is deferred and loaded via tool search.
+
+ - `description: optional string`
+
+ A description of the function. Used by the model to determine whether or not to call the function.
+
+ - `FileSearch object { type, vector_store_ids, filters, 2 more }`
+
+ A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
+
+ - `type: "file_search"`
+
+ The type of the file search tool. Always `file_search`.
+
+ - `"file_search"`
+
+ - `vector_store_ids: array of string`
+
+ The IDs of the vector stores to search.
+
+ - `filters: optional ComparisonFilter or CompoundFilter`
+
+ A filter to apply.
+
+ - `ComparisonFilter object { key, type, value }`
+
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+
+ - `key: string`
+
+ The key to compare against the value.
+
+ - `type: "eq" or "ne" or "gt" or 5 more`
+
+ Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+
+ - `eq`: equals
+ - `ne`: not equal
+ - `gt`: greater than
+ - `gte`: greater than or equal
+ - `lt`: less than
+ - `lte`: less than or equal
+ - `in`: in
+ - `nin`: not in
+
+ - `"eq"`
+
+ - `"ne"`
+
+ - `"gt"`
+
+ - `"gte"`
+
+ - `"lt"`
+
+ - `"lte"`
+
+ - `"in"`
+
+ - `"nin"`
+
+ - `value: string or number or boolean or array of string or number`
+
+ The value to compare against the attribute key; supports string, number, or boolean types.
+
+ - `string`
+
+ - `number`
+
+ - `boolean`
+
+ - `array of string or number`
+
+ - `string`
+
+ - `number`
+
+ - `CompoundFilter object { filters, type }`
+
+ Combine multiple filters using `and` or `or`.
+
+ - `filters: array of ComparisonFilter or unknown`
+
+ Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
+
+ - `ComparisonFilter object { key, type, value }`
+
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+
+ - `unknown`
+
+ - `type: "and" or "or"`
+
+ Type of operation: `and` or `or`.
+
+ - `"and"`
+
+ - `"or"`
+
+ - `max_num_results: optional number`
+
+ The maximum number of results to return. This number should be between 1 and 50 inclusive.
+
+ - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
+
+ Ranking options for search.
+
+ - `hybrid_search: optional object { embedding_weight, text_weight }`
+
+ Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
+
+ - `embedding_weight: number`
+
+ The weight of the embedding in the reciprocal ranking fusion.
+
+ - `text_weight: number`
+
+ The weight of the text in the reciprocal ranking fusion.
+
+ - `ranker: optional "auto" or "default-2024-11-15"`
+
+ The ranker to use for the file search.
+
+ - `"auto"`
+
+ - `"default-2024-11-15"`
+
+ - `score_threshold: optional number`
+
+ The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
+
+ - `Computer object { type }`
+
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+
+ - `type: "computer"`
+
+ The type of the computer tool. Always `computer`.
+
+ - `"computer"`
+
+ - `ComputerUsePreview object { display_height, display_width, environment, type }`
+
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+
+ - `display_height: number`
+
+ The height of the computer display.
+
+ - `display_width: number`
+
+ The width of the computer display.
+
+ - `environment: "windows" or "mac" or "linux" or 2 more`
+
+ The type of computer environment to control.
+
+ - `"windows"`
+
+ - `"mac"`
+
+ - `"linux"`
+
+ - `"ubuntu"`
+
+ - `"browser"`
+
+ - `type: "computer_use_preview"`
+
+ The type of the computer use tool. Always `computer_use_preview`.
+
+ - `"computer_use_preview"`
+
+ - `WebSearch object { type, filters, search_context_size, user_location }`
+
+ Search the Internet for sources related to the prompt. Learn more about the
+ [web search tool](/docs/guides/tools-web-search).
+
+ - `type: "web_search" or "web_search_2025_08_26"`
+
+ The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
+
+ - `"web_search"`
+
+ - `"web_search_2025_08_26"`
+
+ - `filters: optional object { allowed_domains }`
+
+ Filters for the search.
+
+ - `allowed_domains: optional array of string`
+
+ Allowed domains for the search. If not provided, all domains are allowed.
+ Subdomains of the provided domains are allowed as well.
+
+ Example: `["pubmed.ncbi.nlm.nih.gov"]`
+
+ - `search_context_size: optional "low" or "medium" or "high"`
+
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `user_location: optional object { city, country, region, 2 more }`
+
+ The approximate location of the user.
+
+ - `city: optional string`
+
+ Free text input for the city of the user, e.g. `San Francisco`.
+
+ - `country: optional string`
+
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+
+ - `region: optional string`
+
+ Free text input for the region of the user, e.g. `California`.
+
+ - `timezone: optional string`
+
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+
+ - `type: optional "approximate"`
+
+ The type of location approximation. Always `approximate`.
+
+ - `"approximate"`
+
+ - `Mcp object { server_label, type, allowed_tools, 7 more }`
+
+ Give the model access to additional tools via remote Model Context Protocol
+ (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
+
+ - `server_label: string`
+
+ A label for this MCP server, used to identify it in tool calls.
+
+ - `type: "mcp"`
+
+ The type of the MCP tool. Always `mcp`.
+
+ - `"mcp"`
+
+ - `allowed_tools: optional array of string or object { read_only, tool_names }`
+
+ List of allowed tool names or a filter object.
+
+ - `McpAllowedTools = array of string`
+
+ A string array of allowed tool names
+
+ - `McpToolFilter object { read_only, tool_names }`
+
+ A filter object to specify which tools are allowed.
+
+ - `read_only: optional boolean`
+
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
+
+ - `tool_names: optional array of string`
+
+ List of allowed tool names.
+
+ - `authorization: optional string`
+
+ An OAuth access token that can be used with a remote MCP server, either
+ with a custom MCP server URL or a service connector. Your application
+ must handle the OAuth authorization flow and provide the token here.
+
+ - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
+
+ Identifier for service connectors, like those available in ChatGPT. One of
+ `server_url` or `connector_id` must be provided. Learn more about service
+ connectors [here](/docs/guides/tools-remote-mcp#connectors).
+
+ Currently supported `connector_id` values are:
+
+ - Dropbox: `connector_dropbox`
+ - Gmail: `connector_gmail`
+ - Google Calendar: `connector_googlecalendar`
+ - Google Drive: `connector_googledrive`
+ - Microsoft Teams: `connector_microsoftteams`
+ - Outlook Calendar: `connector_outlookcalendar`
+ - Outlook Email: `connector_outlookemail`
+ - SharePoint: `connector_sharepoint`
+
+ - `"connector_dropbox"`
+
+ - `"connector_gmail"`
+
+ - `"connector_googlecalendar"`
+
+ - `"connector_googledrive"`
+
+ - `"connector_microsoftteams"`
+
+ - `"connector_outlookcalendar"`
+
+ - `"connector_outlookemail"`
+
+ - `"connector_sharepoint"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this MCP tool is deferred and discovered via tool search.
+
+ - `headers: optional map[string]`
+
+ Optional HTTP headers to send to the MCP server. Use for authentication
+ or other purposes.
+
+ - `require_approval: optional object { always, never } or "always" or "never"`
+
+ Specify which of the MCP server's tools require approval.
+
+ - `McpToolApprovalFilter object { always, never }`
+
+ Specify which of the MCP server's tools require approval. Can be
+ `always`, `never`, or a filter object associated with tools
+ that require approval.
+
+ - `always: optional object { read_only, tool_names }`
+
+ A filter object to specify which tools are allowed.
+
+ - `read_only: optional boolean`
+
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
+
+ - `tool_names: optional array of string`
+
+ List of allowed tool names.
+
+ - `never: optional object { read_only, tool_names }`
+
+ A filter object to specify which tools are allowed.
+
+ - `read_only: optional boolean`
+
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
+
+ - `tool_names: optional array of string`
+
+ List of allowed tool names.
+
+ - `McpToolApprovalSetting = "always" or "never"`
+
+ Specify a single approval policy for all tools. One of `always` or
+ `never`. When set to `always`, all tools will require approval. When
+ set to `never`, all tools will not require approval.
+
+ - `"always"`
+
+ - `"never"`
+
+ - `server_description: optional string`
+
+ Optional description of the MCP server, used to provide more context.
+
+ - `server_url: optional string`
+
+ The URL for the MCP server. One of `server_url` or `connector_id` must be
+ provided.
+
+ - `CodeInterpreter object { container, type }`
+
+ A tool that runs Python code to help generate a response to a prompt.
+
+ - `container: string or object { type, file_ids, memory_limit, network_policy }`
+
+ The code interpreter container. Can be a container ID or an object that
+ specifies uploaded file IDs to make available to your code, along with an
+ optional `memory_limit` setting.
+
+ - `string`
+
+ The container ID.
+
+ - `CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy }`
+
+ Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
+
+ - `type: "auto"`
+
+ Always `auto`.
+
+ - `"auto"`
+
+ - `file_ids: optional array of string`
+
+ An optional list of uploaded files to make available to your code.
+
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+
+ The memory limit for the code interpreter container.
+
+ - `"1g"`
+
+ - `"4g"`
+
+ - `"16g"`
+
+ - `"64g"`
+
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+
+ Network access policy for the container.
+
+ - `ContainerNetworkPolicyDisabled object { type }`
+
+ - `type: "disabled"`
+
+ Disable outbound network access. Always `disabled`.
+
+ - `"disabled"`
+
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
+
+ - `allowed_domains: array of string`
+
+ A list of allowed domains when type is `allowlist`.
+
+ - `type: "allowlist"`
+
+ Allow outbound network access only to specified domains. Always `allowlist`.
+
+ - `"allowlist"`
+
+ - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
+
+ Optional domain-scoped secrets for allowlisted domains.
+
+ - `domain: string`
+
+ The domain associated with the secret.
+
+ - `name: string`
+
+ The name of the secret to inject for the domain.
+
+ - `value: string`
+
+ The secret value to inject for the domain.
+
+ - `type: "code_interpreter"`
+
+ The type of the code interpreter tool. Always `code_interpreter`.
+
+ - `"code_interpreter"`
+
+ - `ImageGeneration object { type, action, background, 9 more }`
+
+ A tool that generates images using the GPT image models.
+
+ - `type: "image_generation"`
+
+ The type of the image generation tool. Always `image_generation`.
+
+ - `"image_generation"`
+
+ - `action: optional "generate" or "edit" or "auto"`
+
+ Whether to generate a new image or edit an existing image. Default: `auto`.
+
+ - `"generate"`
+
+ - `"edit"`
+
+ - `"auto"`
+
+ - `background: optional "transparent" or "opaque" or "auto"`
+
+ Background type for the generated image. One of `transparent`,
+ `opaque`, or `auto`. Default: `auto`.
+
+ - `"transparent"`
+
+ - `"opaque"`
+
+ - `"auto"`
+
+ - `input_fidelity: optional "high" or "low"`
+
+ Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
+
+ - `"high"`
+
+ - `"low"`
+
+ - `input_image_mask: optional object { file_id, image_url }`
+
+ Optional mask for inpainting. Contains `image_url`
+ (string, optional) and `file_id` (string, optional).
+
+ - `file_id: optional string`
+
+ File ID for the mask image.
+
+ - `image_url: optional string`
+
+ Base64-encoded mask image.
+
+ - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+
+ The image generation model to use. Default: `gpt-image-1`.
+
+ - `string`
+
+ - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+
+ The image generation model to use. Default: `gpt-image-1`.
+
+ - `"gpt-image-1"`
+
+ - `"gpt-image-1-mini"`
+
+ - `"gpt-image-1.5"`
+
+ - `moderation: optional "auto" or "low"`
+
+ Moderation level for the generated image. Default: `auto`.
+
+ - `"auto"`
+
+ - `"low"`
+
+ - `output_compression: optional number`
+
+ Compression level for the output image. Default: 100.
+
+ - `output_format: optional "png" or "webp" or "jpeg"`
+
+ The output format of the generated image. One of `png`, `webp`, or
+ `jpeg`. Default: `png`.
+
+ - `"png"`
+
+ - `"webp"`
+
+ - `"jpeg"`
+
+ - `partial_images: optional number`
+
+ Number of partial images to generate in streaming mode, from 0 (default value) to 3.
+
+ - `quality: optional "low" or "medium" or "high" or "auto"`
+
+ The quality of the generated image. One of `low`, `medium`, `high`,
+ or `auto`. Default: `auto`.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `"auto"`
+
+ - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
+
+ The size of the generated image. One of `1024x1024`, `1024x1536`,
+ `1536x1024`, or `auto`. Default: `auto`.
+
+ - `"1024x1024"`
+
+ - `"1024x1536"`
+
+ - `"1536x1024"`
+
+ - `"auto"`
+
+ - `LocalShell object { type }`
+
+ A tool that allows the model to execute shell commands in a local environment.
+
+ - `type: "local_shell"`
+
+ The type of the local shell tool. Always `local_shell`.
+
+ - `"local_shell"`
+
+ - `Shell object { type, environment }`
+
+ A tool that allows the model to execute shell commands.
+
+ - `type: "shell"`
+
+ The type of the shell tool. Always `shell`.
+
+ - `"shell"`
+
+ - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
+
+ - `ContainerAuto object { type, file_ids, memory_limit, 2 more }`
+
+ - `type: "container_auto"`
+
+ Automatically creates a container for this request
+
+ - `"container_auto"`
+
+ - `file_ids: optional array of string`
+
+ An optional list of uploaded files to make available to your code.
+
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+
+ The memory limit for the container.
+
+ - `"1g"`
+
+ - `"4g"`
+
+ - `"16g"`
+
+ - `"64g"`
+
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+
+ Network access policy for the container.
+
+ - `ContainerNetworkPolicyDisabled object { type }`
+
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
+
+ - `skills: optional array of SkillReference or InlineSkill`
+
+ An optional list of skills referenced by id or inline data.
+
+ - `SkillReference object { skill_id, type, version }`
+
+ - `skill_id: string`
+
+ The ID of the referenced skill.
+
+ - `type: "skill_reference"`
+
+ References a skill created with the /v1/skills endpoint.
+
+ - `"skill_reference"`
+
+ - `version: optional string`
+
+ Optional skill version. Use a positive integer or 'latest'. Omit for default.
+
+ - `InlineSkill object { description, name, source, type }`
+
+ - `description: string`
+
+ The description of the skill.
+
+ - `name: string`
+
+ The name of the skill.
+
+ - `source: InlineSkillSource`
+
+ Inline skill payload
+
+ - `data: string`
+
+ Base64-encoded skill zip bundle.
+
+ - `media_type: "application/zip"`
+
+ The media type of the inline skill payload. Must be `application/zip`.
+
+ - `"application/zip"`
+
+ - `type: "base64"`
+
+ The type of the inline skill source. Must be `base64`.
+
+ - `"base64"`
+
+ - `type: "inline"`
+
+ Defines an inline skill for this request.
+
+ - `"inline"`
+
+ - `LocalEnvironment object { type, skills }`
+
+ - `type: "local"`
+
+ Use a local computer environment.
+
+ - `"local"`
+
+ - `skills: optional array of LocalSkill`
+
+ An optional list of skills.
+
+ - `description: string`
+
+ The description of the skill.
+
+ - `name: string`
+
+ The name of the skill.
+
+ - `path: string`
+
+ The path to the directory containing the skill.
+
+ - `ContainerReference object { container_id, type }`
+
+ - `container_id: string`
+
+ The ID of the referenced container.
+
+ - `type: "container_reference"`
+
+ References a container created with the /v1/containers endpoint
+
+ - `"container_reference"`
+
+ - `Custom object { name, type, defer_loading, 2 more }`
+
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+
+ - `name: string`
+
+ The name of the custom tool, used to identify it in tool calls.
+
+ - `type: "custom"`
+
+ The type of the custom tool. Always `custom`.
+
+ - `"custom"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this tool should be deferred and discovered via tool search.
+
+ - `description: optional string`
+
+ Optional description of the custom tool, used to provide more context.
+
+ - `format: optional CustomToolInputFormat`
+
+ The input format for the custom tool. Default is unconstrained text.
+
+ - `Text object { type }`
+
+ Unconstrained free-form text.
+
+ - `type: "text"`
+
+ Unconstrained text format. Always `text`.
+
+ - `"text"`
+
+ - `Grammar object { definition, syntax, type }`
+
+ A grammar defined by the user.
+
+ - `definition: string`
+
+ The grammar definition.
+
+ - `syntax: "lark" or "regex"`
+
+ The syntax of the grammar definition. One of `lark` or `regex`.
+
+ - `"lark"`
+
+ - `"regex"`
+
+ - `type: "grammar"`
+
+ Grammar format. Always `grammar`.
+
+ - `"grammar"`
+
+ - `Namespace object { description, name, tools, type }`
+
+ Groups function/custom tools under a shared namespace.
+
+ - `description: string`
+
+ A description of the namespace shown to the model.
+
+ - `name: string`
+
+ The namespace name used in tool calls (for example, `crm`).
+
+ - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
+
+ The function/custom tools available inside this namespace.
+
+ - `Function object { name, type, defer_loading, 3 more }`
+
+ - `name: string`
+
+ - `type: "function"`
+
+ - `"function"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this function should be deferred and discovered via tool search.
+
+ - `description: optional string`
+
+ - `parameters: optional unknown`
+
+ - `strict: optional boolean`
+
+ - `Custom object { name, type, defer_loading, 2 more }`
+
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
+
+ - `name: string`
+
+ The name of the custom tool, used to identify it in tool calls.
+
+ - `type: "custom"`
+
+ The type of the custom tool. Always `custom`.
+
+ - `"custom"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this tool should be deferred and discovered via tool search.
+
+ - `description: optional string`
+
+ Optional description of the custom tool, used to provide more context.
+
+ - `format: optional CustomToolInputFormat`
+
+ The input format for the custom tool. Default is unconstrained text.
+
+ - `type: "namespace"`
+
+ The type of the tool. Always `namespace`.
+
+ - `"namespace"`
+
+ - `ToolSearch object { type, description, execution, parameters }`
+
+ Hosted or BYOT tool search configuration for deferred tools.
+
+ - `type: "tool_search"`
+
+ The type of the tool. Always `tool_search`.
+
+ - `"tool_search"`
+
+ - `description: optional string`
+
+ Description shown to the model for a client-executed tool search tool.
+
+ - `execution: optional "server" or "client"`
+
+ Whether tool search is executed by the server or by the client.
+
+ - `"server"`
+
+ - `"client"`
+
+ - `parameters: optional unknown`
+
+ Parameter schema for a client-executed tool search tool.
+
+ - `WebSearchPreview object { type, search_content_types, search_context_size, user_location }`
+
+ This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
+
+ - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
+
+ The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
+
+ - `"web_search_preview"`
+
+ - `"web_search_preview_2025_03_11"`
+
+ - `search_content_types: optional array of "text" or "image"`
+
+ - `"text"`
+
+ - `"image"`
+
+ - `search_context_size: optional "low" or "medium" or "high"`
+
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `user_location: optional object { type, city, country, 2 more }`
+
+ The user's location.
+
+ - `type: "approximate"`
+
+ The type of location approximation. Always `approximate`.
+
+ - `"approximate"`
+
+ - `city: optional string`
+
+ Free text input for the city of the user, e.g. `San Francisco`.
+
+ - `country: optional string`
+
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+
+ - `region: optional string`
+
+ Free text input for the region of the user, e.g. `California`.
+
+ - `timezone: optional string`
+
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+
+ - `ApplyPatch object { type }`
+
+ Allows the assistant to create, delete, or update files using unified diffs.
+
+ - `type: "apply_patch"`
+
+ The type of the tool. Always `apply_patch`.
+
+ - `"apply_patch"`
+
+ - `top_p: optional number`
+
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+
+ - `error: EvalAPIError`
+
+ An object representing an error response from the Eval API.
+
+ - `code: string`
+
+ The error code.
+
+ - `message: string`
+
+ The error message.
+
+ - `eval_id: string`
+
+ The identifier of the associated evaluation.
+
+ - `metadata: Metadata`
+
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
+
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
+
+ - `model: string`
+
+ The model that is evaluated, if applicable.
+
+ - `name: string`
+
+ The name of the evaluation run.
+
+ - `object: "eval.run"`
+
+ The type of the object. Always "eval.run".
+
+ - `"eval.run"`
+
+ - `per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }`
+
+ Usage statistics for each model during the evaluation run.
+
+ - `cached_tokens: number`
+
+ The number of tokens retrieved from cache.
+
+ - `completion_tokens: number`
+
+ The number of completion tokens generated.
+
+ - `invocation_count: number`
+
+ The number of invocations.
+
+ - `model_name: string`
+
+ The name of the model.
+
+ - `prompt_tokens: number`
+
+ The number of prompt tokens used.
+
+ - `total_tokens: number`
+
+ The total number of tokens used.
+
+ - `per_testing_criteria_results: array of object { failed, passed, testing_criteria }`
+
+ Results per testing criteria applied during the evaluation run.
+
+ - `failed: number`
+
+ Number of tests failed for this criteria.
+
+ - `passed: number`
+
+ Number of tests passed for this criteria.
+
+ - `testing_criteria: string`
+
+ A description of the testing criteria.
+
+ - `report_url: string`
+
+ The URL to the rendered evaluation run report on the UI dashboard.
+
+ - `result_counts: object { errored, failed, passed, total }`
+
+ Counters summarizing the outcomes of the evaluation run.
+
+ - `errored: number`
+
+ Number of output items that resulted in an error.
+
+ - `failed: number`
+
+ Number of output items that failed to pass the evaluation.
+
+ - `passed: number`
+
+ Number of output items that passed the evaluation.
+
+ - `total: number`
+
+ Total number of executed output items.
+
+ - `status: string`
+
+ The status of the evaluation run.
+
+### Run Cancel Response
+
+- `RunCancelResponse object { id, created_at, data_source, 11 more }`
+
+ A schema representing an evaluation run.
+
+ - `id: string`
+
+ Unique identifier for the evaluation run.
+
+ - `created_at: number`
+
+ Unix timestamp (in seconds) when the evaluation run was created.
+
+ - `data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }`
+
+ Information about the run's data source.
+
+ - `CreateEvalJSONLRunDataSource object { source, type }`
+
+ A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
+
+ - `source: object { content, type } or object { id, type }`
+
+ Determines what populates the `item` namespace in the data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `type: "jsonl"`
+
+ The type of data source. Always `jsonl`.
+
+ - `"jsonl"`
+
+ - `CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }`
+
+ A CompletionsRunDataSource object describing a model sampling configuration.
+
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
+
+ Determines what populates the `item` namespace in this run's data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more }`
+
+ A StoredCompletionsRunDataSource configuration describing a set of filters
+
+ - `type: "stored_completions"`
+
+ The type of source. Always `stored_completions`.
+
+ - `"stored_completions"`
+
+ - `created_after: optional number`
+
+ An optional Unix timestamp to filter items created after this time.
+
+ - `created_before: optional number`
+
+ An optional Unix timestamp to filter items created before this time.
+
+ - `limit: optional number`
+
+ An optional maximum number of items to return.
+
+ - `metadata: optional Metadata`
+
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
+
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
+
+ - `model: optional string`
+
+ An optional model to filter by (e.g., 'gpt-4o').
+
+ - `type: "completions"`
+
+ The type of run data source. Always `completions`.
+
+ - `"completions"`
+
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
+
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+
+ - `TemplateInputMessages object { template, type }`
+
+ - `template: array of EasyInputMessage or object { content, role, type }`
+
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+
+ - `EasyInputMessage object { content, role, phase, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputMessageContentList`
+
+ Text, image, or audio input to the model, used to generate a response.
+ Can also contain previous assistant responses.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputMessageContentList = array of ResponseInputContent`
+
+ A list of one or many input items to the model, containing different content
+ types.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `text: string`
+
+ The text input to the model.
+
+ - `type: "input_text"`
+
+ The type of the input item. Always `input_text`.
+
+ - `"input_text"`
+
+ - `ResponseInputImage object { detail, type, file_id, image_url }`
+
+ An image input to the model. Learn about [image inputs](/docs/guides/vision).
+
+ - `detail: "low" or "high" or "auto" or "original"`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
+
+ - `"low"`
+
+ - `"high"`
+
+ - `"auto"`
+
+ - `"original"`
+
+ - `type: "input_image"`
+
+ The type of the input item. Always `input_image`.
+
+ - `"input_image"`
+
+ - `file_id: optional string`
+
+ The ID of the file to be sent to the model.
+
+ - `image_url: optional string`
+
+ The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
+
+ - `ResponseInputFile object { type, detail, file_data, 3 more }`
+
+ A file input to the model.
+
+ - `type: "input_file"`
+
+ The type of the input item. Always `input_file`.
+
+ - `"input_file"`
+
+ - `detail: optional "low" or "high"`
+
+ The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
+
+ - `"low"`
+
+ - `"high"`
+
+ - `file_data: optional string`
+
+ The content of the file to be sent to the model.
+
+ - `file_id: optional string`
+
+ The ID of the file to be sent to the model.
+
+ - `file_url: optional string`
+
+ The URL of the file to be sent to the model.
+
+ - `filename: optional string`
+
+ The name of the file to be sent to the model.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `phase: optional "commentary" or "final_answer"`
+
+ Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
+ For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
+ phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
+
+ - `"commentary"`
+
+ - `"final_answer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `EvalMessageObject object { content, role, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
+
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `input_audio: object { data, format }`
+
+ - `data: string`
+
+ Base64-encoded audio data.
+
+ - `format: "mp3" or "wav"`
+
+ The format of the audio data. Currently supported formats are `mp3` and
+ `wav`.
+
+ - `"mp3"`
+
+ - `"wav"`
+
+ - `type: "input_audio"`
+
+ The type of the input item. Always `input_audio`.
+
+ - `"input_audio"`
+
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `type: "template"`
+
+ The type of input messages. Always `template`.
+
+ - `"template"`
+
+ - `ItemReferenceInputMessages object { item_reference, type }`
+
+ - `item_reference: string`
+
+ A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
+
+ - `type: "item_reference"`
+
+ The type of input messages. Always `item_reference`.
+
+ - `"item_reference"`
+
+ - `model: optional string`
+
+ The name of the model to use for generating completions (e.g. "o3-mini").
+
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
+
+ - `max_completion_tokens: optional number`
+
+ The maximum number of tokens in the generated output.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `"none"`
+
+ - `"minimal"`
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `"xhigh"`
+
+ - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
+
+ An object specifying the format that the model must output.
+
+ Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
+ Structured Outputs which ensures the model will match your supplied JSON
+ schema. Learn more in the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
+
+ - `ResponseFormatText object { type }`
+
+ Default response format. Used to generate text responses.
+
+ - `type: "text"`
+
+ The type of response format being defined. Always `text`.
+
+ - `"text"`
+
+ - `ResponseFormatJSONSchema object { json_schema, type }`
+
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+
+ - `json_schema: object { name, description, schema, strict }`
+
+ Structured Outputs configuration options, including a JSON Schema.
+
+ - `name: string`
+
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
+
+ - `description: optional string`
+
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
+
+ - `schema: optional map[unknown]`
+
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ - `type: "json_schema"`
+
+ The type of response format being defined. Always `json_schema`.
+
+ - `"json_schema"`
+
+ - `ResponseFormatJSONObject object { type }`
+
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
+
+ - `type: "json_object"`
+
+ The type of response format being defined. Always `json_object`.
+
+ - `"json_object"`
+
+ - `seed: optional number`
+
+ A seed value to initialize the randomness, during sampling.
+
+ - `temperature: optional number`
+
+ A higher temperature increases randomness in the outputs.
+
+ - `tools: optional array of ChatCompletionFunctionTool`
+
+ A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
+
+ - `function: FunctionDefinition`
+
+ - `name: string`
+
+ The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
+
+ - `description: optional string`
+
+ A description of what the function does, used by the model to choose when and how to call the function.
+
+ - `parameters: optional FunctionParameters`
+
+ The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
+
+ Omitting `parameters` defines a function with an empty parameter list.
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
+
+ - `type: "function"`
+
+ The type of the tool. Currently, only `function` is supported.
+
+ - `"function"`
+
+ - `top_p: optional number`
+
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+
+ - `ResponsesRunDataSource object { source, type, input_messages, 2 more }`
+
+ A ResponsesRunDataSource object describing a model sampling configuration.
+
+ - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }`
+
+ Determines what populates the `item` namespace in this run's data source.
+
+ - `EvalJSONLFileContentSource object { content, type }`
+
+ - `content: array of object { item, sample }`
+
+ The content of the jsonl file.
+
+ - `item: map[unknown]`
+
+ - `sample: optional map[unknown]`
+
+ - `type: "file_content"`
+
+ The type of jsonl source. Always `file_content`.
+
+ - `"file_content"`
+
+ - `EvalJSONLFileIDSource object { id, type }`
+
+ - `id: string`
+
+ The identifier of the file.
+
+ - `type: "file_id"`
+
+ The type of jsonl source. Always `file_id`.
+
+ - `"file_id"`
+
+ - `EvalResponsesSource object { type, created_after, created_before, 8 more }`
+
+ A EvalResponsesSource object describing a run data source configuration.
+
+ - `type: "responses"`
+
+ The type of run data source. Always `responses`.
+
+ - `"responses"`
+
+ - `created_after: optional number`
+
+ Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
+
+ - `created_before: optional number`
+
+ Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
+
+ - `instructions_search: optional string`
+
+ Optional string to search the 'instructions' field. This is a query parameter used to select responses.
+
+ - `metadata: optional unknown`
+
+ Metadata filter for the responses. This is a query parameter used to select responses.
+
+ - `model: optional string`
+
+ The name of the model to find responses for. This is a query parameter used to select responses.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `temperature: optional number`
+
+ Sampling temperature. This is a query parameter used to select responses.
+
+ - `tools: optional array of string`
+
+ List of tool names. This is a query parameter used to select responses.
+
+ - `top_p: optional number`
+
+ Nucleus sampling parameter. This is a query parameter used to select responses.
+
+ - `users: optional array of string`
+
+ List of user identifiers. This is a query parameter used to select responses.
+
+ - `type: "responses"`
+
+ The type of run data source. Always `responses`.
+
+ - `"responses"`
+
+ - `input_messages: optional object { template, type } or object { item_reference, type }`
+
+ Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+
+ - `InputMessagesTemplate object { template, type }`
+
+ - `template: array of object { content, role } or object { content, role, type }`
+
+ A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+
+ - `ChatMessage object { content, role }`
+
+ - `content: string`
+
+ The content of the message.
+
+ - `role: string`
+
+ The role of the message (e.g. "system", "assistant", "user").
+
+ - `EvalMessageObject object { content, role, type }`
+
+ A message input to the model with a role indicating instruction following
+ hierarchy. Instructions given with the `developer` or `system` role take
+ precedence over instructions given with the `user` role. Messages with the
+ `assistant` role are presumed to have been generated by the model in previous
+ interactions.
+
+ - `content: string or ResponseInputText or object { text, type } or 3 more`
+
+ Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+
+ - `TextInput = string`
+
+ A text input to the model.
+
+ - `ResponseInputText object { text, type }`
+
+ A text input to the model.
+
+ - `OutputText object { text, type }`
+
+ A text output from the model.
+
+ - `text: string`
+
+ The text output from the model.
+
+ - `type: "output_text"`
+
+ The type of the output text. Always `output_text`.
+
+ - `"output_text"`
+
+ - `InputImage object { image_url, type, detail }`
+
+ An image input block used within EvalItem content arrays.
+
+ - `image_url: string`
+
+ The URL of the image input.
+
+ - `type: "input_image"`
+
+ The type of the image input. Always `input_image`.
+
+ - `"input_image"`
+
+ - `detail: optional string`
+
+ The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+
+ - `ResponseInputAudio object { input_audio, type }`
+
+ An audio input to the model.
+
+ - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+
+ A list of inputs, each of which may be either an input text, output text, input
+ image, or input audio object.
+
+ - `role: "user" or "assistant" or "system" or "developer"`
+
+ The role of the message input. One of `user`, `assistant`, `system`, or
+ `developer`.
+
+ - `"user"`
+
+ - `"assistant"`
+
+ - `"system"`
+
+ - `"developer"`
+
+ - `type: optional "message"`
+
+ The type of the message input. Always `message`.
+
+ - `"message"`
+
+ - `type: "template"`
+
+ The type of input messages. Always `template`.
+
+ - `"template"`
+
+ - `InputMessagesItemReference object { item_reference, type }`
+
+ - `item_reference: string`
+
+ A reference to a variable in the `item` namespace. Ie, "item.name"
+
+ - `type: "item_reference"`
+
+ The type of input messages. Always `item_reference`.
+
+ - `"item_reference"`
+
+ - `model: optional string`
+
+ The name of the model to use for generating completions (e.g. "o3-mini").
+
+ - `sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }`
+
+ - `max_completion_tokens: optional number`
+
+ The maximum number of tokens in the generated output.
+
+ - `reasoning_effort: optional ReasoningEffort`
+
+ Constrains effort on reasoning for
+ [reasoning models](https://platform.openai.com/docs/guides/reasoning).
+ Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
+ reasoning effort can result in faster responses and fewer tokens used
+ on reasoning in a response.
+
+ - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
+ - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
+ - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
+ - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+
+ - `seed: optional number`
+
+ A seed value to initialize the randomness, during sampling.
+
+ - `temperature: optional number`
+
+ A higher temperature increases randomness in the outputs.
+
+ - `text: optional object { format }`
+
+ Configuration options for a text response from the model. Can be plain
+ text or structured JSON data. Learn more:
+
+ - [Text inputs and outputs](/docs/guides/text)
+ - [Structured Outputs](/docs/guides/structured-outputs)
+
+ - `format: optional ResponseFormatTextConfig`
+
+ An object specifying the format that the model must output.
+
+ Configuring `{ "type": "json_schema" }` enables Structured Outputs,
+ which ensures the model will match your supplied JSON schema. Learn more in the
+ [Structured Outputs guide](/docs/guides/structured-outputs).
+
+ The default format is `{ "type": "text" }` with no additional options.
+
+ **Not recommended for gpt-4o and newer models:**
+
+ Setting to `{ "type": "json_object" }` enables the older JSON mode, which
+ ensures the message the model generates is valid JSON. Using `json_schema`
+ is preferred for models that support it.
+
+ - `ResponseFormatText object { type }`
+
+ Default response format. Used to generate text responses.
+
+ - `ResponseFormatTextJSONSchemaConfig object { name, schema, type, 2 more }`
+
+ JSON Schema response format. Used to generate structured JSON responses.
+ Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+
+ - `name: string`
+
+ The name of the response format. Must be a-z, A-Z, 0-9, or contain
+ underscores and dashes, with a maximum length of 64.
+
+ - `schema: map[unknown]`
+
+ The schema for the response format, described as a JSON Schema object.
+ Learn how to build JSON schemas [here](https://json-schema.org/).
+
+ - `type: "json_schema"`
+
+ The type of response format being defined. Always `json_schema`.
+
+ - `"json_schema"`
+
+ - `description: optional string`
+
+ A description of what the response format is for, used by the model to
+ determine how to respond in the format.
+
+ - `strict: optional boolean`
+
+ Whether to enable strict schema adherence when generating the output.
+ If set to true, the model will always follow the exact schema defined
+ in the `schema` field. Only a subset of JSON Schema is supported when
+ `strict` is `true`. To learn more, read the [Structured Outputs
+ guide](/docs/guides/structured-outputs).
+
+ - `ResponseFormatJSONObject object { type }`
+
+ JSON object response format. An older method of generating JSON responses.
+ Using `json_schema` is recommended for models that support it. Note that the
+ model will not generate JSON without a system or user message instructing it
+ to do so.
+
+ - `tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more`
+
+ An array of tools the model may call while generating a response. You
+ can specify which tool to use by setting the `tool_choice` parameter.
+
+ The two categories of tools you can provide the model are:
+
+ - **Built-in tools**: Tools that are provided by OpenAI that extend the
+ model's capabilities, like [web search](/docs/guides/tools-web-search)
+ or [file search](/docs/guides/tools-file-search). Learn more about
+ [built-in tools](/docs/guides/tools).
+ - **Function calls (custom tools)**: Functions that are defined by you,
+ enabling the model to call your own code. Learn more about
+ [function calling](/docs/guides/function-calling).
+
+ - `Function object { name, parameters, strict, 3 more }`
+
+ Defines a function in your own code the model can choose to call. Learn more about [function calling](https://platform.openai.com/docs/guides/function-calling).
+
+ - `name: string`
+
+ The name of the function to call.
+
+ - `parameters: map[unknown]`
+
+ A JSON schema object describing the parameters of the function.
+
+ - `strict: boolean`
+
+ Whether to enforce strict parameter validation. Default `true`.
+
+ - `type: "function"`
+
+ The type of the function tool. Always `function`.
+
+ - `"function"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this function is deferred and loaded via tool search.
+
+ - `description: optional string`
+
+ A description of the function. Used by the model to determine whether or not to call the function.
+
+ - `FileSearch object { type, vector_store_ids, filters, 2 more }`
+
+ A tool that searches for relevant content from uploaded files. Learn more about the [file search tool](https://platform.openai.com/docs/guides/tools-file-search).
+
+ - `type: "file_search"`
+
+ The type of the file search tool. Always `file_search`.
+
+ - `"file_search"`
+
+ - `vector_store_ids: array of string`
+
+ The IDs of the vector stores to search.
+
+ - `filters: optional ComparisonFilter or CompoundFilter`
+
+ A filter to apply.
+
+ - `ComparisonFilter object { key, type, value }`
+
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+
+ - `key: string`
+
+ The key to compare against the value.
+
+ - `type: "eq" or "ne" or "gt" or 5 more`
+
+ Specifies the comparison operator: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`.
+
+ - `eq`: equals
+ - `ne`: not equal
+ - `gt`: greater than
+ - `gte`: greater than or equal
+ - `lt`: less than
+ - `lte`: less than or equal
+ - `in`: in
+ - `nin`: not in
+
+ - `"eq"`
+
+ - `"ne"`
+
+ - `"gt"`
+
+ - `"gte"`
+
+ - `"lt"`
+
+ - `"lte"`
+
+ - `"in"`
+
+ - `"nin"`
+
+ - `value: string or number or boolean or array of string or number`
+
+ The value to compare against the attribute key; supports string, number, or boolean types.
+
+ - `string`
+
+ - `number`
+
+ - `boolean`
+
+ - `array of string or number`
+
+ - `string`
+
+ - `number`
+
+ - `CompoundFilter object { filters, type }`
+
+ Combine multiple filters using `and` or `or`.
+
+ - `filters: array of ComparisonFilter or unknown`
+
+ Array of filters to combine. Items can be `ComparisonFilter` or `CompoundFilter`.
+
+ - `ComparisonFilter object { key, type, value }`
+
+ A filter used to compare a specified attribute key to a given value using a defined comparison operation.
+
+ - `unknown`
+
+ - `type: "and" or "or"`
+
+ Type of operation: `and` or `or`.
+
+ - `"and"`
+
+ - `"or"`
+
+ - `max_num_results: optional number`
+
+ The maximum number of results to return. This number should be between 1 and 50 inclusive.
+
+ - `ranking_options: optional object { hybrid_search, ranker, score_threshold }`
+
+ Ranking options for search.
+
+ - `hybrid_search: optional object { embedding_weight, text_weight }`
+
+ Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
+
+ - `embedding_weight: number`
+
+ The weight of the embedding in the reciprocal ranking fusion.
+
+ - `text_weight: number`
+
+ The weight of the text in the reciprocal ranking fusion.
+
+ - `ranker: optional "auto" or "default-2024-11-15"`
+
+ The ranker to use for the file search.
+
+ - `"auto"`
+
+ - `"default-2024-11-15"`
+
+ - `score_threshold: optional number`
+
+ The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
+
+ - `Computer object { type }`
+
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+
+ - `type: "computer"`
+
+ The type of the computer tool. Always `computer`.
+
+ - `"computer"`
+
+ - `ComputerUsePreview object { display_height, display_width, environment, type }`
+
+ A tool that controls a virtual computer. Learn more about the [computer tool](https://platform.openai.com/docs/guides/tools-computer-use).
+
+ - `display_height: number`
+
+ The height of the computer display.
+
+ - `display_width: number`
+
+ The width of the computer display.
+
+ - `environment: "windows" or "mac" or "linux" or 2 more`
+
+ The type of computer environment to control.
+
+ - `"windows"`
+
+ - `"mac"`
+
+ - `"linux"`
+
+ - `"ubuntu"`
+
+ - `"browser"`
+
+ - `type: "computer_use_preview"`
+
+ The type of the computer use tool. Always `computer_use_preview`.
+
+ - `"computer_use_preview"`
+
+ - `WebSearch object { type, filters, search_context_size, user_location }`
+
+ Search the Internet for sources related to the prompt. Learn more about the
+ [web search tool](/docs/guides/tools-web-search).
+
+ - `type: "web_search" or "web_search_2025_08_26"`
+
+ The type of the web search tool. One of `web_search` or `web_search_2025_08_26`.
+
+ - `"web_search"`
+
+ - `"web_search_2025_08_26"`
+
+ - `filters: optional object { allowed_domains }`
+
+ Filters for the search.
+
+ - `allowed_domains: optional array of string`
+
+ Allowed domains for the search. If not provided, all domains are allowed.
+ Subdomains of the provided domains are allowed as well.
+
+ Example: `["pubmed.ncbi.nlm.nih.gov"]`
+
+ - `search_context_size: optional "low" or "medium" or "high"`
+
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `user_location: optional object { city, country, region, 2 more }`
+
+ The approximate location of the user.
+
+ - `city: optional string`
+
+ Free text input for the city of the user, e.g. `San Francisco`.
+
+ - `country: optional string`
+
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
+
+ - `region: optional string`
+
+ Free text input for the region of the user, e.g. `California`.
+
+ - `timezone: optional string`
+
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
+
+ - `type: optional "approximate"`
+
+ The type of location approximation. Always `approximate`.
+
+ - `"approximate"`
+
+ - `Mcp object { server_label, type, allowed_tools, 7 more }`
+
+ Give the model access to additional tools via remote Model Context Protocol
+ (MCP) servers. [Learn more about MCP](/docs/guides/tools-remote-mcp).
+
+ - `server_label: string`
+
+ A label for this MCP server, used to identify it in tool calls.
+
+ - `type: "mcp"`
+
+ The type of the MCP tool. Always `mcp`.
+
+ - `"mcp"`
+
+ - `allowed_tools: optional array of string or object { read_only, tool_names }`
+
+ List of allowed tool names or a filter object.
+
+ - `McpAllowedTools = array of string`
+
+ A string array of allowed tool names
+
+ - `McpToolFilter object { read_only, tool_names }`
+
+ A filter object to specify which tools are allowed.
+
+ - `read_only: optional boolean`
+
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
+
+ - `tool_names: optional array of string`
+
+ List of allowed tool names.
+
+ - `authorization: optional string`
+
+ An OAuth access token that can be used with a remote MCP server, either
+ with a custom MCP server URL or a service connector. Your application
+ must handle the OAuth authorization flow and provide the token here.
+
+ - `connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more`
+
+ Identifier for service connectors, like those available in ChatGPT. One of
+ `server_url` or `connector_id` must be provided. Learn more about service
+ connectors [here](/docs/guides/tools-remote-mcp#connectors).
+
+ Currently supported `connector_id` values are:
+
+ - Dropbox: `connector_dropbox`
+ - Gmail: `connector_gmail`
+ - Google Calendar: `connector_googlecalendar`
+ - Google Drive: `connector_googledrive`
+ - Microsoft Teams: `connector_microsoftteams`
+ - Outlook Calendar: `connector_outlookcalendar`
+ - Outlook Email: `connector_outlookemail`
+ - SharePoint: `connector_sharepoint`
+
+ - `"connector_dropbox"`
+
+ - `"connector_gmail"`
+
+ - `"connector_googlecalendar"`
+
+ - `"connector_googledrive"`
+
+ - `"connector_microsoftteams"`
+
+ - `"connector_outlookcalendar"`
+
+ - `"connector_outlookemail"`
+
+ - `"connector_sharepoint"`
+
+ - `defer_loading: optional boolean`
+
+ Whether this MCP tool is deferred and discovered via tool search.
+
+ - `headers: optional map[string]`
+
+ Optional HTTP headers to send to the MCP server. Use for authentication
+ or other purposes.
+
+ - `require_approval: optional object { always, never } or "always" or "never"`
+
+ Specify which of the MCP server's tools require approval.
+
+ - `McpToolApprovalFilter object { always, never }`
+
+ Specify which of the MCP server's tools require approval. Can be
+ `always`, `never`, or a filter object associated with tools
+ that require approval.
+
+ - `always: optional object { read_only, tool_names }`
+
+ A filter object to specify which tools are allowed.
+
+ - `read_only: optional boolean`
+
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
+
+ - `tool_names: optional array of string`
+
+ List of allowed tool names.
+
+ - `never: optional object { read_only, tool_names }`
+
+ A filter object to specify which tools are allowed.
+
+ - `read_only: optional boolean`
+
+ Indicates whether or not a tool modifies data or is read-only. If an
+ MCP server is [annotated with `readOnlyHint`](https://modelcontextprotocol.io/specification/2025-06-18/schema#toolannotations-readonlyhint),
+ it will match this filter.
+
+ - `tool_names: optional array of string`
+
+ List of allowed tool names.
+
+ - `McpToolApprovalSetting = "always" or "never"`
+
+ Specify a single approval policy for all tools. One of `always` or
+ `never`. When set to `always`, all tools will require approval. When
+ set to `never`, all tools will not require approval.
+
+ - `"always"`
+
+ - `"never"`
+
+ - `server_description: optional string`
+
+ Optional description of the MCP server, used to provide more context.
+
+ - `server_url: optional string`
+
+ The URL for the MCP server. One of `server_url` or `connector_id` must be
+ provided.
+
+ - `CodeInterpreter object { container, type }`
+
+ A tool that runs Python code to help generate a response to a prompt.
+
+ - `container: string or object { type, file_ids, memory_limit, network_policy }`
+
+ The code interpreter container. Can be a container ID or an object that
+ specifies uploaded file IDs to make available to your code, along with an
+ optional `memory_limit` setting.
+
+ - `string`
+
+ The container ID.
+
+ - `CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy }`
+
+ Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
+
+ - `type: "auto"`
+
+ Always `auto`.
+
+ - `"auto"`
+
+ - `file_ids: optional array of string`
+
+ An optional list of uploaded files to make available to your code.
+
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+
+ The memory limit for the code interpreter container.
+
+ - `"1g"`
+
+ - `"4g"`
+
+ - `"16g"`
+
+ - `"64g"`
+
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+
+ Network access policy for the container.
+
+ - `ContainerNetworkPolicyDisabled object { type }`
+
+ - `type: "disabled"`
+
+ Disable outbound network access. Always `disabled`.
+
+ - `"disabled"`
-**delete** `/evals/{eval_id}/runs/{run_id}`
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
-Delete an eval run.
+ - `allowed_domains: array of string`
-### Path Parameters
+ A list of allowed domains when type is `allowlist`.
-- `eval_id: string`
+ - `type: "allowlist"`
-- `run_id: string`
+ Allow outbound network access only to specified domains. Always `allowlist`.
-### Returns
+ - `"allowlist"`
-- `deleted: optional boolean`
+ - `domain_secrets: optional array of ContainerNetworkPolicyDomainSecret`
-- `object: optional string`
+ Optional domain-scoped secrets for allowlisted domains.
-- `run_id: optional string`
+ - `domain: string`
-### Example
+ The domain associated with the secret.
-```http
-curl https://api.openai.com/v1/evals/$EVAL_ID/runs/$RUN_ID \
- -X DELETE \
- -H "Authorization: Bearer $OPENAI_API_KEY"
-```
+ - `name: string`
-#### Response
+ The name of the secret to inject for the domain.
-```json
-{
- "deleted": true,
- "object": "eval.run.deleted",
- "run_id": "evalrun_677469f564d48190807532a852da3afb"
-}
-```
+ - `value: string`
-### Example
+ The secret value to inject for the domain.
-```http
-curl https://api.openai.com/v1/evals/eval_123abc/runs/evalrun_abc456 \
- -X DELETE \
- -H "Authorization: Bearer $OPENAI_API_KEY" \
- -H "Content-Type: application/json"
-```
+ - `type: "code_interpreter"`
-#### Response
+ The type of the code interpreter tool. Always `code_interpreter`.
-```json
-{
- "object": "eval.run.deleted",
- "deleted": true,
- "run_id": "evalrun_abc456"
-}
-```
+ - `"code_interpreter"`
-## Domain Types
+ - `ImageGeneration object { type, action, background, 9 more }`
+
+ A tool that generates images using the GPT image models.
+
+ - `type: "image_generation"`
+
+ The type of the image generation tool. Always `image_generation`.
+
+ - `"image_generation"`
+
+ - `action: optional "generate" or "edit" or "auto"`
+
+ Whether to generate a new image or edit an existing image. Default: `auto`.
+
+ - `"generate"`
+
+ - `"edit"`
+
+ - `"auto"`
+
+ - `background: optional "transparent" or "opaque" or "auto"`
+
+ Background type for the generated image. One of `transparent`,
+ `opaque`, or `auto`. Default: `auto`.
+
+ - `"transparent"`
+
+ - `"opaque"`
+
+ - `"auto"`
+
+ - `input_fidelity: optional "high" or "low"`
+
+ Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for `gpt-image-1` and `gpt-image-1.5` and later models, unsupported for `gpt-image-1-mini`. Supports `high` and `low`. Defaults to `low`.
+
+ - `"high"`
+
+ - `"low"`
+
+ - `input_image_mask: optional object { file_id, image_url }`
+
+ Optional mask for inpainting. Contains `image_url`
+ (string, optional) and `file_id` (string, optional).
+
+ - `file_id: optional string`
+
+ File ID for the mask image.
+
+ - `image_url: optional string`
+
+ Base64-encoded mask image.
+
+ - `model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+
+ The image generation model to use. Default: `gpt-image-1`.
+
+ - `string`
+
+ - `"gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"`
+
+ The image generation model to use. Default: `gpt-image-1`.
+
+ - `"gpt-image-1"`
+
+ - `"gpt-image-1-mini"`
+
+ - `"gpt-image-1.5"`
+
+ - `moderation: optional "auto" or "low"`
+
+ Moderation level for the generated image. Default: `auto`.
+
+ - `"auto"`
+
+ - `"low"`
+
+ - `output_compression: optional number`
+
+ Compression level for the output image. Default: 100.
+
+ - `output_format: optional "png" or "webp" or "jpeg"`
+
+ The output format of the generated image. One of `png`, `webp`, or
+ `jpeg`. Default: `png`.
+
+ - `"png"`
+
+ - `"webp"`
+
+ - `"jpeg"`
+
+ - `partial_images: optional number`
+
+ Number of partial images to generate in streaming mode, from 0 (default value) to 3.
+
+ - `quality: optional "low" or "medium" or "high" or "auto"`
+
+ The quality of the generated image. One of `low`, `medium`, `high`,
+ or `auto`. Default: `auto`.
+
+ - `"low"`
+
+ - `"medium"`
+
+ - `"high"`
+
+ - `"auto"`
+
+ - `size: optional "1024x1024" or "1024x1536" or "1536x1024" or "auto"`
+
+ The size of the generated image. One of `1024x1024`, `1024x1536`,
+ `1536x1024`, or `auto`. Default: `auto`.
+
+ - `"1024x1024"`
+
+ - `"1024x1536"`
+
+ - `"1536x1024"`
+
+ - `"auto"`
+
+ - `LocalShell object { type }`
+
+ A tool that allows the model to execute shell commands in a local environment.
+
+ - `type: "local_shell"`
+
+ The type of the local shell tool. Always `local_shell`.
+
+ - `"local_shell"`
+
+ - `Shell object { type, environment }`
+
+ A tool that allows the model to execute shell commands.
+
+ - `type: "shell"`
+
+ The type of the shell tool. Always `shell`.
+
+ - `"shell"`
+
+ - `environment: optional ContainerAuto or LocalEnvironment or ContainerReference`
+
+ - `ContainerAuto object { type, file_ids, memory_limit, 2 more }`
+
+ - `type: "container_auto"`
+
+ Automatically creates a container for this request
+
+ - `"container_auto"`
+
+ - `file_ids: optional array of string`
+
+ An optional list of uploaded files to make available to your code.
+
+ - `memory_limit: optional "1g" or "4g" or "16g" or "64g"`
+
+ The memory limit for the container.
+
+ - `"1g"`
+
+ - `"4g"`
+
+ - `"16g"`
+
+ - `"64g"`
+
+ - `network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist`
+
+ Network access policy for the container.
+
+ - `ContainerNetworkPolicyDisabled object { type }`
+
+ - `ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }`
+
+ - `skills: optional array of SkillReference or InlineSkill`
+
+ An optional list of skills referenced by id or inline data.
+
+ - `SkillReference object { skill_id, type, version }`
+
+ - `skill_id: string`
+
+ The ID of the referenced skill.
+
+ - `type: "skill_reference"`
+
+ References a skill created with the /v1/skills endpoint.
+
+ - `"skill_reference"`
+
+ - `version: optional string`
+
+ Optional skill version. Use a positive integer or 'latest'. Omit for default.
+
+ - `InlineSkill object { description, name, source, type }`
+
+ - `description: string`
+
+ The description of the skill.
+
+ - `name: string`
+
+ The name of the skill.
+
+ - `source: InlineSkillSource`
-### Create Eval Completions Run Data Source
+ Inline skill payload
-- `CreateEvalCompletionsRunDataSource = object { source, type, input_messages, 2 more }`
+ - `data: string`
- A CompletionsRunDataSource object describing a model sampling configuration.
+ Base64-encoded skill zip bundle.
- - `source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }`
+ - `media_type: "application/zip"`
- Determines what populates the `item` namespace in this run's data source.
+ The media type of the inline skill payload. Must be `application/zip`.
- - `EvalJSONLFileContentSource = object { content, type }`
+ - `"application/zip"`
- - `content: array of object { item, sample }`
+ - `type: "base64"`
- The content of the jsonl file.
+ The type of the inline skill source. Must be `base64`.
- - `item: map[unknown]`
+ - `"base64"`
- - `sample: optional map[unknown]`
+ - `type: "inline"`
- - `type: "file_content"`
+ Defines an inline skill for this request.
- The type of jsonl source. Always `file_content`.
+ - `"inline"`
- - `"file_content"`
+ - `LocalEnvironment object { type, skills }`
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `type: "local"`
- - `id: string`
+ Use a local computer environment.
- The identifier of the file.
+ - `"local"`
- - `type: "file_id"`
+ - `skills: optional array of LocalSkill`
- The type of jsonl source. Always `file_id`.
+ An optional list of skills.
- - `"file_id"`
+ - `description: string`
- - `StoredCompletionsRunDataSource = object { type, created_after, created_before, 3 more }`
+ The description of the skill.
- A StoredCompletionsRunDataSource configuration describing a set of filters
+ - `name: string`
- - `type: "stored_completions"`
+ The name of the skill.
- The type of source. Always `stored_completions`.
+ - `path: string`
- - `"stored_completions"`
+ The path to the directory containing the skill.
- - `created_after: optional number`
+ - `ContainerReference object { container_id, type }`
- An optional Unix timestamp to filter items created after this time.
+ - `container_id: string`
- - `created_before: optional number`
+ The ID of the referenced container.
- An optional Unix timestamp to filter items created before this time.
+ - `type: "container_reference"`
- - `limit: optional number`
+ References a container created with the /v1/containers endpoint
- An optional maximum number of items to return.
+ - `"container_reference"`
- - `metadata: optional Metadata`
+ - `Custom object { name, type, defer_loading, 2 more }`
- Set of 16 key-value pairs that can be attached to an object. This can be
- useful for storing additional information about the object in a structured
- format, and querying for objects via API or the dashboard.
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
- Keys are strings with a maximum length of 64 characters. Values are strings
- with a maximum length of 512 characters.
+ - `name: string`
- - `model: optional string`
+ The name of the custom tool, used to identify it in tool calls.
- An optional model to filter by (e.g., 'gpt-4o').
+ - `type: "custom"`
- - `type: "completions"`
+ The type of the custom tool. Always `custom`.
- The type of run data source. Always `completions`.
+ - `"custom"`
- - `"completions"`
+ - `defer_loading: optional boolean`
- - `input_messages: optional object { template, type } or object { item_reference, type }`
+ Whether this tool should be deferred and discovered via tool search.
- Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, `item.input_trajectory`), or a template with variable references to the `item` namespace.
+ - `description: optional string`
- - `TemplateInputMessages = object { template, type }`
+ Optional description of the custom tool, used to provide more context.
- - `template: array of EasyInputMessage or object { content, role, type }`
+ - `format: optional CustomToolInputFormat`
- A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
+ The input format for the custom tool. Default is unconstrained text.
- - `EasyInputMessage = object { content, role, phase, type }`
+ - `Text object { type }`
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ Unconstrained free-form text.
- - `content: string or ResponseInputMessageContentList`
+ - `type: "text"`
- Text, image, or audio input to the model, used to generate a response.
- Can also contain previous assistant responses.
+ Unconstrained text format. Always `text`.
- - `TextInput = string`
+ - `"text"`
- A text input to the model.
+ - `Grammar object { definition, syntax, type }`
- - `ResponseInputMessageContentList = array of ResponseInputContent`
+ A grammar defined by the user.
- A list of one or many input items to the model, containing different content
- types.
+ - `definition: string`
- - `ResponseInputText = object { text, type }`
+ The grammar definition.
- A text input to the model.
+ - `syntax: "lark" or "regex"`
- - `text: string`
+ The syntax of the grammar definition. One of `lark` or `regex`.
- The text input to the model.
+ - `"lark"`
- - `type: "input_text"`
+ - `"regex"`
- The type of the input item. Always `input_text`.
+ - `type: "grammar"`
- - `"input_text"`
+ Grammar format. Always `grammar`.
- - `ResponseInputImage = object { detail, type, file_id, image_url }`
+ - `"grammar"`
- An image input to the model. Learn about [image inputs](/docs/guides/vision).
+ - `Namespace object { description, name, tools, type }`
- - `detail: "low" or "high" or "auto" or "original"`
+ Groups function/custom tools under a shared namespace.
- The detail level of the image to be sent to the model. One of `high`, `low`, `auto`, or `original`. Defaults to `auto`.
+ - `description: string`
- - `"low"`
+ A description of the namespace shown to the model.
- - `"high"`
+ - `name: string`
- - `"auto"`
+ The namespace name used in tool calls (for example, `crm`).
- - `"original"`
+ - `tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }`
- - `type: "input_image"`
+ The function/custom tools available inside this namespace.
- The type of the input item. Always `input_image`.
+ - `Function object { name, type, defer_loading, 3 more }`
- - `"input_image"`
+ - `name: string`
- - `file_id: optional string`
+ - `type: "function"`
- The ID of the file to be sent to the model.
+ - `"function"`
- - `image_url: optional string`
+ - `defer_loading: optional boolean`
- The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
+ Whether this function should be deferred and discovered via tool search.
- - `ResponseInputFile = object { type, detail, file_data, 3 more }`
+ - `description: optional string`
- A file input to the model.
+ - `parameters: optional unknown`
- - `type: "input_file"`
+ - `strict: optional boolean`
- The type of the input item. Always `input_file`.
+ - `Custom object { name, type, defer_loading, 2 more }`
- - `"input_file"`
+ A custom tool that processes input using a specified format. Learn more about [custom tools](/docs/guides/function-calling#custom-tools)
- - `detail: optional "low" or "high"`
+ - `name: string`
- The detail level of the file to be sent to the model. Use `low` for the default rendering behavior, or `high` to render the file at higher quality. Defaults to `low`.
+ The name of the custom tool, used to identify it in tool calls.
- - `"low"`
+ - `type: "custom"`
- - `"high"`
+ The type of the custom tool. Always `custom`.
- - `file_data: optional string`
+ - `"custom"`
- The content of the file to be sent to the model.
+ - `defer_loading: optional boolean`
- - `file_id: optional string`
+ Whether this tool should be deferred and discovered via tool search.
- The ID of the file to be sent to the model.
+ - `description: optional string`
- - `file_url: optional string`
+ Optional description of the custom tool, used to provide more context.
- The URL of the file to be sent to the model.
+ - `format: optional CustomToolInputFormat`
- - `filename: optional string`
+ The input format for the custom tool. Default is unconstrained text.
- The name of the file to be sent to the model.
+ - `type: "namespace"`
- - `role: "user" or "assistant" or "system" or "developer"`
+ The type of the tool. Always `namespace`.
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
+ - `"namespace"`
- - `"user"`
+ - `ToolSearch object { type, description, execution, parameters }`
- - `"assistant"`
+ Hosted or BYOT tool search configuration for deferred tools.
- - `"system"`
+ - `type: "tool_search"`
- - `"developer"`
+ The type of the tool. Always `tool_search`.
- - `phase: optional "commentary" or "final_answer"`
+ - `"tool_search"`
- Labels an `assistant` message as intermediate commentary (`commentary`) or the final answer (`final_answer`).
- For models like `gpt-5.3-codex` and beyond, when sending follow-up requests, preserve and resend
- phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
+ - `description: optional string`
- - `"commentary"`
+ Description shown to the model for a client-executed tool search tool.
- - `"final_answer"`
+ - `execution: optional "server" or "client"`
- - `type: optional "message"`
+ Whether tool search is executed by the server or by the client.
- The type of the message input. Always `message`.
+ - `"server"`
- - `"message"`
+ - `"client"`
- - `EvalMessageObject = object { content, role, type }`
+ - `parameters: optional unknown`
- A message input to the model with a role indicating instruction following
- hierarchy. Instructions given with the `developer` or `system` role take
- precedence over instructions given with the `user` role. Messages with the
- `assistant` role are presumed to have been generated by the model in previous
- interactions.
+ Parameter schema for a client-executed tool search tool.
- - `content: string or ResponseInputText or object { text, type } or 3 more`
+ - `WebSearchPreview object { type, search_content_types, search_context_size, user_location }`
- Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
+ This tool searches the web for relevant results to use in a response. Learn more about the [web search tool](https://platform.openai.com/docs/guides/tools-web-search).
- - `TextInput = string`
+ - `type: "web_search_preview" or "web_search_preview_2025_03_11"`
- A text input to the model.
+ The type of the web search tool. One of `web_search_preview` or `web_search_preview_2025_03_11`.
- - `ResponseInputText = object { text, type }`
+ - `"web_search_preview"`
- A text input to the model.
+ - `"web_search_preview_2025_03_11"`
- - `text: string`
+ - `search_content_types: optional array of "text" or "image"`
- The text input to the model.
+ - `"text"`
- - `type: "input_text"`
+ - `"image"`
- The type of the input item. Always `input_text`.
+ - `search_context_size: optional "low" or "medium" or "high"`
- - `"input_text"`
+ High level guidance for the amount of context window space to use for the search. One of `low`, `medium`, or `high`. `medium` is the default.
- - `OutputText = object { text, type }`
+ - `"low"`
- A text output from the model.
+ - `"medium"`
- - `text: string`
+ - `"high"`
- The text output from the model.
+ - `user_location: optional object { type, city, country, 2 more }`
- - `type: "output_text"`
+ The user's location.
- The type of the output text. Always `output_text`.
+ - `type: "approximate"`
- - `"output_text"`
+ The type of location approximation. Always `approximate`.
- - `InputImage = object { image_url, type, detail }`
+ - `"approximate"`
- An image input block used within EvalItem content arrays.
+ - `city: optional string`
- - `image_url: string`
+ Free text input for the city of the user, e.g. `San Francisco`.
- The URL of the image input.
+ - `country: optional string`
- - `type: "input_image"`
+ The two-letter [ISO country code](https://en.wikipedia.org/wiki/ISO_3166-1) of the user, e.g. `US`.
- The type of the image input. Always `input_image`.
+ - `region: optional string`
- - `"input_image"`
+ Free text input for the region of the user, e.g. `California`.
- - `detail: optional string`
+ - `timezone: optional string`
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ The [IANA timezone](https://timeapi.io/documentation/iana-timezones) of the user, e.g. `America/Los_Angeles`.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `ApplyPatch object { type }`
- An audio input to the model.
+ Allows the assistant to create, delete, or update files using unified diffs.
- - `input_audio: object { data, format }`
+ - `type: "apply_patch"`
- - `data: string`
+ The type of the tool. Always `apply_patch`.
- Base64-encoded audio data.
+ - `"apply_patch"`
- - `format: "mp3" or "wav"`
+ - `top_p: optional number`
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- - `"mp3"`
+ - `error: EvalAPIError`
- - `"wav"`
+ An object representing an error response from the Eval API.
- - `type: "input_audio"`
+ - `code: string`
- The type of the input item. Always `input_audio`.
+ The error code.
- - `"input_audio"`
+ - `message: string`
- - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
+ The error message.
- A list of inputs, each of which may be either an input text, output text, input
- image, or input audio object.
+ - `eval_id: string`
- - `TextInput = string`
+ The identifier of the associated evaluation.
- A text input to the model.
+ - `metadata: Metadata`
- - `ResponseInputText = object { text, type }`
+ Set of 16 key-value pairs that can be attached to an object. This can be
+ useful for storing additional information about the object in a structured
+ format, and querying for objects via API or the dashboard.
- A text input to the model.
+ Keys are strings with a maximum length of 64 characters. Values are strings
+ with a maximum length of 512 characters.
- - `text: string`
+ - `model: string`
- The text input to the model.
+ The model that is evaluated, if applicable.
- - `type: "input_text"`
+ - `name: string`
- The type of the input item. Always `input_text`.
+ The name of the evaluation run.
- - `"input_text"`
+ - `object: "eval.run"`
- - `OutputText = object { text, type }`
+ The type of the object. Always "eval.run".
- A text output from the model.
+ - `"eval.run"`
- - `text: string`
+ - `per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }`
- The text output from the model.
+ Usage statistics for each model during the evaluation run.
- - `type: "output_text"`
+ - `cached_tokens: number`
- The type of the output text. Always `output_text`.
+ The number of tokens retrieved from cache.
- - `"output_text"`
+ - `completion_tokens: number`
- - `InputImage = object { image_url, type, detail }`
+ The number of completion tokens generated.
- An image input block used within EvalItem content arrays.
+ - `invocation_count: number`
- - `image_url: string`
+ The number of invocations.
- The URL of the image input.
+ - `model_name: string`
- - `type: "input_image"`
+ The name of the model.
- The type of the image input. Always `input_image`.
+ - `prompt_tokens: number`
- - `"input_image"`
+ The number of prompt tokens used.
- - `detail: optional string`
+ - `total_tokens: number`
- The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
+ The total number of tokens used.
- - `ResponseInputAudio = object { input_audio, type }`
+ - `per_testing_criteria_results: array of object { failed, passed, testing_criteria }`
- An audio input to the model.
+ Results per testing criteria applied during the evaluation run.
- - `input_audio: object { data, format }`
+ - `failed: number`
- - `data: string`
+ Number of tests failed for this criteria.
- Base64-encoded audio data.
+ - `passed: number`
- - `format: "mp3" or "wav"`
+ Number of tests passed for this criteria.
- The format of the audio data. Currently supported formats are `mp3` and
- `wav`.
+ - `testing_criteria: string`
- - `"mp3"`
+ A description of the testing criteria.
- - `"wav"`
+ - `report_url: string`
- - `type: "input_audio"`
+ The URL to the rendered evaluation run report on the UI dashboard.
- The type of the input item. Always `input_audio`.
+ - `result_counts: object { errored, failed, passed, total }`
- - `"input_audio"`
+ Counters summarizing the outcomes of the evaluation run.
- - `role: "user" or "assistant" or "system" or "developer"`
+ - `errored: number`
- The role of the message input. One of `user`, `assistant`, `system`, or
- `developer`.
+ Number of output items that resulted in an error.
- - `"user"`
+ - `failed: number`
- - `"assistant"`
+ Number of output items that failed to pass the evaluation.
- - `"system"`
+ - `passed: number`
- - `"developer"`
+ Number of output items that passed the evaluation.
- - `type: optional "message"`
+ - `total: number`
- The type of the message input. Always `message`.
+ Total number of executed output items.
- - `"message"`
+ - `status: string`
- - `type: "template"`
+ The status of the evaluation run.
- The type of input messages. Always `template`.
+### Run Delete Response
- - `"template"`
+- `RunDeleteResponse object { deleted, object, run_id }`
- - `ItemReferenceInputMessages = object { item_reference, type }`
+ - `deleted: optional boolean`
- - `item_reference: string`
+ - `object: optional string`
- A reference to a variable in the `item` namespace. Ie, "item.input_trajectory"
+ - `run_id: optional string`
- - `type: "item_reference"`
+# Output Items
- The type of input messages. Always `item_reference`.
+## Get eval run output items
- - `"item_reference"`
+**get** `/evals/{eval_id}/runs/{run_id}/output_items`
- - `model: optional string`
+Get a list of output items for an evaluation run.
- The name of the model to use for generating completions (e.g. "o3-mini").
+### Path Parameters
- - `sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }`
+- `eval_id: string`
- - `max_completion_tokens: optional number`
+- `run_id: string`
- The maximum number of tokens in the generated output.
+### Query Parameters
- - `reasoning_effort: optional ReasoningEffort`
+- `after: optional string`
- Constrains effort on reasoning for
- [reasoning models](https://platform.openai.com/docs/guides/reasoning).
- Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
- reasoning effort can result in faster responses and fewer tokens used
- on reasoning in a response.
+ Identifier for the last output item from the previous pagination request.
- - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
- - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
- - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
- - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
+- `limit: optional number`
- - `"none"`
+ Number of output items to retrieve.
- - `"minimal"`
+- `order: optional "asc" or "desc"`
- - `"low"`
+ Sort order for output items by timestamp. Use `asc` for ascending order or `desc` for descending order. Defaults to `asc`.
- - `"medium"`
+ - `"asc"`
- - `"high"`
+ - `"desc"`
- - `"xhigh"`
+- `status: optional "fail" or "pass"`
- - `response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject`
+ Filter output items by status. Use `failed` to filter by failed output
+ items or `pass` to filter by passed output items.
- An object specifying the format that the model must output.
+ - `"fail"`
- Setting to `{ "type": "json_schema", "json_schema": {...} }` enables
- Structured Outputs which ensures the model will match your supplied JSON
- schema. Learn more in the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ - `"pass"`
- Setting to `{ "type": "json_object" }` enables the older JSON mode, which
- ensures the message the model generates is valid JSON. Using `json_schema`
- is preferred for models that support it.
+### Returns
- - `ResponseFormatText = object { type }`
+- `data: array of object { id, created_at, datasource_item, 7 more }`
- Default response format. Used to generate text responses.
+ An array of eval run output item objects.
- - `type: "text"`
+ - `id: string`
- The type of response format being defined. Always `text`.
+ Unique identifier for the evaluation run output item.
- - `"text"`
+ - `created_at: number`
- - `ResponseFormatJSONSchema = object { json_schema, type }`
+ Unix timestamp (in seconds) when the evaluation run was created.
- JSON Schema response format. Used to generate structured JSON responses.
- Learn more about [Structured Outputs](/docs/guides/structured-outputs).
+ - `datasource_item: map[unknown]`
- - `json_schema: object { name, description, schema, strict }`
+ Details of the input data source item.
- Structured Outputs configuration options, including a JSON Schema.
+ - `datasource_item_id: number`
- - `name: string`
+ The identifier for the data source item.
- The name of the response format. Must be a-z, A-Z, 0-9, or contain
- underscores and dashes, with a maximum length of 64.
+ - `eval_id: string`
- - `description: optional string`
+ The identifier of the evaluation group.
- A description of what the response format is for, used by the model to
- determine how to respond in the format.
+ - `object: "eval.run.output_item"`
- - `schema: optional map[unknown]`
+ The type of the object. Always "eval.run.output_item".
- The schema for the response format, described as a JSON Schema object.
- Learn how to build JSON schemas [here](https://json-schema.org/).
+ - `"eval.run.output_item"`
- - `strict: optional boolean`
+ - `results: array of object { name, passed, score, 2 more }`
- Whether to enable strict schema adherence when generating the output.
- If set to true, the model will always follow the exact schema defined
- in the `schema` field. Only a subset of JSON Schema is supported when
- `strict` is `true`. To learn more, read the [Structured Outputs
- guide](/docs/guides/structured-outputs).
+ A list of grader results for this output item.
- - `type: "json_schema"`
+ - `name: string`
- The type of response format being defined. Always `json_schema`.
+ The name of the grader.
- - `"json_schema"`
+ - `passed: boolean`
- - `ResponseFormatJSONObject = object { type }`
+ Whether the grader considered the output a pass.
- JSON object response format. An older method of generating JSON responses.
- Using `json_schema` is recommended for models that support it. Note that the
- model will not generate JSON without a system or user message instructing it
- to do so.
+ - `score: number`
- - `type: "json_object"`
+ The numeric score produced by the grader.
- The type of response format being defined. Always `json_object`.
+ - `sample: optional map[unknown]`
- - `"json_object"`
+ Optional sample or intermediate data produced by the grader.
- - `seed: optional number`
+ - `type: optional string`
- A seed value to initialize the randomness, during sampling.
+ The grader type (for example, "string-check-grader").
- - `temperature: optional number`
+ - `run_id: string`
- A higher temperature increases randomness in the outputs.
+ The identifier of the evaluation run associated with this output item.
- - `tools: optional array of ChatCompletionFunctionTool`
+ - `sample: object { error, finish_reason, input, 7 more }`
- A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
+ A sample containing the input and output of the evaluation run.
- - `function: FunctionDefinition`
+ - `error: EvalAPIError`
- - `name: string`
+ An object representing an error response from the Eval API.
- The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
+ - `code: string`
- - `description: optional string`
+ The error code.
- A description of what the function does, used by the model to choose when and how to call the function.
+ - `message: string`
- - `parameters: optional FunctionParameters`
+ The error message.
- The parameters the functions accepts, described as a JSON Schema object. See the [guide](/docs/guides/function-calling) for examples, and the [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for documentation about the format.
+ - `finish_reason: string`
- Omitting `parameters` defines a function with an empty parameter list.
+ The reason why the sample generation was finished.
- - `strict: optional boolean`
+ - `input: array of object { content, role }`
- Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the `parameters` field. Only a subset of JSON Schema is supported when `strict` is `true`. Learn more about Structured Outputs in the [function calling guide](/docs/guides/function-calling).
+ An array of input messages.
- - `type: "function"`
+ - `content: string`
- The type of the tool. Currently, only `function` is supported.
+ The content of the message.
- - `"function"`
+ - `role: string`
- - `top_p: optional number`
+ The role of the message sender (e.g., system, user, developer).
- An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
+ - `max_completion_tokens: number`
-### Create Eval JSONL Run Data Source
+ The maximum number of tokens allowed for completion.
-- `CreateEvalJSONLRunDataSource = object { source, type }`
+ - `model: string`
- A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
+ The model used for generating the sample.
- - `source: object { content, type } or object { id, type }`
+ - `output: array of object { content, role }`
- Determines what populates the `item` namespace in the data source.
+ An array of output messages.
- - `EvalJSONLFileContentSource = object { content, type }`
+ - `content: optional string`
- - `content: array of object { item, sample }`
+ The content of the message.
- The content of the jsonl file.
+ - `role: optional string`
- - `item: map[unknown]`
+ The role of the message (e.g. "system", "assistant", "user").
- - `sample: optional map[unknown]`
+ - `seed: number`
- - `type: "file_content"`
+ The seed used for generating the sample.
- The type of jsonl source. Always `file_content`.
+ - `temperature: number`
- - `"file_content"`
+ The sampling temperature used.
- - `EvalJSONLFileIDSource = object { id, type }`
+ - `top_p: number`
- - `id: string`
+ The top_p value used for sampling.
- The identifier of the file.
+ - `usage: object { cached_tokens, completion_tokens, prompt_tokens, total_tokens }`
- - `type: "file_id"`
+ Token usage details for the sample.
- The type of jsonl source. Always `file_id`.
+ - `cached_tokens: number`
- - `"file_id"`
+ The number of tokens retrieved from cache.
- - `type: "jsonl"`
+ - `completion_tokens: number`
- The type of data source. Always `jsonl`.
+ The number of completion tokens generated.
- - `"jsonl"`
+ - `prompt_tokens: number`
-### Eval API Error
+ The number of prompt tokens used.
-- `EvalAPIError = object { code, message }`
+ - `total_tokens: number`
- An object representing an error response from the Eval API.
+ The total number of tokens used.
- - `code: string`
+ - `status: string`
- The error code.
+ The status of the evaluation run.
- - `message: string`
+- `first_id: string`
- The error message.
+ The identifier of the first eval run output item in the data array.
-# Output Items
+- `has_more: boolean`
-## Get eval run output items
+ Indicates whether there are more eval run output items available.
-**get** `/evals/{eval_id}/runs/{run_id}/output_items`
+- `last_id: string`
-Get a list of output items for an evaluation run.
+ The identifier of the last eval run output item in the data array.
-### Path Parameters
+- `object: "list"`
-- `eval_id: string`
+ The type of this object. It is always set to "list".
-- `run_id: string`
+ - `"list"`
-### Query Parameters
+### Example
-- `after: optional string`
+```http
+curl https://api.openai.com/v1/evals/$EVAL_ID/runs/$RUN_ID/output_items \
+ -H "Authorization: Bearer $OPENAI_API_KEY"
+```
- Identifier for the last output item from the previous pagination request.
+#### Response
-- `limit: optional number`
+```json
+{
+ "data": [
+ {
+ "id": "id",
+ "created_at": 0,
+ "datasource_item": {
+ "foo": "bar"
+ },
+ "datasource_item_id": 0,
+ "eval_id": "eval_id",
+ "object": "eval.run.output_item",
+ "results": [
+ {
+ "name": "name",
+ "passed": true,
+ "score": 0,
+ "sample": {
+ "foo": "bar"
+ },
+ "type": "type"
+ }
+ ],
+ "run_id": "run_id",
+ "sample": {
+ "error": {
+ "code": "code",
+ "message": "message"
+ },
+ "finish_reason": "finish_reason",
+ "input": [
+ {
+ "content": "content",
+ "role": "role"
+ }
+ ],
+ "max_completion_tokens": 0,
+ "model": "model",
+ "output": [
+ {
+ "content": "content",
+ "role": "role"
+ }
+ ],
+ "seed": 0,
+ "temperature": 0,
+ "top_p": 0,
+ "usage": {
+ "cached_tokens": 0,
+ "completion_tokens": 0,
+ "prompt_tokens": 0,
+ "total_tokens": 0
+ }
+ },
+ "status": "status"
+ }
+ ],
+ "first_id": "first_id",
+ "has_more": true,
+ "last_id": "last_id",
+ "object": "list"
+}
+```
- Number of output items to retrieve.
+### Example
-- `order: optional "asc" or "desc"`
+```http
+curl https://api.openai.com/v1/evals/egroup_67abd54d9b0081909a86353f6fb9317a/runs/erun_67abd54d60ec8190832b46859da808f7/output_items \
+ -H "Authorization: Bearer $OPENAI_API_KEY" \
+ -H "Content-Type: application/json"
+```
- Sort order for output items by timestamp. Use `asc` for ascending order or `desc` for descending order. Defaults to `asc`.
+#### Response
- - `"asc"`
+```json
+{
+ "object": "list",
+ "data": [
+ {
+ "object": "eval.run.output_item",
+ "id": "outputitem_67e5796c28e081909917bf79f6e6214d",
+ "created_at": 1743092076,
+ "run_id": "evalrun_67abd54d60ec8190832b46859da808f7",
+ "eval_id": "eval_67abd54d9b0081909a86353f6fb9317a",
+ "status": "pass",
+ "datasource_item_id": 5,
+ "datasource_item": {
+ "input": "Stock Markets Rally After Positive Economic Data Released",
+ "ground_truth": "Markets"
+ },
+ "results": [
+ {
+ "name": "String check-a2486074-d803-4445-b431-ad2262e85d47",
+ "sample": null,
+ "passed": true,
+ "score": 1.0
+ }
+ ],
+ "sample": {
+ "input": [
+ {
+ "role": "developer",
+ "content": "Categorize a given news headline into one of the following topics: Technology, Markets, World, Business, or Sports.\n\n# Steps\n\n1. Analyze the content of the news headline to understand its primary focus.\n2. Extract the subject matter, identifying any key indicators or keywords.\n3. Use the identified indicators to determine the most suitable category out of the five options: Technology, Markets, World, Business, or Sports.\n4. Ensure only one category is selected per headline.\n\n# Output Format\n\nRespond with the chosen category as a single word. For instance: \"Technology\", \"Markets\", \"World\", \"Business\", or \"Sports\".\n\n# Examples\n\n**Input**: \"Apple Unveils New iPhone Model, Featuring Advanced AI Features\" \n**Output**: \"Technology\"\n\n**Input**: \"Global Stocks Mixed as Investors Await Central Bank Decisions\" \n**Output**: \"Markets\"\n\n**Input**: \"War in Ukraine: Latest Updates on Negotiation Status\" \n**Output**: \"World\"\n\n**Input**: \"Microsoft in Talks to Acquire Gaming Company for $2 Billion\" \n**Output**: \"Business\"\n\n**Input**: \"Manchester United Secures Win in Premier League Football Match\" \n**Output**: \"Sports\" \n\n# Notes\n\n- If the headline appears to fit into more than one category, choose the most dominant theme.\n- Keywords or phrases such as \"stocks\", \"company acquisition\", \"match\", or technological brands can be good indicators for classification.\n",
+ "tool_call_id": null,
+ "tool_calls": null,
+ "function_call": null
+ },
+ {
+ "role": "user",
+ "content": "Stock Markets Rally After Positive Economic Data Released",
+ "tool_call_id": null,
+ "tool_calls": null,
+ "function_call": null
+ }
+ ],
+ "output": [
+ {
+ "role": "assistant",
+ "content": "Markets",
+ "tool_call_id": null,
+ "tool_calls": null,
+ "function_call": null
+ }
+ ],
+ "finish_reason": "stop",
+ "model": "gpt-4o-mini-2024-07-18",
+ "usage": {
+ "total_tokens": 325,
+ "completion_tokens": 2,
+ "prompt_tokens": 323,
+ "cached_tokens": 0
+ },
+ "error": null,
+ "temperature": 1.0,
+ "max_completion_tokens": 2048,
+ "top_p": 1.0,
+ "seed": 42
+ }
+ }
+ ],
+ "first_id": "outputitem_67e5796c28e081909917bf79f6e6214d",
+ "last_id": "outputitem_67e5796c28e081909917bf79f6e6214d",
+ "has_more": true
+}
+```
- - `"desc"`
+## Get an output item of an eval run
-- `status: optional "fail" or "pass"`
+**get** `/evals/{eval_id}/runs/{run_id}/output_items/{output_item_id}`
- Filter output items by status. Use `failed` to filter by failed output
- items or `pass` to filter by passed output items.
+Get an evaluation run output item by ID.
- - `"fail"`
+### Path Parameters
- - `"pass"`
+- `eval_id: string`
-### Returns
+- `run_id: string`
-- `data: array of object { id, created_at, datasource_item, 7 more }`
+- `output_item_id: string`
- An array of eval run output item objects.
+### Returns
- - `id: string`
+- `id: string`
Unique identifier for the evaluation run output item.
- - `created_at: number`
+- `created_at: number`
Unix timestamp (in seconds) when the evaluation run was created.
- - `datasource_item: map[unknown]`
+- `datasource_item: map[unknown]`
Details of the input data source item.
- - `datasource_item_id: number`
+- `datasource_item_id: number`
The identifier for the data source item.
- - `eval_id: string`
+- `eval_id: string`
The identifier of the evaluation group.
- - `object: "eval.run.output_item"`
+- `object: "eval.run.output_item"`
The type of the object. Always "eval.run.output_item".
- `"eval.run.output_item"`
- - `results: array of object { name, passed, score, 2 more }`
+- `results: array of object { name, passed, score, 2 more }`
A list of grader results for this output item.
@@ -15632,11 +23731,11 @@ Get a list of output items for an evaluation run.
The grader type (for example, "string-check-grader").
- - `run_id: string`
+- `run_id: string`
The identifier of the evaluation run associated with this output item.
- - `sample: object { error, finish_reason, input, 7 more }`
+- `sample: object { error, finish_reason, input, 7 more }`
A sample containing the input and output of the evaluation run.
@@ -15720,32 +23819,14 @@ Get a list of output items for an evaluation run.
The total number of tokens used.
- - `status: string`
+- `status: string`
The status of the evaluation run.
-- `first_id: string`
-
- The identifier of the first eval run output item in the data array.
-
-- `has_more: boolean`
-
- Indicates whether there are more eval run output items available.
-
-- `last_id: string`
-
- The identifier of the last eval run output item in the data array.
-
-- `object: "list"`
-
- The type of this object. It is always set to "list".
-
- - `"list"`
-
### Example
```http
-curl https://api.openai.com/v1/evals/$EVAL_ID/runs/$RUN_ID/output_items \
+curl https://api.openai.com/v1/evals/$EVAL_ID/runs/$RUN_ID/output_items/$OUTPUT_ITEM_ID \
-H "Authorization: Bearer $OPENAI_API_KEY"
```
@@ -15753,8 +23834,6 @@ curl https://api.openai.com/v1/evals/$EVAL_ID/runs/$RUN_ID/output_items \
```json
{
- "data": [
- {
"id": "id",
"created_at": 0,
"datasource_item": {
@@ -15806,140 +23885,266 @@ curl https://api.openai.com/v1/evals/$EVAL_ID/runs/$RUN_ID/output_items \
}
},
"status": "status"
+}
+```
+
+### Example
+
+```http
+curl https://api.openai.com/v1/evals/eval_67abd54d9b0081909a86353f6fb9317a/runs/evalrun_67abd54d60ec8190832b46859da808f7/output_items/outputitem_67abd55eb6548190bb580745d5644a33 \
+ -H "Authorization: Bearer $OPENAI_API_KEY" \
+ -H "Content-Type: application/json"
+```
+
+#### Response
+
+```json
+{
+ "object": "eval.run.output_item",
+ "id": "outputitem_67e5796c28e081909917bf79f6e6214d",
+ "created_at": 1743092076,
+ "run_id": "evalrun_67abd54d60ec8190832b46859da808f7",
+ "eval_id": "eval_67abd54d9b0081909a86353f6fb9317a",
+ "status": "pass",
+ "datasource_item_id": 5,
+ "datasource_item": {
+ "input": "Stock Markets Rally After Positive Economic Data Released",
+ "ground_truth": "Markets"
+ },
+ "results": [
+ {
+ "name": "String check-a2486074-d803-4445-b431-ad2262e85d47",
+ "sample": null,
+ "passed": true,
+ "score": 1.0
}
],
- "first_id": "first_id",
- "has_more": true,
- "last_id": "last_id",
- "object": "list"
+ "sample": {
+ "input": [
+ {
+ "role": "developer",
+ "content": "Categorize a given news headline into one of the following topics: Technology, Markets, World, Business, or Sports.\n\n# Steps\n\n1. Analyze the content of the news headline to understand its primary focus.\n2. Extract the subject matter, identifying any key indicators or keywords.\n3. Use the identified indicators to determine the most suitable category out of the five options: Technology, Markets, World, Business, or Sports.\n4. Ensure only one category is selected per headline.\n\n# Output Format\n\nRespond with the chosen category as a single word. For instance: \"Technology\", \"Markets\", \"World\", \"Business\", or \"Sports\".\n\n# Examples\n\n**Input**: \"Apple Unveils New iPhone Model, Featuring Advanced AI Features\" \n**Output**: \"Technology\"\n\n**Input**: \"Global Stocks Mixed as Investors Await Central Bank Decisions\" \n**Output**: \"Markets\"\n\n**Input**: \"War in Ukraine: Latest Updates on Negotiation Status\" \n**Output**: \"World\"\n\n**Input**: \"Microsoft in Talks to Acquire Gaming Company for $2 Billion\" \n**Output**: \"Business\"\n\n**Input**: \"Manchester United Secures Win in Premier League Football Match\" \n**Output**: \"Sports\" \n\n# Notes\n\n- If the headline appears to fit into more than one category, choose the most dominant theme.\n- Keywords or phrases such as \"stocks\", \"company acquisition\", \"match\", or technological brands can be good indicators for classification.\n",
+ "tool_call_id": null,
+ "tool_calls": null,
+ "function_call": null
+ },
+ {
+ "role": "user",
+ "content": "Stock Markets Rally After Positive Economic Data Released",
+ "tool_call_id": null,
+ "tool_calls": null,
+ "function_call": null
+ }
+ ],
+ "output": [
+ {
+ "role": "assistant",
+ "content": "Markets",
+ "tool_call_id": null,
+ "tool_calls": null,
+ "function_call": null
+ }
+ ],
+ "finish_reason": "stop",
+ "model": "gpt-4o-mini-2024-07-18",
+ "usage": {
+ "total_tokens": 325,
+ "completion_tokens": 2,
+ "prompt_tokens": 323,
+ "cached_tokens": 0
+ },
+ "error": null,
+ "temperature": 1.0,
+ "max_completion_tokens": 2048,
+ "top_p": 1.0,
+ "seed": 42
+ }
}
```
-### Example
+## Domain Types
+
+### Output Item List Response
+
+- `OutputItemListResponse object { id, created_at, datasource_item, 7 more }`
+
+ A schema representing an evaluation run output item.
+
+ - `id: string`
+
+ Unique identifier for the evaluation run output item.
+
+ - `created_at: number`
+
+ Unix timestamp (in seconds) when the evaluation run was created.
+
+ - `datasource_item: map[unknown]`
+
+ Details of the input data source item.
+
+ - `datasource_item_id: number`
+
+ The identifier for the data source item.
+
+ - `eval_id: string`
+
+ The identifier of the evaluation group.
+
+ - `object: "eval.run.output_item"`
+
+ The type of the object. Always "eval.run.output_item".
+
+ - `"eval.run.output_item"`
+
+ - `results: array of object { name, passed, score, 2 more }`
+
+ A list of grader results for this output item.
+
+ - `name: string`
+
+ The name of the grader.
+
+ - `passed: boolean`
+
+ Whether the grader considered the output a pass.
+
+ - `score: number`
+
+ The numeric score produced by the grader.
+
+ - `sample: optional map[unknown]`
+
+ Optional sample or intermediate data produced by the grader.
+
+ - `type: optional string`
+
+ The grader type (for example, "string-check-grader").
+
+ - `run_id: string`
+
+ The identifier of the evaluation run associated with this output item.
+
+ - `sample: object { error, finish_reason, input, 7 more }`
+
+ A sample containing the input and output of the evaluation run.
+
+ - `error: EvalAPIError`
+
+ An object representing an error response from the Eval API.
+
+ - `code: string`
+
+ The error code.
+
+ - `message: string`
+
+ The error message.
+
+ - `finish_reason: string`
+
+ The reason why the sample generation was finished.
+
+ - `input: array of object { content, role }`
+
+ An array of input messages.
+
+ - `content: string`
+
+ The content of the message.
+
+ - `role: string`
+
+ The role of the message sender (e.g., system, user, developer).
+
+ - `max_completion_tokens: number`
+
+ The maximum number of tokens allowed for completion.
+
+ - `model: string`
+
+ The model used for generating the sample.
+
+ - `output: array of object { content, role }`
+
+ An array of output messages.
+
+ - `content: optional string`
+
+ The content of the message.
+
+ - `role: optional string`
+
+ The role of the message (e.g. "system", "assistant", "user").
+
+ - `seed: number`
+
+ The seed used for generating the sample.
+
+ - `temperature: number`
+
+ The sampling temperature used.
+
+ - `top_p: number`
+
+ The top_p value used for sampling.
+
+ - `usage: object { cached_tokens, completion_tokens, prompt_tokens, total_tokens }`
+
+ Token usage details for the sample.
+
+ - `cached_tokens: number`
-```http
-curl https://api.openai.com/v1/evals/egroup_67abd54d9b0081909a86353f6fb9317a/runs/erun_67abd54d60ec8190832b46859da808f7/output_items \
- -H "Authorization: Bearer $OPENAI_API_KEY" \
- -H "Content-Type: application/json"
-```
+ The number of tokens retrieved from cache.
-#### Response
+ - `completion_tokens: number`
-```json
-{
- "object": "list",
- "data": [
- {
- "object": "eval.run.output_item",
- "id": "outputitem_67e5796c28e081909917bf79f6e6214d",
- "created_at": 1743092076,
- "run_id": "evalrun_67abd54d60ec8190832b46859da808f7",
- "eval_id": "eval_67abd54d9b0081909a86353f6fb9317a",
- "status": "pass",
- "datasource_item_id": 5,
- "datasource_item": {
- "input": "Stock Markets Rally After Positive Economic Data Released",
- "ground_truth": "Markets"
- },
- "results": [
- {
- "name": "String check-a2486074-d803-4445-b431-ad2262e85d47",
- "sample": null,
- "passed": true,
- "score": 1.0
- }
- ],
- "sample": {
- "input": [
- {
- "role": "developer",
- "content": "Categorize a given news headline into one of the following topics: Technology, Markets, World, Business, or Sports.\n\n# Steps\n\n1. Analyze the content of the news headline to understand its primary focus.\n2. Extract the subject matter, identifying any key indicators or keywords.\n3. Use the identified indicators to determine the most suitable category out of the five options: Technology, Markets, World, Business, or Sports.\n4. Ensure only one category is selected per headline.\n\n# Output Format\n\nRespond with the chosen category as a single word. For instance: \"Technology\", \"Markets\", \"World\", \"Business\", or \"Sports\".\n\n# Examples\n\n**Input**: \"Apple Unveils New iPhone Model, Featuring Advanced AI Features\" \n**Output**: \"Technology\"\n\n**Input**: \"Global Stocks Mixed as Investors Await Central Bank Decisions\" \n**Output**: \"Markets\"\n\n**Input**: \"War in Ukraine: Latest Updates on Negotiation Status\" \n**Output**: \"World\"\n\n**Input**: \"Microsoft in Talks to Acquire Gaming Company for $2 Billion\" \n**Output**: \"Business\"\n\n**Input**: \"Manchester United Secures Win in Premier League Football Match\" \n**Output**: \"Sports\" \n\n# Notes\n\n- If the headline appears to fit into more than one category, choose the most dominant theme.\n- Keywords or phrases such as \"stocks\", \"company acquisition\", \"match\", or technological brands can be good indicators for classification.\n",
- "tool_call_id": null,
- "tool_calls": null,
- "function_call": null
- },
- {
- "role": "user",
- "content": "Stock Markets Rally After Positive Economic Data Released",
- "tool_call_id": null,
- "tool_calls": null,
- "function_call": null
- }
- ],
- "output": [
- {
- "role": "assistant",
- "content": "Markets",
- "tool_call_id": null,
- "tool_calls": null,
- "function_call": null
- }
- ],
- "finish_reason": "stop",
- "model": "gpt-4o-mini-2024-07-18",
- "usage": {
- "total_tokens": 325,
- "completion_tokens": 2,
- "prompt_tokens": 323,
- "cached_tokens": 0
- },
- "error": null,
- "temperature": 1.0,
- "max_completion_tokens": 2048,
- "top_p": 1.0,
- "seed": 42
- }
- }
- ],
- "first_id": "outputitem_67e5796c28e081909917bf79f6e6214d",
- "last_id": "outputitem_67e5796c28e081909917bf79f6e6214d",
- "has_more": true
-}
-```
+ The number of completion tokens generated.
-## Get an output item of an eval run
+ - `prompt_tokens: number`
-**get** `/evals/{eval_id}/runs/{run_id}/output_items/{output_item_id}`
+ The number of prompt tokens used.
-Get an evaluation run output item by ID.
+ - `total_tokens: number`
-### Path Parameters
+ The total number of tokens used.
-- `eval_id: string`
+ - `status: string`
-- `run_id: string`
+ The status of the evaluation run.
-- `output_item_id: string`
+### Output Item Retrieve Response
-### Returns
+- `OutputItemRetrieveResponse object { id, created_at, datasource_item, 7 more }`
-- `id: string`
+ A schema representing an evaluation run output item.
+
+ - `id: string`
Unique identifier for the evaluation run output item.
-- `created_at: number`
+ - `created_at: number`
Unix timestamp (in seconds) when the evaluation run was created.
-- `datasource_item: map[unknown]`
+ - `datasource_item: map[unknown]`
Details of the input data source item.
-- `datasource_item_id: number`
+ - `datasource_item_id: number`
The identifier for the data source item.
-- `eval_id: string`
+ - `eval_id: string`
The identifier of the evaluation group.
-- `object: "eval.run.output_item"`
+ - `object: "eval.run.output_item"`
The type of the object. Always "eval.run.output_item".
- `"eval.run.output_item"`
-- `results: array of object { name, passed, score, 2 more }`
+ - `results: array of object { name, passed, score, 2 more }`
A list of grader results for this output item.
@@ -15963,11 +24168,11 @@ Get an evaluation run output item by ID.
The grader type (for example, "string-check-grader").
-- `run_id: string`
+ - `run_id: string`
The identifier of the evaluation run associated with this output item.
-- `sample: object { error, finish_reason, input, 7 more }`
+ - `sample: object { error, finish_reason, input, 7 more }`
A sample containing the input and output of the evaluation run.
@@ -16051,145 +24256,6 @@ Get an evaluation run output item by ID.
The total number of tokens used.
-- `status: string`
+ - `status: string`
The status of the evaluation run.
-
-### Example
-
-```http
-curl https://api.openai.com/v1/evals/$EVAL_ID/runs/$RUN_ID/output_items/$OUTPUT_ITEM_ID \
- -H "Authorization: Bearer $OPENAI_API_KEY"
-```
-
-#### Response
-
-```json
-{
- "id": "id",
- "created_at": 0,
- "datasource_item": {
- "foo": "bar"
- },
- "datasource_item_id": 0,
- "eval_id": "eval_id",
- "object": "eval.run.output_item",
- "results": [
- {
- "name": "name",
- "passed": true,
- "score": 0,
- "sample": {
- "foo": "bar"
- },
- "type": "type"
- }
- ],
- "run_id": "run_id",
- "sample": {
- "error": {
- "code": "code",
- "message": "message"
- },
- "finish_reason": "finish_reason",
- "input": [
- {
- "content": "content",
- "role": "role"
- }
- ],
- "max_completion_tokens": 0,
- "model": "model",
- "output": [
- {
- "content": "content",
- "role": "role"
- }
- ],
- "seed": 0,
- "temperature": 0,
- "top_p": 0,
- "usage": {
- "cached_tokens": 0,
- "completion_tokens": 0,
- "prompt_tokens": 0,
- "total_tokens": 0
- }
- },
- "status": "status"
-}
-```
-
-### Example
-
-```http
-curl https://api.openai.com/v1/evals/eval_67abd54d9b0081909a86353f6fb9317a/runs/evalrun_67abd54d60ec8190832b46859da808f7/output_items/outputitem_67abd55eb6548190bb580745d5644a33 \
- -H "Authorization: Bearer $OPENAI_API_KEY" \
- -H "Content-Type: application/json"
-```
-
-#### Response
-
-```json
-{
- "object": "eval.run.output_item",
- "id": "outputitem_67e5796c28e081909917bf79f6e6214d",
- "created_at": 1743092076,
- "run_id": "evalrun_67abd54d60ec8190832b46859da808f7",
- "eval_id": "eval_67abd54d9b0081909a86353f6fb9317a",
- "status": "pass",
- "datasource_item_id": 5,
- "datasource_item": {
- "input": "Stock Markets Rally After Positive Economic Data Released",
- "ground_truth": "Markets"
- },
- "results": [
- {
- "name": "String check-a2486074-d803-4445-b431-ad2262e85d47",
- "sample": null,
- "passed": true,
- "score": 1.0
- }
- ],
- "sample": {
- "input": [
- {
- "role": "developer",
- "content": "Categorize a given news headline into one of the following topics: Technology, Markets, World, Business, or Sports.\n\n# Steps\n\n1. Analyze the content of the news headline to understand its primary focus.\n2. Extract the subject matter, identifying any key indicators or keywords.\n3. Use the identified indicators to determine the most suitable category out of the five options: Technology, Markets, World, Business, or Sports.\n4. Ensure only one category is selected per headline.\n\n# Output Format\n\nRespond with the chosen category as a single word. For instance: \"Technology\", \"Markets\", \"World\", \"Business\", or \"Sports\".\n\n# Examples\n\n**Input**: \"Apple Unveils New iPhone Model, Featuring Advanced AI Features\" \n**Output**: \"Technology\"\n\n**Input**: \"Global Stocks Mixed as Investors Await Central Bank Decisions\" \n**Output**: \"Markets\"\n\n**Input**: \"War in Ukraine: Latest Updates on Negotiation Status\" \n**Output**: \"World\"\n\n**Input**: \"Microsoft in Talks to Acquire Gaming Company for $2 Billion\" \n**Output**: \"Business\"\n\n**Input**: \"Manchester United Secures Win in Premier League Football Match\" \n**Output**: \"Sports\" \n\n# Notes\n\n- If the headline appears to fit into more than one category, choose the most dominant theme.\n- Keywords or phrases such as \"stocks\", \"company acquisition\", \"match\", or technological brands can be good indicators for classification.\n",
- "tool_call_id": null,
- "tool_calls": null,
- "function_call": null
- },
- {
- "role": "user",
- "content": "Stock Markets Rally After Positive Economic Data Released",
- "tool_call_id": null,
- "tool_calls": null,
- "function_call": null
- }
- ],
- "output": [
- {
- "role": "assistant",
- "content": "Markets",
- "tool_call_id": null,
- "tool_calls": null,
- "function_call": null
- }
- ],
- "finish_reason": "stop",
- "model": "gpt-4o-mini-2024-07-18",
- "usage": {
- "total_tokens": 325,
- "completion_tokens": 2,
- "prompt_tokens": 323,
- "cached_tokens": 0
- },
- "error": null,
- "temperature": 1.0,
- "max_completion_tokens": 2048,
- "top_p": 1.0,
- "seed": 42
- }
-}
-```