Evals

List evals

get /evals

List evaluations for a project.

Query Parameters

• after: optional string

  Identifier for the last eval from the previous pagination request.

• limit: optional number

  Number of evals to retrieve.

• order: optional "asc" or "desc"

  Sort order for evals by timestamp. Use asc for ascending order or desc for descending order.

  • "asc"

  • "desc"

• order_by: optional "created_at" or "updated_at"

  Evals can be ordered by creation time or last updated time. Use created_at for creation time or updated_at for last updated time.

  • "created_at"

  • "updated_at"

Returns

• data: array of object { id, created_at, data_source_config, 4 more }

  An array of eval objects.

  • id: string

    Unique identifier for the evaluation.

  • created_at: number

    The Unix timestamp (in seconds) for when the eval was created.

  • data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig

    Configuration of data sources used in runs of the evaluation.

    • EvalCustomDataSourceConfig object { schema, type }

      A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

      • Used to define your testing criteria and

      • What data is required when creating a run

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "custom"

        The type of data source. Always custom.

        • "custom"

    • LogsDataSourceConfig object { schema, type, metadata }

      A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals. item and sample are both defined when using this data source config.

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "logs"

        The type of data source. Always logs.

        • "logs"

      • metadata: optional Metadata

        Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

        Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

    • EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }

      Deprecated in favor of LogsDataSourceConfig.

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "stored_completions"

        The type of data source. Always stored_completions.

        • "stored_completions"

      • metadata: optional Metadata

        Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

        Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • metadata: Metadata

    Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

    Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • name: string

    The name of the evaluation.

  • object: "eval"

    The object type.

    • "eval"

  • testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more

    A list of testing criteria.

    • LabelModelGrader object { input, labels, model, 3 more }

      A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

      • input: array of object { content, role, type }

        • content: string or ResponseInputText or object { text, type } or 3 more

          Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

          • TextInput = string

            A text input to the model.

          • ResponseInputText object { text, type }

            A text input to the model.

            • text: string

              The text input to the model.

            • type: "input_text"

              The type of the input item. Always input_text.

              • "input_text"

          • OutputText object { text, type }

            A text output from the model.

            • text: string

              The text output from the model.

            • type: "output_text"

              The type of the output text. Always output_text.

              • "output_text"

          • InputImage object { image_url, type, detail }

            An image input block used within EvalItem content arrays.

            • image_url: string

              The URL of the image input.

            • type: "input_image"

              The type of the image input. Always input_image.

              • "input_image"

            • detail: optional string

              The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

          • ResponseInputAudio object { input_audio, type }

            An audio input to the model.

            • input_audio: object { data, format }

              • data: string

                Base64-encoded audio data.

              • format: "mp3" or "wav"

                The format of the audio data. Currently supported formats are mp3 and wav.

                • "mp3"

                • "wav"

            • type: "input_audio"

              The type of the input item. Always input_audio.

              • "input_audio"

          • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

            A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

            • TextInput = string

              A text input to the model.

            • ResponseInputText object { text, type }

              A text input to the model.

            • OutputText object { text, type }

              A text output from the model.

              • text: string

                The text output from the model.

              • type: "output_text"

                The type of the output text. Always output_text.

                • "output_text"

            • InputImage object { image_url, type, detail }

              An image input block used within EvalItem content arrays.

              • image_url: string

                The URL of the image input.

              • type: "input_image"

                The type of the image input. Always input_image.

                • "input_image"

              • detail: optional string

                The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

            • ResponseInputAudio object { input_audio, type }

              An audio input to the model.

        • role: "user" or "assistant" or "system" or "developer"

          The role of the message input. One of user, assistant, system, or developer.

          • "user"

          • "assistant"

          • "system"

          • "developer"

        • type: optional "message"

          The type of the message input. Always message.

          • "message"

      • labels: array of string

        The labels to assign to each item in the evaluation.

      • model: string

        The model to use for the evaluation. Must support structured outputs.

      • name: string

        The name of the grader.

      • passing_labels: array of string

        The labels that indicate a passing result. Must be a subset of labels.

      • type: "label_model"

        The object type, which is always label_model.

        • "label_model"

    • StringCheckGrader object { input, name, operation, 2 more }

      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

      • input: string

        The input text. This may include template strings.

      • name: string

        The name of the grader.

      • operation: "eq" or "ne" or "like" or "ilike"

        The string check operation to perform. One of eq, ne, like, or ilike.

        • "eq"

        • "ne"

        • "like"

        • "ilike"

      • reference: string

        The reference text. This may include template strings.

      • type: "string_check"

        The object type, which is always string_check.

        • "string_check"

    • TextSimilarityGrader = TextSimilarityGrader

      A TextSimilarityGrader object which grades text based on similarity metrics.

      • pass_threshold: number

        The threshold for the score.

    • PythonGrader = PythonGrader

      A PythonGrader object that runs a python script on the input.

      • pass_threshold: optional number

        The threshold for the score.

    • ScoreModelGrader = ScoreModelGrader

      A ScoreModelGrader object that uses a model to assign a score to the input.

      • pass_threshold: optional number

        The threshold for the score.

• first_id: string

  The identifier of the first eval in the data array.

• has_more: boolean

  Indicates whether there are more evals available.

• last_id: string

  The identifier of the last eval in the data array.

• object: "list"

  The type of this object. It is always set to "list".

  • "list"

Example

curl https://api.openai.com/v1/evals \
    -H "Authorization: Bearer $OPENAI_API_KEY"

Response

{
  "data": [
    {
      "id": "id",
      "created_at": 0,
      "data_source_config": {
        "schema": {
          "foo": "bar"
        },
        "type": "custom"
      },
      "metadata": {
        "foo": "string"
      },
      "name": "Chatbot effectiveness Evaluation",
      "object": "eval",
      "testing_criteria": [
        {
          "input": [
            {
              "content": "string",
              "role": "user",
              "type": "message"
            }
          ],
          "labels": [
            "string"
          ],
          "model": "model",
          "name": "name",
          "passing_labels": [
            "string"
          ],
          "type": "label_model"
        }
      ]
    }
  ],
  "first_id": "first_id",
  "has_more": true,
  "last_id": "last_id",
  "object": "list"
}

Example

curl https://api.openai.com/v1/evals?limit=1 \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json"

Response

{
  "object": "list",
  "data": [
    {
      "id": "eval_67abd54d9b0081909a86353f6fb9317a",
      "object": "eval",
      "data_source_config": {
        "type": "stored_completions",
        "metadata": {
          "usecase": "push_notifications_summarizer"
        },
        "schema": {
          "type": "object",
          "properties": {
            "item": {
              "type": "object"
            },
            "sample": {
              "type": "object"
            }
          },
          "required": [
            "item",
            "sample"
          ]
        }
      },
      "testing_criteria": [
        {
          "name": "Push Notification Summary Grader",
          "id": "Push Notification Summary Grader-9b876f24-4762-4be9-aff4-db7a9b31c673",
          "type": "label_model",
          "model": "o3-mini",
          "input": [
            {
              "type": "message",
              "role": "developer",
              "content": {
                "type": "input_text",
                "text": "\nLabel the following push notification summary as either correct or incorrect.\nThe push notification and the summary will be provided below.\nA good push notificiation summary is concise and snappy.\nIf it is good, then label it as correct, if not, then incorrect.\n"
              }
            },
            {
              "type": "message",
              "role": "user",
              "content": {
                "type": "input_text",
                "text": "\nPush notifications: {{item.input}}\nSummary: {{sample.output_text}}\n"
              }
            }
          ],
          "passing_labels": [
            "correct"
          ],
          "labels": [
            "correct",
            "incorrect"
          ],
          "sampling_params": null
        }
      ],
      "name": "Push Notification Summary Grader",
      "created_at": 1739314509,
      "metadata": {
        "description": "A stored completions eval for push notification summaries"
      }
    }
  ],
  "first_id": "eval_67abd54d9b0081909a86353f6fb9317a",
  "last_id": "eval_67aa884cf6688190b58f657d4441c8b7",
  "has_more": true
}

Create eval

post /evals

Create the structure of an evaluation that can be used to test a model's performance. An evaluation is a set of testing criteria and the config for a data source, which dictates the schema of the data used in the evaluation. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources. For more information, see the Evals guide.

Body Parameters

• data_source_config: object { item_schema, type, include_sample_schema } or object { type, metadata } or object { type, metadata }

  The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.

  • CustomDataSourceConfig object { item_schema, type, include_sample_schema }

    A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs. This schema is used to define the shape of the data that will be:

    • Used to define your testing criteria and

    • What data is required when creating a run

    • item_schema: map[unknown]

      The json schema for each row in the data source.

    • type: "custom"

      The type of data source. Always custom.

      • "custom"

    • include_sample_schema: optional boolean

      Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)

  • LogsDataSourceConfig object { type, metadata }

    A data source config which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc.

    • type: "logs"

      The type of data source. Always logs.

      • "logs"

    • metadata: optional map[unknown]

      Metadata filters for the logs data source.

  • StoredCompletionsDataSourceConfig object { type, metadata }

    Deprecated in favor of LogsDataSourceConfig.

    • type: "stored_completions"

      The type of data source. Always stored_completions.

      • "stored_completions"

    • metadata: optional map[unknown]

      Metadata filters for the stored completions data source.

• testing_criteria: array of object { input, labels, model, 3 more } or StringCheckGrader or TextSimilarityGrader or 2 more

  A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like {{item.variable_name}}. To reference the model's output, use the sample namespace (ie, {{sample.output_text}}).

  • LabelModelGrader object { input, labels, model, 3 more }

    A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

    • input: array of object { content, role } or object { content, role, type }

      A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.

      • SimpleInputMessage object { content, role }

        • content: string

          The content of the message.

        • role: string

          The role of the message (e.g. "system", "assistant", "user").

      • EvalMessageObject object { content, role, type }

        A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.

        • content: string or ResponseInputText or object { text, type } or 3 more

          Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

          • TextInput = string

            A text input to the model.

          • ResponseInputText object { text, type }

            A text input to the model.

            • text: string

              The text input to the model.

            • type: "input_text"

              The type of the input item. Always input_text.

              • "input_text"

          • OutputText object { text, type }

            A text output from the model.

            • text: string

              The text output from the model.

            • type: "output_text"

              The type of the output text. Always output_text.

              • "output_text"

          • InputImage object { image_url, type, detail }

            An image input block used within EvalItem content arrays.

            • image_url: string

              The URL of the image input.

            • type: "input_image"

              The type of the image input. Always input_image.

              • "input_image"

            • detail: optional string

              The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

          • ResponseInputAudio object { input_audio, type }

            An audio input to the model.

            • input_audio: object { data, format }

              • data: string

                Base64-encoded audio data.

              • format: "mp3" or "wav"

                The format of the audio data. Currently supported formats are mp3 and wav.

                • "mp3"

                • "wav"

            • type: "input_audio"

              The type of the input item. Always input_audio.

              • "input_audio"

          • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

            A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

            • TextInput = string

              A text input to the model.

            • ResponseInputText object { text, type }

              A text input to the model.

            • OutputText object { text, type }

              A text output from the model.

              • text: string

                The text output from the model.

              • type: "output_text"

                The type of the output text. Always output_text.

                • "output_text"

            • InputImage object { image_url, type, detail }

              An image input block used within EvalItem content arrays.

              • image_url: string

                The URL of the image input.

              • type: "input_image"

                The type of the image input. Always input_image.

                • "input_image"

              • detail: optional string

                The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

            • ResponseInputAudio object { input_audio, type }

              An audio input to the model.

        • role: "user" or "assistant" or "system" or "developer"

          The role of the message input. One of user, assistant, system, or developer.

          • "user"

          • "assistant"

          • "system"

          • "developer"

        • type: optional "message"

          The type of the message input. Always message.

          • "message"

    • labels: array of string

      The labels to classify to each item in the evaluation.

    • model: string

      The model to use for the evaluation. Must support structured outputs.

    • name: string

      The name of the grader.

    • passing_labels: array of string

      The labels that indicate a passing result. Must be a subset of labels.

    • type: "label_model"

      The object type, which is always label_model.

      • "label_model"

  • StringCheckGrader object { input, name, operation, 2 more }

    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

    • input: string

      The input text. This may include template strings.

    • name: string

      The name of the grader.

    • operation: "eq" or "ne" or "like" or "ilike"

      The string check operation to perform. One of eq, ne, like, or ilike.

      • "eq"

      • "ne"

      • "like"

      • "ilike"

    • reference: string

      The reference text. This may include template strings.

    • type: "string_check"

      The object type, which is always string_check.

      • "string_check"

  • TextSimilarityGrader = TextSimilarityGrader

    A TextSimilarityGrader object which grades text based on similarity metrics.

    • pass_threshold: number

      The threshold for the score.

  • PythonGrader = PythonGrader

    A PythonGrader object that runs a python script on the input.

    • pass_threshold: optional number

      The threshold for the score.

  • ScoreModelGrader = ScoreModelGrader

    A ScoreModelGrader object that uses a model to assign a score to the input.

    • pass_threshold: optional number

      The threshold for the score.

• metadata: optional Metadata

  Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

  Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

• name: optional string

  The name of the evaluation.

Returns

• id: string

  Unique identifier for the evaluation.

• created_at: number

  The Unix timestamp (in seconds) for when the eval was created.

• data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig

  Configuration of data sources used in runs of the evaluation.

  • EvalCustomDataSourceConfig object { schema, type }

    A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

    • Used to define your testing criteria and

    • What data is required when creating a run

    • schema: map[unknown]

      The json schema for the run data source items. Learn how to build JSON schemas here.

    • type: "custom"

      The type of data source. Always custom.

      • "custom"

  • LogsDataSourceConfig object { schema, type, metadata }

    A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals. item and sample are both defined when using this data source config.

    • schema: map[unknown]

      The json schema for the run data source items. Learn how to build JSON schemas here.

    • type: "logs"

      The type of data source. Always logs.

      • "logs"

    • metadata: optional Metadata

      Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

      Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }

    Deprecated in favor of LogsDataSourceConfig.

    • schema: map[unknown]

      The json schema for the run data source items. Learn how to build JSON schemas here.

    • type: "stored_completions"

      The type of data source. Always stored_completions.

      • "stored_completions"

    • metadata: optional Metadata

      Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

      Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

• metadata: Metadata

  Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

  Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

• name: string

  The name of the evaluation.

• object: "eval"

  The object type.

  • "eval"

• testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more

  A list of testing criteria.

  • LabelModelGrader object { input, labels, model, 3 more }

    A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

    • input: array of object { content, role, type }

      • content: string or ResponseInputText or object { text, type } or 3 more

        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

        • TextInput = string

          A text input to the model.

        • ResponseInputText object { text, type }

          A text input to the model.

          • text: string

            The text input to the model.

          • type: "input_text"

            The type of the input item. Always input_text.

            • "input_text"

        • OutputText object { text, type }

          A text output from the model.

          • text: string

            The text output from the model.

          • type: "output_text"

            The type of the output text. Always output_text.

            • "output_text"

        • InputImage object { image_url, type, detail }

          An image input block used within EvalItem content arrays.

          • image_url: string

            The URL of the image input.

          • type: "input_image"

            The type of the image input. Always input_image.

            • "input_image"

          • detail: optional string

            The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

        • ResponseInputAudio object { input_audio, type }

          An audio input to the model.

          • input_audio: object { data, format }

            • data: string

              Base64-encoded audio data.

            • format: "mp3" or "wav"

              The format of the audio data. Currently supported formats are mp3 and wav.

              • "mp3"

              • "wav"

          • type: "input_audio"

            The type of the input item. Always input_audio.

            • "input_audio"

        • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

          A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

          • TextInput = string

            A text input to the model.

          • ResponseInputText object { text, type }

            A text input to the model.

          • OutputText object { text, type }

            A text output from the model.

            • text: string

              The text output from the model.

            • type: "output_text"

              The type of the output text. Always output_text.

              • "output_text"

          • InputImage object { image_url, type, detail }

            An image input block used within EvalItem content arrays.

            • image_url: string

              The URL of the image input.

            • type: "input_image"

              The type of the image input. Always input_image.

              • "input_image"

            • detail: optional string

              The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

          • ResponseInputAudio object { input_audio, type }

            An audio input to the model.

      • role: "user" or "assistant" or "system" or "developer"

        The role of the message input. One of user, assistant, system, or developer.

        • "user"

        • "assistant"

        • "system"

        • "developer"

      • type: optional "message"

        The type of the message input. Always message.

        • "message"

    • labels: array of string

      The labels to assign to each item in the evaluation.

    • model: string

      The model to use for the evaluation. Must support structured outputs.

    • name: string

      The name of the grader.

    • passing_labels: array of string

      The labels that indicate a passing result. Must be a subset of labels.

    • type: "label_model"

      The object type, which is always label_model.

      • "label_model"

  • StringCheckGrader object { input, name, operation, 2 more }

    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

    • input: string

      The input text. This may include template strings.

    • name: string

      The name of the grader.

    • operation: "eq" or "ne" or "like" or "ilike"

      The string check operation to perform. One of eq, ne, like, or ilike.

      • "eq"

      • "ne"

      • "like"

      • "ilike"

    • reference: string

      The reference text. This may include template strings.

    • type: "string_check"

      The object type, which is always string_check.

      • "string_check"

  • TextSimilarityGrader = TextSimilarityGrader

    A TextSimilarityGrader object which grades text based on similarity metrics.

    • pass_threshold: number

      The threshold for the score.

  • PythonGrader = PythonGrader

    A PythonGrader object that runs a python script on the input.

    • pass_threshold: optional number

      The threshold for the score.

  • ScoreModelGrader = ScoreModelGrader

    A ScoreModelGrader object that uses a model to assign a score to the input.

    • pass_threshold: optional number

      The threshold for the score.

Example

curl https://api.openai.com/v1/evals \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -d '{
          "data_source_config": {
            "item_schema": {
              "foo": "bar"
            },
            "type": "custom"
          },
          "testing_criteria": [
            {
              "input": [
                {
                  "content": "content",
                  "role": "role"
                }
              ],
              "labels": [
                "string"
              ],
              "model": "model",
              "name": "name",
              "passing_labels": [
                "string"
              ],
              "type": "label_model"
            }
          ]
        }'

Response

{
  "id": "id",
  "created_at": 0,
  "data_source_config": {
    "schema": {
      "foo": "bar"
    },
    "type": "custom"
  },
  "metadata": {
    "foo": "string"
  },
  "name": "Chatbot effectiveness Evaluation",
  "object": "eval",
  "testing_criteria": [
    {
      "input": [
        {
          "content": "string",
          "role": "user",
          "type": "message"
        }
      ],
      "labels": [
        "string"
      ],
      "model": "model",
      "name": "name",
      "passing_labels": [
        "string"
      ],
      "type": "label_model"
    }
  ]
}

Example

curl https://api.openai.com/v1/evals \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "Sentiment",
        "data_source_config": {
          "type": "stored_completions",
          "metadata": {
              "usecase": "chatbot"
          }
        },
        "testing_criteria": [
          {
            "type": "label_model",
            "model": "o3-mini",
            "input": [
              {
                "role": "developer",
                "content": "Classify the sentiment of the following statement as one of 'positive', 'neutral', or 'negative'"
              },
              {
                "role": "user",
                "content": "Statement: {{item.input}}"
              }
            ],
            "passing_labels": [
              "positive"
            ],
            "labels": [
              "positive",
              "neutral",
              "negative"
            ],
            "name": "Example label grader"
          }
        ]
      }'

Response

{
  "object": "eval",
  "id": "eval_67b7fa9a81a88190ab4aa417e397ea21",
  "data_source_config": {
    "type": "stored_completions",
    "metadata": {
      "usecase": "chatbot"
    },
    "schema": {
      "type": "object",
      "properties": {
        "item": {
          "type": "object"
        },
        "sample": {
          "type": "object"
        }
      },
      "required": [
        "item",
        "sample"
      ]
  },
  "testing_criteria": [
    {
      "name": "Example label grader",
      "type": "label_model",
      "model": "o3-mini",
      "input": [
        {
          "type": "message",
          "role": "developer",
          "content": {
            "type": "input_text",
            "text": "Classify the sentiment of the following statement as one of positive, neutral, or negative"
          }
        },
        {
          "type": "message",
          "role": "user",
          "content": {
            "type": "input_text",
            "text": "Statement: {{item.input}}"
          }
        }
      ],
      "passing_labels": [
        "positive"
      ],
      "labels": [
        "positive",
        "neutral",
        "negative"
      ]
    }
  ],
  "name": "Sentiment",
  "created_at": 1740110490,
  "metadata": {
    "description": "An eval for sentiment analysis"
  }
}

Get an eval

get /evals/{eval_id}

Get an evaluation by ID.

Path Parameters

• eval_id: string

Returns

• id: string

  Unique identifier for the evaluation.

• created_at: number

  The Unix timestamp (in seconds) for when the eval was created.

• data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig

  Configuration of data sources used in runs of the evaluation.

  • EvalCustomDataSourceConfig object { schema, type }

    A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

    • Used to define your testing criteria and

    • What data is required when creating a run

    • schema: map[unknown]

      The json schema for the run data source items. Learn how to build JSON schemas here.

    • type: "custom"

      The type of data source. Always custom.

      • "custom"

  • LogsDataSourceConfig object { schema, type, metadata }

    A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals. item and sample are both defined when using this data source config.

    • schema: map[unknown]

      The json schema for the run data source items. Learn how to build JSON schemas here.

    • type: "logs"

      The type of data source. Always logs.

      • "logs"

    • metadata: optional Metadata

      Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

      Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }

    Deprecated in favor of LogsDataSourceConfig.

    • schema: map[unknown]

      The json schema for the run data source items. Learn how to build JSON schemas here.

    • type: "stored_completions"

      The type of data source. Always stored_completions.

      • "stored_completions"

    • metadata: optional Metadata

      Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

      Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

• metadata: Metadata

  Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

  Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

• name: string

  The name of the evaluation.

• object: "eval"

  The object type.

  • "eval"

• testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more

  A list of testing criteria.

  • LabelModelGrader object { input, labels, model, 3 more }

    A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

    • input: array of object { content, role, type }

      • content: string or ResponseInputText or object { text, type } or 3 more

        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

        • TextInput = string

          A text input to the model.

        • ResponseInputText object { text, type }

          A text input to the model.

          • text: string

            The text input to the model.

          • type: "input_text"

            The type of the input item. Always input_text.

            • "input_text"

        • OutputText object { text, type }

          A text output from the model.

          • text: string

            The text output from the model.

          • type: "output_text"

            The type of the output text. Always output_text.

            • "output_text"

        • InputImage object { image_url, type, detail }

          An image input block used within EvalItem content arrays.

          • image_url: string

            The URL of the image input.

          • type: "input_image"

            The type of the image input. Always input_image.

            • "input_image"

          • detail: optional string

            The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

        • ResponseInputAudio object { input_audio, type }

          An audio input to the model.

          • input_audio: object { data, format }

            • data: string

              Base64-encoded audio data.

            • format: "mp3" or "wav"

              The format of the audio data. Currently supported formats are mp3 and wav.

              • "mp3"

              • "wav"

          • type: "input_audio"

            The type of the input item. Always input_audio.

            • "input_audio"

        • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

          A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

          • TextInput = string

            A text input to the model.

          • ResponseInputText object { text, type }

            A text input to the model.

          • OutputText object { text, type }

            A text output from the model.

            • text: string

              The text output from the model.

            • type: "output_text"

              The type of the output text. Always output_text.

              • "output_text"

          • InputImage object { image_url, type, detail }

            An image input block used within EvalItem content arrays.

            • image_url: string

              The URL of the image input.

            • type: "input_image"

              The type of the image input. Always input_image.

              • "input_image"

            • detail: optional string

              The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

          • ResponseInputAudio object { input_audio, type }

            An audio input to the model.

      • role: "user" or "assistant" or "system" or "developer"

        The role of the message input. One of user, assistant, system, or developer.

        • "user"

        • "assistant"

        • "system"

        • "developer"

      • type: optional "message"

        The type of the message input. Always message.

        • "message"

    • labels: array of string

      The labels to assign to each item in the evaluation.

    • model: string

      The model to use for the evaluation. Must support structured outputs.

    • name: string

      The name of the grader.

    • passing_labels: array of string

      The labels that indicate a passing result. Must be a subset of labels.

    • type: "label_model"

      The object type, which is always label_model.

      • "label_model"

  • StringCheckGrader object { input, name, operation, 2 more }

    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

    • input: string

      The input text. This may include template strings.

    • name: string

      The name of the grader.

    • operation: "eq" or "ne" or "like" or "ilike"

      The string check operation to perform. One of eq, ne, like, or ilike.

      • "eq"

      • "ne"

      • "like"

      • "ilike"

    • reference: string

      The reference text. This may include template strings.

    • type: "string_check"

      The object type, which is always string_check.

      • "string_check"

  • TextSimilarityGrader = TextSimilarityGrader

    A TextSimilarityGrader object which grades text based on similarity metrics.

    • pass_threshold: number

      The threshold for the score.

  • PythonGrader = PythonGrader

    A PythonGrader object that runs a python script on the input.

    • pass_threshold: optional number

      The threshold for the score.

  • ScoreModelGrader = ScoreModelGrader

    A ScoreModelGrader object that uses a model to assign a score to the input.

    • pass_threshold: optional number

      The threshold for the score.

Example

curl https://api.openai.com/v1/evals/$EVAL_ID \
    -H "Authorization: Bearer $OPENAI_API_KEY"

Response

{
  "id": "id",
  "created_at": 0,
  "data_source_config": {
    "schema": {
      "foo": "bar"
    },
    "type": "custom"
  },
  "metadata": {
    "foo": "string"
  },
  "name": "Chatbot effectiveness Evaluation",
  "object": "eval",
  "testing_criteria": [
    {
      "input": [
        {
          "content": "string",
          "role": "user",
          "type": "message"
        }
      ],
      "labels": [
        "string"
      ],
      "model": "model",
      "name": "name",
      "passing_labels": [
        "string"
      ],
      "type": "label_model"
    }
  ]
}

Example

curl https://api.openai.com/v1/evals/eval_67abd54d9b0081909a86353f6fb9317a \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json"

Response

{
  "object": "eval",
  "id": "eval_67abd54d9b0081909a86353f6fb9317a",
  "data_source_config": {
    "type": "custom",
    "schema": {
      "type": "object",
      "properties": {
        "item": {
          "type": "object",
          "properties": {
            "input": {
              "type": "string"
            },
            "ground_truth": {
              "type": "string"
            }
          },
          "required": [
            "input",
            "ground_truth"
          ]
        }
      },
      "required": [
        "item"
      ]
    }
  },
  "testing_criteria": [
    {
      "name": "String check",
      "id": "String check-2eaf2d8d-d649-4335-8148-9535a7ca73c2",
      "type": "string_check",
      "input": "{{item.input}}",
      "reference": "{{item.ground_truth}}",
      "operation": "eq"
    }
  ],
  "name": "External Data Eval",
  "created_at": 1739314509,
  "metadata": {},
}

Update an eval

post /evals/{eval_id}

Update certain properties of an evaluation.

Path Parameters

• eval_id: string

Body Parameters

• metadata: optional Metadata

  Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

  Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

• name: optional string

  Rename the evaluation.

Returns

• id: string

  Unique identifier for the evaluation.

• created_at: number

  The Unix timestamp (in seconds) for when the eval was created.

• data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig

  Configuration of data sources used in runs of the evaluation.

  • EvalCustomDataSourceConfig object { schema, type }

    A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

    • Used to define your testing criteria and

    • What data is required when creating a run

    • schema: map[unknown]

      The json schema for the run data source items. Learn how to build JSON schemas here.

    • type: "custom"

      The type of data source. Always custom.

      • "custom"

  • LogsDataSourceConfig object { schema, type, metadata }

    A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals. item and sample are both defined when using this data source config.

    • schema: map[unknown]

      The json schema for the run data source items. Learn how to build JSON schemas here.

    • type: "logs"

      The type of data source. Always logs.

      • "logs"

    • metadata: optional Metadata

      Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

      Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }

    Deprecated in favor of LogsDataSourceConfig.

    • schema: map[unknown]

      The json schema for the run data source items. Learn how to build JSON schemas here.

    • type: "stored_completions"

      The type of data source. Always stored_completions.

      • "stored_completions"

    • metadata: optional Metadata

      Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

      Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

• metadata: Metadata

  Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

  Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

• name: string

  The name of the evaluation.

• object: "eval"

  The object type.

  • "eval"

• testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more

  A list of testing criteria.

  • LabelModelGrader object { input, labels, model, 3 more }

    A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

    • input: array of object { content, role, type }

      • content: string or ResponseInputText or object { text, type } or 3 more

        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

        • TextInput = string

          A text input to the model.

        • ResponseInputText object { text, type }

          A text input to the model.

          • text: string

            The text input to the model.

          • type: "input_text"

            The type of the input item. Always input_text.

            • "input_text"

        • OutputText object { text, type }

          A text output from the model.

          • text: string

            The text output from the model.

          • type: "output_text"

            The type of the output text. Always output_text.

            • "output_text"

        • InputImage object { image_url, type, detail }

          An image input block used within EvalItem content arrays.

          • image_url: string

            The URL of the image input.

          • type: "input_image"

            The type of the image input. Always input_image.

            • "input_image"

          • detail: optional string

            The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

        • ResponseInputAudio object { input_audio, type }

          An audio input to the model.

          • input_audio: object { data, format }

            • data: string

              Base64-encoded audio data.

            • format: "mp3" or "wav"

              The format of the audio data. Currently supported formats are mp3 and wav.

              • "mp3"

              • "wav"

          • type: "input_audio"

            The type of the input item. Always input_audio.

            • "input_audio"

        • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

          A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

          • TextInput = string

            A text input to the model.

          • ResponseInputText object { text, type }

            A text input to the model.

          • OutputText object { text, type }

            A text output from the model.

            • text: string

              The text output from the model.

            • type: "output_text"

              The type of the output text. Always output_text.

              • "output_text"

          • InputImage object { image_url, type, detail }

            An image input block used within EvalItem content arrays.

            • image_url: string

              The URL of the image input.

            • type: "input_image"

              The type of the image input. Always input_image.

              • "input_image"

            • detail: optional string

              The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

          • ResponseInputAudio object { input_audio, type }

            An audio input to the model.

      • role: "user" or "assistant" or "system" or "developer"

        The role of the message input. One of user, assistant, system, or developer.

        • "user"

        • "assistant"

        • "system"

        • "developer"

      • type: optional "message"

        The type of the message input. Always message.

        • "message"

    • labels: array of string

      The labels to assign to each item in the evaluation.

    • model: string

      The model to use for the evaluation. Must support structured outputs.

    • name: string

      The name of the grader.

    • passing_labels: array of string

      The labels that indicate a passing result. Must be a subset of labels.

    • type: "label_model"

      The object type, which is always label_model.

      • "label_model"

  • StringCheckGrader object { input, name, operation, 2 more }

    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

    • input: string

      The input text. This may include template strings.

    • name: string

      The name of the grader.

    • operation: "eq" or "ne" or "like" or "ilike"

      The string check operation to perform. One of eq, ne, like, or ilike.

      • "eq"

      • "ne"

      • "like"

      • "ilike"

    • reference: string

      The reference text. This may include template strings.

    • type: "string_check"

      The object type, which is always string_check.

      • "string_check"

  • TextSimilarityGrader = TextSimilarityGrader

    A TextSimilarityGrader object which grades text based on similarity metrics.

    • pass_threshold: number

      The threshold for the score.

  • PythonGrader = PythonGrader

    A PythonGrader object that runs a python script on the input.

    • pass_threshold: optional number

      The threshold for the score.

  • ScoreModelGrader = ScoreModelGrader

    A ScoreModelGrader object that uses a model to assign a score to the input.

    • pass_threshold: optional number

      The threshold for the score.

Example

curl https://api.openai.com/v1/evals/$EVAL_ID \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -d '{}'

Response

{
  "id": "id",
  "created_at": 0,
  "data_source_config": {
    "schema": {
      "foo": "bar"
    },
    "type": "custom"
  },
  "metadata": {
    "foo": "string"
  },
  "name": "Chatbot effectiveness Evaluation",
  "object": "eval",
  "testing_criteria": [
    {
      "input": [
        {
          "content": "string",
          "role": "user",
          "type": "message"
        }
      ],
      "labels": [
        "string"
      ],
      "model": "model",
      "name": "name",
      "passing_labels": [
        "string"
      ],
      "type": "label_model"
    }
  ]
}

Example

curl https://api.openai.com/v1/evals/eval_67abd54d9b0081909a86353f6fb9317a \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "Updated Eval", "metadata": {"description": "Updated description"}}'

Response

{
  "object": "eval",
  "id": "eval_67abd54d9b0081909a86353f6fb9317a",
  "data_source_config": {
    "type": "custom",
    "schema": {
      "type": "object",
      "properties": {
        "item": {
          "type": "object",
          "properties": {
            "input": {
              "type": "string"
            },
            "ground_truth": {
              "type": "string"
            }
          },
          "required": [
            "input",
            "ground_truth"
          ]
        }
      },
      "required": [
        "item"
      ]
    }
  },
  "testing_criteria": [
    {
      "name": "String check",
      "id": "String check-2eaf2d8d-d649-4335-8148-9535a7ca73c2",
      "type": "string_check",
      "input": "{{item.input}}",
      "reference": "{{item.ground_truth}}",
      "operation": "eq"
    }
  ],
  "name": "Updated Eval",
  "created_at": 1739314509,
  "metadata": {"description": "Updated description"},
}

Delete an eval

delete /evals/{eval_id}

Delete an evaluation.

Path Parameters

• eval_id: string

Returns

• deleted: boolean

• eval_id: string

• object: string

Example

curl https://api.openai.com/v1/evals/$EVAL_ID \
    -X DELETE \
    -H "Authorization: Bearer $OPENAI_API_KEY"

Response

{
  "deleted": true,
  "eval_id": "eval_abc123",
  "object": "eval.deleted"
}

Example

curl https://api.openai.com/v1/evals/eval_abc123 \
  -X DELETE \
  -H "Authorization: Bearer $OPENAI_API_KEY"

Response

{
  "object": "eval.deleted",
  "deleted": true,
  "eval_id": "eval_abc123"
}

Domain Types

Eval Custom Data Source Config

• EvalCustomDataSourceConfig object { schema, type }

  A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

  • Used to define your testing criteria and

  • What data is required when creating a run

  • schema: map[unknown]

    The json schema for the run data source items. Learn how to build JSON schemas here.

  • type: "custom"

    The type of data source. Always custom.

    • "custom"

Eval Stored Completions Data Source Config

• EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }

  Deprecated in favor of LogsDataSourceConfig.

  • schema: map[unknown]

    The json schema for the run data source items. Learn how to build JSON schemas here.

  • type: "stored_completions"

    The type of data source. Always stored_completions.

    • "stored_completions"

  • metadata: optional Metadata

    Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

    Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

Eval List Response

• EvalListResponse object { id, created_at, data_source_config, 4 more }

  An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot

  • See how well my chatbot handles customer support

  • Check if o4-mini is better at my usecase than gpt-4o

  • id: string

    Unique identifier for the evaluation.

  • created_at: number

    The Unix timestamp (in seconds) for when the eval was created.

  • data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig

    Configuration of data sources used in runs of the evaluation.

    • EvalCustomDataSourceConfig object { schema, type }

      A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

      • Used to define your testing criteria and

      • What data is required when creating a run

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "custom"

        The type of data source. Always custom.

        • "custom"

    • LogsDataSourceConfig object { schema, type, metadata }

      A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals. item and sample are both defined when using this data source config.

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "logs"

        The type of data source. Always logs.

        • "logs"

      • metadata: optional Metadata

        Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

        Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

    • EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }

      Deprecated in favor of LogsDataSourceConfig.

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "stored_completions"

        The type of data source. Always stored_completions.

        • "stored_completions"

      • metadata: optional Metadata

        Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

        Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • metadata: Metadata

    Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

    Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • name: string

    The name of the evaluation.

  • object: "eval"

    The object type.

    • "eval"

  • testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more

    A list of testing criteria.

    • LabelModelGrader object { input, labels, model, 3 more }

      A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

      • input: array of object { content, role, type }

        • content: string or ResponseInputText or object { text, type } or 3 more

          Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

          • TextInput = string

            A text input to the model.

          • ResponseInputText object { text, type }

            A text input to the model.

            • text: string

              The text input to the model.

            • type: "input_text"

              The type of the input item. Always input_text.

              • "input_text"

          • OutputText object { text, type }

            A text output from the model.

            • text: string

              The text output from the model.

            • type: "output_text"

              The type of the output text. Always output_text.

              • "output_text"

          • InputImage object { image_url, type, detail }

            An image input block used within EvalItem content arrays.

            • image_url: string

              The URL of the image input.

            • type: "input_image"

              The type of the image input. Always input_image.

              • "input_image"

            • detail: optional string

              The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

          • ResponseInputAudio object { input_audio, type }

            An audio input to the model.

            • input_audio: object { data, format }

              • data: string

                Base64-encoded audio data.

              • format: "mp3" or "wav"

                The format of the audio data. Currently supported formats are mp3 and wav.

                • "mp3"

                • "wav"

            • type: "input_audio"

              The type of the input item. Always input_audio.

              • "input_audio"

          • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

            A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

            • TextInput = string

              A text input to the model.

            • ResponseInputText object { text, type }

              A text input to the model.

            • OutputText object { text, type }

              A text output from the model.

              • text: string

                The text output from the model.

              • type: "output_text"

                The type of the output text. Always output_text.

                • "output_text"

            • InputImage object { image_url, type, detail }

              An image input block used within EvalItem content arrays.

              • image_url: string

                The URL of the image input.

              • type: "input_image"

                The type of the image input. Always input_image.

                • "input_image"

              • detail: optional string

                The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

            • ResponseInputAudio object { input_audio, type }

              An audio input to the model.

        • role: "user" or "assistant" or "system" or "developer"

          The role of the message input. One of user, assistant, system, or developer.

          • "user"

          • "assistant"

          • "system"

          • "developer"

        • type: optional "message"

          The type of the message input. Always message.

          • "message"

      • labels: array of string

        The labels to assign to each item in the evaluation.

      • model: string

        The model to use for the evaluation. Must support structured outputs.

      • name: string

        The name of the grader.

      • passing_labels: array of string

        The labels that indicate a passing result. Must be a subset of labels.

      • type: "label_model"

        The object type, which is always label_model.

        • "label_model"

    • StringCheckGrader object { input, name, operation, 2 more }

      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

      • input: string

        The input text. This may include template strings.

      • name: string

        The name of the grader.

      • operation: "eq" or "ne" or "like" or "ilike"

        The string check operation to perform. One of eq, ne, like, or ilike.

        • "eq"

        • "ne"

        • "like"

        • "ilike"

      • reference: string

        The reference text. This may include template strings.

      • type: "string_check"

        The object type, which is always string_check.

        • "string_check"

    • TextSimilarityGrader = TextSimilarityGrader

      A TextSimilarityGrader object which grades text based on similarity metrics.

      • pass_threshold: number

        The threshold for the score.

    • PythonGrader = PythonGrader

      A PythonGrader object that runs a python script on the input.

      • pass_threshold: optional number

        The threshold for the score.

    • ScoreModelGrader = ScoreModelGrader

      A ScoreModelGrader object that uses a model to assign a score to the input.

      • pass_threshold: optional number

        The threshold for the score.

Eval Create Response

• EvalCreateResponse object { id, created_at, data_source_config, 4 more }

  An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot

  • See how well my chatbot handles customer support

  • Check if o4-mini is better at my usecase than gpt-4o

  • id: string

    Unique identifier for the evaluation.

  • created_at: number

    The Unix timestamp (in seconds) for when the eval was created.

  • data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig

    Configuration of data sources used in runs of the evaluation.

    • EvalCustomDataSourceConfig object { schema, type }

      A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

      • Used to define your testing criteria and

      • What data is required when creating a run

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "custom"

        The type of data source. Always custom.

        • "custom"

    • LogsDataSourceConfig object { schema, type, metadata }

      A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals. item and sample are both defined when using this data source config.

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "logs"

        The type of data source. Always logs.

        • "logs"

      • metadata: optional Metadata

        Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

        Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

    • EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }

      Deprecated in favor of LogsDataSourceConfig.

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "stored_completions"

        The type of data source. Always stored_completions.

        • "stored_completions"

      • metadata: optional Metadata

        Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

        Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • metadata: Metadata

    Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

    Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • name: string

    The name of the evaluation.

  • object: "eval"

    The object type.

    • "eval"

  • testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more

    A list of testing criteria.

    • LabelModelGrader object { input, labels, model, 3 more }

      A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

      • input: array of object { content, role, type }

        • content: string or ResponseInputText or object { text, type } or 3 more

          Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

          • TextInput = string

            A text input to the model.

          • ResponseInputText object { text, type }

            A text input to the model.

            • text: string

              The text input to the model.

            • type: "input_text"

              The type of the input item. Always input_text.

              • "input_text"

          • OutputText object { text, type }

            A text output from the model.

            • text: string

              The text output from the model.

            • type: "output_text"

              The type of the output text. Always output_text.

              • "output_text"

          • InputImage object { image_url, type, detail }

            An image input block used within EvalItem content arrays.

            • image_url: string

              The URL of the image input.

            • type: "input_image"

              The type of the image input. Always input_image.

              • "input_image"

            • detail: optional string

              The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

          • ResponseInputAudio object { input_audio, type }

            An audio input to the model.

            • input_audio: object { data, format }

              • data: string

                Base64-encoded audio data.

              • format: "mp3" or "wav"

                The format of the audio data. Currently supported formats are mp3 and wav.

                • "mp3"

                • "wav"

            • type: "input_audio"

              The type of the input item. Always input_audio.

              • "input_audio"

          • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

            A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

            • TextInput = string

              A text input to the model.

            • ResponseInputText object { text, type }

              A text input to the model.

            • OutputText object { text, type }

              A text output from the model.

              • text: string

                The text output from the model.

              • type: "output_text"

                The type of the output text. Always output_text.

                • "output_text"

            • InputImage object { image_url, type, detail }

              An image input block used within EvalItem content arrays.

              • image_url: string

                The URL of the image input.

              • type: "input_image"

                The type of the image input. Always input_image.

                • "input_image"

              • detail: optional string

                The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

            • ResponseInputAudio object { input_audio, type }

              An audio input to the model.

        • role: "user" or "assistant" or "system" or "developer"

          The role of the message input. One of user, assistant, system, or developer.

          • "user"

          • "assistant"

          • "system"

          • "developer"

        • type: optional "message"

          The type of the message input. Always message.

          • "message"

      • labels: array of string

        The labels to assign to each item in the evaluation.

      • model: string

        The model to use for the evaluation. Must support structured outputs.

      • name: string

        The name of the grader.

      • passing_labels: array of string

        The labels that indicate a passing result. Must be a subset of labels.

      • type: "label_model"

        The object type, which is always label_model.

        • "label_model"

    • StringCheckGrader object { input, name, operation, 2 more }

      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

      • input: string

        The input text. This may include template strings.

      • name: string

        The name of the grader.

      • operation: "eq" or "ne" or "like" or "ilike"

        The string check operation to perform. One of eq, ne, like, or ilike.

        • "eq"

        • "ne"

        • "like"

        • "ilike"

      • reference: string

        The reference text. This may include template strings.

      • type: "string_check"

        The object type, which is always string_check.

        • "string_check"

    • TextSimilarityGrader = TextSimilarityGrader

      A TextSimilarityGrader object which grades text based on similarity metrics.

      • pass_threshold: number

        The threshold for the score.

    • PythonGrader = PythonGrader

      A PythonGrader object that runs a python script on the input.

      • pass_threshold: optional number

        The threshold for the score.

    • ScoreModelGrader = ScoreModelGrader

      A ScoreModelGrader object that uses a model to assign a score to the input.

      • pass_threshold: optional number

        The threshold for the score.

Eval Retrieve Response

• EvalRetrieveResponse object { id, created_at, data_source_config, 4 more }

  An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot

  • See how well my chatbot handles customer support

  • Check if o4-mini is better at my usecase than gpt-4o

  • id: string

    Unique identifier for the evaluation.

  • created_at: number

    The Unix timestamp (in seconds) for when the eval was created.

  • data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig

    Configuration of data sources used in runs of the evaluation.

    • EvalCustomDataSourceConfig object { schema, type }

      A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

      • Used to define your testing criteria and

      • What data is required when creating a run

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "custom"

        The type of data source. Always custom.

        • "custom"

    • LogsDataSourceConfig object { schema, type, metadata }

      A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals. item and sample are both defined when using this data source config.

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "logs"

        The type of data source. Always logs.

        • "logs"

      • metadata: optional Metadata

        Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

        Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

    • EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }

      Deprecated in favor of LogsDataSourceConfig.

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "stored_completions"

        The type of data source. Always stored_completions.

        • "stored_completions"

      • metadata: optional Metadata

        Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

        Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • metadata: Metadata

    Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

    Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • name: string

    The name of the evaluation.

  • object: "eval"

    The object type.

    • "eval"

  • testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more

    A list of testing criteria.

    • LabelModelGrader object { input, labels, model, 3 more }

      A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

      • input: array of object { content, role, type }

        • content: string or ResponseInputText or object { text, type } or 3 more

          Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

          • TextInput = string

            A text input to the model.

          • ResponseInputText object { text, type }

            A text input to the model.

            • text: string

              The text input to the model.

            • type: "input_text"

              The type of the input item. Always input_text.

              • "input_text"

          • OutputText object { text, type }

            A text output from the model.

            • text: string

              The text output from the model.

            • type: "output_text"

              The type of the output text. Always output_text.

              • "output_text"

          • InputImage object { image_url, type, detail }

            An image input block used within EvalItem content arrays.

            • image_url: string

              The URL of the image input.

            • type: "input_image"

              The type of the image input. Always input_image.

              • "input_image"

            • detail: optional string

              The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

          • ResponseInputAudio object { input_audio, type }

            An audio input to the model.

            • input_audio: object { data, format }

              • data: string

                Base64-encoded audio data.

              • format: "mp3" or "wav"

                The format of the audio data. Currently supported formats are mp3 and wav.

                • "mp3"

                • "wav"

            • type: "input_audio"

              The type of the input item. Always input_audio.

              • "input_audio"

          • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

            A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

            • TextInput = string

              A text input to the model.

            • ResponseInputText object { text, type }

              A text input to the model.

            • OutputText object { text, type }

              A text output from the model.

              • text: string

                The text output from the model.

              • type: "output_text"

                The type of the output text. Always output_text.

                • "output_text"

            • InputImage object { image_url, type, detail }

              An image input block used within EvalItem content arrays.

              • image_url: string

                The URL of the image input.

              • type: "input_image"

                The type of the image input. Always input_image.

                • "input_image"

              • detail: optional string

                The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

            • ResponseInputAudio object { input_audio, type }

              An audio input to the model.

        • role: "user" or "assistant" or "system" or "developer"

          The role of the message input. One of user, assistant, system, or developer.

          • "user"

          • "assistant"

          • "system"

          • "developer"

        • type: optional "message"

          The type of the message input. Always message.

          • "message"

      • labels: array of string

        The labels to assign to each item in the evaluation.

      • model: string

        The model to use for the evaluation. Must support structured outputs.

      • name: string

        The name of the grader.

      • passing_labels: array of string

        The labels that indicate a passing result. Must be a subset of labels.

      • type: "label_model"

        The object type, which is always label_model.

        • "label_model"

    • StringCheckGrader object { input, name, operation, 2 more }

      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

      • input: string

        The input text. This may include template strings.

      • name: string

        The name of the grader.

      • operation: "eq" or "ne" or "like" or "ilike"

        The string check operation to perform. One of eq, ne, like, or ilike.

        • "eq"

        • "ne"

        • "like"

        • "ilike"

      • reference: string

        The reference text. This may include template strings.

      • type: "string_check"

        The object type, which is always string_check.

        • "string_check"

    • TextSimilarityGrader = TextSimilarityGrader

      A TextSimilarityGrader object which grades text based on similarity metrics.

      • pass_threshold: number

        The threshold for the score.

    • PythonGrader = PythonGrader

      A PythonGrader object that runs a python script on the input.

      • pass_threshold: optional number

        The threshold for the score.

    • ScoreModelGrader = ScoreModelGrader

      A ScoreModelGrader object that uses a model to assign a score to the input.

      • pass_threshold: optional number

        The threshold for the score.

Eval Update Response

• EvalUpdateResponse object { id, created_at, data_source_config, 4 more }

  An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot

  • See how well my chatbot handles customer support

  • Check if o4-mini is better at my usecase than gpt-4o

  • id: string

    Unique identifier for the evaluation.

  • created_at: number

    The Unix timestamp (in seconds) for when the eval was created.

  • data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig

    Configuration of data sources used in runs of the evaluation.

    • EvalCustomDataSourceConfig object { schema, type }

      A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

      • Used to define your testing criteria and

      • What data is required when creating a run

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "custom"

        The type of data source. Always custom.

        • "custom"

    • LogsDataSourceConfig object { schema, type, metadata }

      A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals. item and sample are both defined when using this data source config.

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "logs"

        The type of data source. Always logs.

        • "logs"

      • metadata: optional Metadata

        Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

        Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

    • EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }

      Deprecated in favor of LogsDataSourceConfig.

      • schema: map[unknown]

        The json schema for the run data source items. Learn how to build JSON schemas here.

      • type: "stored_completions"

        The type of data source. Always stored_completions.

        • "stored_completions"

      • metadata: optional Metadata

        Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

        Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • metadata: Metadata

    Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

    Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • name: string

    The name of the evaluation.

  • object: "eval"

    The object type.

    • "eval"

  • testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more

    A list of testing criteria.

    • LabelModelGrader object { input, labels, model, 3 more }

      A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

      • input: array of object { content, role, type }

        • content: string or ResponseInputText or object { text, type } or 3 more

          Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

          • TextInput = string

            A text input to the model.

          • ResponseInputText object { text, type }

            A text input to the model.

            • text: string

              The text input to the model.

            • type: "input_text"

              The type of the input item. Always input_text.

              • "input_text"

          • OutputText object { text, type }

            A text output from the model.

            • text: string

              The text output from the model.

            • type: "output_text"

              The type of the output text. Always output_text.

              • "output_text"

          • InputImage object { image_url, type, detail }

            An image input block used within EvalItem content arrays.

            • image_url: string

              The URL of the image input.

            • type: "input_image"

              The type of the image input. Always input_image.

              • "input_image"

            • detail: optional string

              The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

          • ResponseInputAudio object { input_audio, type }

            An audio input to the model.

            • input_audio: object { data, format }

              • data: string

                Base64-encoded audio data.

              • format: "mp3" or "wav"

                The format of the audio data. Currently supported formats are mp3 and wav.

                • "mp3"

                • "wav"

            • type: "input_audio"

              The type of the input item. Always input_audio.

              • "input_audio"

          • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

            A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

            • TextInput = string

              A text input to the model.

            • ResponseInputText object { text, type }

              A text input to the model.

            • OutputText object { text, type }

              A text output from the model.

              • text: string

                The text output from the model.

              • type: "output_text"

                The type of the output text. Always output_text.

                • "output_text"

            • InputImage object { image_url, type, detail }

              An image input block used within EvalItem content arrays.

              • image_url: string

                The URL of the image input.

              • type: "input_image"

                The type of the image input. Always input_image.

                • "input_image"

              • detail: optional string

                The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

            • ResponseInputAudio object { input_audio, type }

              An audio input to the model.

        • role: "user" or "assistant" or "system" or "developer"

          The role of the message input. One of user, assistant, system, or developer.

          • "user"

          • "assistant"

          • "system"

          • "developer"

        • type: optional "message"

          The type of the message input. Always message.

          • "message"

      • labels: array of string

        The labels to assign to each item in the evaluation.

      • model: string

        The model to use for the evaluation. Must support structured outputs.

      • name: string

        The name of the grader.

      • passing_labels: array of string

        The labels that indicate a passing result. Must be a subset of labels.

      • type: "label_model"

        The object type, which is always label_model.

        • "label_model"

    • StringCheckGrader object { input, name, operation, 2 more }

      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

      • input: string

        The input text. This may include template strings.

      • name: string

        The name of the grader.

      • operation: "eq" or "ne" or "like" or "ilike"

        The string check operation to perform. One of eq, ne, like, or ilike.

        • "eq"

        • "ne"

        • "like"

        • "ilike"

      • reference: string

        The reference text. This may include template strings.

      • type: "string_check"

        The object type, which is always string_check.

        • "string_check"

    • TextSimilarityGrader = TextSimilarityGrader

      A TextSimilarityGrader object which grades text based on similarity metrics.

      • pass_threshold: number

        The threshold for the score.

    • PythonGrader = PythonGrader

      A PythonGrader object that runs a python script on the input.

      • pass_threshold: optional number

        The threshold for the score.

    • ScoreModelGrader = ScoreModelGrader

      A ScoreModelGrader object that uses a model to assign a score to the input.

      • pass_threshold: optional number

        The threshold for the score.

Eval Delete Response

• EvalDeleteResponse object { deleted, eval_id, object }

  • deleted: boolean

  • eval_id: string

  • object: string

Runs

Get eval runs

get /evals/{eval_id}/runs

Get a list of runs for an evaluation.

Path Parameters

• eval_id: string

Query Parameters

• after: optional string

  Identifier for the last run from the previous pagination request.

• limit: optional number

  Number of runs to retrieve.

• order: optional "asc" or "desc"

  Sort order for runs by timestamp. Use asc for ascending order or desc for descending order. Defaults to asc.

  • "asc"

  • "desc"

• status: optional "queued" or "in_progress" or "completed" or 2 more

  Filter runs by status. One of queued | in_progress | failed | completed | canceled.

  • "queued"

  • "in_progress"

  • "completed"

  • "canceled"

  • "failed"

Returns

• data: array of object { id, created_at, data_source, 11 more }

  An array of eval run objects.

  • id: string

    Unique identifier for the evaluation run.

  • created_at: number

    Unix timestamp (in seconds) when the evaluation run was created.

  • data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }

    Information about the run's data source.

    • CreateEvalJSONLRunDataSource object { source, type }

      A JsonlRunDataSource object with that specifies a JSONL file that matches the eval

      • source: object { content, type } or object { id, type }

        Determines what populates the item namespace in the data source.

        • EvalJSONLFileContentSource object { content, type }

          • content: array of object { item, sample }

            The content of the jsonl file.

            • item: map[unknown]

            • sample: optional map[unknown]

          • type: "file_content"

            The type of jsonl source. Always file_content.

            • "file_content"

        • EvalJSONLFileIDSource object { id, type }

          • id: string

            The identifier of the file.

          • type: "file_id"

            The type of jsonl source. Always file_id.

            • "file_id"

      • type: "jsonl"

        The type of data source. Always jsonl.

        • "jsonl"

    • CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }

      A CompletionsRunDataSource object describing a model sampling configuration.

      • source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }

        Determines what populates the item namespace in this run's data source.

        • EvalJSONLFileContentSource object { content, type }

          • content: array of object { item, sample }

            The content of the jsonl file.

            • item: map[unknown]

            • sample: optional map[unknown]

          • type: "file_content"

            The type of jsonl source. Always file_content.

            • "file_content"

        • EvalJSONLFileIDSource object { id, type }

          • id: string

            The identifier of the file.

          • type: "file_id"

            The type of jsonl source. Always file_id.

            • "file_id"

        • StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more }

          A StoredCompletionsRunDataSource configuration describing a set of filters

          • type: "stored_completions"

            The type of source. Always stored_completions.

            • "stored_completions"

          • created_after: optional number

            An optional Unix timestamp to filter items created after this time.

          • created_before: optional number

            An optional Unix timestamp to filter items created before this time.

          • limit: optional number

            An optional maximum number of items to return.

          • metadata: optional Metadata

            Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

            Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

          • model: optional string

            An optional model to filter by (e.g., 'gpt-4o').

      • type: "completions"

        The type of run data source. Always completions.

        • "completions"

      • input_messages: optional object { template, type } or object { item_reference, type }

        Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.

        • TemplateInputMessages object { template, type }

          • template: array of EasyInputMessage or object { content, role, type }

            A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.

            • EasyInputMessage object { content, role, phase, type }

              A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.

              • content: string or ResponseInputMessageContentList

                Text, image, or audio input to the model, used to generate a response. Can also contain previous assistant responses.

                • TextInput = string

                  A text input to the model.

                • ResponseInputMessageContentList = array of ResponseInputContent

                  A list of one or many input items to the model, containing different content types.

                  • ResponseInputText object { text, type }

                    A text input to the model.

                    • text: string

                      The text input to the model.

                    • type: "input_text"

                      The type of the input item. Always input_text.

                      • "input_text"

                  • ResponseInputImage object { detail, type, file_id, image_url }

                    An image input to the model. Learn about image inputs.

                    • detail: "low" or "high" or "auto" or "original"

                      The detail level of the image to be sent to the model. One of high, low, auto, or original. Defaults to auto.

                      • "low"

                      • "high"

                      • "auto"

                      • "original"

                    • type: "input_image"

                      The type of the input item. Always input_image.

                      • "input_image"

                    • file_id: optional string

                      The ID of the file to be sent to the model.

                    • image_url: optional string

                      The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.

                  • ResponseInputFile object { type, detail, file_data, 3 more }

                    A file input to the model.

                    • type: "input_file"

                      The type of the input item. Always input_file.

                      • "input_file"

                    • detail: optional "low" or "high"

                      The detail level of the file to be sent to the model. Use low for the default rendering behavior, or high to render the file at higher quality. Defaults to low.

                      • "low"

                      • "high"

                    • file_data: optional string

                      The content of the file to be sent to the model.

                    • file_id: optional string

                      The ID of the file to be sent to the model.

                    • file_url: optional string

                      The URL of the file to be sent to the model.

                    • filename: optional string

                      The name of the file to be sent to the model.

              • role: "user" or "assistant" or "system" or "developer"

                The role of the message input. One of user, assistant, system, or developer.

                • "user"

                • "assistant"

                • "system"

                • "developer"

              • phase: optional "commentary" or "final_answer"

                Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer). For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend phase on all assistant messages — dropping it can degrade performance. Not used for user messages.

                • "commentary"

                • "final_answer"

              • type: optional "message"

                The type of the message input. Always message.

                • "message"

            • EvalMessageObject object { content, role, type }

              A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.

              • content: string or ResponseInputText or object { text, type } or 3 more

                Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

                • TextInput = string

                  A text input to the model.

                • ResponseInputText object { text, type }

                  A text input to the model.

                • OutputText object { text, type }

                  A text output from the model.

                  • text: string

                    The text output from the model.

                  • type: "output_text"

                    The type of the output text. Always output_text.

                    • "output_text"

                • InputImage object { image_url, type, detail }

                  An image input block used within EvalItem content arrays.

                  • image_url: string

                    The URL of the image input.

                  • type: "input_image"

                    The type of the image input. Always input_image.

                    • "input_image"

                  • detail: optional string

                    The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

                • ResponseInputAudio object { input_audio, type }

                  An audio input to the model.

                  • input_audio: object { data, format }

                    • data: string

                      Base64-encoded audio data.

                    • format: "mp3" or "wav"

                      The format of the audio data. Currently supported formats are mp3 and wav.

                      • "mp3"

                      • "wav"

                  • type: "input_audio"

                    The type of the input item. Always input_audio.

                    • "input_audio"

                • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

                  A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

                  • TextInput = string

                    A text input to the model.

                  • ResponseInputText object { text, type }

                    A text input to the model.

                  • OutputText object { text, type }

                    A text output from the model.

                    • text: string

                      The text output from the model.

                    • type: "output_text"

                      The type of the output text. Always output_text.

                      • "output_text"

                  • InputImage object { image_url, type, detail }

                    An image input block used within EvalItem content arrays.

                    • image_url: string

                      The URL of the image input.

                    • type: "input_image"

                      The type of the image input. Always input_image.

                      • "input_image"

                    • detail: optional string

                      The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

                  • ResponseInputAudio object { input_audio, type }

                    An audio input to the model.

              • role: "user" or "assistant" or "system" or "developer"

                The role of the message input. One of user, assistant, system, or developer.

                • "user"

                • "assistant"

                • "system"

                • "developer"

              • type: optional "message"

                The type of the message input. Always message.

                • "message"

          • type: "template"

            The type of input messages. Always template.

            • "template"

        • ItemReferenceInputMessages object { item_reference, type }

          • item_reference: string

            A reference to a variable in the item namespace. Ie, "item.input_trajectory"

          • type: "item_reference"

            The type of input messages. Always item_reference.

            • "item_reference"

      • model: optional string

        The name of the model to use for generating completions (e.g. "o3-mini").

      • sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }

        • max_completion_tokens: optional number

          The maximum number of tokens in the generated output.

        • reasoning_effort: optional ReasoningEffort

          Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.

          • gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.

          • All models before gpt-5.1 default to medium reasoning effort, and do not support none.

          • The gpt-5-pro model defaults to (and only supports) high reasoning effort.

          • xhigh is supported for all models after gpt-5.1-codex-max.

          • "none"

          • "minimal"

          • "low"

          • "medium"

          • "high"

          • "xhigh"

        • response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject

          An object specifying the format that the model must output.

          Setting to { "type": "json_schema", "json_schema": {...} } enables Structured Outputs which ensures the model will match your supplied JSON schema. Learn more in the Structured Outputs guide.

          Setting to { "type": "json_object" } enables the older JSON mode, which ensures the message the model generates is valid JSON. Using json_schema is preferred for models that support it.

          • ResponseFormatText object { type }

            Default response format. Used to generate text responses.

            • type: "text"

              The type of response format being defined. Always text.

              • "text"

          • ResponseFormatJSONSchema object { json_schema, type }

            JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.

            • json_schema: object { name, description, schema, strict }

              Structured Outputs configuration options, including a JSON Schema.

              • name: string

                The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

              • description: optional string

                A description of what the response format is for, used by the model to determine how to respond in the format.

              • schema: optional map[unknown]

                The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

              • strict: optional boolean

                Whether to enable strict schema adherence when generating the output. If set to true, the model will always follow the exact schema defined in the schema field. Only a subset of JSON Schema is supported when strict is true. To learn more, read the Structured Outputs guide.

            • type: "json_schema"

              The type of response format being defined. Always json_schema.

              • "json_schema"

          • ResponseFormatJSONObject object { type }

            JSON object response format. An older method of generating JSON responses. Using json_schema is recommended for models that support it. Note that the model will not generate JSON without a system or user message instructing it to do so.

            • type: "json_object"

              The type of response format being defined. Always json_object.

              • "json_object"

        • seed: optional number

          A seed value to initialize the randomness, during sampling.

        • temperature: optional number

          A higher temperature increases randomness in the outputs.

        • tools: optional array of ChatCompletionFunctionTool

          A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.

          • function: FunctionDefinition

            • name: string

              The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

            • description: optional string

              A description of what the function does, used by the model to choose when and how to call the function.

            • parameters: optional FunctionParameters

              The parameters the functions accepts, described as a JSON Schema object. See the guide for examples, and the JSON Schema reference for documentation about the format.

              Omitting parameters defines a function with an empty parameter list.

            • strict: optional boolean

              Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the parameters field. Only a subset of JSON Schema is supported when strict is true. Learn more about Structured Outputs in the function calling guide.

          • type: "function"

            The type of the tool. Currently, only function is supported.

            • "function"

        • top_p: optional number

          An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

    • ResponsesRunDataSource object { source, type, input_messages, 2 more }

      A ResponsesRunDataSource object describing a model sampling configuration.

      • source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }

        Determines what populates the item namespace in this run's data source.

        • EvalJSONLFileContentSource object { content, type }

          • content: array of object { item, sample }

            The content of the jsonl file.

            • item: map[unknown]

            • sample: optional map[unknown]

          • type: "file_content"

            The type of jsonl source. Always file_content.

            • "file_content"

        • EvalJSONLFileIDSource object { id, type }

          • id: string

            The identifier of the file.

          • type: "file_id"

            The type of jsonl source. Always file_id.

            • "file_id"

        • EvalResponsesSource object { type, created_after, created_before, 8 more }

          A EvalResponsesSource object describing a run data source configuration.

          • type: "responses"

            The type of run data source. Always responses.

            • "responses"

          • created_after: optional number

            Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.

          • created_before: optional number

            Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.

          • instructions_search: optional string

            Optional string to search the 'instructions' field. This is a query parameter used to select responses.

          • metadata: optional unknown

            Metadata filter for the responses. This is a query parameter used to select responses.

          • model: optional string

            The name of the model to find responses for. This is a query parameter used to select responses.

          • reasoning_effort: optional ReasoningEffort

            Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.

            • gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
            • All models before gpt-5.1 default to medium reasoning effort, and do not support none.
            • The gpt-5-pro model defaults to (and only supports) high reasoning effort.
            • xhigh is supported for all models after gpt-5.1-codex-max.

          • temperature: optional number

            Sampling temperature. This is a query parameter used to select responses.

          • tools: optional array of string

            List of tool names. This is a query parameter used to select responses.

          • top_p: optional number

            Nucleus sampling parameter. This is a query parameter used to select responses.

          • users: optional array of string

            List of user identifiers. This is a query parameter used to select responses.

      • type: "responses"

        The type of run data source. Always responses.

        • "responses"

      • input_messages: optional object { template, type } or object { item_reference, type }

        Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.

        • InputMessagesTemplate object { template, type }

          • template: array of object { content, role } or object { content, role, type }

            A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.

            • ChatMessage object { content, role }

              • content: string

                The content of the message.

              • role: string

                The role of the message (e.g. "system", "assistant", "user").

            • EvalMessageObject object { content, role, type }

              A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.

              • content: string or ResponseInputText or object { text, type } or 3 more

                Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

                • TextInput = string

                  A text input to the model.

                • ResponseInputText object { text, type }

                  A text input to the model.

                • OutputText object { text, type }

                  A text output from the model.

                  • text: string

                    The text output from the model.

                  • type: "output_text"

                    The type of the output text. Always output_text.

                    • "output_text"

                • InputImage object { image_url, type, detail }

                  An image input block used within EvalItem content arrays.

                  • image_url: string

                    The URL of the image input.

                  • type: "input_image"

                    The type of the image input. Always input_image.

                    • "input_image"

                  • detail: optional string

                    The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

                • ResponseInputAudio object { input_audio, type }

                  An audio input to the model.

                • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

                  A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

              • role: "user" or "assistant" or "system" or "developer"

                The role of the message input. One of user, assistant, system, or developer.

                • "user"

                • "assistant"

                • "system"

                • "developer"

              • type: optional "message"

                The type of the message input. Always message.

                • "message"

          • type: "template"

            The type of input messages. Always template.

            • "template"

        • InputMessagesItemReference object { item_reference, type }

          • item_reference: string

            A reference to a variable in the item namespace. Ie, "item.name"

          • type: "item_reference"

            The type of input messages. Always item_reference.

            • "item_reference"

      • model: optional string

        The name of the model to use for generating completions (e.g. "o3-mini").

      • sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }

        • max_completion_tokens: optional number

          The maximum number of tokens in the generated output.

        • reasoning_effort: optional ReasoningEffort

          Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.

          • gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
          • All models before gpt-5.1 default to medium reasoning effort, and do not support none.
          • The gpt-5-pro model defaults to (and only supports) high reasoning effort.
          • xhigh is supported for all models after gpt-5.1-codex-max.

        • seed: optional number

          A seed value to initialize the randomness, during sampling.

        • temperature: optional number

          A higher temperature increases randomness in the outputs.

        • text: optional object { format }

          Configuration options for a text response from the model. Can be plain text or structured JSON data. Learn more:

          • Text inputs and outputs

          • Structured Outputs

          • format: optional ResponseFormatTextConfig

            An object specifying the format that the model must output.

            Configuring { "type": "json_schema" } enables Structured Outputs, which ensures the model will match your supplied JSON schema. Learn more in the Structured Outputs guide.

            The default format is { "type": "text" } with no additional options.

            Not recommended for gpt-4o and newer models:

            Setting to { "type": "json_object" } enables the older JSON mode, which ensures the message the model generates is valid JSON. Using json_schema is preferred for models that support it.

            • ResponseFormatText object { type }

              Default response format. Used to generate text responses.

            • ResponseFormatTextJSONSchemaConfig object { name, schema, type, 2 more }

              JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.

              • name: string

                The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

              • schema: map[unknown]

                The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

              • type: "json_schema"

                The type of response format being defined. Always json_schema.

                • "json_schema"

              • description: optional string

                A description of what the response format is for, used by the model to determine how to respond in the format.

              • strict: optional boolean

                Whether to enable strict schema adherence when generating the output. If set to true, the model will always follow the exact schema defined in the schema field. Only a subset of JSON Schema is supported when strict is true. To learn more, read the Structured Outputs guide.

            • ResponseFormatJSONObject object { type }

              JSON object response format. An older method of generating JSON responses. Using json_schema is recommended for models that support it. Note that the model will not generate JSON without a system or user message instructing it to do so.

        • tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more

          An array of tools the model may call while generating a response. You can specify which tool to use by setting the tool_choice parameter.

          The two categories of tools you can provide the model are:

          • Built-in tools: Tools that are provided by OpenAI that extend the model's capabilities, like web search or file search. Learn more about built-in tools.

          • Function calls (custom tools): Functions that are defined by you, enabling the model to call your own code. Learn more about function calling.

          • Function object { name, parameters, strict, 3 more }

            Defines a function in your own code the model can choose to call. Learn more about function calling.

            • name: string

              The name of the function to call.

            • parameters: map[unknown]

              A JSON schema object describing the parameters of the function.

            • strict: boolean

              Whether to enforce strict parameter validation. Default true.

            • type: "function"

              The type of the function tool. Always function.

              • "function"

            • defer_loading: optional boolean

              Whether this function is deferred and loaded via tool search.

            • description: optional string

              A description of the function. Used by the model to determine whether or not to call the function.

          • FileSearch object { type, vector_store_ids, filters, 2 more }

            A tool that searches for relevant content from uploaded files. Learn more about the file search tool.

            • type: "file_search"

              The type of the file search tool. Always file_search.

              • "file_search"

            • vector_store_ids: array of string

              The IDs of the vector stores to search.

            • filters: optional ComparisonFilter or CompoundFilter

              A filter to apply.

              • ComparisonFilter object { key, type, value }

                A filter used to compare a specified attribute key to a given value using a defined comparison operation.

                • key: string

                  The key to compare against the value.

                • type: "eq" or "ne" or "gt" or 5 more

                  Specifies the comparison operator: eq, ne, gt, gte, lt, lte, in, nin.

                  • eq: equals

                  • ne: not equal

                  • gt: greater than

                  • gte: greater than or equal

                  • lt: less than

                  • lte: less than or equal

                  • in: in

                  • nin: not in

                  • "eq"

                  • "ne"

                  • "gt"

                  • "gte"

                  • "lt"

                  • "lte"

                  • "in"

                  • "nin"

                • value: string or number or boolean or array of string or number

                  The value to compare against the attribute key; supports string, number, or boolean types.

                  • string

                  • number

                  • boolean

                  • array of string or number

                    • string

                    • number

              • CompoundFilter object { filters, type }

                Combine multiple filters using and or or.

                • filters: array of ComparisonFilter or unknown

                  Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.

                  • ComparisonFilter object { key, type, value }

                    A filter used to compare a specified attribute key to a given value using a defined comparison operation.

                  • unknown

                • type: "and" or "or"

                  Type of operation: and or or.

                  • "and"

                  • "or"

            • max_num_results: optional number

              The maximum number of results to return. This number should be between 1 and 50 inclusive.

            • ranking_options: optional object { hybrid_search, ranker, score_threshold }

              Ranking options for search.

              • hybrid_search: optional object { embedding_weight, text_weight }

                Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.

                • embedding_weight: number

                  The weight of the embedding in the reciprocal ranking fusion.

                • text_weight: number

                  The weight of the text in the reciprocal ranking fusion.

              • ranker: optional "auto" or "default-2024-11-15"

                The ranker to use for the file search.

                • "auto"

                • "default-2024-11-15"

              • score_threshold: optional number

                The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.

          • Computer object { type }

            A tool that controls a virtual computer. Learn more about the computer tool.

            • type: "computer"

              The type of the computer tool. Always computer.

              • "computer"

          • ComputerUsePreview object { display_height, display_width, environment, type }

            A tool that controls a virtual computer. Learn more about the computer tool.

            • display_height: number

              The height of the computer display.

            • display_width: number

              The width of the computer display.

            • environment: "windows" or "mac" or "linux" or 2 more

              The type of computer environment to control.

              • "windows"

              • "mac"

              • "linux"

              • "ubuntu"

              • "browser"

            • type: "computer_use_preview"

              The type of the computer use tool. Always computer_use_preview.

              • "computer_use_preview"

          • WebSearch object { type, filters, search_context_size, user_location }

            Search the Internet for sources related to the prompt. Learn more about the web search tool.

            • type: "web_search" or "web_search_2025_08_26"

              The type of the web search tool. One of web_search or web_search_2025_08_26.

              • "web_search"

              • "web_search_2025_08_26"

            • filters: optional object { allowed_domains }

              Filters for the search.

              • allowed_domains: optional array of string

                Allowed domains for the search. If not provided, all domains are allowed. Subdomains of the provided domains are allowed as well.

                Example: ["pubmed.ncbi.nlm.nih.gov"]

            • search_context_size: optional "low" or "medium" or "high"

              High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.

              • "low"

              • "medium"

              • "high"

            • user_location: optional object { city, country, region, 2 more }

              The approximate location of the user.

              • city: optional string

                Free text input for the city of the user, e.g. San Francisco.

              • country: optional string

                The two-letter ISO country code of the user, e.g. US.

              • region: optional string

                Free text input for the region of the user, e.g. California.

              • timezone: optional string

                The IANA timezone of the user, e.g. America/Los_Angeles.

              • type: optional "approximate"

                The type of location approximation. Always approximate.

                • "approximate"

          • Mcp object { server_label, type, allowed_tools, 8 more }

            Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.

            • server_label: string

              A label for this MCP server, used to identify it in tool calls.

            • type: "mcp"

              The type of the MCP tool. Always mcp.

              • "mcp"

            • allowed_tools: optional array of string or object { read_only, tool_names }

              List of allowed tool names or a filter object.

              • McpAllowedTools = array of string

                A string array of allowed tool names

              • McpToolFilter object { read_only, tool_names }

                A filter object to specify which tools are allowed.

                • read_only: optional boolean

                  Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

                • tool_names: optional array of string

                  List of allowed tool names.

            • authorization: optional string

              An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.

            • connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more

              Identifier for service connectors, like those available in ChatGPT. One of server_url, connector_id, or tunnel_id must be provided. Learn more about service connectors here.

              Currently supported connector_id values are:

              • Dropbox: connector_dropbox

              • Gmail: connector_gmail

              • Google Calendar: connector_googlecalendar

              • Google Drive: connector_googledrive

              • Microsoft Teams: connector_microsoftteams

              • Outlook Calendar: connector_outlookcalendar

              • Outlook Email: connector_outlookemail

              • SharePoint: connector_sharepoint

              • "connector_dropbox"

              • "connector_gmail"

              • "connector_googlecalendar"

              • "connector_googledrive"

              • "connector_microsoftteams"

              • "connector_outlookcalendar"

              • "connector_outlookemail"

              • "connector_sharepoint"

            • defer_loading: optional boolean

              Whether this MCP tool is deferred and discovered via tool search.

            • headers: optional map[string]

              Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.

            • require_approval: optional object { always, never } or "always" or "never"

              Specify which of the MCP server's tools require approval.

              • McpToolApprovalFilter object { always, never }

                Specify which of the MCP server's tools require approval. Can be always, never, or a filter object associated with tools that require approval.

                • always: optional object { read_only, tool_names }

                  A filter object to specify which tools are allowed.

                  • read_only: optional boolean

                    Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

                  • tool_names: optional array of string

                    List of allowed tool names.

                • never: optional object { read_only, tool_names }

                  A filter object to specify which tools are allowed.

                  • read_only: optional boolean

                    Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

                  • tool_names: optional array of string

                    List of allowed tool names.

              • McpToolApprovalSetting = "always" or "never"

                Specify a single approval policy for all tools. One of always or never. When set to always, all tools will require approval. When set to never, all tools will not require approval.

                • "always"

                • "never"

            • server_description: optional string

              Optional description of the MCP server, used to provide more context.

            • server_url: optional string

              The URL for the MCP server. One of server_url, connector_id, or tunnel_id must be provided.

            • tunnel_id: optional string

              The Secure MCP Tunnel ID to use instead of a direct server URL. One of server_url, connector_id, or tunnel_id must be provided.

          • CodeInterpreter object { container, type }

            A tool that runs Python code to help generate a response to a prompt.

            • container: string or object { type, file_ids, memory_limit, network_policy }

              The code interpreter container. Can be a container ID or an object that specifies uploaded file IDs to make available to your code, along with an optional memory_limit setting.

              • string

                The container ID.

              • CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy }

                Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.

                • type: "auto"

                  Always auto.

                  • "auto"

                • file_ids: optional array of string

                  An optional list of uploaded files to make available to your code.

                • memory_limit: optional "1g" or "4g" or "16g" or "64g"

                  The memory limit for the code interpreter container.

                  • "1g"

                  • "4g"

                  • "16g"

                  • "64g"

                • network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist

                  Network access policy for the container.

                  • ContainerNetworkPolicyDisabled object { type }

                    • type: "disabled"

                      Disable outbound network access. Always disabled.

                      • "disabled"

                  • ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }

                    • allowed_domains: array of string

                      A list of allowed domains when type is allowlist.

                    • type: "allowlist"

                      Allow outbound network access only to specified domains. Always allowlist.

                      • "allowlist"

                    • domain_secrets: optional array of ContainerNetworkPolicyDomainSecret

                      Optional domain-scoped secrets for allowlisted domains.

                      • domain: string

                        The domain associated with the secret.

                      • name: string

                        The name of the secret to inject for the domain.

                      • value: string

                        The secret value to inject for the domain.

            • type: "code_interpreter"

              The type of the code interpreter tool. Always code_interpreter.

              • "code_interpreter"

          • ImageGeneration object { type, action, background, 9 more }

            A tool that generates images using the GPT image models.

            • type: "image_generation"

              The type of the image generation tool. Always image_generation.

              • "image_generation"

            • action: optional "generate" or "edit" or "auto"

              Whether to generate a new image or edit an existing image. Default: auto.

              • "generate"

              • "edit"

              • "auto"

            • background: optional "transparent" or "opaque" or "auto"

              Background type for the generated image. One of transparent, opaque, or auto. Default: auto.

              • "transparent"

              • "opaque"

              • "auto"

            • input_fidelity: optional "high" or "low"

              Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for gpt-image-1 and gpt-image-1.5 and later models, unsupported for gpt-image-1-mini. Supports high and low. Defaults to low.

              • "high"

              • "low"

            • input_image_mask: optional object { file_id, image_url }

              Optional mask for inpainting. Contains image_url (string, optional) and file_id (string, optional).

              • file_id: optional string

                File ID for the mask image.

              • image_url: optional string

                Base64-encoded mask image.

            • model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"

              The image generation model to use. Default: gpt-image-1.

              • string

              • "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"

                The image generation model to use. Default: gpt-image-1.

                • "gpt-image-1"

                • "gpt-image-1-mini"

                • "gpt-image-1.5"

            • moderation: optional "auto" or "low"

              Moderation level for the generated image. Default: auto.

              • "auto"

              • "low"

            • output_compression: optional number

              Compression level for the output image. Default: 100.

            • output_format: optional "png" or "webp" or "jpeg"

              The output format of the generated image. One of png, webp, or jpeg. Default: png.

              • "png"

              • "webp"

              • "jpeg"

            • partial_images: optional number

              Number of partial images to generate in streaming mode, from 0 (default value) to 3.

            • quality: optional "low" or "medium" or "high" or "auto"

              The quality of the generated image. One of low, medium, high, or auto. Default: auto.

              • "low"

              • "medium"

              • "high"

              • "auto"

            • size: optional string or "1024x1024" or "1024x1536" or "1536x1024" or "auto"

              The size of the generated images. For gpt-image-2 and gpt-image-2-2026-04-21, arbitrary resolutions are supported as WIDTHxHEIGHT strings, for example 1536x864. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above 2560x1440 are experimental, and the maximum supported resolution is 3840x2160. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes 1024x1024, 1536x1024, and 1024x1536 are supported by the GPT image models; auto is supported for models that allow automatic sizing. For dall-e-2, use one of 256x256, 512x512, or 1024x1024. For dall-e-3, use one of 1024x1024, 1792x1024, or 1024x1792.

              • string

              • "1024x1024" or "1024x1536" or "1536x1024" or "auto"

                The size of the generated images. For gpt-image-2 and gpt-image-2-2026-04-21, arbitrary resolutions are supported as WIDTHxHEIGHT strings, for example 1536x864. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above 2560x1440 are experimental, and the maximum supported resolution is 3840x2160. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes 1024x1024, 1536x1024, and 1024x1536 are supported by the GPT image models; auto is supported for models that allow automatic sizing. For dall-e-2, use one of 256x256, 512x512, or 1024x1024. For dall-e-3, use one of 1024x1024, 1792x1024, or 1024x1792.

                • "1024x1024"

                • "1024x1536"

                • "1536x1024"

                • "auto"

          • LocalShell object { type }

            A tool that allows the model to execute shell commands in a local environment.

            • type: "local_shell"

              The type of the local shell tool. Always local_shell.

              • "local_shell"

          • Shell object { type, environment }

            A tool that allows the model to execute shell commands.

            • type: "shell"

              The type of the shell tool. Always shell.

              • "shell"

            • environment: optional ContainerAuto or LocalEnvironment or ContainerReference

              • ContainerAuto object { type, file_ids, memory_limit, 2 more }

                • type: "container_auto"

                  Automatically creates a container for this request

                  • "container_auto"

                • file_ids: optional array of string

                  An optional list of uploaded files to make available to your code.

                • memory_limit: optional "1g" or "4g" or "16g" or "64g"

                  The memory limit for the container.

                  • "1g"

                  • "4g"

                  • "16g"

                  • "64g"

                • network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist

                  Network access policy for the container.

                  • ContainerNetworkPolicyDisabled object { type }

                  • ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }

                • skills: optional array of SkillReference or InlineSkill

                  An optional list of skills referenced by id or inline data.

                  • SkillReference object { skill_id, type, version }

                    • skill_id: string

                      The ID of the referenced skill.

                    • type: "skill_reference"

                      References a skill created with the /v1/skills endpoint.

                      • "skill_reference"

                    • version: optional string

                      Optional skill version. Use a positive integer or 'latest'. Omit for default.

                  • InlineSkill object { description, name, source, type }

                    • description: string

                      The description of the skill.

                    • name: string

                      The name of the skill.

                    • source: InlineSkillSource

                      Inline skill payload

                      • data: string

                        Base64-encoded skill zip bundle.

                      • media_type: "application/zip"

                        The media type of the inline skill payload. Must be application/zip.

                        • "application/zip"

                      • type: "base64"

                        The type of the inline skill source. Must be base64.

                        • "base64"

                    • type: "inline"

                      Defines an inline skill for this request.

                      • "inline"

              • LocalEnvironment object { type, skills }

                • type: "local"

                  Use a local computer environment.

                  • "local"

                • skills: optional array of LocalSkill

                  An optional list of skills.

                  • description: string

                    The description of the skill.

                  • name: string

                    The name of the skill.

                  • path: string

                    The path to the directory containing the skill.

              • ContainerReference object { container_id, type }

                • container_id: string

                  The ID of the referenced container.

                • type: "container_reference"

                  References a container created with the /v1/containers endpoint

                  • "container_reference"

          • Custom object { name, type, defer_loading, 2 more }

            A custom tool that processes input using a specified format. Learn more about custom tools

            • name: string

              The name of the custom tool, used to identify it in tool calls.

            • type: "custom"

              The type of the custom tool. Always custom.

              • "custom"

            • defer_loading: optional boolean

              Whether this tool should be deferred and discovered via tool search.

            • description: optional string

              Optional description of the custom tool, used to provide more context.

            • format: optional CustomToolInputFormat

              The input format for the custom tool. Default is unconstrained text.

              • Text object { type }

                Unconstrained free-form text.

                • type: "text"

                  Unconstrained text format. Always text.

                  • "text"

              • Grammar object { definition, syntax, type }

                A grammar defined by the user.

                • definition: string

                  The grammar definition.

                • syntax: "lark" or "regex"

                  The syntax of the grammar definition. One of lark or regex.

                  • "lark"

                  • "regex"

                • type: "grammar"

                  Grammar format. Always grammar.

                  • "grammar"

          • Namespace object { description, name, tools, type }

            Groups function/custom tools under a shared namespace.

            • description: string

              A description of the namespace shown to the model.

            • name: string

              The namespace name used in tool calls (for example, crm).

            • tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }

              The function/custom tools available inside this namespace.

              • Function object { name, type, defer_loading, 3 more }

                • name: string

                • type: "function"

                  • "function"

                • defer_loading: optional boolean

                  Whether this function should be deferred and discovered via tool search.

                • description: optional string

                • parameters: optional unknown

                • strict: optional boolean

              • Custom object { name, type, defer_loading, 2 more }

                A custom tool that processes input using a specified format. Learn more about custom tools

                • name: string

                  The name of the custom tool, used to identify it in tool calls.

                • type: "custom"

                  The type of the custom tool. Always custom.

                  • "custom"

                • defer_loading: optional boolean

                  Whether this tool should be deferred and discovered via tool search.

                • description: optional string

                  Optional description of the custom tool, used to provide more context.

                • format: optional CustomToolInputFormat

                  The input format for the custom tool. Default is unconstrained text.

            • type: "namespace"

              The type of the tool. Always namespace.

              • "namespace"

          • ToolSearch object { type, description, execution, parameters }

            Hosted or BYOT tool search configuration for deferred tools.

            • type: "tool_search"

              The type of the tool. Always tool_search.

              • "tool_search"

            • description: optional string

              Description shown to the model for a client-executed tool search tool.

            • execution: optional "server" or "client"

              Whether tool search is executed by the server or by the client.

              • "server"

              • "client"

            • parameters: optional unknown

              Parameter schema for a client-executed tool search tool.

          • WebSearchPreview object { type, search_content_types, search_context_size, user_location }

            This tool searches the web for relevant results to use in a response. Learn more about the web search tool.

            • type: "web_search_preview" or "web_search_preview_2025_03_11"

              The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.

              • "web_search_preview"

              • "web_search_preview_2025_03_11"

            • search_content_types: optional array of "text" or "image"

              • "text"

              • "image"

            • search_context_size: optional "low" or "medium" or "high"

              High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.

              • "low"

              • "medium"

              • "high"

            • user_location: optional object { type, city, country, 2 more }

              The user's location.

              • type: "approximate"

                The type of location approximation. Always approximate.

                • "approximate"

              • city: optional string

                Free text input for the city of the user, e.g. San Francisco.

              • country: optional string

                The two-letter ISO country code of the user, e.g. US.

              • region: optional string

                Free text input for the region of the user, e.g. California.

              • timezone: optional string

                The IANA timezone of the user, e.g. America/Los_Angeles.

          • ApplyPatch object { type }

            Allows the assistant to create, delete, or update files using unified diffs.

            • type: "apply_patch"

              The type of the tool. Always apply_patch.

              • "apply_patch"

        • top_p: optional number

          An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

  • error: EvalAPIError

    An object representing an error response from the Eval API.

    • code: string

      The error code.

    • message: string

      The error message.

  • eval_id: string

    The identifier of the associated evaluation.

  • metadata: Metadata

    Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

    Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

  • model: string

    The model that is evaluated, if applicable.

  • name: string

    The name of the evaluation run.

  • object: "eval.run"

    The type of the object. Always "eval.run".

    • "eval.run"

  • per_model_usage: array of object { cached_tokens, completion_tokens, invocation_count, 3 more }

    Usage statistics for each model during the evaluation run.

    • cached_tokens: number

      The number of tokens retrieved from cache.

    • completion_tokens: number

      The number of completion tokens generated.

    • invocation_count: number

      The number of invocations.

    • model_name: string

      The name of the model.

    • prompt_tokens: number

      The number of prompt tokens used.

    • total_tokens: number

      The total number of tokens used.

  • per_testing_criteria_results: array of object { failed, passed, testing_criteria }

    Results per testing criteria applied during the evaluation run.

    • failed: number

      Number of tests failed for this criteria.

    • passed: number

      Number of tests passed for this criteria.

    • testing_criteria: string

      A description of the testing criteria.

  • report_url: string

    The URL to the rendered evaluation run report on the UI dashboard.

  • result_counts: object { errored, failed, passed, total }

    Counters summarizing the outcomes of the evaluation run.

    • errored: number

      Number of output items that resulted in an error.

    • failed: number

      Number of output items that failed to pass the evaluation.

    • passed: number

      Number of output items that passed the evaluation.

    • total: number

      Total number of executed output items.

  • status: string

    The status of the evaluation run.

• first_id: string

  The identifier of the first eval run in the data array.

• has_more: boolean

  Indicates whether there are more evals available.

• last_id: string

  The identifier of the last eval run in the data array.

• object: "list"

  The type of this object. It is always set to "list".

  • "list"

Example

curl https://api.openai.com/v1/evals/$EVAL_ID/runs \
    -H "Authorization: Bearer $OPENAI_API_KEY"

Response

{
  "data": [
    {
      "id": "id",
      "created_at": 0,
      "data_source": {
        "source": {
          "content": [
            {
              "item": {
                "foo": "bar"
              },
              "sample": {
                "foo": "bar"
              }
            }
          ],
          "type": "file_content"
        },
        "type": "jsonl"
      },
      "error": {
        "code": "code",
        "message": "message"
      },
      "eval_id": "eval_id",
      "metadata": {
        "foo": "string"
      },
      "model": "model",
      "name": "name",
      "object": "eval.run",
      "per_model_usage": [
        {
          "cached_tokens": 0,
          "completion_tokens": 0,
          "invocation_count": 0,
          "model_name": "model_name",
          "prompt_tokens": 0,
          "total_tokens": 0
        }
      ],
      "per_testing_criteria_results": [
        {
          "failed": 0,
          "passed": 0,
          "testing_criteria": "testing_criteria"
        }
      ],
      "report_url": "https://example.com",
      "result_counts": {
        "errored": 0,
        "failed": 0,
        "passed": 0,
        "total": 0
      },
      "status": "status"
    }
  ],
  "first_id": "first_id",
  "has_more": true,
  "last_id": "last_id",
  "object": "list"
}

Example

curl https://api.openai.com/v1/evals/egroup_67abd54d9b0081909a86353f6fb9317a/runs \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json"

Response

{
  "object": "list",
  "data": [
    {
      "object": "eval.run",
      "id": "evalrun_67e0c7d31560819090d60c0780591042",
      "eval_id": "eval_67e0c726d560819083f19a957c4c640b",
      "report_url": "https://platform.openai.com/evaluations/eval_67e0c726d560819083f19a957c4c640b",
      "status": "completed",
      "model": "o3-mini",
      "name": "bulk_with_negative_examples_o3-mini",
      "created_at": 1742784467,
      "result_counts": {
        "total": 1,
        "errored": 0,
        "failed": 0,
        "passed": 1
      },
      "per_model_usage": [
        {
          "model_name": "o3-mini",
          "invocation_count": 1,
          "prompt_tokens": 563,
          "completion_tokens": 874,
          "total_tokens": 1437,
          "cached_tokens": 0
        }
      ],
      "per_testing_criteria_results": [
        {
          "testing_criteria": "Push Notification Summary Grader-1808cd0b-eeec-4e0b-a519-337e79f4f5d1",
          "passed": 1,
          "failed": 0
        }
      ],
      "data_source": {
        "type": "completions",
        "source": {
          "type": "file_content",
          "content": [
            {
              "item": {
                "notifications": "\n- New message from Sarah: \"Can you call me later?\"\n- Your package has been delivered!\n- Flash sale: 20% off electronics for the next 2 hours!\n"
              }
            }
          ]
        },
        "input_messages": {
          "type": "template",
          "template": [
            {
              "type": "message",
              "role": "developer",
              "content": {
                "type": "input_text",
                "text": "\n\n\n\nYou are a helpful assistant that takes in an array of push notifications and returns a collapsed summary of them.\nThe push notification will be provided as follows:\n<push_notifications>\n...notificationlist...\n</push_notifications>\n\nYou should return just the summary and nothing else.\n\n\nYou should return a summary that is concise and snappy.\n\n\nHere is an example of a good summary:\n<push_notifications>\n- Traffic alert: Accident reported on Main Street.- Package out for delivery: Expected by 5 PM.- New friend suggestion: Connect with Emma.\n</push_notifications>\n<summary>\nTraffic alert, package expected by 5pm, suggestion for new friend (Emily).\n</summary>\n\n\nHere is an example of a bad summary:\n<push_notifications>\n- Traffic alert: Accident reported on Main Street.- Package out for delivery: Expected by 5 PM.- New friend suggestion: Connect with Emma.\n</push_notifications>\n<summary>\nTraffic alert reported on main street. You have a package that will arrive by 5pm, Emily is a new friend suggested for you.\n</summary>\n"
              }
            },
            {
              "type": "message",
              "role": "user",
              "content": {
                "type": "input_text",
                "text": "<push_notifications>{{item.notifications}}</push_notifications>"
              }
            }
          ]
        },
        "model": "o3-mini",
        "sampling_params": null
      },
      "error": null,
      "metadata": {}
    }
  ],
  "first_id": "evalrun_67e0c7d31560819090d60c0780591042",
  "last_id": "evalrun_67e0c7d31560819090d60c0780591042",
  "has_more": true
}

Create eval run

post /evals/{eval_id}/runs

Kicks off a new run for a given evaluation, specifying the data source, and what model configuration to use to test. The datasource will be validated against the schema specified in the config of the evaluation.

Path Parameters

• eval_id: string

Body Parameters

• data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or object { source, type, input_messages, 2 more }

  Details about the run's data source.

  • CreateEvalJSONLRunDataSource object { source, type }

    A JsonlRunDataSource object with that specifies a JSONL file that matches the eval

    • source: object { content, type } or object { id, type }

      Determines what populates the item namespace in the data source.

      • EvalJSONLFileContentSource object { content, type }

        • content: array of object { item, sample }

          The content of the jsonl file.

          • item: map[unknown]

          • sample: optional map[unknown]

        • type: "file_content"

          The type of jsonl source. Always file_content.

          • "file_content"

      • EvalJSONLFileIDSource object { id, type }

        • id: string

          The identifier of the file.

        • type: "file_id"

          The type of jsonl source. Always file_id.

          • "file_id"

    • type: "jsonl"

      The type of data source. Always jsonl.

      • "jsonl"

  • CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }

    A CompletionsRunDataSource object describing a model sampling configuration.

    • source: object { content, type } or object { id, type } or object { type, created_after, created_before, 3 more }

      Determines what populates the item namespace in this run's data source.

      • EvalJSONLFileContentSource object { content, type }

        • content: array of object { item, sample }

          The content of the jsonl file.

          • item: map[unknown]

          • sample: optional map[unknown]

        • type: "file_content"

          The type of jsonl source. Always file_content.

          • "file_content"

      • EvalJSONLFileIDSource object { id, type }

        • id: string

          The identifier of the file.

        • type: "file_id"

          The type of jsonl source. Always file_id.

          • "file_id"

      • StoredCompletionsRunDataSource object { type, created_after, created_before, 3 more }

        A StoredCompletionsRunDataSource configuration describing a set of filters

        • type: "stored_completions"

          The type of source. Always stored_completions.

          • "stored_completions"

        • created_after: optional number

          An optional Unix timestamp to filter items created after this time.

        • created_before: optional number

          An optional Unix timestamp to filter items created before this time.

        • limit: optional number

          An optional maximum number of items to return.

        • metadata: optional Metadata

          Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

          Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

        • model: optional string

          An optional model to filter by (e.g., 'gpt-4o').

    • type: "completions"

      The type of run data source. Always completions.

      • "completions"

    • input_messages: optional object { template, type } or object { item_reference, type }

      Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.

      • TemplateInputMessages object { template, type }

        • template: array of EasyInputMessage or object { content, role, type }

          A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.

          • EasyInputMessage object { content, role, phase, type }

            A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.

            • content: string or ResponseInputMessageContentList

              Text, image, or audio input to the model, used to generate a response. Can also contain previous assistant responses.

              • TextInput = string

                A text input to the model.

              • ResponseInputMessageContentList = array of ResponseInputContent

                A list of one or many input items to the model, containing different content types.

                • ResponseInputText object { text, type }

                  A text input to the model.

                  • text: string

                    The text input to the model.

                  • type: "input_text"

                    The type of the input item. Always input_text.

                    • "input_text"

                • ResponseInputImage object { detail, type, file_id, image_url }

                  An image input to the model. Learn about image inputs.

                  • detail: "low" or "high" or "auto" or "original"

                    The detail level of the image to be sent to the model. One of high, low, auto, or original. Defaults to auto.

                    • "low"

                    • "high"

                    • "auto"

                    • "original"

                  • type: "input_image"

                    The type of the input item. Always input_image.

                    • "input_image"

                  • file_id: optional string

                    The ID of the file to be sent to the model.

                  • image_url: optional string

                    The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.

                • ResponseInputFile object { type, detail, file_data, 3 more }

                  A file input to the model.

                  • type: "input_file"

                    The type of the input item. Always input_file.

                    • "input_file"

                  • detail: optional "low" or "high"

                    The detail level of the file to be sent to the model. Use low for the default rendering behavior, or high to render the file at higher quality. Defaults to low.

                    • "low"

                    • "high"

                  • file_data: optional string

                    The content of the file to be sent to the model.

                  • file_id: optional string

                    The ID of the file to be sent to the model.

                  • file_url: optional string

                    The URL of the file to be sent to the model.

                  • filename: optional string

                    The name of the file to be sent to the model.

            • role: "user" or "assistant" or "system" or "developer"

              The role of the message input. One of user, assistant, system, or developer.

              • "user"

              • "assistant"

              • "system"

              • "developer"

            • phase: optional "commentary" or "final_answer"

              Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer). For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend phase on all assistant messages — dropping it can degrade performance. Not used for user messages.

              • "commentary"

              • "final_answer"

            • type: optional "message"

              The type of the message input. Always message.

              • "message"

          • EvalMessageObject object { content, role, type }

            A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.

            • content: string or ResponseInputText or object { text, type } or 3 more

              Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

              • TextInput = string

                A text input to the model.

              • ResponseInputText object { text, type }

                A text input to the model.

              • OutputText object { text, type }

                A text output from the model.

                • text: string

                  The text output from the model.

                • type: "output_text"

                  The type of the output text. Always output_text.

                  • "output_text"

              • InputImage object { image_url, type, detail }

                An image input block used within EvalItem content arrays.

                • image_url: string

                  The URL of the image input.

                • type: "input_image"

                  The type of the image input. Always input_image.

                  • "input_image"

                • detail: optional string

                  The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

              • ResponseInputAudio object { input_audio, type }

                An audio input to the model.

                • input_audio: object { data, format }

                  • data: string

                    Base64-encoded audio data.

                  • format: "mp3" or "wav"

                    The format of the audio data. Currently supported formats are mp3 and wav.

                    • "mp3"

                    • "wav"

                • type: "input_audio"

                  The type of the input item. Always input_audio.

                  • "input_audio"

              • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

                A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

                • TextInput = string

                  A text input to the model.

                • ResponseInputText object { text, type }

                  A text input to the model.

                • OutputText object { text, type }

                  A text output from the model.

                  • text: string

                    The text output from the model.

                  • type: "output_text"

                    The type of the output text. Always output_text.

                    • "output_text"

                • InputImage object { image_url, type, detail }

                  An image input block used within EvalItem content arrays.

                  • image_url: string

                    The URL of the image input.

                  • type: "input_image"

                    The type of the image input. Always input_image.

                    • "input_image"

                  • detail: optional string

                    The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

                • ResponseInputAudio object { input_audio, type }

                  An audio input to the model.

            • role: "user" or "assistant" or "system" or "developer"

              The role of the message input. One of user, assistant, system, or developer.

              • "user"

              • "assistant"

              • "system"

              • "developer"

            • type: optional "message"

              The type of the message input. Always message.

              • "message"

        • type: "template"

          The type of input messages. Always template.

          • "template"

      • ItemReferenceInputMessages object { item_reference, type }

        • item_reference: string

          A reference to a variable in the item namespace. Ie, "item.input_trajectory"

        • type: "item_reference"

          The type of input messages. Always item_reference.

          • "item_reference"

    • model: optional string

      The name of the model to use for generating completions (e.g. "o3-mini").

    • sampling_params: optional object { max_completion_tokens, reasoning_effort, response_format, 4 more }

      • max_completion_tokens: optional number

        The maximum number of tokens in the generated output.

      • reasoning_effort: optional ReasoningEffort

        Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.

        • gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.

        • All models before gpt-5.1 default to medium reasoning effort, and do not support none.

        • The gpt-5-pro model defaults to (and only supports) high reasoning effort.

        • xhigh is supported for all models after gpt-5.1-codex-max.

        • "none"

        • "minimal"

        • "low"

        • "medium"

        • "high"

        • "xhigh"

      • response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject

        An object specifying the format that the model must output.

        Setting to { "type": "json_schema", "json_schema": {...} } enables Structured Outputs which ensures the model will match your supplied JSON schema. Learn more in the Structured Outputs guide.

        Setting to { "type": "json_object" } enables the older JSON mode, which ensures the message the model generates is valid JSON. Using json_schema is preferred for models that support it.

        • ResponseFormatText object { type }

          Default response format. Used to generate text responses.

          • type: "text"

            The type of response format being defined. Always text.

            • "text"

        • ResponseFormatJSONSchema object { json_schema, type }

          JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.

          • json_schema: object { name, description, schema, strict }

            Structured Outputs configuration options, including a JSON Schema.

            • name: string

              The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

            • description: optional string

              A description of what the response format is for, used by the model to determine how to respond in the format.

            • schema: optional map[unknown]

              The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

            • strict: optional boolean

              Whether to enable strict schema adherence when generating the output. If set to true, the model will always follow the exact schema defined in the schema field. Only a subset of JSON Schema is supported when strict is true. To learn more, read the Structured Outputs guide.

          • type: "json_schema"

            The type of response format being defined. Always json_schema.

            • "json_schema"

        • ResponseFormatJSONObject object { type }

          JSON object response format. An older method of generating JSON responses. Using json_schema is recommended for models that support it. Note that the model will not generate JSON without a system or user message instructing it to do so.

          • type: "json_object"

            The type of response format being defined. Always json_object.

            • "json_object"

      • seed: optional number

        A seed value to initialize the randomness, during sampling.

      • temperature: optional number

        A higher temperature increases randomness in the outputs.

      • tools: optional array of ChatCompletionFunctionTool

        A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.

        • function: FunctionDefinition

          • name: string

            The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

          • description: optional string

            A description of what the function does, used by the model to choose when and how to call the function.

          • parameters: optional FunctionParameters

            The parameters the functions accepts, described as a JSON Schema object. See the guide for examples, and the JSON Schema reference for documentation about the format.

            Omitting parameters defines a function with an empty parameter list.

          • strict: optional boolean

            Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the parameters field. Only a subset of JSON Schema is supported when strict is true. Learn more about Structured Outputs in the function calling guide.

        • type: "function"

          The type of the tool. Currently, only function is supported.

          • "function"

      • top_p: optional number

        An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

  • ResponsesRunDataSource object { source, type, input_messages, 2 more }

    A ResponsesRunDataSource object describing a model sampling configuration.

    • source: object { content, type } or object { id, type } or object { type, created_after, created_before, 8 more }

      Determines what populates the item namespace in this run's data source.

      • EvalJSONLFileContentSource object { content, type }

        • content: array of object { item, sample }

          The content of the jsonl file.

          • item: map[unknown]

          • sample: optional map[unknown]

        • type: "file_content"

          The type of jsonl source. Always file_content.

          • "file_content"

      • EvalJSONLFileIDSource object { id, type }

        • id: string

          The identifier of the file.

        • type: "file_id"

          The type of jsonl source. Always file_id.

          • "file_id"

      • EvalResponsesSource object { type, created_after, created_before, 8 more }

        A EvalResponsesSource object describing a run data source configuration.

        • type: "responses"

          The type of run data source. Always responses.

          • "responses"

        • created_after: optional number

          Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.

        • created_before: optional number

          Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.

        • instructions_search: optional string

          Optional string to search the 'instructions' field. This is a query parameter used to select responses.

        • metadata: optional unknown

          Metadata filter for the responses. This is a query parameter used to select responses.

        • model: optional string

          The name of the model to find responses for. This is a query parameter used to select responses.

        • reasoning_effort: optional ReasoningEffort

          Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.

          • gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
          • All models before gpt-5.1 default to medium reasoning effort, and do not support none.
          • The gpt-5-pro model defaults to (and only supports) high reasoning effort.
          • xhigh is supported for all models after gpt-5.1-codex-max.

        • temperature: optional number

          Sampling temperature. This is a query parameter used to select responses.

        • tools: optional array of string

          List of tool names. This is a query parameter used to select responses.

        • top_p: optional number

          Nucleus sampling parameter. This is a query parameter used to select responses.

        • users: optional array of string

          List of user identifiers. This is a query parameter used to select responses.

    • type: "responses"

      The type of run data source. Always responses.

      • "responses"

    • input_messages: optional object { template, type } or object { item_reference, type }

      Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.

      • InputMessagesTemplate object { template, type }

        • template: array of object { content, role } or object { content, role, type }

          A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.

          • ChatMessage object { content, role }

            • content: string

              The content of the message.

            • role: string

              The role of the message (e.g. "system", "assistant", "user").

          • EvalMessageObject object { content, role, type }

            A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.

            • content: string or ResponseInputText or object { text, type } or 3 more

              Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

              • TextInput = string

                A text input to the model.

              • ResponseInputText object { text, type }

                A text input to the model.

              • OutputText object { text, type }

                A text output from the model.

                • text: string

                  The text output from the model.

                • type: "output_text"

                  The type of the output text. Always output_text.

                  • "output_text"

              • InputImage object { image_url, type, detail }

                An image input block used within EvalItem content arrays.

                • image_url: string

                  The URL of the image input.

                • type: "input_image"

                  The type of the image input. Always input_image.

                  • "input_image"

                • detail: optional string

                  The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

              • ResponseInputAudio object { input_audio, type }

                An audio input to the model.

              • GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more

                A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

            • role: "user" or "assistant" or "system" or "developer"

              The role of the message input. One of user, assistant, system, or developer.

              • "user"

              • "assistant"

              • "system"

              • "developer"

            • type: optional "message"

              The type of the message input. Always message.

              • "message"

        • type: "template"

          The type of input messages. Always template.

          • "template"

      • InputMessagesItemReference object { item_reference, type }

        • item_reference: string

          A reference to a variable in the item namespace. Ie, "item.name"

        • type: "item_reference"

          The type of input messages. Always item_reference.

          • "item_reference"

    • model: optional string

      The name of the model to use for generating completions (e.g. "o3-mini").

    • sampling_params: optional object { max_completion_tokens, reasoning_effort, seed, 4 more }

      • max_completion_tokens: optional number

        The maximum number of tokens in the generated output.

      • reasoning_effort: optional ReasoningEffort

        Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.

        • gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
        • All models before gpt-5.1 default to medium reasoning effort, and do not support none.
        • The gpt-5-pro model defaults to (and only supports) high reasoning effort.
        • xhigh is supported for all models after gpt-5.1-codex-max.

      • seed: optional number

        A seed value to initialize the randomness, during sampling.

      • temperature: optional number

        A higher temperature increases randomness in the outputs.

      • text: optional object { format }

        Configuration options for a text response from the model. Can be plain text or structured JSON data. Learn more:

        • Text inputs and outputs

        • Structured Outputs

        • format: optional ResponseFormatTextConfig

          An object specifying the format that the model must output.

          Configuring { "type": "json_schema" } enables Structured Outputs, which ensures the model will match your supplied JSON schema. Learn more in the Structured Outputs guide.

          The default format is { "type": "text" } with no additional options.

          Not recommended for gpt-4o and newer models:

          Setting to { "type": "json_object" } enables the older JSON mode, which ensures the message the model generates is valid JSON. Using json_schema is preferred for models that support it.

          • ResponseFormatText object { type }

            Default response format. Used to generate text responses.

          • ResponseFormatTextJSONSchemaConfig object { name, schema, type, 2 more }

            JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.

            • name: string

              The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

            • schema: map[unknown]

              The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

            • type: "json_schema"

              The type of response format being defined. Always json_schema.

              • "json_schema"

            • description: optional string

              A description of what the response format is for, used by the model to determine how to respond in the format.

            • strict: optional boolean

              Whether to enable strict schema adherence when generating the output. If set to true, the model will always follow the exact schema defined in the schema field. Only a subset of JSON Schema is supported when strict is true. To learn more, read the Structured Outputs guide.

          • ResponseFormatJSONObject object { type }

            JSON object response format. An older method of generating JSON responses. Using json_schema is recommended for models that support it. Note that the model will not generate JSON without a system or user message instructing it to do so.

      • tools: optional array of object { name, parameters, strict, 3 more } or object { type, vector_store_ids, filters, 2 more } or object { type } or 12 more

        An array of tools the model may call while generating a response. You can specify which tool to use by setting the tool_choice parameter.

        The two categories of tools you can provide the model are:

        • Built-in tools: Tools that are provided by OpenAI that extend the model's capabilities, like web search or file search. Learn more about built-in tools.

        • Function calls (custom tools): Functions that are defined by you, enabling the model to call your own code. Learn more about function calling.

        • Function object { name, parameters, strict, 3 more }

          Defines a function in your own code the model can choose to call. Learn more about function calling.

          • name: string

            The name of the function to call.

          • parameters: map[unknown]

            A JSON schema object describing the parameters of the function.

          • strict: boolean

            Whether to enforce strict parameter validation. Default true.

          • type: "function"

            The type of the function tool. Always function.

            • "function"

          • defer_loading: optional boolean

            Whether this function is deferred and loaded via tool search.

          • description: optional string

            A description of the function. Used by the model to determine whether or not to call the function.

        • FileSearch object { type, vector_store_ids, filters, 2 more }

          A tool that searches for relevant content from uploaded files. Learn more about the file search tool.

          • type: "file_search"

            The type of the file search tool. Always file_search.

            • "file_search"

          • vector_store_ids: array of string

            The IDs of the vector stores to search.

          • filters: optional ComparisonFilter or CompoundFilter

            A filter to apply.

            • ComparisonFilter object { key, type, value }

              A filter used to compare a specified attribute key to a given value using a defined comparison operation.

              • key: string

                The key to compare against the value.

              • type: "eq" or "ne" or "gt" or 5 more

                Specifies the comparison operator: eq, ne, gt, gte, lt, lte, in, nin.

                • eq: equals

                • ne: not equal

                • gt: greater than

                • gte: greater than or equal

                • lt: less than

                • lte: less than or equal

                • in: in

                • nin: not in

                • "eq"

                • "ne"

                • "gt"

                • "gte"

                • "lt"

                • "lte"

                • "in"

                • "nin"

              • value: string or number or boolean or array of string or number

                The value to compare against the attribute key; supports string, number, or boolean types.

                • string

                • number

                • boolean

                • array of string or number

                  • string

                  • number

            • CompoundFilter object { filters, type }

              Combine multiple filters using and or or.

              • filters: array of ComparisonFilter or unknown

                Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.

                • ComparisonFilter object { key, type, value }

                  A filter used to compare a specified attribute key to a given value using a defined comparison operation.

                • unknown

              • type: "and" or "or"

                Type of operation: and or or.

                • "and"

                • "or"

          • max_num_results: optional number

            The maximum number of results to return. This number should be between 1 and 50 inclusive.

          • ranking_options: optional object { hybrid_search, ranker, score_threshold }

            Ranking options for search.

            • hybrid_search: optional object { embedding_weight, text_weight }

              Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.

              • embedding_weight: number

                The weight of the embedding in the reciprocal ranking fusion.

              • text_weight: number

                The weight of the text in the reciprocal ranking fusion.

            • ranker: optional "auto" or "default-2024-11-15"

              The ranker to use for the file search.

              • "auto"

              • "default-2024-11-15"

            • score_threshold: optional number

              The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.

        • Computer object { type }

          A tool that controls a virtual computer. Learn more about the computer tool.

          • type: "computer"

            The type of the computer tool. Always computer.

            • "computer"

        • ComputerUsePreview object { display_height, display_width, environment, type }

          A tool that controls a virtual computer. Learn more about the computer tool.

          • display_height: number

            The height of the computer display.

          • display_width: number

            The width of the computer display.

          • environment: "windows" or "mac" or "linux" or 2 more

            The type of computer environment to control.

            • "windows"

            • "mac"

            • "linux"

            • "ubuntu"

            • "browser"

          • type: "computer_use_preview"

            The type of the computer use tool. Always computer_use_preview.

            • "computer_use_preview"

        • WebSearch object { type, filters, search_context_size, user_location }

          Search the Internet for sources related to the prompt. Learn more about the web search tool.

          • type: "web_search" or "web_search_2025_08_26"

            The type of the web search tool. One of web_search or web_search_2025_08_26.

            • "web_search"

            • "web_search_2025_08_26"

          • filters: optional object { allowed_domains }

            Filters for the search.

            • allowed_domains: optional array of string

              Allowed domains for the search. If not provided, all domains are allowed. Subdomains of the provided domains are allowed as well.

              Example: ["pubmed.ncbi.nlm.nih.gov"]

          • search_context_size: optional "low" or "medium" or "high"

            High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.

            • "low"

            • "medium"

            • "high"

          • user_location: optional object { city, country, region, 2 more }

            The approximate location of the user.

            • city: optional string

              Free text input for the city of the user, e.g. San Francisco.

            • country: optional string

              The two-letter ISO country code of the user, e.g. US.

            • region: optional string

              Free text input for the region of the user, e.g. California.

            • timezone: optional string

              The IANA timezone of the user, e.g. America/Los_Angeles.

            • type: optional "approximate"

              The type of location approximation. Always approximate.

              • "approximate"

        • Mcp object { server_label, type, allowed_tools, 8 more }

          Give the model access to additional tools via remote Model Context Protocol (MCP) servers. Learn more about MCP.

          • server_label: string

            A label for this MCP server, used to identify it in tool calls.

          • type: "mcp"

            The type of the MCP tool. Always mcp.

            • "mcp"

          • allowed_tools: optional array of string or object { read_only, tool_names }

            List of allowed tool names or a filter object.

            • McpAllowedTools = array of string

              A string array of allowed tool names

            • McpToolFilter object { read_only, tool_names }

              A filter object to specify which tools are allowed.

              • read_only: optional boolean

                Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

              • tool_names: optional array of string

                List of allowed tool names.

          • authorization: optional string

            An OAuth access token that can be used with a remote MCP server, either with a custom MCP server URL or a service connector. Your application must handle the OAuth authorization flow and provide the token here.

          • connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more

            Identifier for service connectors, like those available in ChatGPT. One of server_url, connector_id, or tunnel_id must be provided. Learn more about service connectors here.

            Currently supported connector_id values are:

            • Dropbox: connector_dropbox

            • Gmail: connector_gmail

            • Google Calendar: connector_googlecalendar

            • Google Drive: connector_googledrive

            • Microsoft Teams: connector_microsoftteams

            • Outlook Calendar: connector_outlookcalendar

            • Outlook Email: connector_outlookemail

            • SharePoint: connector_sharepoint

            • "connector_dropbox"

            • "connector_gmail"

            • "connector_googlecalendar"

            • "connector_googledrive"

            • "connector_microsoftteams"

            • "connector_outlookcalendar"

            • "connector_outlookemail"

            • "connector_sharepoint"

          • defer_loading: optional boolean

            Whether this MCP tool is deferred and discovered via tool search.

          • headers: optional map[string]

            Optional HTTP headers to send to the MCP server. Use for authentication or other purposes.

          • require_approval: optional object { always, never } or "always" or "never"

            Specify which of the MCP server's tools require approval.

            • McpToolApprovalFilter object { always, never }

              Specify which of the MCP server's tools require approval. Can be always, never, or a filter object associated with tools that require approval.

              • always: optional object { read_only, tool_names }

                A filter object to specify which tools are allowed.

                • read_only: optional boolean

                  Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

                • tool_names: optional array of string

                  List of allowed tool names.

              • never: optional object { read_only, tool_names }

                A filter object to specify which tools are allowed.

                • read_only: optional boolean

                  Indicates whether or not a tool modifies data or is read-only. If an MCP server is annotated with readOnlyHint, it will match this filter.

                • tool_names: optional array of string

                  List of allowed tool names.

            • McpToolApprovalSetting = "always" or "never"

              Specify a single approval policy for all tools. One of always or never. When set to always, all tools will require approval. When set to never, all tools will not require approval.

              • "always"

              • "never"

          • server_description: optional string

            Optional description of the MCP server, used to provide more context.

          • server_url: optional string

            The URL for the MCP server. One of server_url, connector_id, or tunnel_id must be provided.

          • tunnel_id: optional string

            The Secure MCP Tunnel ID to use instead of a direct server URL. One of server_url, connector_id, or tunnel_id must be provided.

        • CodeInterpreter object { container, type }

          A tool that runs Python code to help generate a response to a prompt.

          • container: string or object { type, file_ids, memory_limit, network_policy }

            The code interpreter container. Can be a container ID or an object that specifies uploaded file IDs to make available to your code, along with an optional memory_limit setting.

            • string

              The container ID.

            • CodeInterpreterToolAuto object { type, file_ids, memory_limit, network_policy }

              Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.

              • type: "auto"

                Always auto.

                • "auto"

              • file_ids: optional array of string

                An optional list of uploaded files to make available to your code.

              • memory_limit: optional "1g" or "4g" or "16g" or "64g"

                The memory limit for the code interpreter container.

                • "1g"

                • "4g"

                • "16g"

                • "64g"

              • network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist

                Network access policy for the container.

                • ContainerNetworkPolicyDisabled object { type }

                  • type: "disabled"

                    Disable outbound network access. Always disabled.

                    • "disabled"

                • ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }

                  • allowed_domains: array of string

                    A list of allowed domains when type is allowlist.

                  • type: "allowlist"

                    Allow outbound network access only to specified domains. Always allowlist.

                    • "allowlist"

                  • domain_secrets: optional array of ContainerNetworkPolicyDomainSecret

                    Optional domain-scoped secrets for allowlisted domains.

                    • domain: string

                      The domain associated with the secret.

                    • name: string

                      The name of the secret to inject for the domain.

                    • value: string

                      The secret value to inject for the domain.

          • type: "code_interpreter"

            The type of the code interpreter tool. Always code_interpreter.

            • "code_interpreter"

        • ImageGeneration object { type, action, background, 9 more }

          A tool that generates images using the GPT image models.

          • type: "image_generation"

            The type of the image generation tool. Always image_generation.

            • "image_generation"

          • action: optional "generate" or "edit" or "auto"

            Whether to generate a new image or edit an existing image. Default: auto.

            • "generate"

            • "edit"

            • "auto"

          • background: optional "transparent" or "opaque" or "auto"

            Background type for the generated image. One of transparent, opaque, or auto. Default: auto.

            • "transparent"

            • "opaque"

            • "auto"

          • input_fidelity: optional "high" or "low"

            Control how much effort the model will exert to match the style and features, especially facial features, of input images. This parameter is only supported for gpt-image-1 and gpt-image-1.5 and later models, unsupported for gpt-image-1-mini. Supports high and low. Defaults to low.

            • "high"

            • "low"

          • input_image_mask: optional object { file_id, image_url }

            Optional mask for inpainting. Contains image_url (string, optional) and file_id (string, optional).

            • file_id: optional string

              File ID for the mask image.

            • image_url: optional string

              Base64-encoded mask image.

          • model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"

            The image generation model to use. Default: gpt-image-1.

            • string

            • "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-1.5"

              The image generation model to use. Default: gpt-image-1.

              • "gpt-image-1"

              • "gpt-image-1-mini"

              • "gpt-image-1.5"

          • moderation: optional "auto" or "low"

            Moderation level for the generated image. Default: auto.

            • "auto"

            • "low"

          • output_compression: optional number

            Compression level for the output image. Default: 100.

          • output_format: optional "png" or "webp" or "jpeg"

            The output format of the generated image. One of png, webp, or jpeg. Default: png.

            • "png"

            • "webp"

            • "jpeg"

          • partial_images: optional number

            Number of partial images to generate in streaming mode, from 0 (default value) to 3.

          • quality: optional "low" or "medium" or "high" or "auto"

            The quality of the generated image. One of low, medium, high, or auto. Default: auto.

            • "low"

            • "medium"

            • "high"

            • "auto"

          • size: optional string or "1024x1024" or "1024x1536" or "1536x1024" or "auto"

            The size of the generated images. For gpt-image-2 and gpt-image-2-2026-04-21, arbitrary resolutions are supported as WIDTHxHEIGHT strings, for example 1536x864. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above 2560x1440 are experimental, and the maximum supported resolution is 3840x2160. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes 1024x1024, 1536x1024, and 1024x1536 are supported by the GPT image models; auto is supported for models that allow automatic sizing. For dall-e-2, use one of 256x256, 512x512, or 1024x1024. For dall-e-3, use one of 1024x1024, 1792x1024, or 1024x1792.

            • string

            • "1024x1024" or "1024x1536" or "1536x1024" or "auto"

              The size of the generated images. For gpt-image-2 and gpt-image-2-2026-04-21, arbitrary resolutions are supported as WIDTHxHEIGHT strings, for example 1536x864. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above 2560x1440 are experimental, and the maximum supported resolution is 3840x2160. The requested size must also satisfy the model's current pixel and edge limits. The standard sizes 1024x1024, 1536x1024, and 1024x1536 are supported by the GPT image models; auto is supported for models that allow automatic sizing. For dall-e-2, use one of 256x256, 512x512, or 1024x1024. For dall-e-3, use one of 1024x1024, 1792x1024, or 1024x1792.

              • "1024x1024"

              • "1024x1536"

              • "1536x1024"

              • "auto"

        • LocalShell object { type }

          A tool that allows the model to execute shell commands in a local environment.

          • type: "local_shell"

            The type of the local shell tool. Always local_shell.

            • "local_shell"

        • Shell object { type, environment }

          A tool that allows the model to execute shell commands.

          • type: "shell"

            The type of the shell tool. Always shell.

            • "shell"

          • environment: optional ContainerAuto or LocalEnvironment or ContainerReference

            • ContainerAuto object { type, file_ids, memory_limit, 2 more }

              • type: "container_auto"

                Automatically creates a container for this request

                • "container_auto"

              • file_ids: optional array of string

                An optional list of uploaded files to make available to your code.

              • memory_limit: optional "1g" or "4g" or "16g" or "64g"

                The memory limit for the container.

                • "1g"

                • "4g"

                • "16g"

                • "64g"

              • network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist

                Network access policy for the container.

                • ContainerNetworkPolicyDisabled object { type }

                • ContainerNetworkPolicyAllowlist object { allowed_domains, type, domain_secrets }

              • skills: optional array of SkillReference or InlineSkill

                An optional list of skills referenced by id or inline data.

                • SkillReference object { skill_id, type, version }

                  • skill_id: string

                    The ID of the referenced skill.

                  • type: "skill_reference"

                    References a skill created with the /v1/skills endpoint.

                    • "skill_reference"

                  • version: optional string

                    Optional skill version. Use a positive integer or 'latest'. Omit for default.

                • InlineSkill object { description, name, source, type }

                  • description: string

                    The description of the skill.

                  • name: string

                    The name of the skill.

                  • source: InlineSkillSource

                    Inline skill payload

                    • data: string

                      Base64-encoded skill zip bundle.

                    • media_type: "application/zip"

                      The media type of the inline skill payload. Must be application/zip.

                      • "application/zip"

                    • type: "base64"

                      The type of the inline skill source. Must be base64.

                      • "base64"

                  • type: "inline"

                    Defines an inline skill for this request.

                    • "inline"

            • LocalEnvironment object { type, skills }

              • type: "local"

                Use a local computer environment.

                • "local"

              • skills: optional array of LocalSkill

                An optional list of skills.

                • description: string

                  The description of the skill.

                • name: string

                  The name of the skill.

                • path: string

                  The path to the directory containing the skill.

            • ContainerReference object { container_id, type }

              • container_id: string

                The ID of the referenced container.

              • type: "container_reference"

                References a container created with the /v1/containers endpoint

                • "container_reference"

        • Custom object { name, type, defer_loading, 2 more }

          A custom tool that processes input using a specified format. Learn more about custom tools

          • name: string

            The name of the custom tool, used to identify it in tool calls.

          • type: "custom"

            The type of the custom tool. Always custom.

            • "custom"

          • defer_loading: optional boolean

            Whether this tool should be deferred and discovered via tool search.

          • description: optional string

            Optional description of the custom tool, used to provide more context.

          • format: optional CustomToolInputFormat

            The input format for the custom tool. Default is unconstrained text.

            • Text object { type }

              Unconstrained free-form text.

              • type: "text"

                Unconstrained text format. Always text.

                • "text"

            • Grammar object { definition, syntax, type }

              A grammar defined by the user.

              • definition: string

                The grammar definition.

              • syntax: "lark" or "regex"

                The syntax of the grammar definition. One of lark or regex.

                • "lark"

                • "regex"

              • type: "grammar"

                Grammar format. Always grammar.

                • "grammar"

        • Namespace object { description, name, tools, type }

          Groups function/custom tools under a shared namespace.

          • description: string

            A description of the namespace shown to the model.

          • name: string

            The namespace name used in tool calls (for example, crm).

          • tools: array of object { name, type, defer_loading, 3 more } or object { name, type, defer_loading, 2 more }

            The function/custom tools available inside this namespace.

            • Function object { name, type, defer_loading, 3 more }

              • name: string

              • type: "function"

                • "function"

              • defer_loading: optional boolean

                Whether this function should be deferred and discovered via tool search.

              • description: optional string

              • parameters: optional unknown

              • strict: optional boolean

            • Custom object { name, type, defer_loading, 2 more }

              A custom tool that processes input using a specified format. Learn more about custom tools

              • name: string

                The name of the custom tool, used to identify it in tool calls.

              • type: "custom"

                The type of the custom tool. Always custom.

                • "custom"

              • defer_loading: optional boolean

                Whether this tool should be deferred and discovered via tool search.

              • description: optional string

                Optional description of the custom tool, used to provide more context.

              • format: optional CustomToolInputFormat

                The input format for the custom tool. Default is unconstrained text.

          • type: "namespace"

            The type of the tool. Always namespace.

            • "namespace"

        • ToolSearch object { type, description, execution, parameters }

          Hosted or BYOT tool search configuration for deferred tools.

          • type: "tool_search"

            The type of the tool. Always tool_search.

            • "tool_search"

          • description: optional string

            Description shown to the model for a client-executed tool search tool.

          • execution: optional "server" or "client"

            Whether tool search is executed by the server or by the client.

            • "server"

            • "client"

          • parameters: optional unknown

            Parameter schema for a client-executed tool search tool.

        • WebSearchPreview object { type, search_content_types, search_context_size, user_location }

          This tool searches the web for relevant results to use in a response. Learn more about the web search tool.

          • type: "web_search_preview" or "web_search_preview_2025_03_11"

            The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.

            • "web_search_preview"

            • "web_search_preview_2025_03_11"

          • search_content_types: optional array of "text" or "image"

            • "text"

            • "image"

          • search_context_size: optional "low" or "medium" or "high"

            High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.

            • "low"

            • "medium"

            • "high"

          • user_location: optional object { type, city, country, 2 more }

            The user's location.

            • type: "approximate"

              The type of location approximation. Always approximate.

              • "approximate"

            • city: optional string

              Free text input for the city of the user, e.g. San Francisco.

            • country: optional string

              The two-letter ISO country code of the user, e.g. US.

            • region: optional string

              Free text input for the region of the user, e.g. California.

            • timezone: optional string

              The IANA timezone of the user, e.g. America/Los_Angeles.

        • ApplyPatch object { type }

          Allows the assistant to create, delete, or update files using unified diffs.

          • type: "apply_patch"

            The type of the tool. Always apply_patch.

            • "apply_patch"

      • top_p: optional number

        An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

• metadata: optional Metadata

  Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

  Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

• name: optional string

  The name of the run.