Graders

Grader Models

Domain Types

Grader Inputs

grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more

A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
- Text input: string
  
  A text input to the model.
- response_input_text: object { text, type }
  
  A text input to the model.
  - text: string
    
    The text input to the model.
  - type: "input_text"
    
    The type of the input item. Always input_text.
- Output text: object { text, type }
  
  A text output from the model.
  - text: string
    
    The text output from the model.
  - type: "output_text"
    
    The type of the output text. Always output_text.
- Input image: object { image_url, type, detail }
  
  An image input block used within EvalItem content arrays.
  - image_url: string
    
    The URL of the image input.
  - type: "input_image"
    
    The type of the image input. Always input_image.
  - detail: optional string
    
    The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
- response_input_audio: object { input_audio, type }
  
  An audio input to the model.
  - input_audio: object { data, format }
    - data: string
      
      Base64-encoded audio data.
    - format: "mp3" or "wav"
      
      The format of the audio data. Currently supported formats are mp3 and wav.
      - "mp3"
      - "wav"
  - type: "input_audio"
    
    The type of the input item. Always input_audio.

Label Model Grader

label_model_grader: object { input, labels, model, 3 more }

A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
- input: array of object { content, role, type }
  - content: string or ResponseInputText or object { text, type } or 3 more
    
    Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
    - Text input: string
      
      A text input to the model.
    - response_input_text: object { text, type }
      
      A text input to the model.
      - text: string
        
        The text input to the model.
      - type: "input_text"
        
        The type of the input item. Always input_text.
    - Output text: object { text, type }
      
      A text output from the model.
      - text: string
        
        The text output from the model.
      - type: "output_text"
        
        The type of the output text. Always output_text.
    - Input image: object { image_url, type, detail }
      
      An image input block used within EvalItem content arrays.
      - image_url: string
        
        The URL of the image input.
      - type: "input_image"
        
        The type of the image input. Always input_image.
      - detail: optional string
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
    - response_input_audio: object { input_audio, type }
      
      An audio input to the model.
      - input_audio: object { data, format }
        
        data: string
        
        Base64-encoded audio data.
        
        format: "mp3" or "wav"
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        "mp3"
        
        "wav"
      - type: "input_audio"
        
        The type of the input item. Always input_audio.
    - grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more
      
      A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
      - Text input: string
        
        A text input to the model.
      - response_input_text: object { text, type }
        
        A text input to the model.
      - Output text: object { text, type }
        
        A text output from the model.
        
        text: string
        
        The text output from the model.
        
        type: "output_text"
        
        The type of the output text. Always output_text.
      - Input image: object { image_url, type, detail }
        
        An image input block used within EvalItem content arrays.
        
        image_url: string
        
        The URL of the image input.
        
        type: "input_image"
        
        The type of the image input. Always input_image.
        
        detail: optional string
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
      - response_input_audio: object { input_audio, type }
        
        An audio input to the model.
  - role: "user" or "assistant" or "system" or "developer"
    
    The role of the message input. One of user, assistant, system, or developer.
    - "user"
    - "assistant"
    - "system"
    - "developer"
  - type: optional "message"
    
    The type of the message input. Always message.
    - "message"
- labels: array of string
  
  The labels to assign to each item in the evaluation.
- model: string
  
  The model to use for the evaluation. Must support structured outputs.
- name: string
  
  The name of the grader.
- passing_labels: array of string
  
  The labels that indicate a passing result. Must be a subset of labels.
- type: "label_model"
  
  The object type, which is always label_model.

Multi Grader

multi_grader: object { calculate_output, graders, name, type }

A MultiGrader object combines the output of multiple graders to produce a single score.
- calculate_output: string
  
  A formula to calculate the output based on grader results.
- graders: StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more
  
  A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
  - string_check_grader: object { input, name, operation, 2 more }
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - input: string
      
      The input text. This may include template strings.
    - name: string
      
      The name of the grader.
    - operation: "eq" or "ne" or "like" or "ilike"
      
      The string check operation to perform. One of eq, ne, like, or ilike.
      - "eq"
      - "ne"
      - "like"
      - "ilike"
    - reference: string
      
      The reference text. This may include template strings.
    - type: "string_check"
      
      The object type, which is always string_check.
  - text_similarity_grader: object { evaluation_metric, input, name, 2 more }
    
    A TextSimilarityGrader object which grades text based on similarity metrics.
    - evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more
      
      The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
      - "cosine"
      - "fuzzy_match"
      - "bleu"
      - "gleu"
      - "meteor"
      - "rouge_1"
      - "rouge_2"
      - "rouge_3"
      - "rouge_4"
      - "rouge_5"
      - "rouge_l"
    - input: string
      
      The text being graded.
    - name: string
      
      The name of the grader.
    - reference: string
      
      The text being graded against.
    - type: "text_similarity"
      
      The type of grader.
  - python_grader: object { name, source, type, image_tag }
    
    A PythonGrader object that runs a python script on the input.
    - name: string
      
      The name of the grader.
    - source: string
      
      The source code of the python script.
    - type: "python"
      
      The object type, which is always python.
    - image_tag: optional string
      
      The image tag to use for the python script.
  - score_model_grader: object { input, model, name, 3 more }
    
    A ScoreModelGrader object that uses a model to assign a score to the input.
    - input: array of object { content, role, type }
      
      The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
      - content: string or ResponseInputText or object { text, type } or 3 more
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        Text input: string
        
        A text input to the model.
        
        response_input_text: object { text, type }
        
        A text input to the model.
        
        text: string
        
        The text input to the model.
        
        type: "input_text"
        
        The type of the input item. Always input_text.
        
        Output text: object { text, type }
        
        A text output from the model.
        
        text: string
        
        The text output from the model.
        
        type: "output_text"
        
        The type of the output text. Always output_text.
        
        Input image: object { image_url, type, detail }
        
        An image input block used within EvalItem content arrays.
        
        image_url: string
        
        The URL of the image input.
        
        type: "input_image"
        
        The type of the image input. Always input_image.
        
        detail: optional string
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        response_input_audio: object { input_audio, type }
        
        An audio input to the model.
        
        input_audio: object { data, format }
        
        data: string
        
        Base64-encoded audio data.
        
        format: "mp3" or "wav"
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        "mp3"
        
        "wav"
        
        type: "input_audio"
        
        The type of the input item. Always input_audio.
        
        grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        Text input: string
        
        A text input to the model.
        
        response_input_text: object { text, type }
        
        A text input to the model.
        
        Output text: object { text, type }
        
        A text output from the model.
        
        text: string
        
        The text output from the model.
        
        type: "output_text"
        
        The type of the output text. Always output_text.
        
        Input image: object { image_url, type, detail }
        
        An image input block used within EvalItem content arrays.
        
        image_url: string
        
        The URL of the image input.
        
        type: "input_image"
        
        The type of the image input. Always input_image.
        
        detail: optional string
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        response_input_audio: object { input_audio, type }
        
        An audio input to the model.
      - role: "user" or "assistant" or "system" or "developer"
        
        The role of the message input. One of user, assistant, system, or developer.
        
        "user"
        
        "assistant"
        
        "system"
        
        "developer"
      - type: optional "message"
        
        The type of the message input. Always message.
        
        "message"
    - model: string
      
      The model to use for the evaluation.
    - name: string
      
      The name of the grader.
    - type: "score_model"
      
      The object type, which is always score_model.
    - range: optional array of number
      
      The range of the score. Defaults to [0, 1].
    - sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }
      
      The sampling parameters for the model.
      - max_completions_tokens: optional number
        
        The maximum number of tokens the grader model may generate in its response.
      - reasoning_effort: optional "none" or "minimal" or "low" or 3 more
        
        Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.
        
        gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
        
        All models before gpt-5.1 default to medium reasoning effort, and do not support none.
        
        The gpt-5-pro model defaults to (and only supports) high reasoning effort.
        
        xhigh is supported for all models after gpt-5.1-codex-max.
        
        "none"
        
        "minimal"
        
        "low"
        
        "medium"
        
        "high"
        
        "xhigh"
      - seed: optional number
        
        A seed value to initialize the randomness, during sampling.
      - temperature: optional number
        
        A higher temperature increases randomness in the outputs.
      - top_p: optional number
        
        An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
  - label_model_grader: object { input, labels, model, 3 more }
    
    A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
    - input: array of object { content, role, type }
      - content: string or ResponseInputText or object { text, type } or 3 more
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        Text input: string
        
        A text input to the model.
        
        response_input_text: object { text, type }
        
        A text input to the model.
        
        Output text: object { text, type }
        
        A text output from the model.
        
        text: string
        
        The text output from the model.
        
        type: "output_text"
        
        The type of the output text. Always output_text.
        
        Input image: object { image_url, type, detail }
        
        An image input block used within EvalItem content arrays.
        
        image_url: string
        
        The URL of the image input.
        
        type: "input_image"
        
        The type of the image input. Always input_image.
        
        detail: optional string
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        response_input_audio: object { input_audio, type }
        
        An audio input to the model.
        
        grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
      - role: "user" or "assistant" or "system" or "developer"
        
        The role of the message input. One of user, assistant, system, or developer.
        
        "user"
        
        "assistant"
        
        "system"
        
        "developer"
      - type: optional "message"
        
        The type of the message input. Always message.
        
        "message"
    - labels: array of string
      
      The labels to assign to each item in the evaluation.
    - model: string
      
      The model to use for the evaluation. Must support structured outputs.
    - name: string
      
      The name of the grader.
    - passing_labels: array of string
      
      The labels that indicate a passing result. Must be a subset of labels.
    - type: "label_model"
      
      The object type, which is always label_model.
- name: string
  
  The name of the grader.
- type: "multi"
  
  The object type, which is always multi.

Python Grader

python_grader: object { name, source, type, image_tag }

A PythonGrader object that runs a python script on the input.
- name: string
  
  The name of the grader.
- source: string
  
  The source code of the python script.
- type: "python"
  
  The object type, which is always python.
- image_tag: optional string
  
  The image tag to use for the python script.

Score Model Grader

score_model_grader: object { input, model, name, 3 more }

A ScoreModelGrader object that uses a model to assign a score to the input.
- input: array of object { content, role, type }
  
  The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
  - content: string or ResponseInputText or object { text, type } or 3 more
    
    Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
    - Text input: string
      
      A text input to the model.
    - response_input_text: object { text, type }
      
      A text input to the model.
      - text: string
        
        The text input to the model.
      - type: "input_text"
        
        The type of the input item. Always input_text.
    - Output text: object { text, type }
      
      A text output from the model.
      - text: string
        
        The text output from the model.
      - type: "output_text"
        
        The type of the output text. Always output_text.
    - Input image: object { image_url, type, detail }
      
      An image input block used within EvalItem content arrays.
      - image_url: string
        
        The URL of the image input.
      - type: "input_image"
        
        The type of the image input. Always input_image.
      - detail: optional string
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
    - response_input_audio: object { input_audio, type }
      
      An audio input to the model.
      - input_audio: object { data, format }
        
        data: string
        
        Base64-encoded audio data.
        
        format: "mp3" or "wav"
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        "mp3"
        
        "wav"
      - type: "input_audio"
        
        The type of the input item. Always input_audio.
    - grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more
      
      A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
      - Text input: string
        
        A text input to the model.
      - response_input_text: object { text, type }
        
        A text input to the model.
      - Output text: object { text, type }
        
        A text output from the model.
        
        text: string
        
        The text output from the model.
        
        type: "output_text"
        
        The type of the output text. Always output_text.
      - Input image: object { image_url, type, detail }
        
        An image input block used within EvalItem content arrays.
        
        image_url: string
        
        The URL of the image input.
        
        type: "input_image"
        
        The type of the image input. Always input_image.
        
        detail: optional string
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
      - response_input_audio: object { input_audio, type }
        
        An audio input to the model.
  - role: "user" or "assistant" or "system" or "developer"
    
    The role of the message input. One of user, assistant, system, or developer.
    - "user"
    - "assistant"
    - "system"
    - "developer"
  - type: optional "message"
    
    The type of the message input. Always message.
    - "message"
- model: string
  
  The model to use for the evaluation.
- name: string
  
  The name of the grader.
- type: "score_model"
  
  The object type, which is always score_model.
- range: optional array of number
  
  The range of the score. Defaults to [0, 1].
- sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }
  
  The sampling parameters for the model.
  - max_completions_tokens: optional number
    
    The maximum number of tokens the grader model may generate in its response.
  - reasoning_effort: optional "none" or "minimal" or "low" or 3 more
    
    Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.
    - gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
    - All models before gpt-5.1 default to medium reasoning effort, and do not support none.
    - The gpt-5-pro model defaults to (and only supports) high reasoning effort.
    - xhigh is supported for all models after gpt-5.1-codex-max.
    - "none"
    - "minimal"
    - "low"
    - "medium"
    - "high"
    - "xhigh"
  - seed: optional number
    
    A seed value to initialize the randomness, during sampling.
  - temperature: optional number
    
    A higher temperature increases randomness in the outputs.
  - top_p: optional number
    
    An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

String Check Grader

string_check_grader: object { input, name, operation, 2 more }

A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
- input: string
  
  The input text. This may include template strings.
- name: string
  
  The name of the grader.
- operation: "eq" or "ne" or "like" or "ilike"
  
  The string check operation to perform. One of eq, ne, like, or ilike.
  - "eq"
  - "ne"
  - "like"
  - "ilike"
- reference: string
  
  The reference text. This may include template strings.
- type: "string_check"
  
  The object type, which is always string_check.

Text Similarity Grader

text_similarity_grader: object { evaluation_metric, input, name, 2 more }

A TextSimilarityGrader object which grades text based on similarity metrics.
- evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more
  
  The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
  - "cosine"
  - "fuzzy_match"
  - "bleu"
  - "gleu"
  - "meteor"
  - "rouge_1"
  - "rouge_2"
  - "rouge_3"
  - "rouge_4"
  - "rouge_5"
  - "rouge_l"
- input: string
  
  The text being graded.
- name: string
  
  The name of the grader.
- reference: string
  
  The text being graded against.
- type: "text_similarity"
  
  The type of grader.

cli/resources/graders/index.md +976 −0 created

1# Graders

3# Grader Models

5## Domain Types

7### Grader Inputs

9- `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`

11 A list of inputs, each of which may be either an input text, output text, input

12 image, or input audio object.

14 - `Text input: string`

16 A text input to the model.

18 - `response_input_text: object { text, type }`

20 A text input to the model.

22 - `text: string`

24 The text input to the model.

26 - `type: "input_text"`

28 The type of the input item. Always `input_text`.

30 - `Output text: object { text, type }`

32 A text output from the model.

34 - `text: string`

36 The text output from the model.

38 - `type: "output_text"`

40 The type of the output text. Always `output_text`.

42 - `Input image: object { image_url, type, detail }`

44 An image input block used within EvalItem content arrays.

46 - `image_url: string`

48 The URL of the image input.

50 - `type: "input_image"`

52 The type of the image input. Always `input_image`.

54 - `detail: optional string`

56 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

58 - `response_input_audio: object { input_audio, type }`

60 An audio input to the model.

62 - `input_audio: object { data, format }`

64 - `data: string`

66 Base64-encoded audio data.

68 - `format: "mp3" or "wav"`

70 The format of the audio data. Currently supported formats are `mp3` and

71 `wav`.

73 - `"mp3"`

75 - `"wav"`

77 - `type: "input_audio"`

79 The type of the input item. Always `input_audio`.

81### Label Model Grader

83- `label_model_grader: object { input, labels, model, 3 more }`

85 A LabelModelGrader object which uses a model to assign labels to each item

86 in the evaluation.

88 - `input: array of object { content, role, type }`

90 - `content: string or ResponseInputText or object { text, type } or 3 more`

92 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

94 - `Text input: string`

96 A text input to the model.

98 - `response_input_text: object { text, type }`

100 A text input to the model.

101

102 - `text: string`

103

104 The text input to the model.

105

106 - `type: "input_text"`

107

108 The type of the input item. Always `input_text`.

109

110 - `Output text: object { text, type }`

111

112 A text output from the model.

113

114 - `text: string`

115

116 The text output from the model.

117

118 - `type: "output_text"`

119

120 The type of the output text. Always `output_text`.

121

122 - `Input image: object { image_url, type, detail }`

123

124 An image input block used within EvalItem content arrays.

125

126 - `image_url: string`

127

128 The URL of the image input.

129

130 - `type: "input_image"`

131

132 The type of the image input. Always `input_image`.

133

134 - `detail: optional string`

135

136 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

137

138 - `response_input_audio: object { input_audio, type }`

139

140 An audio input to the model.

141

142 - `input_audio: object { data, format }`

143

144 - `data: string`

145

146 Base64-encoded audio data.

147

148 - `format: "mp3" or "wav"`

149

150 The format of the audio data. Currently supported formats are `mp3` and

151 `wav`.

152

153 - `"mp3"`

154

155 - `"wav"`

156

157 - `type: "input_audio"`

158

159 The type of the input item. Always `input_audio`.

160

161 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`

162

163 A list of inputs, each of which may be either an input text, output text, input

164 image, or input audio object.

165

166 - `Text input: string`

167

168 A text input to the model.

169

170 - `response_input_text: object { text, type }`

171

172 A text input to the model.

173

174 - `Output text: object { text, type }`

175

176 A text output from the model.

177

178 - `text: string`

179

180 The text output from the model.

181

182 - `type: "output_text"`

183

184 The type of the output text. Always `output_text`.

185

186 - `Input image: object { image_url, type, detail }`

187

188 An image input block used within EvalItem content arrays.

189

190 - `image_url: string`

191

192 The URL of the image input.

193

194 - `type: "input_image"`

195

196 The type of the image input. Always `input_image`.

197

198 - `detail: optional string`

199

200 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

201

202 - `response_input_audio: object { input_audio, type }`

203

204 An audio input to the model.

205

206 - `role: "user" or "assistant" or "system" or "developer"`

207

208 The role of the message input. One of `user`, `assistant`, `system`, or

209 `developer`.

210

211 - `"user"`

212

213 - `"assistant"`

214

215 - `"system"`

216

217 - `"developer"`

218

219 - `type: optional "message"`

220

221 The type of the message input. Always `message`.

222

223 - `"message"`

224

225 - `labels: array of string`

226

227 The labels to assign to each item in the evaluation.

228

229 - `model: string`

230

231 The model to use for the evaluation. Must support structured outputs.

232

233 - `name: string`

234

235 The name of the grader.

236

237 - `passing_labels: array of string`

238

239 The labels that indicate a passing result. Must be a subset of labels.

240

241 - `type: "label_model"`

242

243 The object type, which is always `label_model`.

244

245### Multi Grader

246

247- `multi_grader: object { calculate_output, graders, name, type }`

248

249 A MultiGrader object combines the output of multiple graders to produce a single score.

250

251 - `calculate_output: string`

252

253 A formula to calculate the output based on grader results.

254

255 - `graders: StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more`

256

257 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

258

259 - `string_check_grader: object { input, name, operation, 2 more }`

260

261 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

262

263 - `input: string`

264

265 The input text. This may include template strings.

266

267 - `name: string`

268

269 The name of the grader.

270

271 - `operation: "eq" or "ne" or "like" or "ilike"`

272

273 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

274

275 - `"eq"`

276

277 - `"ne"`

278

279 - `"like"`

280

281 - `"ilike"`

282

283 - `reference: string`

284

285 The reference text. This may include template strings.

286

287 - `type: "string_check"`

288

289 The object type, which is always `string_check`.

290

291 - `text_similarity_grader: object { evaluation_metric, input, name, 2 more }`

292

293 A TextSimilarityGrader object which grades text based on similarity metrics.

294

295 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`

296

297 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

298 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

299 or `rouge_l`.

300

301 - `"cosine"`

302

303 - `"fuzzy_match"`

304

305 - `"bleu"`

306

307 - `"gleu"`

308

309 - `"meteor"`

310

311 - `"rouge_1"`

312

313 - `"rouge_2"`

314

315 - `"rouge_3"`

316

317 - `"rouge_4"`

318

319 - `"rouge_5"`

320

321 - `"rouge_l"`

322

323 - `input: string`

324

325 The text being graded.

326

327 - `name: string`

328

329 The name of the grader.

330

331 - `reference: string`

332

333 The text being graded against.

334

335 - `type: "text_similarity"`

336

337 The type of grader.

338

339 - `python_grader: object { name, source, type, image_tag }`

340

341 A PythonGrader object that runs a python script on the input.

342

343 - `name: string`

344

345 The name of the grader.

346

347 - `source: string`

348

349 The source code of the python script.

350

351 - `type: "python"`

352

353 The object type, which is always `python`.

354

355 - `image_tag: optional string`

356

357 The image tag to use for the python script.

358

359 - `score_model_grader: object { input, model, name, 3 more }`

360

361 A ScoreModelGrader object that uses a model to assign a score to the input.

362

363 - `input: array of object { content, role, type }`

364

365 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

366

367 - `content: string or ResponseInputText or object { text, type } or 3 more`

368

369 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

370

371 - `Text input: string`

372

373 A text input to the model.

374

375 - `response_input_text: object { text, type }`

376

377 A text input to the model.

378

379 - `text: string`

380

381 The text input to the model.

382

383 - `type: "input_text"`

384

385 The type of the input item. Always `input_text`.

386

387 - `Output text: object { text, type }`

388

389 A text output from the model.

390

391 - `text: string`

392

393 The text output from the model.

394

395 - `type: "output_text"`

396

397 The type of the output text. Always `output_text`.

398

399 - `Input image: object { image_url, type, detail }`

400

401 An image input block used within EvalItem content arrays.

402

403 - `image_url: string`

404

405 The URL of the image input.

406

407 - `type: "input_image"`

408

409 The type of the image input. Always `input_image`.

410

411 - `detail: optional string`

412

413 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

414

415 - `response_input_audio: object { input_audio, type }`

416

417 An audio input to the model.

418

419 - `input_audio: object { data, format }`

420

421 - `data: string`

422

423 Base64-encoded audio data.

424

425 - `format: "mp3" or "wav"`

426

427 The format of the audio data. Currently supported formats are `mp3` and

428 `wav`.

429

430 - `"mp3"`

431

432 - `"wav"`

433

434 - `type: "input_audio"`

435

436 The type of the input item. Always `input_audio`.

437

438 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`

439

440 A list of inputs, each of which may be either an input text, output text, input

441 image, or input audio object.

442

443 - `Text input: string`

444

445 A text input to the model.

446

447 - `response_input_text: object { text, type }`

448

449 A text input to the model.

450

451 - `Output text: object { text, type }`

452

453 A text output from the model.

454

455 - `text: string`

456

457 The text output from the model.

458

459 - `type: "output_text"`

460

461 The type of the output text. Always `output_text`.

462

463 - `Input image: object { image_url, type, detail }`

464

465 An image input block used within EvalItem content arrays.

466

467 - `image_url: string`

468

469 The URL of the image input.

470

471 - `type: "input_image"`

472

473 The type of the image input. Always `input_image`.

474

475 - `detail: optional string`

476

477 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

478

479 - `response_input_audio: object { input_audio, type }`

480

481 An audio input to the model.

482

483 - `role: "user" or "assistant" or "system" or "developer"`

484

485 The role of the message input. One of `user`, `assistant`, `system`, or

486 `developer`.

487

488 - `"user"`

489

490 - `"assistant"`

491

492 - `"system"`

493

494 - `"developer"`

495

496 - `type: optional "message"`

497

498 The type of the message input. Always `message`.

499

500 - `"message"`

501

502 - `model: string`

503

504 The model to use for the evaluation.

505

506 - `name: string`

507

508 The name of the grader.

509

510 - `type: "score_model"`

511

512 The object type, which is always `score_model`.

513

514 - `range: optional array of number`

515

516 The range of the score. Defaults to `[0, 1]`.

517

518 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`

519

520 The sampling parameters for the model.

521

522 - `max_completions_tokens: optional number`

523

524 The maximum number of tokens the grader model may generate in its response.

525

526 - `reasoning_effort: optional "none" or "minimal" or "low" or 3 more`

527

528 Constrains effort on reasoning for

529 [reasoning models](https://platform.openai.com/docs/guides/reasoning).

530 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing

531 reasoning effort can result in faster responses and fewer tokens used

532 on reasoning in a response.

533

534 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.

535 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.

536 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.

537 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.

538

539 - `"none"`

540

541 - `"minimal"`

542

543 - `"low"`

544

545 - `"medium"`

546

547 - `"high"`

548

549 - `"xhigh"`

550

551 - `seed: optional number`

552

553 A seed value to initialize the randomness, during sampling.

554

555 - `temperature: optional number`

556

557 A higher temperature increases randomness in the outputs.

558

559 - `top_p: optional number`

560

561 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

562

563 - `label_model_grader: object { input, labels, model, 3 more }`

564

565 A LabelModelGrader object which uses a model to assign labels to each item

566 in the evaluation.

567

568 - `input: array of object { content, role, type }`

569

570 - `content: string or ResponseInputText or object { text, type } or 3 more`

571

572 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

573

574 - `Text input: string`

575

576 A text input to the model.

577

578 - `response_input_text: object { text, type }`

579

580 A text input to the model.

581

582 - `Output text: object { text, type }`

583

584 A text output from the model.

585

586 - `text: string`

587

588 The text output from the model.

589

590 - `type: "output_text"`

591

592 The type of the output text. Always `output_text`.

593

594 - `Input image: object { image_url, type, detail }`

595

596 An image input block used within EvalItem content arrays.

597

598 - `image_url: string`

599

600 The URL of the image input.

601

602 - `type: "input_image"`

603

604 The type of the image input. Always `input_image`.

605

606 - `detail: optional string`

607

608 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

609

610 - `response_input_audio: object { input_audio, type }`

611

612 An audio input to the model.

613

614 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`

615

616 A list of inputs, each of which may be either an input text, output text, input

617 image, or input audio object.

618

619 - `role: "user" or "assistant" or "system" or "developer"`

620

621 The role of the message input. One of `user`, `assistant`, `system`, or

622 `developer`.

623

624 - `"user"`

625

626 - `"assistant"`

627

628 - `"system"`

629

630 - `"developer"`

631

632 - `type: optional "message"`

633

634 The type of the message input. Always `message`.

635

636 - `"message"`

637

638 - `labels: array of string`

639

640 The labels to assign to each item in the evaluation.

641

642 - `model: string`

643

644 The model to use for the evaluation. Must support structured outputs.

645

646 - `name: string`

647

648 The name of the grader.

649

650 - `passing_labels: array of string`

651

652 The labels that indicate a passing result. Must be a subset of labels.

653

654 - `type: "label_model"`

655

656 The object type, which is always `label_model`.

657

658 - `name: string`

659

660 The name of the grader.

661

662 - `type: "multi"`

663

664 The object type, which is always `multi`.

665

666### Python Grader

667

668- `python_grader: object { name, source, type, image_tag }`

669

670 A PythonGrader object that runs a python script on the input.

671

672 - `name: string`

673

674 The name of the grader.

675

676 - `source: string`

677

678 The source code of the python script.

679

680 - `type: "python"`

681

682 The object type, which is always `python`.

683

684 - `image_tag: optional string`

685

686 The image tag to use for the python script.

687

688### Score Model Grader

689

690- `score_model_grader: object { input, model, name, 3 more }`

691

692 A ScoreModelGrader object that uses a model to assign a score to the input.

693

694 - `input: array of object { content, role, type }`

695

696 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

697

698 - `content: string or ResponseInputText or object { text, type } or 3 more`

699

700 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

701

702 - `Text input: string`

703

704 A text input to the model.

705

706 - `response_input_text: object { text, type }`

707

708 A text input to the model.

709

710 - `text: string`

711

712 The text input to the model.

713

714 - `type: "input_text"`

715

716 The type of the input item. Always `input_text`.

717

718 - `Output text: object { text, type }`

719

720 A text output from the model.

721

722 - `text: string`

723

724 The text output from the model.

725

726 - `type: "output_text"`

727

728 The type of the output text. Always `output_text`.

729

730 - `Input image: object { image_url, type, detail }`

731

732 An image input block used within EvalItem content arrays.

733

734 - `image_url: string`

735

736 The URL of the image input.

737

738 - `type: "input_image"`

739

740 The type of the image input. Always `input_image`.

741

742 - `detail: optional string`

743

744 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

745

746 - `response_input_audio: object { input_audio, type }`

747

748 An audio input to the model.

749

750 - `input_audio: object { data, format }`

751

752 - `data: string`

753

754 Base64-encoded audio data.

755

756 - `format: "mp3" or "wav"`

757

758 The format of the audio data. Currently supported formats are `mp3` and

759 `wav`.

760

761 - `"mp3"`

762

763 - `"wav"`

764

765 - `type: "input_audio"`

766

767 The type of the input item. Always `input_audio`.

768

769 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`

770

771 A list of inputs, each of which may be either an input text, output text, input

772 image, or input audio object.

773

774 - `Text input: string`

775

776 A text input to the model.

777

778 - `response_input_text: object { text, type }`

779

780 A text input to the model.

781

782 - `Output text: object { text, type }`

783

784 A text output from the model.

785

786 - `text: string`

787

788 The text output from the model.

789

790 - `type: "output_text"`

791

792 The type of the output text. Always `output_text`.

793

794 - `Input image: object { image_url, type, detail }`

795

796 An image input block used within EvalItem content arrays.

797

798 - `image_url: string`

799

800 The URL of the image input.

801

802 - `type: "input_image"`

803

804 The type of the image input. Always `input_image`.

805

806 - `detail: optional string`

807

808 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

809

810 - `response_input_audio: object { input_audio, type }`

811

812 An audio input to the model.

813

814 - `role: "user" or "assistant" or "system" or "developer"`

815

816 The role of the message input. One of `user`, `assistant`, `system`, or

817 `developer`.

818

819 - `"user"`

820

821 - `"assistant"`

822

823 - `"system"`

824

825 - `"developer"`

826

827 - `type: optional "message"`

828

829 The type of the message input. Always `message`.

830

831 - `"message"`

832

833 - `model: string`

834

835 The model to use for the evaluation.

836

837 - `name: string`

838

839 The name of the grader.

840

841 - `type: "score_model"`

842

843 The object type, which is always `score_model`.

844

845 - `range: optional array of number`

846

847 The range of the score. Defaults to `[0, 1]`.

848

849 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`

850

851 The sampling parameters for the model.

852

853 - `max_completions_tokens: optional number`

854

855 The maximum number of tokens the grader model may generate in its response.

856

857 - `reasoning_effort: optional "none" or "minimal" or "low" or 3 more`

858

859 Constrains effort on reasoning for

860 [reasoning models](https://platform.openai.com/docs/guides/reasoning).

861 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing

862 reasoning effort can result in faster responses and fewer tokens used

863 on reasoning in a response.

864

865 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.

866 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.

867 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.

868 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.

869

870 - `"none"`

871

872 - `"minimal"`

873

874 - `"low"`

875

876 - `"medium"`

877

878 - `"high"`

879

880 - `"xhigh"`

881

882 - `seed: optional number`

883

884 A seed value to initialize the randomness, during sampling.

885

886 - `temperature: optional number`

887

888 A higher temperature increases randomness in the outputs.

889

890 - `top_p: optional number`

891

892 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

893

894### String Check Grader

895

896- `string_check_grader: object { input, name, operation, 2 more }`

897

898 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

899

900 - `input: string`

901

902 The input text. This may include template strings.

903

904 - `name: string`

905

906 The name of the grader.

907

908 - `operation: "eq" or "ne" or "like" or "ilike"`

909

910 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

911

912 - `"eq"`

913

914 - `"ne"`

915

916 - `"like"`

917

918 - `"ilike"`

919

920 - `reference: string`

921

922 The reference text. This may include template strings.

923

924 - `type: "string_check"`

925

926 The object type, which is always `string_check`.

927

928### Text Similarity Grader

929

930- `text_similarity_grader: object { evaluation_metric, input, name, 2 more }`

931

932 A TextSimilarityGrader object which grades text based on similarity metrics.

933

934 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`

935

936 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

937 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

938 or `rouge_l`.

939

940 - `"cosine"`

941

942 - `"fuzzy_match"`

943

944 - `"bleu"`

945

946 - `"gleu"`

947

948 - `"meteor"`

949

950 - `"rouge_1"`

951

952 - `"rouge_2"`

953

954 - `"rouge_3"`

955

956 - `"rouge_4"`

957

958 - `"rouge_5"`

959

960 - `"rouge_l"`

961

962 - `input: string`

963

964 The text being graded.

965

966 - `name: string`

967

968 The name of the grader.

969

970 - `reference: string`

971

972 The text being graded against.

973

974 - `type: "text_similarity"`

975

976 The type of grader.