ruby/resources/evals/methods/create/index.md

Create eval

evals.create(**kwargs) -> EvalCreateResponse

post /evals

Create eval

Parameters

data_source_config: Custom{ item_schema, type, include_sample_schema} | Logs{ type, metadata} | StoredCompletions{ type, metadata}

The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
- class Custom
  
  A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs. This schema is used to define the shape of the data that will be:
  - Used to define your testing criteria and
  - What data is required when creating a run
  - item_schema: Hash[Symbol, untyped]
    
    The json schema for each row in the data source.
  - type: :custom
    
    The type of data source. Always custom.
    - :custom
  - include_sample_schema: bool
    
    Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)
- class Logs
  
  A data source config which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc.
  - type: :logs
    
    The type of data source. Always logs.
    - :logs
  - metadata: Hash[Symbol, untyped]
    
    Metadata filters for the logs data source.
- class StoredCompletions
  
  Deprecated in favor of LogsDataSourceConfig.
  - type: :stored_completions
    
    The type of data source. Always stored_completions.
    - :stored_completions
  - metadata: Hash[Symbol, untyped]
    
    Metadata filters for the stored completions data source.
testing_criteria: Array[LabelModel{ input, labels, model, 3 more} | StringCheckGrader | TextSimilarityGrader & { pass_threshold} | 2 more]

A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like {{item.variable_name}}. To reference the model's output, use the sample namespace (ie, {{sample.output_text}}).
- class LabelModel
  
  A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
  - input: Array[SimpleInputMessage{ content, role} | EvalItem{ content, role, type}]
    
    A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
    - class SimpleInputMessage
      - content: String
        
        The content of the message.
      - role: String
        
        The role of the message (e.g. "system", "assistant", "user").
    - class EvalItem
      
      A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.
      - content: String | ResponseInputText | OutputText{ text, type} | 3 more
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        text: String
        
        The text input to the model.
        
        type: :input_text
        
        The type of the input item. Always input_text.
        
        :input_text
        
        prompt_cache_breakpoint: PromptCacheBreakpoint{ mode}
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        mode: :explicit
        
        The breakpoint mode. Always explicit.
        
        :explicit
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
        
        input_audio: InputAudio{ data, format_}
        
        data: String
        
        Base64-encoded audio data.
        
        format_: :mp3 | :wav
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        :mp3
        
        :wav
        
        type: :input_audio
        
        The type of the input item. Always input_audio.
        
        :input_audio
        
        GraderInputs = Array[GraderInputItem]
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
      - role: :user | :assistant | :system | :developer
        
        The role of the message input. One of user, assistant, system, or developer.
        
        :user
        
        :assistant
        
        :system
        
        :developer
      - type: :message
        
        The type of the message input. Always message.
        
        :message
  - labels: Array[String]
    
    The labels to classify to each item in the evaluation.
  - model: String
    
    The model to use for the evaluation. Must support structured outputs.
  - name: String
    
    The name of the grader.
  - passing_labels: Array[String]
    
    The labels that indicate a passing result. Must be a subset of labels.
  - type: :label_model
    
    The object type, which is always label_model.
    - :label_model
- class StringCheckGrader
  
  A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
  - input: String
    
    The input text. This may include template strings.
  - name: String
    
    The name of the grader.
  - operation: :eq | :ne | :like | :ilike
    
    The string check operation to perform. One of eq, ne, like, or ilike.
    - :eq
    - :ne
    - :like
    - :ilike
  - reference: String
    
    The reference text. This may include template strings.
  - type: :string_check
    
    The object type, which is always string_check.
    - :string_check
- class TextSimilarity
  
  A TextSimilarityGrader object which grades text based on similarity metrics.
  - pass_threshold: Float
    
    The threshold for the score.
- class Python
  
  A PythonGrader object that runs a python script on the input.
  - pass_threshold: Float
    
    The threshold for the score.
- class ScoreModel
  
  A ScoreModelGrader object that uses a model to assign a score to the input.
  - pass_threshold: Float
    
    The threshold for the score.
metadata: Metadata

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
name: String

The name of the evaluation.

Returns

class EvalCreateResponse

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:
- Improve the quality of my chatbot
- See how well my chatbot handles customer support
- Check if o4-mini is better at my usecase than gpt-4o
- id: String
  
  Unique identifier for the evaluation.
- created_at: Integer
  
  The Unix timestamp (in seconds) for when the eval was created.
- data_source_config: EvalCustomDataSourceConfig | Logs{ schema, type, metadata} | EvalStoredCompletionsDataSourceConfig
  
  Configuration of data sources used in runs of the evaluation.
  - class EvalCustomDataSourceConfig
    
    A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:
    - Used to define your testing criteria and
    - What data is required when creating a run
    - schema: Hash[Symbol, untyped]
      
      The json schema for the run data source items. Learn how to build JSON schemas here.
    - type: :custom
      
      The type of data source. Always custom.
      - :custom
  - class Logs
    
    A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals. item and sample are both defined when using this data source config.
    - schema: Hash[Symbol, untyped]
      
      The json schema for the run data source items. Learn how to build JSON schemas here.
    - type: :logs
      
      The type of data source. Always logs.
      - :logs
    - metadata: Metadata
      
      Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
      
      Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
  - class EvalStoredCompletionsDataSourceConfig
    
    Deprecated in favor of LogsDataSourceConfig.
    - schema: Hash[Symbol, untyped]
      
      The json schema for the run data source items. Learn how to build JSON schemas here.
    - type: :stored_completions
      
      The type of data source. Always stored_completions.
      - :stored_completions
    - metadata: Metadata
      
      Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
      
      Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
- metadata: Metadata
  
  Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
  
  Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
- name: String
  
  The name of the evaluation.
- object: :eval
  
  The object type.
  - :eval
- testing_criteria: Array[LabelModelGrader | StringCheckGrader | TextSimilarityGrader & { pass_threshold} | 2 more]
  
  A list of testing criteria.
  - class LabelModelGrader
    
    A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
    - input: Array[Input{ content, role, type}]
      - content: String | ResponseInputText | OutputText{ text, type} | 3 more
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        text: String
        
        The text input to the model.
        
        type: :input_text
        
        The type of the input item. Always input_text.
        
        :input_text
        
        prompt_cache_breakpoint: PromptCacheBreakpoint{ mode}
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        mode: :explicit
        
        The breakpoint mode. Always explicit.
        
        :explicit
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
        
        input_audio: InputAudio{ data, format_}
        
        data: String
        
        Base64-encoded audio data.
        
        format_: :mp3 | :wav
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        :mp3
        
        :wav
        
        type: :input_audio
        
        The type of the input item. Always input_audio.
        
        :input_audio
        
        GraderInputs = Array[GraderInputItem]
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
      - role: :user | :assistant | :system | :developer
        
        The role of the message input. One of user, assistant, system, or developer.
        
        :user
        
        :assistant
        
        :system
        
        :developer
      - type: :message
        
        The type of the message input. Always message.
        
        :message
    - labels: Array[String]
      
      The labels to assign to each item in the evaluation.
    - model: String
      
      The model to use for the evaluation. Must support structured outputs.
    - name: String
      
      The name of the grader.
    - passing_labels: Array[String]
      
      The labels that indicate a passing result. Must be a subset of labels.
    - type: :label_model
      
      The object type, which is always label_model.
      - :label_model
  - class StringCheckGrader
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - input: String
      
      The input text. This may include template strings.
    - name: String
      
      The name of the grader.
    - operation: :eq | :ne | :like | :ilike
      
      The string check operation to perform. One of eq, ne, like, or ilike.
      - :eq
      - :ne
      - :like
      - :ilike
    - reference: String
      
      The reference text. This may include template strings.
    - type: :string_check
      
      The object type, which is always string_check.
      - :string_check
  - class EvalGraderTextSimilarity
    
    A TextSimilarityGrader object which grades text based on similarity metrics.
    - pass_threshold: Float
      
      The threshold for the score.
  - class EvalGraderPython
    
    A PythonGrader object that runs a python script on the input.
    - pass_threshold: Float
      
      The threshold for the score.
  - class EvalGraderScoreModel
    
    A ScoreModelGrader object that uses a model to assign a score to the input.
    - pass_threshold: Float
      
      The threshold for the score.

Example

require "openai"

openai = OpenAI::Client.new(api_key: "My API Key")

eval_ = openai.evals.create(
  data_source_config: {item_schema: {foo: "bar"}, type: :custom},
  testing_criteria: [
    {
      input: [{content: "content", role: "role"}],
      labels: ["string"],
      model: "model",
      name: "name",
      passing_labels: ["string"],
      type: :label_model
    }
  ]
)

puts(eval_)

Response

{
  "id": "id",
  "created_at": 0,
  "data_source_config": {
    "schema": {
      "foo": "bar"
    },
    "type": "custom"
  },
  "metadata": {
    "foo": "string"
  },
  "name": "Chatbot effectiveness Evaluation",
  "object": "eval",
  "testing_criteria": [
    {
      "input": [
        {
          "content": "string",
          "role": "user",
          "type": "message"
        }
      ],
      "labels": [
        "string"
      ],
      "model": "model",
      "name": "name",
      "passing_labels": [
        "string"
      ],
      "type": "label_model"
    }
  ]
}

ruby/resources/evals/methods/create/index.md +770 −0 created

1## Create eval

3`evals.create(**kwargs) -> EvalCreateResponse`

5**post** `/evals`

7Create eval

9### Parameters

11- `data_source_config: Custom{ item_schema, type, include_sample_schema} | Logs{ type, metadata} | StoredCompletions{ type, metadata}`

13 The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.

15 - `class Custom`

17 A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs.

18 This schema is used to define the shape of the data that will be:

20 - Used to define your testing criteria and

21 - What data is required when creating a run

23 - `item_schema: Hash[Symbol, untyped]`

25 The json schema for each row in the data source.

27 - `type: :custom`

29 The type of data source. Always `custom`.

31 - `:custom`

33 - `include_sample_schema: bool`

35 Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)

37 - `class Logs`

39 A data source config which specifies the metadata property of your logs query.

40 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.

42 - `type: :logs`

44 The type of data source. Always `logs`.

46 - `:logs`

48 - `metadata: Hash[Symbol, untyped]`

50 Metadata filters for the logs data source.

52 - `class StoredCompletions`

54 Deprecated in favor of LogsDataSourceConfig.

56 - `type: :stored_completions`

58 The type of data source. Always `stored_completions`.

60 - `:stored_completions`

62 - `metadata: Hash[Symbol, untyped]`

64 Metadata filters for the stored completions data source.

66- `testing_criteria: Array[LabelModel{ input, labels, model, 3 more} | StringCheckGrader | TextSimilarityGrader & { pass_threshold} | 2 more]`

68 A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).

70 - `class LabelModel`

72 A LabelModelGrader object which uses a model to assign labels to each item

73 in the evaluation.

75 - `input: Array[SimpleInputMessage{ content, role} | EvalItem{ content, role, type}]`

77 A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.

79 - `class SimpleInputMessage`

81 - `content: String`

83 The content of the message.

85 - `role: String`

87 The role of the message (e.g. "system", "assistant", "user").

89 - `class EvalItem`

91 A message input to the model with a role indicating instruction following

92 hierarchy. Instructions given with the `developer` or `system` role take

93 precedence over instructions given with the `user` role. Messages with the

94 `assistant` role are presumed to have been generated by the model in previous

95 interactions.

97 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`

99 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

100

101 - `String = String`

102

103 A text input to the model.

104

105 - `class ResponseInputText`

106

107 A text input to the model.

108

109 - `text: String`

110

111 The text input to the model.

112

113 - `type: :input_text`

114

115 The type of the input item. Always `input_text`.

116

117 - `:input_text`

118

119 - `prompt_cache_breakpoint: PromptCacheBreakpoint{ mode}`

120

121 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.

122

123 - `mode: :explicit`

124

125 The breakpoint mode. Always `explicit`.

126

127 - `:explicit`

128

129 - `class OutputText`

130

131 A text output from the model.

132

133 - `text: String`

134

135 The text output from the model.

136

137 - `type: :output_text`

138

139 The type of the output text. Always `output_text`.

140

141 - `:output_text`

142

143 - `class InputImage`

144

145 An image input block used within EvalItem content arrays.

146

147 - `image_url: String`

148

149 The URL of the image input.

150

151 - `type: :input_image`

152

153 The type of the image input. Always `input_image`.

154

155 - `:input_image`

156

157 - `detail: String`

158

159 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

160

161 - `class ResponseInputAudio`

162

163 An audio input to the model.

164

165 - `input_audio: InputAudio{ data, format_}`

166

167 - `data: String`

168

169 Base64-encoded audio data.

170

171 - `format_: :mp3 | :wav`

172

173 The format of the audio data. Currently supported formats are `mp3` and

174 `wav`.

175

176 - `:mp3`

177

178 - `:wav`

179

180 - `type: :input_audio`

181

182 The type of the input item. Always `input_audio`.

183

184 - `:input_audio`

185

186 - `GraderInputs = Array[GraderInputItem]`

187

188 A list of inputs, each of which may be either an input text, output text, input

189 image, or input audio object.

190

191 - `String = String`

192

193 A text input to the model.

194

195 - `class ResponseInputText`

196

197 A text input to the model.

198

199 - `class OutputText`

200

201 A text output from the model.

202

203 - `text: String`

204

205 The text output from the model.

206

207 - `type: :output_text`

208

209 The type of the output text. Always `output_text`.

210

211 - `:output_text`

212

213 - `class InputImage`

214

215 An image input block used within EvalItem content arrays.

216

217 - `image_url: String`

218

219 The URL of the image input.

220

221 - `type: :input_image`

222

223 The type of the image input. Always `input_image`.

224

225 - `:input_image`

226

227 - `detail: String`

228

229 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

230

231 - `class ResponseInputAudio`

232

233 An audio input to the model.

234

235 - `role: :user | :assistant | :system | :developer`

236

237 The role of the message input. One of `user`, `assistant`, `system`, or

238 `developer`.

239

240 - `:user`

241

242 - `:assistant`

243

244 - `:system`

245

246 - `:developer`

247

248 - `type: :message`

249

250 The type of the message input. Always `message`.

251

252 - `:message`

253

254 - `labels: Array[String]`

255

256 The labels to classify to each item in the evaluation.

257

258 - `model: String`

259

260 The model to use for the evaluation. Must support structured outputs.

261

262 - `name: String`

263

264 The name of the grader.

265

266 - `passing_labels: Array[String]`

267

268 The labels that indicate a passing result. Must be a subset of labels.

269

270 - `type: :label_model`

271

272 The object type, which is always `label_model`.

273

274 - `:label_model`

275

276 - `class StringCheckGrader`

277

278 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

279

280 - `input: String`

281

282 The input text. This may include template strings.

283

284 - `name: String`

285

286 The name of the grader.

287

288 - `operation: :eq | :ne | :like | :ilike`

289

290 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

291

292 - `:eq`

293

294 - `:ne`

295

296 - `:like`

297

298 - `:ilike`

299

300 - `reference: String`

301

302 The reference text. This may include template strings.

303

304 - `type: :string_check`

305

306 The object type, which is always `string_check`.

307

308 - `:string_check`

309

310 - `class TextSimilarity`

311

312 A TextSimilarityGrader object which grades text based on similarity metrics.

313

314 - `pass_threshold: Float`

315

316 The threshold for the score.

317

318 - `class Python`

319

320 A PythonGrader object that runs a python script on the input.

321

322 - `pass_threshold: Float`

323

324 The threshold for the score.

325

326 - `class ScoreModel`

327

328 A ScoreModelGrader object that uses a model to assign a score to the input.

329

330 - `pass_threshold: Float`

331

332 The threshold for the score.

333

334- `metadata: Metadata`

335

336 Set of 16 key-value pairs that can be attached to an object. This can be

337 useful for storing additional information about the object in a structured

338 format, and querying for objects via API or the dashboard.

339

340 Keys are strings with a maximum length of 64 characters. Values are strings

341 with a maximum length of 512 characters.

342

343- `name: String`

344

345 The name of the evaluation.

346

347### Returns

348

349- `class EvalCreateResponse`

350

351 An Eval object with a data source config and testing criteria.

352 An Eval represents a task to be done for your LLM integration.

353 Like:

354

355 - Improve the quality of my chatbot

356 - See how well my chatbot handles customer support

357 - Check if o4-mini is better at my usecase than gpt-4o

358

359 - `id: String`

360

361 Unique identifier for the evaluation.

362

363 - `created_at: Integer`

364

365 The Unix timestamp (in seconds) for when the eval was created.

366

367 - `data_source_config: EvalCustomDataSourceConfig | Logs{ schema, type, metadata} | EvalStoredCompletionsDataSourceConfig`

368

369 Configuration of data sources used in runs of the evaluation.

370

371 - `class EvalCustomDataSourceConfig`

372

373 A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.

374 The response schema defines the shape of the data that will be:

375

376 - Used to define your testing criteria and

377 - What data is required when creating a run

378

379 - `schema: Hash[Symbol, untyped]`

380

381 The json schema for the run data source items.

382 Learn how to build JSON schemas [here](https://json-schema.org/).

383

384 - `type: :custom`

385

386 The type of data source. Always `custom`.

387

388 - `:custom`

389

390 - `class Logs`

391

392 A LogsDataSourceConfig which specifies the metadata property of your logs query.

393 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.

394 The schema returned by this data source config is used to defined what variables are available in your evals.

395 `item` and `sample` are both defined when using this data source config.

396

397 - `schema: Hash[Symbol, untyped]`

398

399 The json schema for the run data source items.

400 Learn how to build JSON schemas [here](https://json-schema.org/).

401

402 - `type: :logs`

403

404 The type of data source. Always `logs`.

405

406 - `:logs`

407

408 - `metadata: Metadata`

409

410 Set of 16 key-value pairs that can be attached to an object. This can be

411 useful for storing additional information about the object in a structured

412 format, and querying for objects via API or the dashboard.

413

414 Keys are strings with a maximum length of 64 characters. Values are strings

415 with a maximum length of 512 characters.

416

417 - `class EvalStoredCompletionsDataSourceConfig`

418

419 Deprecated in favor of LogsDataSourceConfig.

420

421 - `schema: Hash[Symbol, untyped]`

422

423 The json schema for the run data source items.

424 Learn how to build JSON schemas [here](https://json-schema.org/).

425

426 - `type: :stored_completions`

427

428 The type of data source. Always `stored_completions`.

429

430 - `:stored_completions`

431

432 - `metadata: Metadata`

433

434 Set of 16 key-value pairs that can be attached to an object. This can be

435 useful for storing additional information about the object in a structured

436 format, and querying for objects via API or the dashboard.

437

438 Keys are strings with a maximum length of 64 characters. Values are strings

439 with a maximum length of 512 characters.

440

441 - `metadata: Metadata`

442

443 Set of 16 key-value pairs that can be attached to an object. This can be

444 useful for storing additional information about the object in a structured

445 format, and querying for objects via API or the dashboard.

446

447 Keys are strings with a maximum length of 64 characters. Values are strings

448 with a maximum length of 512 characters.

449

450 - `name: String`

451

452 The name of the evaluation.

453

454 - `object: :eval`

455

456 The object type.

457

458 - `:eval`

459

460 - `testing_criteria: Array[LabelModelGrader | StringCheckGrader | TextSimilarityGrader & { pass_threshold} | 2 more]`

461

462 A list of testing criteria.

463

464 - `class LabelModelGrader`

465

466 A LabelModelGrader object which uses a model to assign labels to each item

467 in the evaluation.

468

469 - `input: Array[Input{ content, role, type}]`

470

471 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`

472

473 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

474

475 - `String = String`

476

477 A text input to the model.

478

479 - `class ResponseInputText`

480

481 A text input to the model.

482

483 - `text: String`

484

485 The text input to the model.

486

487 - `type: :input_text`

488

489 The type of the input item. Always `input_text`.

490

491 - `:input_text`

492

493 - `prompt_cache_breakpoint: PromptCacheBreakpoint{ mode}`

494

495 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.

496

497 - `mode: :explicit`

498

499 The breakpoint mode. Always `explicit`.

500

501 - `:explicit`

502

503 - `class OutputText`

504

505 A text output from the model.

506

507 - `text: String`

508

509 The text output from the model.

510

511 - `type: :output_text`

512

513 The type of the output text. Always `output_text`.

514

515 - `:output_text`

516

517 - `class InputImage`

518

519 An image input block used within EvalItem content arrays.

520

521 - `image_url: String`

522

523 The URL of the image input.

524

525 - `type: :input_image`

526

527 The type of the image input. Always `input_image`.

528

529 - `:input_image`

530

531 - `detail: String`

532

533 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

534

535 - `class ResponseInputAudio`

536

537 An audio input to the model.

538

539 - `input_audio: InputAudio{ data, format_}`

540

541 - `data: String`

542

543 Base64-encoded audio data.

544

545 - `format_: :mp3 | :wav`

546

547 The format of the audio data. Currently supported formats are `mp3` and

548 `wav`.

549

550 - `:mp3`

551

552 - `:wav`

553

554 - `type: :input_audio`

555

556 The type of the input item. Always `input_audio`.

557

558 - `:input_audio`

559

560 - `GraderInputs = Array[GraderInputItem]`

561

562 A list of inputs, each of which may be either an input text, output text, input

563 image, or input audio object.

564

565 - `String = String`

566

567 A text input to the model.

568

569 - `class ResponseInputText`

570

571 A text input to the model.

572

573 - `class OutputText`

574

575 A text output from the model.

576

577 - `text: String`

578

579 The text output from the model.

580

581 - `type: :output_text`

582

583 The type of the output text. Always `output_text`.

584

585 - `:output_text`

586

587 - `class InputImage`

588

589 An image input block used within EvalItem content arrays.

590

591 - `image_url: String`

592

593 The URL of the image input.

594

595 - `type: :input_image`

596

597 The type of the image input. Always `input_image`.

598

599 - `:input_image`

600

601 - `detail: String`

602

603 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

604

605 - `class ResponseInputAudio`

606

607 An audio input to the model.

608

609 - `role: :user | :assistant | :system | :developer`

610

611 The role of the message input. One of `user`, `assistant`, `system`, or

612 `developer`.

613

614 - `:user`

615

616 - `:assistant`

617

618 - `:system`

619

620 - `:developer`

621

622 - `type: :message`

623

624 The type of the message input. Always `message`.

625

626 - `:message`

627

628 - `labels: Array[String]`

629

630 The labels to assign to each item in the evaluation.

631

632 - `model: String`

633

634 The model to use for the evaluation. Must support structured outputs.

635

636 - `name: String`

637

638 The name of the grader.

639

640 - `passing_labels: Array[String]`

641

642 The labels that indicate a passing result. Must be a subset of labels.

643

644 - `type: :label_model`

645

646 The object type, which is always `label_model`.

647

648 - `:label_model`

649

650 - `class StringCheckGrader`

651

652 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

653

654 - `input: String`

655

656 The input text. This may include template strings.

657

658 - `name: String`

659

660 The name of the grader.

661

662 - `operation: :eq | :ne | :like | :ilike`

663

664 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

665

666 - `:eq`

667

668 - `:ne`

669

670 - `:like`

671

672 - `:ilike`

673

674 - `reference: String`

675

676 The reference text. This may include template strings.

677

678 - `type: :string_check`

679

680 The object type, which is always `string_check`.

681

682 - `:string_check`

683

684 - `class EvalGraderTextSimilarity`

685

686 A TextSimilarityGrader object which grades text based on similarity metrics.

687

688 - `pass_threshold: Float`

689

690 The threshold for the score.

691

692 - `class EvalGraderPython`

693

694 A PythonGrader object that runs a python script on the input.

695

696 - `pass_threshold: Float`

697

698 The threshold for the score.

699

700 - `class EvalGraderScoreModel`

701

702 A ScoreModelGrader object that uses a model to assign a score to the input.

703

704 - `pass_threshold: Float`

705

706 The threshold for the score.

707

708### Example

709

710```ruby

711require "openai"

712

713openai = OpenAI::Client.new(api_key: "My API Key")

714

715eval_ = openai.evals.create(

716 data_source_config: {item_schema: {foo: "bar"}, type: :custom},

717 testing_criteria: [

718 {

719 input: [{content: "content", role: "role"}],

720 labels: ["string"],

721 model: "model",

722 name: "name",

723 passing_labels: ["string"],

724 type: :label_model

725 }

726 ]

727)

728

729puts(eval_)

730```

731

732#### Response

733

734```json

735{

736 "id": "id",

737 "created_at": 0,

738 "data_source_config": {

739 "schema": {

740 "foo": "bar"

741 },

742 "type": "custom"

743 },

744 "metadata": {

745 "foo": "string"

746 },

747 "name": "Chatbot effectiveness Evaluation",

748 "object": "eval",

749 "testing_criteria": [

750 {

751 "input": [

752 {

753 "content": "string",

754 "role": "user",

755 "type": "message"

756 }

757 ],

758 "labels": [

759 "string"

760 ],

761 "model": "model",

762 "name": "name",

763 "passing_labels": [

764 "string"

765 ],

766 "type": "label_model"

767 }

768 ]

769}

770```

ruby/resources/evals/methods/create/index.md 2026-07-07 08:02 UTC to 2026-07-09 20:58 UTC

Create eval

Parameters

Returns

Example

Response