ruby/resources/evals/methods/create/index.md +0 −752 deleted
File Deleted View Diff
1## Create eval
2
3`evals.create(**kwargs) -> EvalCreateResponse`
4
5**post** `/evals`
6
7Create the structure of an evaluation that can be used to test a model's performance.
8An evaluation is a set of testing criteria and the config for a data source, which dictates the schema of the data used in the evaluation. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources.
9For more information, see the [Evals guide](https://platform.openai.com/docs/guides/evals).
10
11### Parameters
12
13- `data_source_config: Custom{ item_schema, type, include_sample_schema} | Logs{ type, metadata} | StoredCompletions{ type, metadata}`
14
15 The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
16
17 - `class Custom`
18
19 A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs.
20 This schema is used to define the shape of the data that will be:
21
22 - Used to define your testing criteria and
23 - What data is required when creating a run
24
25 - `item_schema: Hash[Symbol, untyped]`
26
27 The json schema for each row in the data source.
28
29 - `type: :custom`
30
31 The type of data source. Always `custom`.
32
33 - `:custom`
34
35 - `include_sample_schema: bool`
36
37 Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)
38
39 - `class Logs`
40
41 A data source config which specifies the metadata property of your logs query.
42 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
43
44 - `type: :logs`
45
46 The type of data source. Always `logs`.
47
48 - `:logs`
49
50 - `metadata: Hash[Symbol, untyped]`
51
52 Metadata filters for the logs data source.
53
54 - `class StoredCompletions`
55
56 Deprecated in favor of LogsDataSourceConfig.
57
58 - `type: :stored_completions`
59
60 The type of data source. Always `stored_completions`.
61
62 - `:stored_completions`
63
64 - `metadata: Hash[Symbol, untyped]`
65
66 Metadata filters for the stored completions data source.
67
68- `testing_criteria: Array[LabelModel{ input, labels, model, 3 more} | StringCheckGrader | TextSimilarityGrader & { pass_threshold} | 2 more]`
69
70 A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).
71
72 - `class LabelModel`
73
74 A LabelModelGrader object which uses a model to assign labels to each item
75 in the evaluation.
76
77 - `input: Array[SimpleInputMessage{ content, role} | EvalItem{ content, role, type}]`
78
79 A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
80
81 - `class SimpleInputMessage`
82
83 - `content: String`
84
85 The content of the message.
86
87 - `role: String`
88
89 The role of the message (e.g. "system", "assistant", "user").
90
91 - `class EvalItem`
92
93 A message input to the model with a role indicating instruction following
94 hierarchy. Instructions given with the `developer` or `system` role take
95 precedence over instructions given with the `user` role. Messages with the
96 `assistant` role are presumed to have been generated by the model in previous
97 interactions.
98
99 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`
100
101 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
102
103 - `String = String`
104
105 A text input to the model.
106
107 - `class ResponseInputText`
108
109 A text input to the model.
110
111 - `text: String`
112
113 The text input to the model.
114
115 - `type: :input_text`
116
117 The type of the input item. Always `input_text`.
118
119 - `:input_text`
120
121 - `class OutputText`
122
123 A text output from the model.
124
125 - `text: String`
126
127 The text output from the model.
128
129 - `type: :output_text`
130
131 The type of the output text. Always `output_text`.
132
133 - `:output_text`
134
135 - `class InputImage`
136
137 An image input block used within EvalItem content arrays.
138
139 - `image_url: String`
140
141 The URL of the image input.
142
143 - `type: :input_image`
144
145 The type of the image input. Always `input_image`.
146
147 - `:input_image`
148
149 - `detail: String`
150
151 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
152
153 - `class ResponseInputAudio`
154
155 An audio input to the model.
156
157 - `input_audio: InputAudio{ data, format_}`
158
159 - `data: String`
160
161 Base64-encoded audio data.
162
163 - `format_: :mp3 | :wav`
164
165 The format of the audio data. Currently supported formats are `mp3` and
166 `wav`.
167
168 - `:mp3`
169
170 - `:wav`
171
172 - `type: :input_audio`
173
174 The type of the input item. Always `input_audio`.
175
176 - `:input_audio`
177
178 - `GraderInputs = Array[GraderInputItem]`
179
180 A list of inputs, each of which may be either an input text, output text, input
181 image, or input audio object.
182
183 - `String = String`
184
185 A text input to the model.
186
187 - `class ResponseInputText`
188
189 A text input to the model.
190
191 - `class OutputText`
192
193 A text output from the model.
194
195 - `text: String`
196
197 The text output from the model.
198
199 - `type: :output_text`
200
201 The type of the output text. Always `output_text`.
202
203 - `:output_text`
204
205 - `class InputImage`
206
207 An image input block used within EvalItem content arrays.
208
209 - `image_url: String`
210
211 The URL of the image input.
212
213 - `type: :input_image`
214
215 The type of the image input. Always `input_image`.
216
217 - `:input_image`
218
219 - `detail: String`
220
221 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
222
223 - `class ResponseInputAudio`
224
225 An audio input to the model.
226
227 - `role: :user | :assistant | :system | :developer`
228
229 The role of the message input. One of `user`, `assistant`, `system`, or
230 `developer`.
231
232 - `:user`
233
234 - `:assistant`
235
236 - `:system`
237
238 - `:developer`
239
240 - `type: :message`
241
242 The type of the message input. Always `message`.
243
244 - `:message`
245
246 - `labels: Array[String]`
247
248 The labels to classify to each item in the evaluation.
249
250 - `model: String`
251
252 The model to use for the evaluation. Must support structured outputs.
253
254 - `name: String`
255
256 The name of the grader.
257
258 - `passing_labels: Array[String]`
259
260 The labels that indicate a passing result. Must be a subset of labels.
261
262 - `type: :label_model`
263
264 The object type, which is always `label_model`.
265
266 - `:label_model`
267
268 - `class StringCheckGrader`
269
270 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
271
272 - `input: String`
273
274 The input text. This may include template strings.
275
276 - `name: String`
277
278 The name of the grader.
279
280 - `operation: :eq | :ne | :like | :ilike`
281
282 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
283
284 - `:eq`
285
286 - `:ne`
287
288 - `:like`
289
290 - `:ilike`
291
292 - `reference: String`
293
294 The reference text. This may include template strings.
295
296 - `type: :string_check`
297
298 The object type, which is always `string_check`.
299
300 - `:string_check`
301
302 - `class TextSimilarity`
303
304 A TextSimilarityGrader object which grades text based on similarity metrics.
305
306 - `pass_threshold: Float`
307
308 The threshold for the score.
309
310 - `class Python`
311
312 A PythonGrader object that runs a python script on the input.
313
314 - `pass_threshold: Float`
315
316 The threshold for the score.
317
318 - `class ScoreModel`
319
320 A ScoreModelGrader object that uses a model to assign a score to the input.
321
322 - `pass_threshold: Float`
323
324 The threshold for the score.
325
326- `metadata: Metadata`
327
328 Set of 16 key-value pairs that can be attached to an object. This can be
329 useful for storing additional information about the object in a structured
330 format, and querying for objects via API or the dashboard.
331
332 Keys are strings with a maximum length of 64 characters. Values are strings
333 with a maximum length of 512 characters.
334
335- `name: String`
336
337 The name of the evaluation.
338
339### Returns
340
341- `class EvalCreateResponse`
342
343 An Eval object with a data source config and testing criteria.
344 An Eval represents a task to be done for your LLM integration.
345 Like:
346
347 - Improve the quality of my chatbot
348 - See how well my chatbot handles customer support
349 - Check if o4-mini is better at my usecase than gpt-4o
350
351 - `id: String`
352
353 Unique identifier for the evaluation.
354
355 - `created_at: Integer`
356
357 The Unix timestamp (in seconds) for when the eval was created.
358
359 - `data_source_config: EvalCustomDataSourceConfig | Logs{ schema, type, metadata} | EvalStoredCompletionsDataSourceConfig`
360
361 Configuration of data sources used in runs of the evaluation.
362
363 - `class EvalCustomDataSourceConfig`
364
365 A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
366 The response schema defines the shape of the data that will be:
367
368 - Used to define your testing criteria and
369 - What data is required when creating a run
370
371 - `schema: Hash[Symbol, untyped]`
372
373 The json schema for the run data source items.
374 Learn how to build JSON schemas [here](https://json-schema.org/).
375
376 - `type: :custom`
377
378 The type of data source. Always `custom`.
379
380 - `:custom`
381
382 - `class Logs`
383
384 A LogsDataSourceConfig which specifies the metadata property of your logs query.
385 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
386 The schema returned by this data source config is used to defined what variables are available in your evals.
387 `item` and `sample` are both defined when using this data source config.
388
389 - `schema: Hash[Symbol, untyped]`
390
391 The json schema for the run data source items.
392 Learn how to build JSON schemas [here](https://json-schema.org/).
393
394 - `type: :logs`
395
396 The type of data source. Always `logs`.
397
398 - `:logs`
399
400 - `metadata: Metadata`
401
402 Set of 16 key-value pairs that can be attached to an object. This can be
403 useful for storing additional information about the object in a structured
404 format, and querying for objects via API or the dashboard.
405
406 Keys are strings with a maximum length of 64 characters. Values are strings
407 with a maximum length of 512 characters.
408
409 - `class EvalStoredCompletionsDataSourceConfig`
410
411 Deprecated in favor of LogsDataSourceConfig.
412
413 - `schema: Hash[Symbol, untyped]`
414
415 The json schema for the run data source items.
416 Learn how to build JSON schemas [here](https://json-schema.org/).
417
418 - `type: :stored_completions`
419
420 The type of data source. Always `stored_completions`.
421
422 - `:stored_completions`
423
424 - `metadata: Metadata`
425
426 Set of 16 key-value pairs that can be attached to an object. This can be
427 useful for storing additional information about the object in a structured
428 format, and querying for objects via API or the dashboard.
429
430 Keys are strings with a maximum length of 64 characters. Values are strings
431 with a maximum length of 512 characters.
432
433 - `metadata: Metadata`
434
435 Set of 16 key-value pairs that can be attached to an object. This can be
436 useful for storing additional information about the object in a structured
437 format, and querying for objects via API or the dashboard.
438
439 Keys are strings with a maximum length of 64 characters. Values are strings
440 with a maximum length of 512 characters.
441
442 - `name: String`
443
444 The name of the evaluation.
445
446 - `object: :eval`
447
448 The object type.
449
450 - `:eval`
451
452 - `testing_criteria: Array[LabelModelGrader | StringCheckGrader | TextSimilarityGrader & { pass_threshold} | 2 more]`
453
454 A list of testing criteria.
455
456 - `class LabelModelGrader`
457
458 A LabelModelGrader object which uses a model to assign labels to each item
459 in the evaluation.
460
461 - `input: Array[Input{ content, role, type}]`
462
463 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`
464
465 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
466
467 - `String = String`
468
469 A text input to the model.
470
471 - `class ResponseInputText`
472
473 A text input to the model.
474
475 - `text: String`
476
477 The text input to the model.
478
479 - `type: :input_text`
480
481 The type of the input item. Always `input_text`.
482
483 - `:input_text`
484
485 - `class OutputText`
486
487 A text output from the model.
488
489 - `text: String`
490
491 The text output from the model.
492
493 - `type: :output_text`
494
495 The type of the output text. Always `output_text`.
496
497 - `:output_text`
498
499 - `class InputImage`
500
501 An image input block used within EvalItem content arrays.
502
503 - `image_url: String`
504
505 The URL of the image input.
506
507 - `type: :input_image`
508
509 The type of the image input. Always `input_image`.
510
511 - `:input_image`
512
513 - `detail: String`
514
515 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
516
517 - `class ResponseInputAudio`
518
519 An audio input to the model.
520
521 - `input_audio: InputAudio{ data, format_}`
522
523 - `data: String`
524
525 Base64-encoded audio data.
526
527 - `format_: :mp3 | :wav`
528
529 The format of the audio data. Currently supported formats are `mp3` and
530 `wav`.
531
532 - `:mp3`
533
534 - `:wav`
535
536 - `type: :input_audio`
537
538 The type of the input item. Always `input_audio`.
539
540 - `:input_audio`
541
542 - `GraderInputs = Array[GraderInputItem]`
543
544 A list of inputs, each of which may be either an input text, output text, input
545 image, or input audio object.
546
547 - `String = String`
548
549 A text input to the model.
550
551 - `class ResponseInputText`
552
553 A text input to the model.
554
555 - `class OutputText`
556
557 A text output from the model.
558
559 - `text: String`
560
561 The text output from the model.
562
563 - `type: :output_text`
564
565 The type of the output text. Always `output_text`.
566
567 - `:output_text`
568
569 - `class InputImage`
570
571 An image input block used within EvalItem content arrays.
572
573 - `image_url: String`
574
575 The URL of the image input.
576
577 - `type: :input_image`
578
579 The type of the image input. Always `input_image`.
580
581 - `:input_image`
582
583 - `detail: String`
584
585 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
586
587 - `class ResponseInputAudio`
588
589 An audio input to the model.
590
591 - `role: :user | :assistant | :system | :developer`
592
593 The role of the message input. One of `user`, `assistant`, `system`, or
594 `developer`.
595
596 - `:user`
597
598 - `:assistant`
599
600 - `:system`
601
602 - `:developer`
603
604 - `type: :message`
605
606 The type of the message input. Always `message`.
607
608 - `:message`
609
610 - `labels: Array[String]`
611
612 The labels to assign to each item in the evaluation.
613
614 - `model: String`
615
616 The model to use for the evaluation. Must support structured outputs.
617
618 - `name: String`
619
620 The name of the grader.
621
622 - `passing_labels: Array[String]`
623
624 The labels that indicate a passing result. Must be a subset of labels.
625
626 - `type: :label_model`
627
628 The object type, which is always `label_model`.
629
630 - `:label_model`
631
632 - `class StringCheckGrader`
633
634 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
635
636 - `input: String`
637
638 The input text. This may include template strings.
639
640 - `name: String`
641
642 The name of the grader.
643
644 - `operation: :eq | :ne | :like | :ilike`
645
646 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
647
648 - `:eq`
649
650 - `:ne`
651
652 - `:like`
653
654 - `:ilike`
655
656 - `reference: String`
657
658 The reference text. This may include template strings.
659
660 - `type: :string_check`
661
662 The object type, which is always `string_check`.
663
664 - `:string_check`
665
666 - `class EvalGraderTextSimilarity`
667
668 A TextSimilarityGrader object which grades text based on similarity metrics.
669
670 - `pass_threshold: Float`
671
672 The threshold for the score.
673
674 - `class EvalGraderPython`
675
676 A PythonGrader object that runs a python script on the input.
677
678 - `pass_threshold: Float`
679
680 The threshold for the score.
681
682 - `class EvalGraderScoreModel`
683
684 A ScoreModelGrader object that uses a model to assign a score to the input.
685
686 - `pass_threshold: Float`
687
688 The threshold for the score.
689
690### Example
691
692```ruby
693require "openai"
694
695openai = OpenAI::Client.new(api_key: "My API Key")
696
697eval_ = openai.evals.create(
698 data_source_config: {item_schema: {foo: "bar"}, type: :custom},
699 testing_criteria: [
700 {
701 input: [{content: "content", role: "role"}],
702 labels: ["string"],
703 model: "model",
704 name: "name",
705 passing_labels: ["string"],
706 type: :label_model
707 }
708 ]
709)
710
711puts(eval_)
712```
713
714#### Response
715
716```json
717{
718 "id": "id",
719 "created_at": 0,
720 "data_source_config": {
721 "schema": {
722 "foo": "bar"
723 },
724 "type": "custom"
725 },
726 "metadata": {
727 "foo": "string"
728 },
729 "name": "Chatbot effectiveness Evaluation",
730 "object": "eval",
731 "testing_criteria": [
732 {
733 "input": [
734 {
735 "content": "string",
736 "role": "user",
737 "type": "message"
738 }
739 ],
740 "labels": [
741 "string"
742 ],
743 "model": "model",
744 "name": "name",
745 "passing_labels": [
746 "string"
747 ],
748 "type": "label_model"
749 }
750 ]
751}
752```