ruby/resources/evals/methods/create/index.md +0 −772 deleted
File Deleted View Diff
1## Create eval
2
3`evals.create(**kwargs) -> EvalCreateResponse`
4
5**post** `/evals`
6
7Create the structure of an evaluation that can be used to test a model's performance.
8An evaluation is a set of testing criteria and the config for a data source, which dictates the schema of the data used in the evaluation. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources.
9For more information, see the [Evals guide](https://platform.openai.com/docs/guides/evals).
10
11### Parameters
12
13- `data_source_config: Custom{ item_schema, type, include_sample_schema} | Logs{ type, metadata} | StoredCompletions{ type, metadata}`
14
15 The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
16
17 - `class Custom`
18
19 A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs.
20 This schema is used to define the shape of the data that will be:
21
22 - Used to define your testing criteria and
23 - What data is required when creating a run
24
25 - `item_schema: Hash[Symbol, untyped]`
26
27 The json schema for each row in the data source.
28
29 - `type: :custom`
30
31 The type of data source. Always `custom`.
32
33 - `:custom`
34
35 - `include_sample_schema: bool`
36
37 Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)
38
39 - `class Logs`
40
41 A data source config which specifies the metadata property of your logs query.
42 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
43
44 - `type: :logs`
45
46 The type of data source. Always `logs`.
47
48 - `:logs`
49
50 - `metadata: Hash[Symbol, untyped]`
51
52 Metadata filters for the logs data source.
53
54 - `class StoredCompletions`
55
56 Deprecated in favor of LogsDataSourceConfig.
57
58 - `type: :stored_completions`
59
60 The type of data source. Always `stored_completions`.
61
62 - `:stored_completions`
63
64 - `metadata: Hash[Symbol, untyped]`
65
66 Metadata filters for the stored completions data source.
67
68- `testing_criteria: Array[LabelModel{ input, labels, model, 3 more} | StringCheckGrader | TextSimilarityGrader & { pass_threshold} | 2 more]`
69
70 A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).
71
72 - `class LabelModel`
73
74 A LabelModelGrader object which uses a model to assign labels to each item
75 in the evaluation.
76
77 - `input: Array[SimpleInputMessage{ content, role} | EvalItem{ content, role, type}]`
78
79 A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
80
81 - `class SimpleInputMessage`
82
83 - `content: String`
84
85 The content of the message.
86
87 - `role: String`
88
89 The role of the message (e.g. "system", "assistant", "user").
90
91 - `class EvalItem`
92
93 A message input to the model with a role indicating instruction following
94 hierarchy. Instructions given with the `developer` or `system` role take
95 precedence over instructions given with the `user` role. Messages with the
96 `assistant` role are presumed to have been generated by the model in previous
97 interactions.
98
99 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`
100
101 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
102
103 - `String = String`
104
105 A text input to the model.
106
107 - `class ResponseInputText`
108
109 A text input to the model.
110
111 - `text: String`
112
113 The text input to the model.
114
115 - `type: :input_text`
116
117 The type of the input item. Always `input_text`.
118
119 - `:input_text`
120
121 - `prompt_cache_breakpoint: PromptCacheBreakpoint{ mode}`
122
123 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
124
125 - `mode: :explicit`
126
127 The breakpoint mode. Always `explicit`.
128
129 - `:explicit`
130
131 - `class OutputText`
132
133 A text output from the model.
134
135 - `text: String`
136
137 The text output from the model.
138
139 - `type: :output_text`
140
141 The type of the output text. Always `output_text`.
142
143 - `:output_text`
144
145 - `class InputImage`
146
147 An image input block used within EvalItem content arrays.
148
149 - `image_url: String`
150
151 The URL of the image input.
152
153 - `type: :input_image`
154
155 The type of the image input. Always `input_image`.
156
157 - `:input_image`
158
159 - `detail: String`
160
161 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
162
163 - `class ResponseInputAudio`
164
165 An audio input to the model.
166
167 - `input_audio: InputAudio{ data, format_}`
168
169 - `data: String`
170
171 Base64-encoded audio data.
172
173 - `format_: :mp3 | :wav`
174
175 The format of the audio data. Currently supported formats are `mp3` and
176 `wav`.
177
178 - `:mp3`
179
180 - `:wav`
181
182 - `type: :input_audio`
183
184 The type of the input item. Always `input_audio`.
185
186 - `:input_audio`
187
188 - `GraderInputs = Array[GraderInputItem]`
189
190 A list of inputs, each of which may be either an input text, output text, input
191 image, or input audio object.
192
193 - `String = String`
194
195 A text input to the model.
196
197 - `class ResponseInputText`
198
199 A text input to the model.
200
201 - `class OutputText`
202
203 A text output from the model.
204
205 - `text: String`
206
207 The text output from the model.
208
209 - `type: :output_text`
210
211 The type of the output text. Always `output_text`.
212
213 - `:output_text`
214
215 - `class InputImage`
216
217 An image input block used within EvalItem content arrays.
218
219 - `image_url: String`
220
221 The URL of the image input.
222
223 - `type: :input_image`
224
225 The type of the image input. Always `input_image`.
226
227 - `:input_image`
228
229 - `detail: String`
230
231 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
232
233 - `class ResponseInputAudio`
234
235 An audio input to the model.
236
237 - `role: :user | :assistant | :system | :developer`
238
239 The role of the message input. One of `user`, `assistant`, `system`, or
240 `developer`.
241
242 - `:user`
243
244 - `:assistant`
245
246 - `:system`
247
248 - `:developer`
249
250 - `type: :message`
251
252 The type of the message input. Always `message`.
253
254 - `:message`
255
256 - `labels: Array[String]`
257
258 The labels to classify to each item in the evaluation.
259
260 - `model: String`
261
262 The model to use for the evaluation. Must support structured outputs.
263
264 - `name: String`
265
266 The name of the grader.
267
268 - `passing_labels: Array[String]`
269
270 The labels that indicate a passing result. Must be a subset of labels.
271
272 - `type: :label_model`
273
274 The object type, which is always `label_model`.
275
276 - `:label_model`
277
278 - `class StringCheckGrader`
279
280 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
281
282 - `input: String`
283
284 The input text. This may include template strings.
285
286 - `name: String`
287
288 The name of the grader.
289
290 - `operation: :eq | :ne | :like | :ilike`
291
292 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
293
294 - `:eq`
295
296 - `:ne`
297
298 - `:like`
299
300 - `:ilike`
301
302 - `reference: String`
303
304 The reference text. This may include template strings.
305
306 - `type: :string_check`
307
308 The object type, which is always `string_check`.
309
310 - `:string_check`
311
312 - `class TextSimilarity`
313
314 A TextSimilarityGrader object which grades text based on similarity metrics.
315
316 - `pass_threshold: Float`
317
318 The threshold for the score.
319
320 - `class Python`
321
322 A PythonGrader object that runs a python script on the input.
323
324 - `pass_threshold: Float`
325
326 The threshold for the score.
327
328 - `class ScoreModel`
329
330 A ScoreModelGrader object that uses a model to assign a score to the input.
331
332 - `pass_threshold: Float`
333
334 The threshold for the score.
335
336- `metadata: Metadata`
337
338 Set of 16 key-value pairs that can be attached to an object. This can be
339 useful for storing additional information about the object in a structured
340 format, and querying for objects via API or the dashboard.
341
342 Keys are strings with a maximum length of 64 characters. Values are strings
343 with a maximum length of 512 characters.
344
345- `name: String`
346
347 The name of the evaluation.
348
349### Returns
350
351- `class EvalCreateResponse`
352
353 An Eval object with a data source config and testing criteria.
354 An Eval represents a task to be done for your LLM integration.
355 Like:
356
357 - Improve the quality of my chatbot
358 - See how well my chatbot handles customer support
359 - Check if o4-mini is better at my usecase than gpt-4o
360
361 - `id: String`
362
363 Unique identifier for the evaluation.
364
365 - `created_at: Integer`
366
367 The Unix timestamp (in seconds) for when the eval was created.
368
369 - `data_source_config: EvalCustomDataSourceConfig | Logs{ schema, type, metadata} | EvalStoredCompletionsDataSourceConfig`
370
371 Configuration of data sources used in runs of the evaluation.
372
373 - `class EvalCustomDataSourceConfig`
374
375 A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
376 The response schema defines the shape of the data that will be:
377
378 - Used to define your testing criteria and
379 - What data is required when creating a run
380
381 - `schema: Hash[Symbol, untyped]`
382
383 The json schema for the run data source items.
384 Learn how to build JSON schemas [here](https://json-schema.org/).
385
386 - `type: :custom`
387
388 The type of data source. Always `custom`.
389
390 - `:custom`
391
392 - `class Logs`
393
394 A LogsDataSourceConfig which specifies the metadata property of your logs query.
395 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
396 The schema returned by this data source config is used to defined what variables are available in your evals.
397 `item` and `sample` are both defined when using this data source config.
398
399 - `schema: Hash[Symbol, untyped]`
400
401 The json schema for the run data source items.
402 Learn how to build JSON schemas [here](https://json-schema.org/).
403
404 - `type: :logs`
405
406 The type of data source. Always `logs`.
407
408 - `:logs`
409
410 - `metadata: Metadata`
411
412 Set of 16 key-value pairs that can be attached to an object. This can be
413 useful for storing additional information about the object in a structured
414 format, and querying for objects via API or the dashboard.
415
416 Keys are strings with a maximum length of 64 characters. Values are strings
417 with a maximum length of 512 characters.
418
419 - `class EvalStoredCompletionsDataSourceConfig`
420
421 Deprecated in favor of LogsDataSourceConfig.
422
423 - `schema: Hash[Symbol, untyped]`
424
425 The json schema for the run data source items.
426 Learn how to build JSON schemas [here](https://json-schema.org/).
427
428 - `type: :stored_completions`
429
430 The type of data source. Always `stored_completions`.
431
432 - `:stored_completions`
433
434 - `metadata: Metadata`
435
436 Set of 16 key-value pairs that can be attached to an object. This can be
437 useful for storing additional information about the object in a structured
438 format, and querying for objects via API or the dashboard.
439
440 Keys are strings with a maximum length of 64 characters. Values are strings
441 with a maximum length of 512 characters.
442
443 - `metadata: Metadata`
444
445 Set of 16 key-value pairs that can be attached to an object. This can be
446 useful for storing additional information about the object in a structured
447 format, and querying for objects via API or the dashboard.
448
449 Keys are strings with a maximum length of 64 characters. Values are strings
450 with a maximum length of 512 characters.
451
452 - `name: String`
453
454 The name of the evaluation.
455
456 - `object: :eval`
457
458 The object type.
459
460 - `:eval`
461
462 - `testing_criteria: Array[LabelModelGrader | StringCheckGrader | TextSimilarityGrader & { pass_threshold} | 2 more]`
463
464 A list of testing criteria.
465
466 - `class LabelModelGrader`
467
468 A LabelModelGrader object which uses a model to assign labels to each item
469 in the evaluation.
470
471 - `input: Array[Input{ content, role, type}]`
472
473 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`
474
475 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
476
477 - `String = String`
478
479 A text input to the model.
480
481 - `class ResponseInputText`
482
483 A text input to the model.
484
485 - `text: String`
486
487 The text input to the model.
488
489 - `type: :input_text`
490
491 The type of the input item. Always `input_text`.
492
493 - `:input_text`
494
495 - `prompt_cache_breakpoint: PromptCacheBreakpoint{ mode}`
496
497 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
498
499 - `mode: :explicit`
500
501 The breakpoint mode. Always `explicit`.
502
503 - `:explicit`
504
505 - `class OutputText`
506
507 A text output from the model.
508
509 - `text: String`
510
511 The text output from the model.
512
513 - `type: :output_text`
514
515 The type of the output text. Always `output_text`.
516
517 - `:output_text`
518
519 - `class InputImage`
520
521 An image input block used within EvalItem content arrays.
522
523 - `image_url: String`
524
525 The URL of the image input.
526
527 - `type: :input_image`
528
529 The type of the image input. Always `input_image`.
530
531 - `:input_image`
532
533 - `detail: String`
534
535 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
536
537 - `class ResponseInputAudio`
538
539 An audio input to the model.
540
541 - `input_audio: InputAudio{ data, format_}`
542
543 - `data: String`
544
545 Base64-encoded audio data.
546
547 - `format_: :mp3 | :wav`
548
549 The format of the audio data. Currently supported formats are `mp3` and
550 `wav`.
551
552 - `:mp3`
553
554 - `:wav`
555
556 - `type: :input_audio`
557
558 The type of the input item. Always `input_audio`.
559
560 - `:input_audio`
561
562 - `GraderInputs = Array[GraderInputItem]`
563
564 A list of inputs, each of which may be either an input text, output text, input
565 image, or input audio object.
566
567 - `String = String`
568
569 A text input to the model.
570
571 - `class ResponseInputText`
572
573 A text input to the model.
574
575 - `class OutputText`
576
577 A text output from the model.
578
579 - `text: String`
580
581 The text output from the model.
582
583 - `type: :output_text`
584
585 The type of the output text. Always `output_text`.
586
587 - `:output_text`
588
589 - `class InputImage`
590
591 An image input block used within EvalItem content arrays.
592
593 - `image_url: String`
594
595 The URL of the image input.
596
597 - `type: :input_image`
598
599 The type of the image input. Always `input_image`.
600
601 - `:input_image`
602
603 - `detail: String`
604
605 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
606
607 - `class ResponseInputAudio`
608
609 An audio input to the model.
610
611 - `role: :user | :assistant | :system | :developer`
612
613 The role of the message input. One of `user`, `assistant`, `system`, or
614 `developer`.
615
616 - `:user`
617
618 - `:assistant`
619
620 - `:system`
621
622 - `:developer`
623
624 - `type: :message`
625
626 The type of the message input. Always `message`.
627
628 - `:message`
629
630 - `labels: Array[String]`
631
632 The labels to assign to each item in the evaluation.
633
634 - `model: String`
635
636 The model to use for the evaluation. Must support structured outputs.
637
638 - `name: String`
639
640 The name of the grader.
641
642 - `passing_labels: Array[String]`
643
644 The labels that indicate a passing result. Must be a subset of labels.
645
646 - `type: :label_model`
647
648 The object type, which is always `label_model`.
649
650 - `:label_model`
651
652 - `class StringCheckGrader`
653
654 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
655
656 - `input: String`
657
658 The input text. This may include template strings.
659
660 - `name: String`
661
662 The name of the grader.
663
664 - `operation: :eq | :ne | :like | :ilike`
665
666 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
667
668 - `:eq`
669
670 - `:ne`
671
672 - `:like`
673
674 - `:ilike`
675
676 - `reference: String`
677
678 The reference text. This may include template strings.
679
680 - `type: :string_check`
681
682 The object type, which is always `string_check`.
683
684 - `:string_check`
685
686 - `class EvalGraderTextSimilarity`
687
688 A TextSimilarityGrader object which grades text based on similarity metrics.
689
690 - `pass_threshold: Float`
691
692 The threshold for the score.
693
694 - `class EvalGraderPython`
695
696 A PythonGrader object that runs a python script on the input.
697
698 - `pass_threshold: Float`
699
700 The threshold for the score.
701
702 - `class EvalGraderScoreModel`
703
704 A ScoreModelGrader object that uses a model to assign a score to the input.
705
706 - `pass_threshold: Float`
707
708 The threshold for the score.
709
710### Example
711
712```ruby
713require "openai"
714
715openai = OpenAI::Client.new(api_key: "My API Key")
716
717eval_ = openai.evals.create(
718 data_source_config: {item_schema: {foo: "bar"}, type: :custom},
719 testing_criteria: [
720 {
721 input: [{content: "content", role: "role"}],
722 labels: ["string"],
723 model: "model",
724 name: "name",
725 passing_labels: ["string"],
726 type: :label_model
727 }
728 ]
729)
730
731puts(eval_)
732```
733
734#### Response
735
736```json
737{
738 "id": "id",
739 "created_at": 0,
740 "data_source_config": {
741 "schema": {
742 "foo": "bar"
743 },
744 "type": "custom"
745 },
746 "metadata": {
747 "foo": "string"
748 },
749 "name": "Chatbot effectiveness Evaluation",
750 "object": "eval",
751 "testing_criteria": [
752 {
753 "input": [
754 {
755 "content": "string",
756 "role": "user",
757 "type": "message"
758 }
759 ],
760 "labels": [
761 "string"
762 ],
763 "model": "model",
764 "name": "name",
765 "passing_labels": [
766 "string"
767 ],
768 "type": "label_model"
769 }
770 ]
771}
772```