python/resources/evals/methods/create/index.md +0 −869 deleted
File Deleted View Diff
1## Create eval
2
3`evals.create(EvalCreateParams**kwargs) -> EvalCreateResponse`
4
5**post** `/evals`
6
7Create the structure of an evaluation that can be used to test a model's performance.
8An evaluation is a set of testing criteria and the config for a data source, which dictates the schema of the data used in the evaluation. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources.
9For more information, see the [Evals guide](https://platform.openai.com/docs/guides/evals).
10
11### Parameters
12
13- `data_source_config: DataSourceConfig`
14
15 The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
16
17 - `class DataSourceConfigCustom: …`
18
19 A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs.
20 This schema is used to define the shape of the data that will be:
21
22 - Used to define your testing criteria and
23 - What data is required when creating a run
24
25 - `item_schema: Dict[str, object]`
26
27 The json schema for each row in the data source.
28
29 - `type: Literal["custom"]`
30
31 The type of data source. Always `custom`.
32
33 - `"custom"`
34
35 - `include_sample_schema: Optional[bool]`
36
37 Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)
38
39 - `class DataSourceConfigLogs: …`
40
41 A data source config which specifies the metadata property of your logs query.
42 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
43
44 - `type: Literal["logs"]`
45
46 The type of data source. Always `logs`.
47
48 - `"logs"`
49
50 - `metadata: Optional[Dict[str, object]]`
51
52 Metadata filters for the logs data source.
53
54 - `class DataSourceConfigStoredCompletions: …`
55
56 Deprecated in favor of LogsDataSourceConfig.
57
58 - `type: Literal["stored_completions"]`
59
60 The type of data source. Always `stored_completions`.
61
62 - `"stored_completions"`
63
64 - `metadata: Optional[Dict[str, object]]`
65
66 Metadata filters for the stored completions data source.
67
68- `testing_criteria: Iterable[TestingCriterion]`
69
70 A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).
71
72 - `class TestingCriterionLabelModel: …`
73
74 A LabelModelGrader object which uses a model to assign labels to each item
75 in the evaluation.
76
77 - `input: Iterable[TestingCriterionLabelModelInput]`
78
79 A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
80
81 - `class TestingCriterionLabelModelInputSimpleInputMessage: …`
82
83 - `content: str`
84
85 The content of the message.
86
87 - `role: str`
88
89 The role of the message (e.g. "system", "assistant", "user").
90
91 - `class TestingCriterionLabelModelInputEvalItem: …`
92
93 A message input to the model with a role indicating instruction following
94 hierarchy. Instructions given with the `developer` or `system` role take
95 precedence over instructions given with the `user` role. Messages with the
96 `assistant` role are presumed to have been generated by the model in previous
97 interactions.
98
99 - `content: TestingCriterionLabelModelInputEvalItemContent`
100
101 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
102
103 - `str`
104
105 A text input to the model.
106
107 - `class ResponseInputText: …`
108
109 A text input to the model.
110
111 - `text: str`
112
113 The text input to the model.
114
115 - `type: Literal["input_text"]`
116
117 The type of the input item. Always `input_text`.
118
119 - `"input_text"`
120
121 - `prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]`
122
123 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
124
125 - `mode: Literal["explicit"]`
126
127 The breakpoint mode. Always `explicit`.
128
129 - `"explicit"`
130
131 - `class TestingCriterionLabelModelInputEvalItemContentOutputText: …`
132
133 A text output from the model.
134
135 - `text: str`
136
137 The text output from the model.
138
139 - `type: Literal["output_text"]`
140
141 The type of the output text. Always `output_text`.
142
143 - `"output_text"`
144
145 - `class TestingCriterionLabelModelInputEvalItemContentInputImage: …`
146
147 An image input block used within EvalItem content arrays.
148
149 - `image_url: str`
150
151 The URL of the image input.
152
153 - `type: Literal["input_image"]`
154
155 The type of the image input. Always `input_image`.
156
157 - `"input_image"`
158
159 - `detail: Optional[str]`
160
161 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
162
163 - `class ResponseInputAudio: …`
164
165 An audio input to the model.
166
167 - `input_audio: InputAudio`
168
169 - `data: str`
170
171 Base64-encoded audio data.
172
173 - `format: Literal["mp3", "wav"]`
174
175 The format of the audio data. Currently supported formats are `mp3` and
176 `wav`.
177
178 - `"mp3"`
179
180 - `"wav"`
181
182 - `type: Literal["input_audio"]`
183
184 The type of the input item. Always `input_audio`.
185
186 - `"input_audio"`
187
188 - `Sequence[GraderInputsParamItem]`
189
190 - `str`
191
192 A text input to the model.
193
194 - `class ResponseInputText: …`
195
196 A text input to the model.
197
198 - `class GraderInputItemOutputText: …`
199
200 A text output from the model.
201
202 - `text: str`
203
204 The text output from the model.
205
206 - `type: Literal["output_text"]`
207
208 The type of the output text. Always `output_text`.
209
210 - `"output_text"`
211
212 - `class GraderInputItemInputImage: …`
213
214 An image input block used within EvalItem content arrays.
215
216 - `image_url: str`
217
218 The URL of the image input.
219
220 - `type: Literal["input_image"]`
221
222 The type of the image input. Always `input_image`.
223
224 - `"input_image"`
225
226 - `detail: Optional[str]`
227
228 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
229
230 - `class ResponseInputAudio: …`
231
232 An audio input to the model.
233
234 - `role: Literal["user", "assistant", "system", "developer"]`
235
236 The role of the message input. One of `user`, `assistant`, `system`, or
237 `developer`.
238
239 - `"user"`
240
241 - `"assistant"`
242
243 - `"system"`
244
245 - `"developer"`
246
247 - `type: Optional[Literal["message"]]`
248
249 The type of the message input. Always `message`.
250
251 - `"message"`
252
253 - `labels: Sequence[str]`
254
255 The labels to classify to each item in the evaluation.
256
257 - `model: str`
258
259 The model to use for the evaluation. Must support structured outputs.
260
261 - `name: str`
262
263 The name of the grader.
264
265 - `passing_labels: Sequence[str]`
266
267 The labels that indicate a passing result. Must be a subset of labels.
268
269 - `type: Literal["label_model"]`
270
271 The object type, which is always `label_model`.
272
273 - `"label_model"`
274
275 - `class StringCheckGrader: …`
276
277 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
278
279 - `input: str`
280
281 The input text. This may include template strings.
282
283 - `name: str`
284
285 The name of the grader.
286
287 - `operation: Literal["eq", "ne", "like", "ilike"]`
288
289 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
290
291 - `"eq"`
292
293 - `"ne"`
294
295 - `"like"`
296
297 - `"ilike"`
298
299 - `reference: str`
300
301 The reference text. This may include template strings.
302
303 - `type: Literal["string_check"]`
304
305 The object type, which is always `string_check`.
306
307 - `"string_check"`
308
309 - `class TestingCriterionTextSimilarity: …`
310
311 A TextSimilarityGrader object which grades text based on similarity metrics.
312
313 - `pass_threshold: float`
314
315 The threshold for the score.
316
317 - `class TestingCriterionPython: …`
318
319 A PythonGrader object that runs a python script on the input.
320
321 - `pass_threshold: Optional[float]`
322
323 The threshold for the score.
324
325 - `class TestingCriterionScoreModel: …`
326
327 A ScoreModelGrader object that uses a model to assign a score to the input.
328
329 - `pass_threshold: Optional[float]`
330
331 The threshold for the score.
332
333- `metadata: Optional[Metadata]`
334
335 Set of 16 key-value pairs that can be attached to an object. This can be
336 useful for storing additional information about the object in a structured
337 format, and querying for objects via API or the dashboard.
338
339 Keys are strings with a maximum length of 64 characters. Values are strings
340 with a maximum length of 512 characters.
341
342- `name: Optional[str]`
343
344 The name of the evaluation.
345
346### Returns
347
348- `class EvalCreateResponse: …`
349
350 An Eval object with a data source config and testing criteria.
351 An Eval represents a task to be done for your LLM integration.
352 Like:
353
354 - Improve the quality of my chatbot
355 - See how well my chatbot handles customer support
356 - Check if o4-mini is better at my usecase than gpt-4o
357
358 - `id: str`
359
360 Unique identifier for the evaluation.
361
362 - `created_at: int`
363
364 The Unix timestamp (in seconds) for when the eval was created.
365
366 - `data_source_config: DataSourceConfig`
367
368 Configuration of data sources used in runs of the evaluation.
369
370 - `class EvalCustomDataSourceConfig: …`
371
372 A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
373 The response schema defines the shape of the data that will be:
374
375 - Used to define your testing criteria and
376 - What data is required when creating a run
377
378 - `schema: Dict[str, object]`
379
380 The json schema for the run data source items.
381 Learn how to build JSON schemas [here](https://json-schema.org/).
382
383 - `type: Literal["custom"]`
384
385 The type of data source. Always `custom`.
386
387 - `"custom"`
388
389 - `class DataSourceConfigLogs: …`
390
391 A LogsDataSourceConfig which specifies the metadata property of your logs query.
392 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
393 The schema returned by this data source config is used to defined what variables are available in your evals.
394 `item` and `sample` are both defined when using this data source config.
395
396 - `schema: Dict[str, object]`
397
398 The json schema for the run data source items.
399 Learn how to build JSON schemas [here](https://json-schema.org/).
400
401 - `type: Literal["logs"]`
402
403 The type of data source. Always `logs`.
404
405 - `"logs"`
406
407 - `metadata: Optional[Metadata]`
408
409 Set of 16 key-value pairs that can be attached to an object. This can be
410 useful for storing additional information about the object in a structured
411 format, and querying for objects via API or the dashboard.
412
413 Keys are strings with a maximum length of 64 characters. Values are strings
414 with a maximum length of 512 characters.
415
416 - `class EvalStoredCompletionsDataSourceConfig: …`
417
418 Deprecated in favor of LogsDataSourceConfig.
419
420 - `schema: Dict[str, object]`
421
422 The json schema for the run data source items.
423 Learn how to build JSON schemas [here](https://json-schema.org/).
424
425 - `type: Literal["stored_completions"]`
426
427 The type of data source. Always `stored_completions`.
428
429 - `"stored_completions"`
430
431 - `metadata: Optional[Metadata]`
432
433 Set of 16 key-value pairs that can be attached to an object. This can be
434 useful for storing additional information about the object in a structured
435 format, and querying for objects via API or the dashboard.
436
437 Keys are strings with a maximum length of 64 characters. Values are strings
438 with a maximum length of 512 characters.
439
440 - `metadata: Optional[Metadata]`
441
442 Set of 16 key-value pairs that can be attached to an object. This can be
443 useful for storing additional information about the object in a structured
444 format, and querying for objects via API or the dashboard.
445
446 Keys are strings with a maximum length of 64 characters. Values are strings
447 with a maximum length of 512 characters.
448
449 - `name: str`
450
451 The name of the evaluation.
452
453 - `object: Literal["eval"]`
454
455 The object type.
456
457 - `"eval"`
458
459 - `testing_criteria: List[TestingCriterion]`
460
461 A list of testing criteria.
462
463 - `class LabelModelGrader: …`
464
465 A LabelModelGrader object which uses a model to assign labels to each item
466 in the evaluation.
467
468 - `input: List[Input]`
469
470 - `content: InputContent`
471
472 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
473
474 - `str`
475
476 A text input to the model.
477
478 - `class ResponseInputText: …`
479
480 A text input to the model.
481
482 - `text: str`
483
484 The text input to the model.
485
486 - `type: Literal["input_text"]`
487
488 The type of the input item. Always `input_text`.
489
490 - `"input_text"`
491
492 - `prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]`
493
494 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
495
496 - `mode: Literal["explicit"]`
497
498 The breakpoint mode. Always `explicit`.
499
500 - `"explicit"`
501
502 - `class InputContentOutputText: …`
503
504 A text output from the model.
505
506 - `text: str`
507
508 The text output from the model.
509
510 - `type: Literal["output_text"]`
511
512 The type of the output text. Always `output_text`.
513
514 - `"output_text"`
515
516 - `class InputContentInputImage: …`
517
518 An image input block used within EvalItem content arrays.
519
520 - `image_url: str`
521
522 The URL of the image input.
523
524 - `type: Literal["input_image"]`
525
526 The type of the image input. Always `input_image`.
527
528 - `"input_image"`
529
530 - `detail: Optional[str]`
531
532 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
533
534 - `class ResponseInputAudio: …`
535
536 An audio input to the model.
537
538 - `input_audio: InputAudio`
539
540 - `data: str`
541
542 Base64-encoded audio data.
543
544 - `format: Literal["mp3", "wav"]`
545
546 The format of the audio data. Currently supported formats are `mp3` and
547 `wav`.
548
549 - `"mp3"`
550
551 - `"wav"`
552
553 - `type: Literal["input_audio"]`
554
555 The type of the input item. Always `input_audio`.
556
557 - `"input_audio"`
558
559 - `List[GraderInputItem]`
560
561 - `str`
562
563 A text input to the model.
564
565 - `class ResponseInputText: …`
566
567 A text input to the model.
568
569 - `class GraderInputItemOutputText: …`
570
571 A text output from the model.
572
573 - `text: str`
574
575 The text output from the model.
576
577 - `type: Literal["output_text"]`
578
579 The type of the output text. Always `output_text`.
580
581 - `"output_text"`
582
583 - `class GraderInputItemInputImage: …`
584
585 An image input block used within EvalItem content arrays.
586
587 - `image_url: str`
588
589 The URL of the image input.
590
591 - `type: Literal["input_image"]`
592
593 The type of the image input. Always `input_image`.
594
595 - `"input_image"`
596
597 - `detail: Optional[str]`
598
599 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
600
601 - `class ResponseInputAudio: …`
602
603 An audio input to the model.
604
605 - `role: Literal["user", "assistant", "system", "developer"]`
606
607 The role of the message input. One of `user`, `assistant`, `system`, or
608 `developer`.
609
610 - `"user"`
611
612 - `"assistant"`
613
614 - `"system"`
615
616 - `"developer"`
617
618 - `type: Optional[Literal["message"]]`
619
620 The type of the message input. Always `message`.
621
622 - `"message"`
623
624 - `labels: List[str]`
625
626 The labels to assign to each item in the evaluation.
627
628 - `model: str`
629
630 The model to use for the evaluation. Must support structured outputs.
631
632 - `name: str`
633
634 The name of the grader.
635
636 - `passing_labels: List[str]`
637
638 The labels that indicate a passing result. Must be a subset of labels.
639
640 - `type: Literal["label_model"]`
641
642 The object type, which is always `label_model`.
643
644 - `"label_model"`
645
646 - `class StringCheckGrader: …`
647
648 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
649
650 - `input: str`
651
652 The input text. This may include template strings.
653
654 - `name: str`
655
656 The name of the grader.
657
658 - `operation: Literal["eq", "ne", "like", "ilike"]`
659
660 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
661
662 - `"eq"`
663
664 - `"ne"`
665
666 - `"like"`
667
668 - `"ilike"`
669
670 - `reference: str`
671
672 The reference text. This may include template strings.
673
674 - `type: Literal["string_check"]`
675
676 The object type, which is always `string_check`.
677
678 - `"string_check"`
679
680 - `class TestingCriterionEvalGraderTextSimilarity: …`
681
682 A TextSimilarityGrader object which grades text based on similarity metrics.
683
684 - `pass_threshold: float`
685
686 The threshold for the score.
687
688 - `class TestingCriterionEvalGraderPython: …`
689
690 A PythonGrader object that runs a python script on the input.
691
692 - `pass_threshold: Optional[float]`
693
694 The threshold for the score.
695
696 - `class TestingCriterionEvalGraderScoreModel: …`
697
698 A ScoreModelGrader object that uses a model to assign a score to the input.
699
700 - `pass_threshold: Optional[float]`
701
702 The threshold for the score.
703
704### Example
705
706```python
707import os
708from openai import OpenAI
709
710client = OpenAI(
711 api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted
712)
713eval = client.evals.create(
714 data_source_config={
715 "item_schema": {
716 "foo": "bar"
717 },
718 "type": "custom",
719 },
720 testing_criteria=[{
721 "input": [{
722 "content": "content",
723 "role": "role",
724 }],
725 "labels": ["string"],
726 "model": "model",
727 "name": "name",
728 "passing_labels": ["string"],
729 "type": "label_model",
730 }],
731)
732print(eval.id)
733```
734
735#### Response
736
737```json
738{
739 "id": "id",
740 "created_at": 0,
741 "data_source_config": {
742 "schema": {
743 "foo": "bar"
744 },
745 "type": "custom"
746 },
747 "metadata": {
748 "foo": "string"
749 },
750 "name": "Chatbot effectiveness Evaluation",
751 "object": "eval",
752 "testing_criteria": [
753 {
754 "input": [
755 {
756 "content": "string",
757 "role": "user",
758 "type": "message"
759 }
760 ],
761 "labels": [
762 "string"
763 ],
764 "model": "model",
765 "name": "name",
766 "passing_labels": [
767 "string"
768 ],
769 "type": "label_model"
770 }
771 ]
772}
773```
774
775### Example
776
777```python
778from openai import OpenAI
779client = OpenAI()
780
781eval_obj = client.evals.create(
782 name="Sentiment",
783 data_source_config={
784 "type": "stored_completions",
785 "metadata": {"usecase": "chatbot"}
786 },
787 testing_criteria=[
788 {
789 "type": "label_model",
790 "model": "o3-mini",
791 "input": [
792 {"role": "developer", "content": "Classify the sentiment of the following statement as one of 'positive', 'neutral', or 'negative'"},
793 {"role": "user", "content": "Statement: {{item.input}}"}
794 ],
795 "passing_labels": ["positive"],
796 "labels": ["positive", "neutral", "negative"],
797 "name": "Example label grader"
798 }
799 ]
800)
801print(eval_obj)
802```
803
804#### Response
805
806```json
807{
808 "object": "eval",
809 "id": "eval_67b7fa9a81a88190ab4aa417e397ea21",
810 "data_source_config": {
811 "type": "stored_completions",
812 "metadata": {
813 "usecase": "chatbot"
814 },
815 "schema": {
816 "type": "object",
817 "properties": {
818 "item": {
819 "type": "object"
820 },
821 "sample": {
822 "type": "object"
823 }
824 },
825 "required": [
826 "item",
827 "sample"
828 ]
829 },
830 "testing_criteria": [
831 {
832 "name": "Example label grader",
833 "type": "label_model",
834 "model": "o3-mini",
835 "input": [
836 {
837 "type": "message",
838 "role": "developer",
839 "content": {
840 "type": "input_text",
841 "text": "Classify the sentiment of the following statement as one of positive, neutral, or negative"
842 }
843 },
844 {
845 "type": "message",
846 "role": "user",
847 "content": {
848 "type": "input_text",
849 "text": "Statement: {{item.input}}"
850 }
851 }
852 ],
853 "passing_labels": [
854 "positive"
855 ],
856 "labels": [
857 "positive",
858 "neutral",
859 "negative"
860 ]
861 }
862 ],
863 "name": "Sentiment",
864 "created_at": 1740110490,
865 "metadata": {
866 "description": "An eval for sentiment analysis"
867 }
868}
869```