python/resources/evals/methods/create/index.md +0 −849 deleted
File Deleted View Diff
1## Create eval
2
3`evals.create(EvalCreateParams**kwargs) -> EvalCreateResponse`
4
5**post** `/evals`
6
7Create the structure of an evaluation that can be used to test a model's performance.
8An evaluation is a set of testing criteria and the config for a data source, which dictates the schema of the data used in the evaluation. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources.
9For more information, see the [Evals guide](https://platform.openai.com/docs/guides/evals).
10
11### Parameters
12
13- `data_source_config: DataSourceConfig`
14
15 The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
16
17 - `class DataSourceConfigCustom: …`
18
19 A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs.
20 This schema is used to define the shape of the data that will be:
21
22 - Used to define your testing criteria and
23 - What data is required when creating a run
24
25 - `item_schema: Dict[str, object]`
26
27 The json schema for each row in the data source.
28
29 - `type: Literal["custom"]`
30
31 The type of data source. Always `custom`.
32
33 - `"custom"`
34
35 - `include_sample_schema: Optional[bool]`
36
37 Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)
38
39 - `class DataSourceConfigLogs: …`
40
41 A data source config which specifies the metadata property of your logs query.
42 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
43
44 - `type: Literal["logs"]`
45
46 The type of data source. Always `logs`.
47
48 - `"logs"`
49
50 - `metadata: Optional[Dict[str, object]]`
51
52 Metadata filters for the logs data source.
53
54 - `class DataSourceConfigStoredCompletions: …`
55
56 Deprecated in favor of LogsDataSourceConfig.
57
58 - `type: Literal["stored_completions"]`
59
60 The type of data source. Always `stored_completions`.
61
62 - `"stored_completions"`
63
64 - `metadata: Optional[Dict[str, object]]`
65
66 Metadata filters for the stored completions data source.
67
68- `testing_criteria: Iterable[TestingCriterion]`
69
70 A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).
71
72 - `class TestingCriterionLabelModel: …`
73
74 A LabelModelGrader object which uses a model to assign labels to each item
75 in the evaluation.
76
77 - `input: Iterable[TestingCriterionLabelModelInput]`
78
79 A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
80
81 - `class TestingCriterionLabelModelInputSimpleInputMessage: …`
82
83 - `content: str`
84
85 The content of the message.
86
87 - `role: str`
88
89 The role of the message (e.g. "system", "assistant", "user").
90
91 - `class TestingCriterionLabelModelInputEvalItem: …`
92
93 A message input to the model with a role indicating instruction following
94 hierarchy. Instructions given with the `developer` or `system` role take
95 precedence over instructions given with the `user` role. Messages with the
96 `assistant` role are presumed to have been generated by the model in previous
97 interactions.
98
99 - `content: TestingCriterionLabelModelInputEvalItemContent`
100
101 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
102
103 - `str`
104
105 A text input to the model.
106
107 - `class ResponseInputText: …`
108
109 A text input to the model.
110
111 - `text: str`
112
113 The text input to the model.
114
115 - `type: Literal["input_text"]`
116
117 The type of the input item. Always `input_text`.
118
119 - `"input_text"`
120
121 - `class TestingCriterionLabelModelInputEvalItemContentOutputText: …`
122
123 A text output from the model.
124
125 - `text: str`
126
127 The text output from the model.
128
129 - `type: Literal["output_text"]`
130
131 The type of the output text. Always `output_text`.
132
133 - `"output_text"`
134
135 - `class TestingCriterionLabelModelInputEvalItemContentInputImage: …`
136
137 An image input block used within EvalItem content arrays.
138
139 - `image_url: str`
140
141 The URL of the image input.
142
143 - `type: Literal["input_image"]`
144
145 The type of the image input. Always `input_image`.
146
147 - `"input_image"`
148
149 - `detail: Optional[str]`
150
151 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
152
153 - `class ResponseInputAudio: …`
154
155 An audio input to the model.
156
157 - `input_audio: InputAudio`
158
159 - `data: str`
160
161 Base64-encoded audio data.
162
163 - `format: Literal["mp3", "wav"]`
164
165 The format of the audio data. Currently supported formats are `mp3` and
166 `wav`.
167
168 - `"mp3"`
169
170 - `"wav"`
171
172 - `type: Literal["input_audio"]`
173
174 The type of the input item. Always `input_audio`.
175
176 - `"input_audio"`
177
178 - `Sequence[GraderInputsParamItem]`
179
180 - `str`
181
182 A text input to the model.
183
184 - `class ResponseInputText: …`
185
186 A text input to the model.
187
188 - `class GraderInputItemOutputText: …`
189
190 A text output from the model.
191
192 - `text: str`
193
194 The text output from the model.
195
196 - `type: Literal["output_text"]`
197
198 The type of the output text. Always `output_text`.
199
200 - `"output_text"`
201
202 - `class GraderInputItemInputImage: …`
203
204 An image input block used within EvalItem content arrays.
205
206 - `image_url: str`
207
208 The URL of the image input.
209
210 - `type: Literal["input_image"]`
211
212 The type of the image input. Always `input_image`.
213
214 - `"input_image"`
215
216 - `detail: Optional[str]`
217
218 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
219
220 - `class ResponseInputAudio: …`
221
222 An audio input to the model.
223
224 - `role: Literal["user", "assistant", "system", "developer"]`
225
226 The role of the message input. One of `user`, `assistant`, `system`, or
227 `developer`.
228
229 - `"user"`
230
231 - `"assistant"`
232
233 - `"system"`
234
235 - `"developer"`
236
237 - `type: Optional[Literal["message"]]`
238
239 The type of the message input. Always `message`.
240
241 - `"message"`
242
243 - `labels: Sequence[str]`
244
245 The labels to classify to each item in the evaluation.
246
247 - `model: str`
248
249 The model to use for the evaluation. Must support structured outputs.
250
251 - `name: str`
252
253 The name of the grader.
254
255 - `passing_labels: Sequence[str]`
256
257 The labels that indicate a passing result. Must be a subset of labels.
258
259 - `type: Literal["label_model"]`
260
261 The object type, which is always `label_model`.
262
263 - `"label_model"`
264
265 - `class StringCheckGrader: …`
266
267 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
268
269 - `input: str`
270
271 The input text. This may include template strings.
272
273 - `name: str`
274
275 The name of the grader.
276
277 - `operation: Literal["eq", "ne", "like", "ilike"]`
278
279 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
280
281 - `"eq"`
282
283 - `"ne"`
284
285 - `"like"`
286
287 - `"ilike"`
288
289 - `reference: str`
290
291 The reference text. This may include template strings.
292
293 - `type: Literal["string_check"]`
294
295 The object type, which is always `string_check`.
296
297 - `"string_check"`
298
299 - `class TestingCriterionTextSimilarity: …`
300
301 A TextSimilarityGrader object which grades text based on similarity metrics.
302
303 - `pass_threshold: float`
304
305 The threshold for the score.
306
307 - `class TestingCriterionPython: …`
308
309 A PythonGrader object that runs a python script on the input.
310
311 - `pass_threshold: Optional[float]`
312
313 The threshold for the score.
314
315 - `class TestingCriterionScoreModel: …`
316
317 A ScoreModelGrader object that uses a model to assign a score to the input.
318
319 - `pass_threshold: Optional[float]`
320
321 The threshold for the score.
322
323- `metadata: Optional[Metadata]`
324
325 Set of 16 key-value pairs that can be attached to an object. This can be
326 useful for storing additional information about the object in a structured
327 format, and querying for objects via API or the dashboard.
328
329 Keys are strings with a maximum length of 64 characters. Values are strings
330 with a maximum length of 512 characters.
331
332- `name: Optional[str]`
333
334 The name of the evaluation.
335
336### Returns
337
338- `class EvalCreateResponse: …`
339
340 An Eval object with a data source config and testing criteria.
341 An Eval represents a task to be done for your LLM integration.
342 Like:
343
344 - Improve the quality of my chatbot
345 - See how well my chatbot handles customer support
346 - Check if o4-mini is better at my usecase than gpt-4o
347
348 - `id: str`
349
350 Unique identifier for the evaluation.
351
352 - `created_at: int`
353
354 The Unix timestamp (in seconds) for when the eval was created.
355
356 - `data_source_config: DataSourceConfig`
357
358 Configuration of data sources used in runs of the evaluation.
359
360 - `class EvalCustomDataSourceConfig: …`
361
362 A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
363 The response schema defines the shape of the data that will be:
364
365 - Used to define your testing criteria and
366 - What data is required when creating a run
367
368 - `schema: Dict[str, object]`
369
370 The json schema for the run data source items.
371 Learn how to build JSON schemas [here](https://json-schema.org/).
372
373 - `type: Literal["custom"]`
374
375 The type of data source. Always `custom`.
376
377 - `"custom"`
378
379 - `class DataSourceConfigLogs: …`
380
381 A LogsDataSourceConfig which specifies the metadata property of your logs query.
382 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
383 The schema returned by this data source config is used to defined what variables are available in your evals.
384 `item` and `sample` are both defined when using this data source config.
385
386 - `schema: Dict[str, object]`
387
388 The json schema for the run data source items.
389 Learn how to build JSON schemas [here](https://json-schema.org/).
390
391 - `type: Literal["logs"]`
392
393 The type of data source. Always `logs`.
394
395 - `"logs"`
396
397 - `metadata: Optional[Metadata]`
398
399 Set of 16 key-value pairs that can be attached to an object. This can be
400 useful for storing additional information about the object in a structured
401 format, and querying for objects via API or the dashboard.
402
403 Keys are strings with a maximum length of 64 characters. Values are strings
404 with a maximum length of 512 characters.
405
406 - `class EvalStoredCompletionsDataSourceConfig: …`
407
408 Deprecated in favor of LogsDataSourceConfig.
409
410 - `schema: Dict[str, object]`
411
412 The json schema for the run data source items.
413 Learn how to build JSON schemas [here](https://json-schema.org/).
414
415 - `type: Literal["stored_completions"]`
416
417 The type of data source. Always `stored_completions`.
418
419 - `"stored_completions"`
420
421 - `metadata: Optional[Metadata]`
422
423 Set of 16 key-value pairs that can be attached to an object. This can be
424 useful for storing additional information about the object in a structured
425 format, and querying for objects via API or the dashboard.
426
427 Keys are strings with a maximum length of 64 characters. Values are strings
428 with a maximum length of 512 characters.
429
430 - `metadata: Optional[Metadata]`
431
432 Set of 16 key-value pairs that can be attached to an object. This can be
433 useful for storing additional information about the object in a structured
434 format, and querying for objects via API or the dashboard.
435
436 Keys are strings with a maximum length of 64 characters. Values are strings
437 with a maximum length of 512 characters.
438
439 - `name: str`
440
441 The name of the evaluation.
442
443 - `object: Literal["eval"]`
444
445 The object type.
446
447 - `"eval"`
448
449 - `testing_criteria: List[TestingCriterion]`
450
451 A list of testing criteria.
452
453 - `class LabelModelGrader: …`
454
455 A LabelModelGrader object which uses a model to assign labels to each item
456 in the evaluation.
457
458 - `input: List[Input]`
459
460 - `content: InputContent`
461
462 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
463
464 - `str`
465
466 A text input to the model.
467
468 - `class ResponseInputText: …`
469
470 A text input to the model.
471
472 - `text: str`
473
474 The text input to the model.
475
476 - `type: Literal["input_text"]`
477
478 The type of the input item. Always `input_text`.
479
480 - `"input_text"`
481
482 - `class InputContentOutputText: …`
483
484 A text output from the model.
485
486 - `text: str`
487
488 The text output from the model.
489
490 - `type: Literal["output_text"]`
491
492 The type of the output text. Always `output_text`.
493
494 - `"output_text"`
495
496 - `class InputContentInputImage: …`
497
498 An image input block used within EvalItem content arrays.
499
500 - `image_url: str`
501
502 The URL of the image input.
503
504 - `type: Literal["input_image"]`
505
506 The type of the image input. Always `input_image`.
507
508 - `"input_image"`
509
510 - `detail: Optional[str]`
511
512 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
513
514 - `class ResponseInputAudio: …`
515
516 An audio input to the model.
517
518 - `input_audio: InputAudio`
519
520 - `data: str`
521
522 Base64-encoded audio data.
523
524 - `format: Literal["mp3", "wav"]`
525
526 The format of the audio data. Currently supported formats are `mp3` and
527 `wav`.
528
529 - `"mp3"`
530
531 - `"wav"`
532
533 - `type: Literal["input_audio"]`
534
535 The type of the input item. Always `input_audio`.
536
537 - `"input_audio"`
538
539 - `List[GraderInputItem]`
540
541 - `str`
542
543 A text input to the model.
544
545 - `class ResponseInputText: …`
546
547 A text input to the model.
548
549 - `class GraderInputItemOutputText: …`
550
551 A text output from the model.
552
553 - `text: str`
554
555 The text output from the model.
556
557 - `type: Literal["output_text"]`
558
559 The type of the output text. Always `output_text`.
560
561 - `"output_text"`
562
563 - `class GraderInputItemInputImage: …`
564
565 An image input block used within EvalItem content arrays.
566
567 - `image_url: str`
568
569 The URL of the image input.
570
571 - `type: Literal["input_image"]`
572
573 The type of the image input. Always `input_image`.
574
575 - `"input_image"`
576
577 - `detail: Optional[str]`
578
579 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
580
581 - `class ResponseInputAudio: …`
582
583 An audio input to the model.
584
585 - `role: Literal["user", "assistant", "system", "developer"]`
586
587 The role of the message input. One of `user`, `assistant`, `system`, or
588 `developer`.
589
590 - `"user"`
591
592 - `"assistant"`
593
594 - `"system"`
595
596 - `"developer"`
597
598 - `type: Optional[Literal["message"]]`
599
600 The type of the message input. Always `message`.
601
602 - `"message"`
603
604 - `labels: List[str]`
605
606 The labels to assign to each item in the evaluation.
607
608 - `model: str`
609
610 The model to use for the evaluation. Must support structured outputs.
611
612 - `name: str`
613
614 The name of the grader.
615
616 - `passing_labels: List[str]`
617
618 The labels that indicate a passing result. Must be a subset of labels.
619
620 - `type: Literal["label_model"]`
621
622 The object type, which is always `label_model`.
623
624 - `"label_model"`
625
626 - `class StringCheckGrader: …`
627
628 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
629
630 - `input: str`
631
632 The input text. This may include template strings.
633
634 - `name: str`
635
636 The name of the grader.
637
638 - `operation: Literal["eq", "ne", "like", "ilike"]`
639
640 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
641
642 - `"eq"`
643
644 - `"ne"`
645
646 - `"like"`
647
648 - `"ilike"`
649
650 - `reference: str`
651
652 The reference text. This may include template strings.
653
654 - `type: Literal["string_check"]`
655
656 The object type, which is always `string_check`.
657
658 - `"string_check"`
659
660 - `class TestingCriterionEvalGraderTextSimilarity: …`
661
662 A TextSimilarityGrader object which grades text based on similarity metrics.
663
664 - `pass_threshold: float`
665
666 The threshold for the score.
667
668 - `class TestingCriterionEvalGraderPython: …`
669
670 A PythonGrader object that runs a python script on the input.
671
672 - `pass_threshold: Optional[float]`
673
674 The threshold for the score.
675
676 - `class TestingCriterionEvalGraderScoreModel: …`
677
678 A ScoreModelGrader object that uses a model to assign a score to the input.
679
680 - `pass_threshold: Optional[float]`
681
682 The threshold for the score.
683
684### Example
685
686```python
687import os
688from openai import OpenAI
689
690client = OpenAI(
691 api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted
692)
693eval = client.evals.create(
694 data_source_config={
695 "item_schema": {
696 "foo": "bar"
697 },
698 "type": "custom",
699 },
700 testing_criteria=[{
701 "input": [{
702 "content": "content",
703 "role": "role",
704 }],
705 "labels": ["string"],
706 "model": "model",
707 "name": "name",
708 "passing_labels": ["string"],
709 "type": "label_model",
710 }],
711)
712print(eval.id)
713```
714
715#### Response
716
717```json
718{
719 "id": "id",
720 "created_at": 0,
721 "data_source_config": {
722 "schema": {
723 "foo": "bar"
724 },
725 "type": "custom"
726 },
727 "metadata": {
728 "foo": "string"
729 },
730 "name": "Chatbot effectiveness Evaluation",
731 "object": "eval",
732 "testing_criteria": [
733 {
734 "input": [
735 {
736 "content": "string",
737 "role": "user",
738 "type": "message"
739 }
740 ],
741 "labels": [
742 "string"
743 ],
744 "model": "model",
745 "name": "name",
746 "passing_labels": [
747 "string"
748 ],
749 "type": "label_model"
750 }
751 ]
752}
753```
754
755### Example
756
757```python
758from openai import OpenAI
759client = OpenAI()
760
761eval_obj = client.evals.create(
762 name="Sentiment",
763 data_source_config={
764 "type": "stored_completions",
765 "metadata": {"usecase": "chatbot"}
766 },
767 testing_criteria=[
768 {
769 "type": "label_model",
770 "model": "o3-mini",
771 "input": [
772 {"role": "developer", "content": "Classify the sentiment of the following statement as one of 'positive', 'neutral', or 'negative'"},
773 {"role": "user", "content": "Statement: {{item.input}}"}
774 ],
775 "passing_labels": ["positive"],
776 "labels": ["positive", "neutral", "negative"],
777 "name": "Example label grader"
778 }
779 ]
780)
781print(eval_obj)
782```
783
784#### Response
785
786```json
787{
788 "object": "eval",
789 "id": "eval_67b7fa9a81a88190ab4aa417e397ea21",
790 "data_source_config": {
791 "type": "stored_completions",
792 "metadata": {
793 "usecase": "chatbot"
794 },
795 "schema": {
796 "type": "object",
797 "properties": {
798 "item": {
799 "type": "object"
800 },
801 "sample": {
802 "type": "object"
803 }
804 },
805 "required": [
806 "item",
807 "sample"
808 ]
809 },
810 "testing_criteria": [
811 {
812 "name": "Example label grader",
813 "type": "label_model",
814 "model": "o3-mini",
815 "input": [
816 {
817 "type": "message",
818 "role": "developer",
819 "content": {
820 "type": "input_text",
821 "text": "Classify the sentiment of the following statement as one of positive, neutral, or negative"
822 }
823 },
824 {
825 "type": "message",
826 "role": "user",
827 "content": {
828 "type": "input_text",
829 "text": "Statement: {{item.input}}"
830 }
831 }
832 ],
833 "passing_labels": [
834 "positive"
835 ],
836 "labels": [
837 "positive",
838 "neutral",
839 "negative"
840 ]
841 }
842 ],
843 "name": "Sentiment",
844 "created_at": 1740110490,
845 "metadata": {
846 "description": "An eval for sentiment analysis"
847 }
848}
849```