cli/resources/graders/index.md +0 −976 deleted
File Deleted View Diff
1# Graders
2
3# Grader Models
4
5## Domain Types
6
7### Grader Inputs
8
9- `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
10
11 A list of inputs, each of which may be either an input text, output text, input
12 image, or input audio object.
13
14 - `Text input: string`
15
16 A text input to the model.
17
18 - `response_input_text: object { text, type }`
19
20 A text input to the model.
21
22 - `text: string`
23
24 The text input to the model.
25
26 - `type: "input_text"`
27
28 The type of the input item. Always `input_text`.
29
30 - `Output text: object { text, type }`
31
32 A text output from the model.
33
34 - `text: string`
35
36 The text output from the model.
37
38 - `type: "output_text"`
39
40 The type of the output text. Always `output_text`.
41
42 - `Input image: object { image_url, type, detail }`
43
44 An image input block used within EvalItem content arrays.
45
46 - `image_url: string`
47
48 The URL of the image input.
49
50 - `type: "input_image"`
51
52 The type of the image input. Always `input_image`.
53
54 - `detail: optional string`
55
56 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
57
58 - `response_input_audio: object { input_audio, type }`
59
60 An audio input to the model.
61
62 - `input_audio: object { data, format }`
63
64 - `data: string`
65
66 Base64-encoded audio data.
67
68 - `format: "mp3" or "wav"`
69
70 The format of the audio data. Currently supported formats are `mp3` and
71 `wav`.
72
73 - `"mp3"`
74
75 - `"wav"`
76
77 - `type: "input_audio"`
78
79 The type of the input item. Always `input_audio`.
80
81### Label Model Grader
82
83- `label_model_grader: object { input, labels, model, 3 more }`
84
85 A LabelModelGrader object which uses a model to assign labels to each item
86 in the evaluation.
87
88 - `input: array of object { content, role, type }`
89
90 - `content: string or ResponseInputText or object { text, type } or 3 more`
91
92 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
93
94 - `Text input: string`
95
96 A text input to the model.
97
98 - `response_input_text: object { text, type }`
99
100 A text input to the model.
101
102 - `text: string`
103
104 The text input to the model.
105
106 - `type: "input_text"`
107
108 The type of the input item. Always `input_text`.
109
110 - `Output text: object { text, type }`
111
112 A text output from the model.
113
114 - `text: string`
115
116 The text output from the model.
117
118 - `type: "output_text"`
119
120 The type of the output text. Always `output_text`.
121
122 - `Input image: object { image_url, type, detail }`
123
124 An image input block used within EvalItem content arrays.
125
126 - `image_url: string`
127
128 The URL of the image input.
129
130 - `type: "input_image"`
131
132 The type of the image input. Always `input_image`.
133
134 - `detail: optional string`
135
136 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
137
138 - `response_input_audio: object { input_audio, type }`
139
140 An audio input to the model.
141
142 - `input_audio: object { data, format }`
143
144 - `data: string`
145
146 Base64-encoded audio data.
147
148 - `format: "mp3" or "wav"`
149
150 The format of the audio data. Currently supported formats are `mp3` and
151 `wav`.
152
153 - `"mp3"`
154
155 - `"wav"`
156
157 - `type: "input_audio"`
158
159 The type of the input item. Always `input_audio`.
160
161 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
162
163 A list of inputs, each of which may be either an input text, output text, input
164 image, or input audio object.
165
166 - `Text input: string`
167
168 A text input to the model.
169
170 - `response_input_text: object { text, type }`
171
172 A text input to the model.
173
174 - `Output text: object { text, type }`
175
176 A text output from the model.
177
178 - `text: string`
179
180 The text output from the model.
181
182 - `type: "output_text"`
183
184 The type of the output text. Always `output_text`.
185
186 - `Input image: object { image_url, type, detail }`
187
188 An image input block used within EvalItem content arrays.
189
190 - `image_url: string`
191
192 The URL of the image input.
193
194 - `type: "input_image"`
195
196 The type of the image input. Always `input_image`.
197
198 - `detail: optional string`
199
200 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
201
202 - `response_input_audio: object { input_audio, type }`
203
204 An audio input to the model.
205
206 - `role: "user" or "assistant" or "system" or "developer"`
207
208 The role of the message input. One of `user`, `assistant`, `system`, or
209 `developer`.
210
211 - `"user"`
212
213 - `"assistant"`
214
215 - `"system"`
216
217 - `"developer"`
218
219 - `type: optional "message"`
220
221 The type of the message input. Always `message`.
222
223 - `"message"`
224
225 - `labels: array of string`
226
227 The labels to assign to each item in the evaluation.
228
229 - `model: string`
230
231 The model to use for the evaluation. Must support structured outputs.
232
233 - `name: string`
234
235 The name of the grader.
236
237 - `passing_labels: array of string`
238
239 The labels that indicate a passing result. Must be a subset of labels.
240
241 - `type: "label_model"`
242
243 The object type, which is always `label_model`.
244
245### Multi Grader
246
247- `multi_grader: object { calculate_output, graders, name, type }`
248
249 A MultiGrader object combines the output of multiple graders to produce a single score.
250
251 - `calculate_output: string`
252
253 A formula to calculate the output based on grader results.
254
255 - `graders: StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more`
256
257 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
258
259 - `string_check_grader: object { input, name, operation, 2 more }`
260
261 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
262
263 - `input: string`
264
265 The input text. This may include template strings.
266
267 - `name: string`
268
269 The name of the grader.
270
271 - `operation: "eq" or "ne" or "like" or "ilike"`
272
273 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
274
275 - `"eq"`
276
277 - `"ne"`
278
279 - `"like"`
280
281 - `"ilike"`
282
283 - `reference: string`
284
285 The reference text. This may include template strings.
286
287 - `type: "string_check"`
288
289 The object type, which is always `string_check`.
290
291 - `text_similarity_grader: object { evaluation_metric, input, name, 2 more }`
292
293 A TextSimilarityGrader object which grades text based on similarity metrics.
294
295 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`
296
297 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,
298 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,
299 or `rouge_l`.
300
301 - `"cosine"`
302
303 - `"fuzzy_match"`
304
305 - `"bleu"`
306
307 - `"gleu"`
308
309 - `"meteor"`
310
311 - `"rouge_1"`
312
313 - `"rouge_2"`
314
315 - `"rouge_3"`
316
317 - `"rouge_4"`
318
319 - `"rouge_5"`
320
321 - `"rouge_l"`
322
323 - `input: string`
324
325 The text being graded.
326
327 - `name: string`
328
329 The name of the grader.
330
331 - `reference: string`
332
333 The text being graded against.
334
335 - `type: "text_similarity"`
336
337 The type of grader.
338
339 - `python_grader: object { name, source, type, image_tag }`
340
341 A PythonGrader object that runs a python script on the input.
342
343 - `name: string`
344
345 The name of the grader.
346
347 - `source: string`
348
349 The source code of the python script.
350
351 - `type: "python"`
352
353 The object type, which is always `python`.
354
355 - `image_tag: optional string`
356
357 The image tag to use for the python script.
358
359 - `score_model_grader: object { input, model, name, 3 more }`
360
361 A ScoreModelGrader object that uses a model to assign a score to the input.
362
363 - `input: array of object { content, role, type }`
364
365 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
366
367 - `content: string or ResponseInputText or object { text, type } or 3 more`
368
369 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
370
371 - `Text input: string`
372
373 A text input to the model.
374
375 - `response_input_text: object { text, type }`
376
377 A text input to the model.
378
379 - `text: string`
380
381 The text input to the model.
382
383 - `type: "input_text"`
384
385 The type of the input item. Always `input_text`.
386
387 - `Output text: object { text, type }`
388
389 A text output from the model.
390
391 - `text: string`
392
393 The text output from the model.
394
395 - `type: "output_text"`
396
397 The type of the output text. Always `output_text`.
398
399 - `Input image: object { image_url, type, detail }`
400
401 An image input block used within EvalItem content arrays.
402
403 - `image_url: string`
404
405 The URL of the image input.
406
407 - `type: "input_image"`
408
409 The type of the image input. Always `input_image`.
410
411 - `detail: optional string`
412
413 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
414
415 - `response_input_audio: object { input_audio, type }`
416
417 An audio input to the model.
418
419 - `input_audio: object { data, format }`
420
421 - `data: string`
422
423 Base64-encoded audio data.
424
425 - `format: "mp3" or "wav"`
426
427 The format of the audio data. Currently supported formats are `mp3` and
428 `wav`.
429
430 - `"mp3"`
431
432 - `"wav"`
433
434 - `type: "input_audio"`
435
436 The type of the input item. Always `input_audio`.
437
438 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
439
440 A list of inputs, each of which may be either an input text, output text, input
441 image, or input audio object.
442
443 - `Text input: string`
444
445 A text input to the model.
446
447 - `response_input_text: object { text, type }`
448
449 A text input to the model.
450
451 - `Output text: object { text, type }`
452
453 A text output from the model.
454
455 - `text: string`
456
457 The text output from the model.
458
459 - `type: "output_text"`
460
461 The type of the output text. Always `output_text`.
462
463 - `Input image: object { image_url, type, detail }`
464
465 An image input block used within EvalItem content arrays.
466
467 - `image_url: string`
468
469 The URL of the image input.
470
471 - `type: "input_image"`
472
473 The type of the image input. Always `input_image`.
474
475 - `detail: optional string`
476
477 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
478
479 - `response_input_audio: object { input_audio, type }`
480
481 An audio input to the model.
482
483 - `role: "user" or "assistant" or "system" or "developer"`
484
485 The role of the message input. One of `user`, `assistant`, `system`, or
486 `developer`.
487
488 - `"user"`
489
490 - `"assistant"`
491
492 - `"system"`
493
494 - `"developer"`
495
496 - `type: optional "message"`
497
498 The type of the message input. Always `message`.
499
500 - `"message"`
501
502 - `model: string`
503
504 The model to use for the evaluation.
505
506 - `name: string`
507
508 The name of the grader.
509
510 - `type: "score_model"`
511
512 The object type, which is always `score_model`.
513
514 - `range: optional array of number`
515
516 The range of the score. Defaults to `[0, 1]`.
517
518 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`
519
520 The sampling parameters for the model.
521
522 - `max_completions_tokens: optional number`
523
524 The maximum number of tokens the grader model may generate in its response.
525
526 - `reasoning_effort: optional "none" or "minimal" or "low" or 3 more`
527
528 Constrains effort on reasoning for
529 [reasoning models](https://platform.openai.com/docs/guides/reasoning).
530 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
531 reasoning effort can result in faster responses and fewer tokens used
532 on reasoning in a response.
533
534 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
535 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
536 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
537 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
538
539 - `"none"`
540
541 - `"minimal"`
542
543 - `"low"`
544
545 - `"medium"`
546
547 - `"high"`
548
549 - `"xhigh"`
550
551 - `seed: optional number`
552
553 A seed value to initialize the randomness, during sampling.
554
555 - `temperature: optional number`
556
557 A higher temperature increases randomness in the outputs.
558
559 - `top_p: optional number`
560
561 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
562
563 - `label_model_grader: object { input, labels, model, 3 more }`
564
565 A LabelModelGrader object which uses a model to assign labels to each item
566 in the evaluation.
567
568 - `input: array of object { content, role, type }`
569
570 - `content: string or ResponseInputText or object { text, type } or 3 more`
571
572 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
573
574 - `Text input: string`
575
576 A text input to the model.
577
578 - `response_input_text: object { text, type }`
579
580 A text input to the model.
581
582 - `Output text: object { text, type }`
583
584 A text output from the model.
585
586 - `text: string`
587
588 The text output from the model.
589
590 - `type: "output_text"`
591
592 The type of the output text. Always `output_text`.
593
594 - `Input image: object { image_url, type, detail }`
595
596 An image input block used within EvalItem content arrays.
597
598 - `image_url: string`
599
600 The URL of the image input.
601
602 - `type: "input_image"`
603
604 The type of the image input. Always `input_image`.
605
606 - `detail: optional string`
607
608 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
609
610 - `response_input_audio: object { input_audio, type }`
611
612 An audio input to the model.
613
614 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
615
616 A list of inputs, each of which may be either an input text, output text, input
617 image, or input audio object.
618
619 - `role: "user" or "assistant" or "system" or "developer"`
620
621 The role of the message input. One of `user`, `assistant`, `system`, or
622 `developer`.
623
624 - `"user"`
625
626 - `"assistant"`
627
628 - `"system"`
629
630 - `"developer"`
631
632 - `type: optional "message"`
633
634 The type of the message input. Always `message`.
635
636 - `"message"`
637
638 - `labels: array of string`
639
640 The labels to assign to each item in the evaluation.
641
642 - `model: string`
643
644 The model to use for the evaluation. Must support structured outputs.
645
646 - `name: string`
647
648 The name of the grader.
649
650 - `passing_labels: array of string`
651
652 The labels that indicate a passing result. Must be a subset of labels.
653
654 - `type: "label_model"`
655
656 The object type, which is always `label_model`.
657
658 - `name: string`
659
660 The name of the grader.
661
662 - `type: "multi"`
663
664 The object type, which is always `multi`.
665
666### Python Grader
667
668- `python_grader: object { name, source, type, image_tag }`
669
670 A PythonGrader object that runs a python script on the input.
671
672 - `name: string`
673
674 The name of the grader.
675
676 - `source: string`
677
678 The source code of the python script.
679
680 - `type: "python"`
681
682 The object type, which is always `python`.
683
684 - `image_tag: optional string`
685
686 The image tag to use for the python script.
687
688### Score Model Grader
689
690- `score_model_grader: object { input, model, name, 3 more }`
691
692 A ScoreModelGrader object that uses a model to assign a score to the input.
693
694 - `input: array of object { content, role, type }`
695
696 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
697
698 - `content: string or ResponseInputText or object { text, type } or 3 more`
699
700 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
701
702 - `Text input: string`
703
704 A text input to the model.
705
706 - `response_input_text: object { text, type }`
707
708 A text input to the model.
709
710 - `text: string`
711
712 The text input to the model.
713
714 - `type: "input_text"`
715
716 The type of the input item. Always `input_text`.
717
718 - `Output text: object { text, type }`
719
720 A text output from the model.
721
722 - `text: string`
723
724 The text output from the model.
725
726 - `type: "output_text"`
727
728 The type of the output text. Always `output_text`.
729
730 - `Input image: object { image_url, type, detail }`
731
732 An image input block used within EvalItem content arrays.
733
734 - `image_url: string`
735
736 The URL of the image input.
737
738 - `type: "input_image"`
739
740 The type of the image input. Always `input_image`.
741
742 - `detail: optional string`
743
744 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
745
746 - `response_input_audio: object { input_audio, type }`
747
748 An audio input to the model.
749
750 - `input_audio: object { data, format }`
751
752 - `data: string`
753
754 Base64-encoded audio data.
755
756 - `format: "mp3" or "wav"`
757
758 The format of the audio data. Currently supported formats are `mp3` and
759 `wav`.
760
761 - `"mp3"`
762
763 - `"wav"`
764
765 - `type: "input_audio"`
766
767 The type of the input item. Always `input_audio`.
768
769 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
770
771 A list of inputs, each of which may be either an input text, output text, input
772 image, or input audio object.
773
774 - `Text input: string`
775
776 A text input to the model.
777
778 - `response_input_text: object { text, type }`
779
780 A text input to the model.
781
782 - `Output text: object { text, type }`
783
784 A text output from the model.
785
786 - `text: string`
787
788 The text output from the model.
789
790 - `type: "output_text"`
791
792 The type of the output text. Always `output_text`.
793
794 - `Input image: object { image_url, type, detail }`
795
796 An image input block used within EvalItem content arrays.
797
798 - `image_url: string`
799
800 The URL of the image input.
801
802 - `type: "input_image"`
803
804 The type of the image input. Always `input_image`.
805
806 - `detail: optional string`
807
808 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
809
810 - `response_input_audio: object { input_audio, type }`
811
812 An audio input to the model.
813
814 - `role: "user" or "assistant" or "system" or "developer"`
815
816 The role of the message input. One of `user`, `assistant`, `system`, or
817 `developer`.
818
819 - `"user"`
820
821 - `"assistant"`
822
823 - `"system"`
824
825 - `"developer"`
826
827 - `type: optional "message"`
828
829 The type of the message input. Always `message`.
830
831 - `"message"`
832
833 - `model: string`
834
835 The model to use for the evaluation.
836
837 - `name: string`
838
839 The name of the grader.
840
841 - `type: "score_model"`
842
843 The object type, which is always `score_model`.
844
845 - `range: optional array of number`
846
847 The range of the score. Defaults to `[0, 1]`.
848
849 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`
850
851 The sampling parameters for the model.
852
853 - `max_completions_tokens: optional number`
854
855 The maximum number of tokens the grader model may generate in its response.
856
857 - `reasoning_effort: optional "none" or "minimal" or "low" or 3 more`
858
859 Constrains effort on reasoning for
860 [reasoning models](https://platform.openai.com/docs/guides/reasoning).
861 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
862 reasoning effort can result in faster responses and fewer tokens used
863 on reasoning in a response.
864
865 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
866 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
867 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
868 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
869
870 - `"none"`
871
872 - `"minimal"`
873
874 - `"low"`
875
876 - `"medium"`
877
878 - `"high"`
879
880 - `"xhigh"`
881
882 - `seed: optional number`
883
884 A seed value to initialize the randomness, during sampling.
885
886 - `temperature: optional number`
887
888 A higher temperature increases randomness in the outputs.
889
890 - `top_p: optional number`
891
892 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
893
894### String Check Grader
895
896- `string_check_grader: object { input, name, operation, 2 more }`
897
898 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
899
900 - `input: string`
901
902 The input text. This may include template strings.
903
904 - `name: string`
905
906 The name of the grader.
907
908 - `operation: "eq" or "ne" or "like" or "ilike"`
909
910 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
911
912 - `"eq"`
913
914 - `"ne"`
915
916 - `"like"`
917
918 - `"ilike"`
919
920 - `reference: string`
921
922 The reference text. This may include template strings.
923
924 - `type: "string_check"`
925
926 The object type, which is always `string_check`.
927
928### Text Similarity Grader
929
930- `text_similarity_grader: object { evaluation_metric, input, name, 2 more }`
931
932 A TextSimilarityGrader object which grades text based on similarity metrics.
933
934 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`
935
936 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,
937 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,
938 or `rouge_l`.
939
940 - `"cosine"`
941
942 - `"fuzzy_match"`
943
944 - `"bleu"`
945
946 - `"gleu"`
947
948 - `"meteor"`
949
950 - `"rouge_1"`
951
952 - `"rouge_2"`
953
954 - `"rouge_3"`
955
956 - `"rouge_4"`
957
958 - `"rouge_5"`
959
960 - `"rouge_l"`
961
962 - `input: string`
963
964 The text being graded.
965
966 - `name: string`
967
968 The name of the grader.
969
970 - `reference: string`
971
972 The text being graded against.
973
974 - `type: "text_similarity"`
975
976 The type of grader.