cli/resources/graders/subresources/grader_models/index.md +0 −974 deleted
File Deleted View Diff
1# Grader Models
2
3## Domain Types
4
5### Grader Inputs
6
7- `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
8
9 A list of inputs, each of which may be either an input text, output text, input
10 image, or input audio object.
11
12 - `Text input: string`
13
14 A text input to the model.
15
16 - `response_input_text: object { text, type }`
17
18 A text input to the model.
19
20 - `text: string`
21
22 The text input to the model.
23
24 - `type: "input_text"`
25
26 The type of the input item. Always `input_text`.
27
28 - `Output text: object { text, type }`
29
30 A text output from the model.
31
32 - `text: string`
33
34 The text output from the model.
35
36 - `type: "output_text"`
37
38 The type of the output text. Always `output_text`.
39
40 - `Input image: object { image_url, type, detail }`
41
42 An image input block used within EvalItem content arrays.
43
44 - `image_url: string`
45
46 The URL of the image input.
47
48 - `type: "input_image"`
49
50 The type of the image input. Always `input_image`.
51
52 - `detail: optional string`
53
54 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
55
56 - `response_input_audio: object { input_audio, type }`
57
58 An audio input to the model.
59
60 - `input_audio: object { data, format }`
61
62 - `data: string`
63
64 Base64-encoded audio data.
65
66 - `format: "mp3" or "wav"`
67
68 The format of the audio data. Currently supported formats are `mp3` and
69 `wav`.
70
71 - `"mp3"`
72
73 - `"wav"`
74
75 - `type: "input_audio"`
76
77 The type of the input item. Always `input_audio`.
78
79### Label Model Grader
80
81- `label_model_grader: object { input, labels, model, 3 more }`
82
83 A LabelModelGrader object which uses a model to assign labels to each item
84 in the evaluation.
85
86 - `input: array of object { content, role, type }`
87
88 - `content: string or ResponseInputText or object { text, type } or 3 more`
89
90 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
91
92 - `Text input: string`
93
94 A text input to the model.
95
96 - `response_input_text: object { text, type }`
97
98 A text input to the model.
99
100 - `text: string`
101
102 The text input to the model.
103
104 - `type: "input_text"`
105
106 The type of the input item. Always `input_text`.
107
108 - `Output text: object { text, type }`
109
110 A text output from the model.
111
112 - `text: string`
113
114 The text output from the model.
115
116 - `type: "output_text"`
117
118 The type of the output text. Always `output_text`.
119
120 - `Input image: object { image_url, type, detail }`
121
122 An image input block used within EvalItem content arrays.
123
124 - `image_url: string`
125
126 The URL of the image input.
127
128 - `type: "input_image"`
129
130 The type of the image input. Always `input_image`.
131
132 - `detail: optional string`
133
134 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
135
136 - `response_input_audio: object { input_audio, type }`
137
138 An audio input to the model.
139
140 - `input_audio: object { data, format }`
141
142 - `data: string`
143
144 Base64-encoded audio data.
145
146 - `format: "mp3" or "wav"`
147
148 The format of the audio data. Currently supported formats are `mp3` and
149 `wav`.
150
151 - `"mp3"`
152
153 - `"wav"`
154
155 - `type: "input_audio"`
156
157 The type of the input item. Always `input_audio`.
158
159 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
160
161 A list of inputs, each of which may be either an input text, output text, input
162 image, or input audio object.
163
164 - `Text input: string`
165
166 A text input to the model.
167
168 - `response_input_text: object { text, type }`
169
170 A text input to the model.
171
172 - `Output text: object { text, type }`
173
174 A text output from the model.
175
176 - `text: string`
177
178 The text output from the model.
179
180 - `type: "output_text"`
181
182 The type of the output text. Always `output_text`.
183
184 - `Input image: object { image_url, type, detail }`
185
186 An image input block used within EvalItem content arrays.
187
188 - `image_url: string`
189
190 The URL of the image input.
191
192 - `type: "input_image"`
193
194 The type of the image input. Always `input_image`.
195
196 - `detail: optional string`
197
198 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
199
200 - `response_input_audio: object { input_audio, type }`
201
202 An audio input to the model.
203
204 - `role: "user" or "assistant" or "system" or "developer"`
205
206 The role of the message input. One of `user`, `assistant`, `system`, or
207 `developer`.
208
209 - `"user"`
210
211 - `"assistant"`
212
213 - `"system"`
214
215 - `"developer"`
216
217 - `type: optional "message"`
218
219 The type of the message input. Always `message`.
220
221 - `"message"`
222
223 - `labels: array of string`
224
225 The labels to assign to each item in the evaluation.
226
227 - `model: string`
228
229 The model to use for the evaluation. Must support structured outputs.
230
231 - `name: string`
232
233 The name of the grader.
234
235 - `passing_labels: array of string`
236
237 The labels that indicate a passing result. Must be a subset of labels.
238
239 - `type: "label_model"`
240
241 The object type, which is always `label_model`.
242
243### Multi Grader
244
245- `multi_grader: object { calculate_output, graders, name, type }`
246
247 A MultiGrader object combines the output of multiple graders to produce a single score.
248
249 - `calculate_output: string`
250
251 A formula to calculate the output based on grader results.
252
253 - `graders: StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more`
254
255 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
256
257 - `string_check_grader: object { input, name, operation, 2 more }`
258
259 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
260
261 - `input: string`
262
263 The input text. This may include template strings.
264
265 - `name: string`
266
267 The name of the grader.
268
269 - `operation: "eq" or "ne" or "like" or "ilike"`
270
271 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
272
273 - `"eq"`
274
275 - `"ne"`
276
277 - `"like"`
278
279 - `"ilike"`
280
281 - `reference: string`
282
283 The reference text. This may include template strings.
284
285 - `type: "string_check"`
286
287 The object type, which is always `string_check`.
288
289 - `text_similarity_grader: object { evaluation_metric, input, name, 2 more }`
290
291 A TextSimilarityGrader object which grades text based on similarity metrics.
292
293 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`
294
295 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,
296 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,
297 or `rouge_l`.
298
299 - `"cosine"`
300
301 - `"fuzzy_match"`
302
303 - `"bleu"`
304
305 - `"gleu"`
306
307 - `"meteor"`
308
309 - `"rouge_1"`
310
311 - `"rouge_2"`
312
313 - `"rouge_3"`
314
315 - `"rouge_4"`
316
317 - `"rouge_5"`
318
319 - `"rouge_l"`
320
321 - `input: string`
322
323 The text being graded.
324
325 - `name: string`
326
327 The name of the grader.
328
329 - `reference: string`
330
331 The text being graded against.
332
333 - `type: "text_similarity"`
334
335 The type of grader.
336
337 - `python_grader: object { name, source, type, image_tag }`
338
339 A PythonGrader object that runs a python script on the input.
340
341 - `name: string`
342
343 The name of the grader.
344
345 - `source: string`
346
347 The source code of the python script.
348
349 - `type: "python"`
350
351 The object type, which is always `python`.
352
353 - `image_tag: optional string`
354
355 The image tag to use for the python script.
356
357 - `score_model_grader: object { input, model, name, 3 more }`
358
359 A ScoreModelGrader object that uses a model to assign a score to the input.
360
361 - `input: array of object { content, role, type }`
362
363 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
364
365 - `content: string or ResponseInputText or object { text, type } or 3 more`
366
367 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
368
369 - `Text input: string`
370
371 A text input to the model.
372
373 - `response_input_text: object { text, type }`
374
375 A text input to the model.
376
377 - `text: string`
378
379 The text input to the model.
380
381 - `type: "input_text"`
382
383 The type of the input item. Always `input_text`.
384
385 - `Output text: object { text, type }`
386
387 A text output from the model.
388
389 - `text: string`
390
391 The text output from the model.
392
393 - `type: "output_text"`
394
395 The type of the output text. Always `output_text`.
396
397 - `Input image: object { image_url, type, detail }`
398
399 An image input block used within EvalItem content arrays.
400
401 - `image_url: string`
402
403 The URL of the image input.
404
405 - `type: "input_image"`
406
407 The type of the image input. Always `input_image`.
408
409 - `detail: optional string`
410
411 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
412
413 - `response_input_audio: object { input_audio, type }`
414
415 An audio input to the model.
416
417 - `input_audio: object { data, format }`
418
419 - `data: string`
420
421 Base64-encoded audio data.
422
423 - `format: "mp3" or "wav"`
424
425 The format of the audio data. Currently supported formats are `mp3` and
426 `wav`.
427
428 - `"mp3"`
429
430 - `"wav"`
431
432 - `type: "input_audio"`
433
434 The type of the input item. Always `input_audio`.
435
436 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
437
438 A list of inputs, each of which may be either an input text, output text, input
439 image, or input audio object.
440
441 - `Text input: string`
442
443 A text input to the model.
444
445 - `response_input_text: object { text, type }`
446
447 A text input to the model.
448
449 - `Output text: object { text, type }`
450
451 A text output from the model.
452
453 - `text: string`
454
455 The text output from the model.
456
457 - `type: "output_text"`
458
459 The type of the output text. Always `output_text`.
460
461 - `Input image: object { image_url, type, detail }`
462
463 An image input block used within EvalItem content arrays.
464
465 - `image_url: string`
466
467 The URL of the image input.
468
469 - `type: "input_image"`
470
471 The type of the image input. Always `input_image`.
472
473 - `detail: optional string`
474
475 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
476
477 - `response_input_audio: object { input_audio, type }`
478
479 An audio input to the model.
480
481 - `role: "user" or "assistant" or "system" or "developer"`
482
483 The role of the message input. One of `user`, `assistant`, `system`, or
484 `developer`.
485
486 - `"user"`
487
488 - `"assistant"`
489
490 - `"system"`
491
492 - `"developer"`
493
494 - `type: optional "message"`
495
496 The type of the message input. Always `message`.
497
498 - `"message"`
499
500 - `model: string`
501
502 The model to use for the evaluation.
503
504 - `name: string`
505
506 The name of the grader.
507
508 - `type: "score_model"`
509
510 The object type, which is always `score_model`.
511
512 - `range: optional array of number`
513
514 The range of the score. Defaults to `[0, 1]`.
515
516 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`
517
518 The sampling parameters for the model.
519
520 - `max_completions_tokens: optional number`
521
522 The maximum number of tokens the grader model may generate in its response.
523
524 - `reasoning_effort: optional "none" or "minimal" or "low" or 3 more`
525
526 Constrains effort on reasoning for
527 [reasoning models](https://platform.openai.com/docs/guides/reasoning).
528 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
529 reasoning effort can result in faster responses and fewer tokens used
530 on reasoning in a response.
531
532 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
533 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
534 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
535 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
536
537 - `"none"`
538
539 - `"minimal"`
540
541 - `"low"`
542
543 - `"medium"`
544
545 - `"high"`
546
547 - `"xhigh"`
548
549 - `seed: optional number`
550
551 A seed value to initialize the randomness, during sampling.
552
553 - `temperature: optional number`
554
555 A higher temperature increases randomness in the outputs.
556
557 - `top_p: optional number`
558
559 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
560
561 - `label_model_grader: object { input, labels, model, 3 more }`
562
563 A LabelModelGrader object which uses a model to assign labels to each item
564 in the evaluation.
565
566 - `input: array of object { content, role, type }`
567
568 - `content: string or ResponseInputText or object { text, type } or 3 more`
569
570 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
571
572 - `Text input: string`
573
574 A text input to the model.
575
576 - `response_input_text: object { text, type }`
577
578 A text input to the model.
579
580 - `Output text: object { text, type }`
581
582 A text output from the model.
583
584 - `text: string`
585
586 The text output from the model.
587
588 - `type: "output_text"`
589
590 The type of the output text. Always `output_text`.
591
592 - `Input image: object { image_url, type, detail }`
593
594 An image input block used within EvalItem content arrays.
595
596 - `image_url: string`
597
598 The URL of the image input.
599
600 - `type: "input_image"`
601
602 The type of the image input. Always `input_image`.
603
604 - `detail: optional string`
605
606 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
607
608 - `response_input_audio: object { input_audio, type }`
609
610 An audio input to the model.
611
612 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
613
614 A list of inputs, each of which may be either an input text, output text, input
615 image, or input audio object.
616
617 - `role: "user" or "assistant" or "system" or "developer"`
618
619 The role of the message input. One of `user`, `assistant`, `system`, or
620 `developer`.
621
622 - `"user"`
623
624 - `"assistant"`
625
626 - `"system"`
627
628 - `"developer"`
629
630 - `type: optional "message"`
631
632 The type of the message input. Always `message`.
633
634 - `"message"`
635
636 - `labels: array of string`
637
638 The labels to assign to each item in the evaluation.
639
640 - `model: string`
641
642 The model to use for the evaluation. Must support structured outputs.
643
644 - `name: string`
645
646 The name of the grader.
647
648 - `passing_labels: array of string`
649
650 The labels that indicate a passing result. Must be a subset of labels.
651
652 - `type: "label_model"`
653
654 The object type, which is always `label_model`.
655
656 - `name: string`
657
658 The name of the grader.
659
660 - `type: "multi"`
661
662 The object type, which is always `multi`.
663
664### Python Grader
665
666- `python_grader: object { name, source, type, image_tag }`
667
668 A PythonGrader object that runs a python script on the input.
669
670 - `name: string`
671
672 The name of the grader.
673
674 - `source: string`
675
676 The source code of the python script.
677
678 - `type: "python"`
679
680 The object type, which is always `python`.
681
682 - `image_tag: optional string`
683
684 The image tag to use for the python script.
685
686### Score Model Grader
687
688- `score_model_grader: object { input, model, name, 3 more }`
689
690 A ScoreModelGrader object that uses a model to assign a score to the input.
691
692 - `input: array of object { content, role, type }`
693
694 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
695
696 - `content: string or ResponseInputText or object { text, type } or 3 more`
697
698 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
699
700 - `Text input: string`
701
702 A text input to the model.
703
704 - `response_input_text: object { text, type }`
705
706 A text input to the model.
707
708 - `text: string`
709
710 The text input to the model.
711
712 - `type: "input_text"`
713
714 The type of the input item. Always `input_text`.
715
716 - `Output text: object { text, type }`
717
718 A text output from the model.
719
720 - `text: string`
721
722 The text output from the model.
723
724 - `type: "output_text"`
725
726 The type of the output text. Always `output_text`.
727
728 - `Input image: object { image_url, type, detail }`
729
730 An image input block used within EvalItem content arrays.
731
732 - `image_url: string`
733
734 The URL of the image input.
735
736 - `type: "input_image"`
737
738 The type of the image input. Always `input_image`.
739
740 - `detail: optional string`
741
742 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
743
744 - `response_input_audio: object { input_audio, type }`
745
746 An audio input to the model.
747
748 - `input_audio: object { data, format }`
749
750 - `data: string`
751
752 Base64-encoded audio data.
753
754 - `format: "mp3" or "wav"`
755
756 The format of the audio data. Currently supported formats are `mp3` and
757 `wav`.
758
759 - `"mp3"`
760
761 - `"wav"`
762
763 - `type: "input_audio"`
764
765 The type of the input item. Always `input_audio`.
766
767 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
768
769 A list of inputs, each of which may be either an input text, output text, input
770 image, or input audio object.
771
772 - `Text input: string`
773
774 A text input to the model.
775
776 - `response_input_text: object { text, type }`
777
778 A text input to the model.
779
780 - `Output text: object { text, type }`
781
782 A text output from the model.
783
784 - `text: string`
785
786 The text output from the model.
787
788 - `type: "output_text"`
789
790 The type of the output text. Always `output_text`.
791
792 - `Input image: object { image_url, type, detail }`
793
794 An image input block used within EvalItem content arrays.
795
796 - `image_url: string`
797
798 The URL of the image input.
799
800 - `type: "input_image"`
801
802 The type of the image input. Always `input_image`.
803
804 - `detail: optional string`
805
806 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
807
808 - `response_input_audio: object { input_audio, type }`
809
810 An audio input to the model.
811
812 - `role: "user" or "assistant" or "system" or "developer"`
813
814 The role of the message input. One of `user`, `assistant`, `system`, or
815 `developer`.
816
817 - `"user"`
818
819 - `"assistant"`
820
821 - `"system"`
822
823 - `"developer"`
824
825 - `type: optional "message"`
826
827 The type of the message input. Always `message`.
828
829 - `"message"`
830
831 - `model: string`
832
833 The model to use for the evaluation.
834
835 - `name: string`
836
837 The name of the grader.
838
839 - `type: "score_model"`
840
841 The object type, which is always `score_model`.
842
843 - `range: optional array of number`
844
845 The range of the score. Defaults to `[0, 1]`.
846
847 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`
848
849 The sampling parameters for the model.
850
851 - `max_completions_tokens: optional number`
852
853 The maximum number of tokens the grader model may generate in its response.
854
855 - `reasoning_effort: optional "none" or "minimal" or "low" or 3 more`
856
857 Constrains effort on reasoning for
858 [reasoning models](https://platform.openai.com/docs/guides/reasoning).
859 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
860 reasoning effort can result in faster responses and fewer tokens used
861 on reasoning in a response.
862
863 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
864 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
865 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
866 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
867
868 - `"none"`
869
870 - `"minimal"`
871
872 - `"low"`
873
874 - `"medium"`
875
876 - `"high"`
877
878 - `"xhigh"`
879
880 - `seed: optional number`
881
882 A seed value to initialize the randomness, during sampling.
883
884 - `temperature: optional number`
885
886 A higher temperature increases randomness in the outputs.
887
888 - `top_p: optional number`
889
890 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
891
892### String Check Grader
893
894- `string_check_grader: object { input, name, operation, 2 more }`
895
896 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
897
898 - `input: string`
899
900 The input text. This may include template strings.
901
902 - `name: string`
903
904 The name of the grader.
905
906 - `operation: "eq" or "ne" or "like" or "ilike"`
907
908 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
909
910 - `"eq"`
911
912 - `"ne"`
913
914 - `"like"`
915
916 - `"ilike"`
917
918 - `reference: string`
919
920 The reference text. This may include template strings.
921
922 - `type: "string_check"`
923
924 The object type, which is always `string_check`.
925
926### Text Similarity Grader
927
928- `text_similarity_grader: object { evaluation_metric, input, name, 2 more }`
929
930 A TextSimilarityGrader object which grades text based on similarity metrics.
931
932 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`
933
934 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,
935 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,
936 or `rouge_l`.
937
938 - `"cosine"`
939
940 - `"fuzzy_match"`
941
942 - `"bleu"`
943
944 - `"gleu"`
945
946 - `"meteor"`
947
948 - `"rouge_1"`
949
950 - `"rouge_2"`
951
952 - `"rouge_3"`
953
954 - `"rouge_4"`
955
956 - `"rouge_5"`
957
958 - `"rouge_l"`
959
960 - `input: string`
961
962 The text being graded.
963
964 - `name: string`
965
966 The name of the grader.
967
968 - `reference: string`
969
970 The text being graded against.
971
972 - `type: "text_similarity"`
973
974 The type of grader.