resources/graders/subresources/grader_models/index.md +0 −1044 deleted
File Deleted View Diff
1# Grader Models
2
3## Domain Types
4
5### Grader Inputs
6
7- `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
8
9 A list of inputs, each of which may be either an input text, output text, input
10 image, or input audio object.
11
12 - `TextInput = string`
13
14 A text input to the model.
15
16 - `ResponseInputText object { text, type }`
17
18 A text input to the model.
19
20 - `text: string`
21
22 The text input to the model.
23
24 - `type: "input_text"`
25
26 The type of the input item. Always `input_text`.
27
28 - `"input_text"`
29
30 - `OutputText object { text, type }`
31
32 A text output from the model.
33
34 - `text: string`
35
36 The text output from the model.
37
38 - `type: "output_text"`
39
40 The type of the output text. Always `output_text`.
41
42 - `"output_text"`
43
44 - `InputImage object { image_url, type, detail }`
45
46 An image input block used within EvalItem content arrays.
47
48 - `image_url: string`
49
50 The URL of the image input.
51
52 - `type: "input_image"`
53
54 The type of the image input. Always `input_image`.
55
56 - `"input_image"`
57
58 - `detail: optional string`
59
60 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
61
62 - `ResponseInputAudio object { input_audio, type }`
63
64 An audio input to the model.
65
66 - `input_audio: object { data, format }`
67
68 - `data: string`
69
70 Base64-encoded audio data.
71
72 - `format: "mp3" or "wav"`
73
74 The format of the audio data. Currently supported formats are `mp3` and
75 `wav`.
76
77 - `"mp3"`
78
79 - `"wav"`
80
81 - `type: "input_audio"`
82
83 The type of the input item. Always `input_audio`.
84
85 - `"input_audio"`
86
87### Label Model Grader
88
89- `LabelModelGrader object { input, labels, model, 3 more }`
90
91 A LabelModelGrader object which uses a model to assign labels to each item
92 in the evaluation.
93
94 - `input: array of object { content, role, type }`
95
96 - `content: string or ResponseInputText or object { text, type } or 3 more`
97
98 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
99
100 - `TextInput = string`
101
102 A text input to the model.
103
104 - `ResponseInputText object { text, type }`
105
106 A text input to the model.
107
108 - `text: string`
109
110 The text input to the model.
111
112 - `type: "input_text"`
113
114 The type of the input item. Always `input_text`.
115
116 - `"input_text"`
117
118 - `OutputText object { text, type }`
119
120 A text output from the model.
121
122 - `text: string`
123
124 The text output from the model.
125
126 - `type: "output_text"`
127
128 The type of the output text. Always `output_text`.
129
130 - `"output_text"`
131
132 - `InputImage object { image_url, type, detail }`
133
134 An image input block used within EvalItem content arrays.
135
136 - `image_url: string`
137
138 The URL of the image input.
139
140 - `type: "input_image"`
141
142 The type of the image input. Always `input_image`.
143
144 - `"input_image"`
145
146 - `detail: optional string`
147
148 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
149
150 - `ResponseInputAudio object { input_audio, type }`
151
152 An audio input to the model.
153
154 - `input_audio: object { data, format }`
155
156 - `data: string`
157
158 Base64-encoded audio data.
159
160 - `format: "mp3" or "wav"`
161
162 The format of the audio data. Currently supported formats are `mp3` and
163 `wav`.
164
165 - `"mp3"`
166
167 - `"wav"`
168
169 - `type: "input_audio"`
170
171 The type of the input item. Always `input_audio`.
172
173 - `"input_audio"`
174
175 - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
176
177 A list of inputs, each of which may be either an input text, output text, input
178 image, or input audio object.
179
180 - `TextInput = string`
181
182 A text input to the model.
183
184 - `ResponseInputText object { text, type }`
185
186 A text input to the model.
187
188 - `OutputText object { text, type }`
189
190 A text output from the model.
191
192 - `text: string`
193
194 The text output from the model.
195
196 - `type: "output_text"`
197
198 The type of the output text. Always `output_text`.
199
200 - `"output_text"`
201
202 - `InputImage object { image_url, type, detail }`
203
204 An image input block used within EvalItem content arrays.
205
206 - `image_url: string`
207
208 The URL of the image input.
209
210 - `type: "input_image"`
211
212 The type of the image input. Always `input_image`.
213
214 - `"input_image"`
215
216 - `detail: optional string`
217
218 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
219
220 - `ResponseInputAudio object { input_audio, type }`
221
222 An audio input to the model.
223
224 - `role: "user" or "assistant" or "system" or "developer"`
225
226 The role of the message input. One of `user`, `assistant`, `system`, or
227 `developer`.
228
229 - `"user"`
230
231 - `"assistant"`
232
233 - `"system"`
234
235 - `"developer"`
236
237 - `type: optional "message"`
238
239 The type of the message input. Always `message`.
240
241 - `"message"`
242
243 - `labels: array of string`
244
245 The labels to assign to each item in the evaluation.
246
247 - `model: string`
248
249 The model to use for the evaluation. Must support structured outputs.
250
251 - `name: string`
252
253 The name of the grader.
254
255 - `passing_labels: array of string`
256
257 The labels that indicate a passing result. Must be a subset of labels.
258
259 - `type: "label_model"`
260
261 The object type, which is always `label_model`.
262
263 - `"label_model"`
264
265### Multi Grader
266
267- `MultiGrader object { calculate_output, graders, name, type }`
268
269 A MultiGrader object combines the output of multiple graders to produce a single score.
270
271 - `calculate_output: string`
272
273 A formula to calculate the output based on grader results.
274
275 - `graders: StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more`
276
277 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
278
279 - `StringCheckGrader object { input, name, operation, 2 more }`
280
281 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
282
283 - `input: string`
284
285 The input text. This may include template strings.
286
287 - `name: string`
288
289 The name of the grader.
290
291 - `operation: "eq" or "ne" or "like" or "ilike"`
292
293 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
294
295 - `"eq"`
296
297 - `"ne"`
298
299 - `"like"`
300
301 - `"ilike"`
302
303 - `reference: string`
304
305 The reference text. This may include template strings.
306
307 - `type: "string_check"`
308
309 The object type, which is always `string_check`.
310
311 - `"string_check"`
312
313 - `TextSimilarityGrader object { evaluation_metric, input, name, 2 more }`
314
315 A TextSimilarityGrader object which grades text based on similarity metrics.
316
317 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`
318
319 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,
320 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,
321 or `rouge_l`.
322
323 - `"cosine"`
324
325 - `"fuzzy_match"`
326
327 - `"bleu"`
328
329 - `"gleu"`
330
331 - `"meteor"`
332
333 - `"rouge_1"`
334
335 - `"rouge_2"`
336
337 - `"rouge_3"`
338
339 - `"rouge_4"`
340
341 - `"rouge_5"`
342
343 - `"rouge_l"`
344
345 - `input: string`
346
347 The text being graded.
348
349 - `name: string`
350
351 The name of the grader.
352
353 - `reference: string`
354
355 The text being graded against.
356
357 - `type: "text_similarity"`
358
359 The type of grader.
360
361 - `"text_similarity"`
362
363 - `PythonGrader object { name, source, type, image_tag }`
364
365 A PythonGrader object that runs a python script on the input.
366
367 - `name: string`
368
369 The name of the grader.
370
371 - `source: string`
372
373 The source code of the python script.
374
375 - `type: "python"`
376
377 The object type, which is always `python`.
378
379 - `"python"`
380
381 - `image_tag: optional string`
382
383 The image tag to use for the python script.
384
385 - `ScoreModelGrader object { input, model, name, 3 more }`
386
387 A ScoreModelGrader object that uses a model to assign a score to the input.
388
389 - `input: array of object { content, role, type }`
390
391 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
392
393 - `content: string or ResponseInputText or object { text, type } or 3 more`
394
395 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
396
397 - `TextInput = string`
398
399 A text input to the model.
400
401 - `ResponseInputText object { text, type }`
402
403 A text input to the model.
404
405 - `text: string`
406
407 The text input to the model.
408
409 - `type: "input_text"`
410
411 The type of the input item. Always `input_text`.
412
413 - `"input_text"`
414
415 - `OutputText object { text, type }`
416
417 A text output from the model.
418
419 - `text: string`
420
421 The text output from the model.
422
423 - `type: "output_text"`
424
425 The type of the output text. Always `output_text`.
426
427 - `"output_text"`
428
429 - `InputImage object { image_url, type, detail }`
430
431 An image input block used within EvalItem content arrays.
432
433 - `image_url: string`
434
435 The URL of the image input.
436
437 - `type: "input_image"`
438
439 The type of the image input. Always `input_image`.
440
441 - `"input_image"`
442
443 - `detail: optional string`
444
445 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
446
447 - `ResponseInputAudio object { input_audio, type }`
448
449 An audio input to the model.
450
451 - `input_audio: object { data, format }`
452
453 - `data: string`
454
455 Base64-encoded audio data.
456
457 - `format: "mp3" or "wav"`
458
459 The format of the audio data. Currently supported formats are `mp3` and
460 `wav`.
461
462 - `"mp3"`
463
464 - `"wav"`
465
466 - `type: "input_audio"`
467
468 The type of the input item. Always `input_audio`.
469
470 - `"input_audio"`
471
472 - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
473
474 A list of inputs, each of which may be either an input text, output text, input
475 image, or input audio object.
476
477 - `TextInput = string`
478
479 A text input to the model.
480
481 - `ResponseInputText object { text, type }`
482
483 A text input to the model.
484
485 - `OutputText object { text, type }`
486
487 A text output from the model.
488
489 - `text: string`
490
491 The text output from the model.
492
493 - `type: "output_text"`
494
495 The type of the output text. Always `output_text`.
496
497 - `"output_text"`
498
499 - `InputImage object { image_url, type, detail }`
500
501 An image input block used within EvalItem content arrays.
502
503 - `image_url: string`
504
505 The URL of the image input.
506
507 - `type: "input_image"`
508
509 The type of the image input. Always `input_image`.
510
511 - `"input_image"`
512
513 - `detail: optional string`
514
515 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
516
517 - `ResponseInputAudio object { input_audio, type }`
518
519 An audio input to the model.
520
521 - `role: "user" or "assistant" or "system" or "developer"`
522
523 The role of the message input. One of `user`, `assistant`, `system`, or
524 `developer`.
525
526 - `"user"`
527
528 - `"assistant"`
529
530 - `"system"`
531
532 - `"developer"`
533
534 - `type: optional "message"`
535
536 The type of the message input. Always `message`.
537
538 - `"message"`
539
540 - `model: string`
541
542 The model to use for the evaluation.
543
544 - `name: string`
545
546 The name of the grader.
547
548 - `type: "score_model"`
549
550 The object type, which is always `score_model`.
551
552 - `"score_model"`
553
554 - `range: optional array of number`
555
556 The range of the score. Defaults to `[0, 1]`.
557
558 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`
559
560 The sampling parameters for the model.
561
562 - `max_completions_tokens: optional number`
563
564 The maximum number of tokens the grader model may generate in its response.
565
566 - `reasoning_effort: optional ReasoningEffort`
567
568 Constrains effort on reasoning for
569 [reasoning models](https://platform.openai.com/docs/guides/reasoning).
570 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
571 reasoning effort can result in faster responses and fewer tokens used
572 on reasoning in a response.
573
574 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
575 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
576 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
577 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
578
579 - `"none"`
580
581 - `"minimal"`
582
583 - `"low"`
584
585 - `"medium"`
586
587 - `"high"`
588
589 - `"xhigh"`
590
591 - `seed: optional number`
592
593 A seed value to initialize the randomness, during sampling.
594
595 - `temperature: optional number`
596
597 A higher temperature increases randomness in the outputs.
598
599 - `top_p: optional number`
600
601 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
602
603 - `LabelModelGrader object { input, labels, model, 3 more }`
604
605 A LabelModelGrader object which uses a model to assign labels to each item
606 in the evaluation.
607
608 - `input: array of object { content, role, type }`
609
610 - `content: string or ResponseInputText or object { text, type } or 3 more`
611
612 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
613
614 - `TextInput = string`
615
616 A text input to the model.
617
618 - `ResponseInputText object { text, type }`
619
620 A text input to the model.
621
622 - `OutputText object { text, type }`
623
624 A text output from the model.
625
626 - `text: string`
627
628 The text output from the model.
629
630 - `type: "output_text"`
631
632 The type of the output text. Always `output_text`.
633
634 - `"output_text"`
635
636 - `InputImage object { image_url, type, detail }`
637
638 An image input block used within EvalItem content arrays.
639
640 - `image_url: string`
641
642 The URL of the image input.
643
644 - `type: "input_image"`
645
646 The type of the image input. Always `input_image`.
647
648 - `"input_image"`
649
650 - `detail: optional string`
651
652 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
653
654 - `ResponseInputAudio object { input_audio, type }`
655
656 An audio input to the model.
657
658 - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
659
660 A list of inputs, each of which may be either an input text, output text, input
661 image, or input audio object.
662
663 - `role: "user" or "assistant" or "system" or "developer"`
664
665 The role of the message input. One of `user`, `assistant`, `system`, or
666 `developer`.
667
668 - `"user"`
669
670 - `"assistant"`
671
672 - `"system"`
673
674 - `"developer"`
675
676 - `type: optional "message"`
677
678 The type of the message input. Always `message`.
679
680 - `"message"`
681
682 - `labels: array of string`
683
684 The labels to assign to each item in the evaluation.
685
686 - `model: string`
687
688 The model to use for the evaluation. Must support structured outputs.
689
690 - `name: string`
691
692 The name of the grader.
693
694 - `passing_labels: array of string`
695
696 The labels that indicate a passing result. Must be a subset of labels.
697
698 - `type: "label_model"`
699
700 The object type, which is always `label_model`.
701
702 - `"label_model"`
703
704 - `name: string`
705
706 The name of the grader.
707
708 - `type: "multi"`
709
710 The object type, which is always `multi`.
711
712 - `"multi"`
713
714### Python Grader
715
716- `PythonGrader object { name, source, type, image_tag }`
717
718 A PythonGrader object that runs a python script on the input.
719
720 - `name: string`
721
722 The name of the grader.
723
724 - `source: string`
725
726 The source code of the python script.
727
728 - `type: "python"`
729
730 The object type, which is always `python`.
731
732 - `"python"`
733
734 - `image_tag: optional string`
735
736 The image tag to use for the python script.
737
738### Score Model Grader
739
740- `ScoreModelGrader object { input, model, name, 3 more }`
741
742 A ScoreModelGrader object that uses a model to assign a score to the input.
743
744 - `input: array of object { content, role, type }`
745
746 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
747
748 - `content: string or ResponseInputText or object { text, type } or 3 more`
749
750 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
751
752 - `TextInput = string`
753
754 A text input to the model.
755
756 - `ResponseInputText object { text, type }`
757
758 A text input to the model.
759
760 - `text: string`
761
762 The text input to the model.
763
764 - `type: "input_text"`
765
766 The type of the input item. Always `input_text`.
767
768 - `"input_text"`
769
770 - `OutputText object { text, type }`
771
772 A text output from the model.
773
774 - `text: string`
775
776 The text output from the model.
777
778 - `type: "output_text"`
779
780 The type of the output text. Always `output_text`.
781
782 - `"output_text"`
783
784 - `InputImage object { image_url, type, detail }`
785
786 An image input block used within EvalItem content arrays.
787
788 - `image_url: string`
789
790 The URL of the image input.
791
792 - `type: "input_image"`
793
794 The type of the image input. Always `input_image`.
795
796 - `"input_image"`
797
798 - `detail: optional string`
799
800 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
801
802 - `ResponseInputAudio object { input_audio, type }`
803
804 An audio input to the model.
805
806 - `input_audio: object { data, format }`
807
808 - `data: string`
809
810 Base64-encoded audio data.
811
812 - `format: "mp3" or "wav"`
813
814 The format of the audio data. Currently supported formats are `mp3` and
815 `wav`.
816
817 - `"mp3"`
818
819 - `"wav"`
820
821 - `type: "input_audio"`
822
823 The type of the input item. Always `input_audio`.
824
825 - `"input_audio"`
826
827 - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`
828
829 A list of inputs, each of which may be either an input text, output text, input
830 image, or input audio object.
831
832 - `TextInput = string`
833
834 A text input to the model.
835
836 - `ResponseInputText object { text, type }`
837
838 A text input to the model.
839
840 - `OutputText object { text, type }`
841
842 A text output from the model.
843
844 - `text: string`
845
846 The text output from the model.
847
848 - `type: "output_text"`
849
850 The type of the output text. Always `output_text`.
851
852 - `"output_text"`
853
854 - `InputImage object { image_url, type, detail }`
855
856 An image input block used within EvalItem content arrays.
857
858 - `image_url: string`
859
860 The URL of the image input.
861
862 - `type: "input_image"`
863
864 The type of the image input. Always `input_image`.
865
866 - `"input_image"`
867
868 - `detail: optional string`
869
870 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
871
872 - `ResponseInputAudio object { input_audio, type }`
873
874 An audio input to the model.
875
876 - `role: "user" or "assistant" or "system" or "developer"`
877
878 The role of the message input. One of `user`, `assistant`, `system`, or
879 `developer`.
880
881 - `"user"`
882
883 - `"assistant"`
884
885 - `"system"`
886
887 - `"developer"`
888
889 - `type: optional "message"`
890
891 The type of the message input. Always `message`.
892
893 - `"message"`
894
895 - `model: string`
896
897 The model to use for the evaluation.
898
899 - `name: string`
900
901 The name of the grader.
902
903 - `type: "score_model"`
904
905 The object type, which is always `score_model`.
906
907 - `"score_model"`
908
909 - `range: optional array of number`
910
911 The range of the score. Defaults to `[0, 1]`.
912
913 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`
914
915 The sampling parameters for the model.
916
917 - `max_completions_tokens: optional number`
918
919 The maximum number of tokens the grader model may generate in its response.
920
921 - `reasoning_effort: optional ReasoningEffort`
922
923 Constrains effort on reasoning for
924 [reasoning models](https://platform.openai.com/docs/guides/reasoning).
925 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing
926 reasoning effort can result in faster responses and fewer tokens used
927 on reasoning in a response.
928
929 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.
930 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.
931 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.
932 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.
933
934 - `"none"`
935
936 - `"minimal"`
937
938 - `"low"`
939
940 - `"medium"`
941
942 - `"high"`
943
944 - `"xhigh"`
945
946 - `seed: optional number`
947
948 A seed value to initialize the randomness, during sampling.
949
950 - `temperature: optional number`
951
952 A higher temperature increases randomness in the outputs.
953
954 - `top_p: optional number`
955
956 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
957
958### String Check Grader
959
960- `StringCheckGrader object { input, name, operation, 2 more }`
961
962 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
963
964 - `input: string`
965
966 The input text. This may include template strings.
967
968 - `name: string`
969
970 The name of the grader.
971
972 - `operation: "eq" or "ne" or "like" or "ilike"`
973
974 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
975
976 - `"eq"`
977
978 - `"ne"`
979
980 - `"like"`
981
982 - `"ilike"`
983
984 - `reference: string`
985
986 The reference text. This may include template strings.
987
988 - `type: "string_check"`
989
990 The object type, which is always `string_check`.
991
992 - `"string_check"`
993
994### Text Similarity Grader
995
996- `TextSimilarityGrader object { evaluation_metric, input, name, 2 more }`
997
998 A TextSimilarityGrader object which grades text based on similarity metrics.
999
1000 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`
1001
1002 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,
1003 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,
1004 or `rouge_l`.
1005
1006 - `"cosine"`
1007
1008 - `"fuzzy_match"`
1009
1010 - `"bleu"`
1011
1012 - `"gleu"`
1013
1014 - `"meteor"`
1015
1016 - `"rouge_1"`
1017
1018 - `"rouge_2"`
1019
1020 - `"rouge_3"`
1021
1022 - `"rouge_4"`
1023
1024 - `"rouge_5"`
1025
1026 - `"rouge_l"`
1027
1028 - `input: string`
1029
1030 The text being graded.
1031
1032 - `name: string`
1033
1034 The name of the grader.
1035
1036 - `reference: string`
1037
1038 The text being graded against.
1039
1040 - `type: "text_similarity"`
1041
1042 The type of grader.
1043
1044 - `"text_similarity"`