cli/resources/graders/subresources/grader_models/index.md +0 −1004 deleted
File Deleted View Diff
1# Grader Models
2
3## Domain Types
4
5### Grader Inputs
6
7- `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
8
9 A list of inputs, each of which may be either an input text, output text, input
10 image, or input audio object.
11
12 - `Text input: string`
13
14 A text input to the model.
15
16 - `response_input_text: object { text, type, prompt_cache_breakpoint }`
17
18 A text input to the model.
19
20 - `text: string`
21
22 The text input to the model.
23
24 - `type: "input_text"`
25
26 The type of the input item. Always `input_text`.
27
28 - `prompt_cache_breakpoint: optional object { mode }`
29
30 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
31
32 - `mode: "explicit"`
33
34 The breakpoint mode. Always `explicit`.
35
36 - `Output text: object { text, type }`
37
38 A text output from the model.
39
40 - `text: string`
41
42 The text output from the model.
43
44 - `type: "output_text"`
45
46 The type of the output text. Always `output_text`.
47
48 - `Input image: object { image_url, type, detail }`
49
50 An image input block used within EvalItem content arrays.
51
52 - `image_url: string`
53
54 The URL of the image input.
55
56 - `type: "input_image"`
57
58 The type of the image input. Always `input_image`.
59
60 - `detail: optional string`
61
62 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
63
64 - `response_input_audio: object { input_audio, type }`
65
66 An audio input to the model.
67
68 - `input_audio: object { data, format }`
69
70 - `data: string`
71
72 Base64-encoded audio data.
73
74 - `format: "mp3" or "wav"`
75
76 The format of the audio data. Currently supported formats are `mp3` and
77 `wav`.
78
79 - `"mp3"`
80
81 - `"wav"`
82
83 - `type: "input_audio"`
84
85 The type of the input item. Always `input_audio`.
86
87### Label Model Grader
88
89- `label_model_grader: object { input, labels, model, 3 more }`
90
91 A LabelModelGrader object which uses a model to assign labels to each item
92 in the evaluation.
93
94 - `input: array of object { content, role, type }`
95
96 - `content: string or ResponseInputText or object { text, type } or 3 more`
97
98 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
99
100 - `Text input: string`
101
102 A text input to the model.
103
104 - `response_input_text: object { text, type, prompt_cache_breakpoint }`
105
106 A text input to the model.
107
108 - `text: string`
109
110 The text input to the model.
111
112 - `type: "input_text"`
113
114 The type of the input item. Always `input_text`.
115
116 - `prompt_cache_breakpoint: optional object { mode }`
117
118 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
119
120 - `mode: "explicit"`
121
122 The breakpoint mode. Always `explicit`.
123
124 - `Output text: object { text, type }`
125
126 A text output from the model.
127
128 - `text: string`
129
130 The text output from the model.
131
132 - `type: "output_text"`
133
134 The type of the output text. Always `output_text`.
135
136 - `Input image: object { image_url, type, detail }`
137
138 An image input block used within EvalItem content arrays.
139
140 - `image_url: string`
141
142 The URL of the image input.
143
144 - `type: "input_image"`
145
146 The type of the image input. Always `input_image`.
147
148 - `detail: optional string`
149
150 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
151
152 - `response_input_audio: object { input_audio, type }`
153
154 An audio input to the model.
155
156 - `input_audio: object { data, format }`
157
158 - `data: string`
159
160 Base64-encoded audio data.
161
162 - `format: "mp3" or "wav"`
163
164 The format of the audio data. Currently supported formats are `mp3` and
165 `wav`.
166
167 - `"mp3"`
168
169 - `"wav"`
170
171 - `type: "input_audio"`
172
173 The type of the input item. Always `input_audio`.
174
175 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
176
177 A list of inputs, each of which may be either an input text, output text, input
178 image, or input audio object.
179
180 - `Text input: string`
181
182 A text input to the model.
183
184 - `response_input_text: object { text, type, prompt_cache_breakpoint }`
185
186 A text input to the model.
187
188 - `Output text: object { text, type }`
189
190 A text output from the model.
191
192 - `text: string`
193
194 The text output from the model.
195
196 - `type: "output_text"`
197
198 The type of the output text. Always `output_text`.
199
200 - `Input image: object { image_url, type, detail }`
201
202 An image input block used within EvalItem content arrays.
203
204 - `image_url: string`
205
206 The URL of the image input.
207
208 - `type: "input_image"`
209
210 The type of the image input. Always `input_image`.
211
212 - `detail: optional string`
213
214 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
215
216 - `response_input_audio: object { input_audio, type }`
217
218 An audio input to the model.
219
220 - `role: "user" or "assistant" or "system" or "developer"`
221
222 The role of the message input. One of `user`, `assistant`, `system`, or
223 `developer`.
224
225 - `"user"`
226
227 - `"assistant"`
228
229 - `"system"`
230
231 - `"developer"`
232
233 - `type: optional "message"`
234
235 The type of the message input. Always `message`.
236
237 - `"message"`
238
239 - `labels: array of string`
240
241 The labels to assign to each item in the evaluation.
242
243 - `model: string`
244
245 The model to use for the evaluation. Must support structured outputs.
246
247 - `name: string`
248
249 The name of the grader.
250
251 - `passing_labels: array of string`
252
253 The labels that indicate a passing result. Must be a subset of labels.
254
255 - `type: "label_model"`
256
257 The object type, which is always `label_model`.
258
259### Multi Grader
260
261- `multi_grader: object { calculate_output, graders, name, type }`
262
263 A MultiGrader object combines the output of multiple graders to produce a single score.
264
265 - `calculate_output: string`
266
267 A formula to calculate the output based on grader results.
268
269 - `graders: StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more`
270
271 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
272
273 - `string_check_grader: object { input, name, operation, 2 more }`
274
275 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
276
277 - `input: string`
278
279 The input text. This may include template strings.
280
281 - `name: string`
282
283 The name of the grader.
284
285 - `operation: "eq" or "ne" or "like" or "ilike"`
286
287 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
288
289 - `"eq"`
290
291 - `"ne"`
292
293 - `"like"`
294
295 - `"ilike"`
296
297 - `reference: string`
298
299 The reference text. This may include template strings.
300
301 - `type: "string_check"`
302
303 The object type, which is always `string_check`.
304
305 - `text_similarity_grader: object { evaluation_metric, input, name, 2 more }`
306
307 A TextSimilarityGrader object which grades text based on similarity metrics.
308
309 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`
310
311 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,
312 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,
313 or `rouge_l`.
314
315 - `"cosine"`
316
317 - `"fuzzy_match"`
318
319 - `"bleu"`
320
321 - `"gleu"`
322
323 - `"meteor"`
324
325 - `"rouge_1"`
326
327 - `"rouge_2"`
328
329 - `"rouge_3"`
330
331 - `"rouge_4"`
332
333 - `"rouge_5"`
334
335 - `"rouge_l"`
336
337 - `input: string`
338
339 The text being graded.
340
341 - `name: string`
342
343 The name of the grader.
344
345 - `reference: string`
346
347 The text being graded against.
348
349 - `type: "text_similarity"`
350
351 The type of grader.
352
353 - `python_grader: object { name, source, type, image_tag }`
354
355 A PythonGrader object that runs a python script on the input.
356
357 - `name: string`
358
359 The name of the grader.
360
361 - `source: string`
362
363 The source code of the python script.
364
365 - `type: "python"`
366
367 The object type, which is always `python`.
368
369 - `image_tag: optional string`
370
371 The image tag to use for the python script.
372
373 - `score_model_grader: object { input, model, name, 3 more }`
374
375 A ScoreModelGrader object that uses a model to assign a score to the input.
376
377 - `input: array of object { content, role, type }`
378
379 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
380
381 - `content: string or ResponseInputText or object { text, type } or 3 more`
382
383 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
384
385 - `Text input: string`
386
387 A text input to the model.
388
389 - `response_input_text: object { text, type, prompt_cache_breakpoint }`
390
391 A text input to the model.
392
393 - `text: string`
394
395 The text input to the model.
396
397 - `type: "input_text"`
398
399 The type of the input item. Always `input_text`.
400
401 - `prompt_cache_breakpoint: optional object { mode }`
402
403 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
404
405 - `mode: "explicit"`
406
407 The breakpoint mode. Always `explicit`.
408
409 - `Output text: object { text, type }`
410
411 A text output from the model.
412
413 - `text: string`
414
415 The text output from the model.
416
417 - `type: "output_text"`
418
419 The type of the output text. Always `output_text`.
420
421 - `Input image: object { image_url, type, detail }`
422
423 An image input block used within EvalItem content arrays.
424
425 - `image_url: string`
426
427 The URL of the image input.
428
429 - `type: "input_image"`
430
431 The type of the image input. Always `input_image`.
432
433 - `detail: optional string`
434
435 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
436
437 - `response_input_audio: object { input_audio, type }`
438
439 An audio input to the model.
440
441 - `input_audio: object { data, format }`
442
443 - `data: string`
444
445 Base64-encoded audio data.
446
447 - `format: "mp3" or "wav"`
448
449 The format of the audio data. Currently supported formats are `mp3` and
450 `wav`.
451
452 - `"mp3"`
453
454 - `"wav"`
455
456 - `type: "input_audio"`
457
458 The type of the input item. Always `input_audio`.
459
460 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
461
462 A list of inputs, each of which may be either an input text, output text, input
463 image, or input audio object.
464
465 - `Text input: string`
466
467 A text input to the model.
468
469 - `response_input_text: object { text, type, prompt_cache_breakpoint }`
470
471 A text input to the model.
472
473 - `Output text: object { text, type }`
474
475 A text output from the model.
476
477 - `text: string`
478
479 The text output from the model.
480
481 - `type: "output_text"`
482
483 The type of the output text. Always `output_text`.
484
485 - `Input image: object { image_url, type, detail }`
486
487 An image input block used within EvalItem content arrays.
488
489 - `image_url: string`
490
491 The URL of the image input.
492
493 - `type: "input_image"`
494
495 The type of the image input. Always `input_image`.
496
497 - `detail: optional string`
498
499 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
500
501 - `response_input_audio: object { input_audio, type }`
502
503 An audio input to the model.
504
505 - `role: "user" or "assistant" or "system" or "developer"`
506
507 The role of the message input. One of `user`, `assistant`, `system`, or
508 `developer`.
509
510 - `"user"`
511
512 - `"assistant"`
513
514 - `"system"`
515
516 - `"developer"`
517
518 - `type: optional "message"`
519
520 The type of the message input. Always `message`.
521
522 - `"message"`
523
524 - `model: string`
525
526 The model to use for the evaluation.
527
528 - `name: string`
529
530 The name of the grader.
531
532 - `type: "score_model"`
533
534 The object type, which is always `score_model`.
535
536 - `range: optional array of number`
537
538 The range of the score. Defaults to `[0, 1]`.
539
540 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`
541
542 The sampling parameters for the model.
543
544 - `max_completions_tokens: optional number`
545
546 The maximum number of tokens the grader model may generate in its response.
547
548 - `reasoning_effort: optional "none" or "minimal" or "low" or 4 more`
549
550 Constrains effort on reasoning for reasoning models. Currently supported
551 values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`.
552 Reducing reasoning effort can result in faster responses and fewer tokens
553 used on reasoning in a response. Not all reasoning models support every
554 value. See the
555 [reasoning guide](https://platform.openai.com/docs/guides/reasoning)
556 for model-specific support.
557
558 - `"none"`
559
560 - `"minimal"`
561
562 - `"low"`
563
564 - `"medium"`
565
566 - `"high"`
567
568 - `"xhigh"`
569
570 - `"max"`
571
572 - `seed: optional number`
573
574 A seed value to initialize the randomness, during sampling.
575
576 - `temperature: optional number`
577
578 A higher temperature increases randomness in the outputs.
579
580 - `top_p: optional number`
581
582 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
583
584 - `label_model_grader: object { input, labels, model, 3 more }`
585
586 A LabelModelGrader object which uses a model to assign labels to each item
587 in the evaluation.
588
589 - `input: array of object { content, role, type }`
590
591 - `content: string or ResponseInputText or object { text, type } or 3 more`
592
593 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
594
595 - `Text input: string`
596
597 A text input to the model.
598
599 - `response_input_text: object { text, type, prompt_cache_breakpoint }`
600
601 A text input to the model.
602
603 - `Output text: object { text, type }`
604
605 A text output from the model.
606
607 - `text: string`
608
609 The text output from the model.
610
611 - `type: "output_text"`
612
613 The type of the output text. Always `output_text`.
614
615 - `Input image: object { image_url, type, detail }`
616
617 An image input block used within EvalItem content arrays.
618
619 - `image_url: string`
620
621 The URL of the image input.
622
623 - `type: "input_image"`
624
625 The type of the image input. Always `input_image`.
626
627 - `detail: optional string`
628
629 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
630
631 - `response_input_audio: object { input_audio, type }`
632
633 An audio input to the model.
634
635 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
636
637 A list of inputs, each of which may be either an input text, output text, input
638 image, or input audio object.
639
640 - `role: "user" or "assistant" or "system" or "developer"`
641
642 The role of the message input. One of `user`, `assistant`, `system`, or
643 `developer`.
644
645 - `"user"`
646
647 - `"assistant"`
648
649 - `"system"`
650
651 - `"developer"`
652
653 - `type: optional "message"`
654
655 The type of the message input. Always `message`.
656
657 - `"message"`
658
659 - `labels: array of string`
660
661 The labels to assign to each item in the evaluation.
662
663 - `model: string`
664
665 The model to use for the evaluation. Must support structured outputs.
666
667 - `name: string`
668
669 The name of the grader.
670
671 - `passing_labels: array of string`
672
673 The labels that indicate a passing result. Must be a subset of labels.
674
675 - `type: "label_model"`
676
677 The object type, which is always `label_model`.
678
679 - `name: string`
680
681 The name of the grader.
682
683 - `type: "multi"`
684
685 The object type, which is always `multi`.
686
687### Python Grader
688
689- `python_grader: object { name, source, type, image_tag }`
690
691 A PythonGrader object that runs a python script on the input.
692
693 - `name: string`
694
695 The name of the grader.
696
697 - `source: string`
698
699 The source code of the python script.
700
701 - `type: "python"`
702
703 The object type, which is always `python`.
704
705 - `image_tag: optional string`
706
707 The image tag to use for the python script.
708
709### Score Model Grader
710
711- `score_model_grader: object { input, model, name, 3 more }`
712
713 A ScoreModelGrader object that uses a model to assign a score to the input.
714
715 - `input: array of object { content, role, type }`
716
717 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
718
719 - `content: string or ResponseInputText or object { text, type } or 3 more`
720
721 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
722
723 - `Text input: string`
724
725 A text input to the model.
726
727 - `response_input_text: object { text, type, prompt_cache_breakpoint }`
728
729 A text input to the model.
730
731 - `text: string`
732
733 The text input to the model.
734
735 - `type: "input_text"`
736
737 The type of the input item. Always `input_text`.
738
739 - `prompt_cache_breakpoint: optional object { mode }`
740
741 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
742
743 - `mode: "explicit"`
744
745 The breakpoint mode. Always `explicit`.
746
747 - `Output text: object { text, type }`
748
749 A text output from the model.
750
751 - `text: string`
752
753 The text output from the model.
754
755 - `type: "output_text"`
756
757 The type of the output text. Always `output_text`.
758
759 - `Input image: object { image_url, type, detail }`
760
761 An image input block used within EvalItem content arrays.
762
763 - `image_url: string`
764
765 The URL of the image input.
766
767 - `type: "input_image"`
768
769 The type of the image input. Always `input_image`.
770
771 - `detail: optional string`
772
773 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
774
775 - `response_input_audio: object { input_audio, type }`
776
777 An audio input to the model.
778
779 - `input_audio: object { data, format }`
780
781 - `data: string`
782
783 Base64-encoded audio data.
784
785 - `format: "mp3" or "wav"`
786
787 The format of the audio data. Currently supported formats are `mp3` and
788 `wav`.
789
790 - `"mp3"`
791
792 - `"wav"`
793
794 - `type: "input_audio"`
795
796 The type of the input item. Always `input_audio`.
797
798 - `grader_inputs: array of string or ResponseInputText or object { text, type } or 2 more`
799
800 A list of inputs, each of which may be either an input text, output text, input
801 image, or input audio object.
802
803 - `Text input: string`
804
805 A text input to the model.
806
807 - `response_input_text: object { text, type, prompt_cache_breakpoint }`
808
809 A text input to the model.
810
811 - `Output text: object { text, type }`
812
813 A text output from the model.
814
815 - `text: string`
816
817 The text output from the model.
818
819 - `type: "output_text"`
820
821 The type of the output text. Always `output_text`.
822
823 - `Input image: object { image_url, type, detail }`
824
825 An image input block used within EvalItem content arrays.
826
827 - `image_url: string`
828
829 The URL of the image input.
830
831 - `type: "input_image"`
832
833 The type of the image input. Always `input_image`.
834
835 - `detail: optional string`
836
837 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
838
839 - `response_input_audio: object { input_audio, type }`
840
841 An audio input to the model.
842
843 - `role: "user" or "assistant" or "system" or "developer"`
844
845 The role of the message input. One of `user`, `assistant`, `system`, or
846 `developer`.
847
848 - `"user"`
849
850 - `"assistant"`
851
852 - `"system"`
853
854 - `"developer"`
855
856 - `type: optional "message"`
857
858 The type of the message input. Always `message`.
859
860 - `"message"`
861
862 - `model: string`
863
864 The model to use for the evaluation.
865
866 - `name: string`
867
868 The name of the grader.
869
870 - `type: "score_model"`
871
872 The object type, which is always `score_model`.
873
874 - `range: optional array of number`
875
876 The range of the score. Defaults to `[0, 1]`.
877
878 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`
879
880 The sampling parameters for the model.
881
882 - `max_completions_tokens: optional number`
883
884 The maximum number of tokens the grader model may generate in its response.
885
886 - `reasoning_effort: optional "none" or "minimal" or "low" or 4 more`
887
888 Constrains effort on reasoning for reasoning models. Currently supported
889 values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`.
890 Reducing reasoning effort can result in faster responses and fewer tokens
891 used on reasoning in a response. Not all reasoning models support every
892 value. See the
893 [reasoning guide](https://platform.openai.com/docs/guides/reasoning)
894 for model-specific support.
895
896 - `"none"`
897
898 - `"minimal"`
899
900 - `"low"`
901
902 - `"medium"`
903
904 - `"high"`
905
906 - `"xhigh"`
907
908 - `"max"`
909
910 - `seed: optional number`
911
912 A seed value to initialize the randomness, during sampling.
913
914 - `temperature: optional number`
915
916 A higher temperature increases randomness in the outputs.
917
918 - `top_p: optional number`
919
920 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
921
922### String Check Grader
923
924- `string_check_grader: object { input, name, operation, 2 more }`
925
926 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
927
928 - `input: string`
929
930 The input text. This may include template strings.
931
932 - `name: string`
933
934 The name of the grader.
935
936 - `operation: "eq" or "ne" or "like" or "ilike"`
937
938 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
939
940 - `"eq"`
941
942 - `"ne"`
943
944 - `"like"`
945
946 - `"ilike"`
947
948 - `reference: string`
949
950 The reference text. This may include template strings.
951
952 - `type: "string_check"`
953
954 The object type, which is always `string_check`.
955
956### Text Similarity Grader
957
958- `text_similarity_grader: object { evaluation_metric, input, name, 2 more }`
959
960 A TextSimilarityGrader object which grades text based on similarity metrics.
961
962 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`
963
964 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,
965 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,
966 or `rouge_l`.
967
968 - `"cosine"`
969
970 - `"fuzzy_match"`
971
972 - `"bleu"`
973
974 - `"gleu"`
975
976 - `"meteor"`
977
978 - `"rouge_1"`
979
980 - `"rouge_2"`
981
982 - `"rouge_3"`
983
984 - `"rouge_4"`
985
986 - `"rouge_5"`
987
988 - `"rouge_l"`
989
990 - `input: string`
991
992 The text being graded.
993
994 - `name: string`
995
996 The name of the grader.
997
998 - `reference: string`
999
1000 The text being graded against.
1001
1002 - `type: "text_similarity"`
1003
1004 The type of grader.