resources/fine_tuning/subresources/alpha/index.md 2026-05-18 22:01 UTC to 2026-05-19 06:34 UTC

1# Alpha

2 

3# Graders

4 

5## Run grader

6 

7**post** `/fine_tuning/alpha/graders/run`

8 

9Run a grader.

10 

11### Body Parameters

12 

13- `grader: StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more`

14 

15 The grader used for the fine-tuning job.

16 

17 - `StringCheckGrader object { input, name, operation, 2 more }`

18 

19 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

20 

21 - `input: string`

22 

23 The input text. This may include template strings.

24 

25 - `name: string`

26 

27 The name of the grader.

28 

29 - `operation: "eq" or "ne" or "like" or "ilike"`

30 

31 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

32 

33 - `"eq"`

34 

35 - `"ne"`

36 

37 - `"like"`

38 

39 - `"ilike"`

40 

41 - `reference: string`

42 

43 The reference text. This may include template strings.

44 

45 - `type: "string_check"`

46 

47 The object type, which is always `string_check`.

48 

49 - `"string_check"`

50 

51 - `TextSimilarityGrader object { evaluation_metric, input, name, 2 more }`

52 

53 A TextSimilarityGrader object which grades text based on similarity metrics.

54 

55 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`

56 

57 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

58 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

59 or `rouge_l`.

60 

61 - `"cosine"`

62 

63 - `"fuzzy_match"`

64 

65 - `"bleu"`

66 

67 - `"gleu"`

68 

69 - `"meteor"`

70 

71 - `"rouge_1"`

72 

73 - `"rouge_2"`

74 

75 - `"rouge_3"`

76 

77 - `"rouge_4"`

78 

79 - `"rouge_5"`

80 

81 - `"rouge_l"`

82 

83 - `input: string`

84 

85 The text being graded.

86 

87 - `name: string`

88 

89 The name of the grader.

90 

91 - `reference: string`

92 

93 The text being graded against.

94 

95 - `type: "text_similarity"`

96 

97 The type of grader.

98 

99 - `"text_similarity"`

100 

101 - `PythonGrader object { name, source, type, image_tag }`

102 

103 A PythonGrader object that runs a python script on the input.

104 

105 - `name: string`

106 

107 The name of the grader.

108 

109 - `source: string`

110 

111 The source code of the python script.

112 

113 - `type: "python"`

114 

115 The object type, which is always `python`.

116 

117 - `"python"`

118 

119 - `image_tag: optional string`

120 

121 The image tag to use for the python script.

122 

123 - `ScoreModelGrader object { input, model, name, 3 more }`

124 

125 A ScoreModelGrader object that uses a model to assign a score to the input.

126 

127 - `input: array of object { content, role, type }`

128 

129 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

130 

131 - `content: string or ResponseInputText or object { text, type } or 3 more`

132 

133 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

134 

135 - `TextInput = string`

136 

137 A text input to the model.

138 

139 - `ResponseInputText object { text, type }`

140 

141 A text input to the model.

142 

143 - `text: string`

144 

145 The text input to the model.

146 

147 - `type: "input_text"`

148 

149 The type of the input item. Always `input_text`.

150 

151 - `"input_text"`

152 

153 - `OutputText object { text, type }`

154 

155 A text output from the model.

156 

157 - `text: string`

158 

159 The text output from the model.

160 

161 - `type: "output_text"`

162 

163 The type of the output text. Always `output_text`.

164 

165 - `"output_text"`

166 

167 - `InputImage object { image_url, type, detail }`

168 

169 An image input block used within EvalItem content arrays.

170 

171 - `image_url: string`

172 

173 The URL of the image input.

174 

175 - `type: "input_image"`

176 

177 The type of the image input. Always `input_image`.

178 

179 - `"input_image"`

180 

181 - `detail: optional string`

182 

183 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

184 

185 - `ResponseInputAudio object { input_audio, type }`

186 

187 An audio input to the model.

188 

189 - `input_audio: object { data, format }`

190 

191 - `data: string`

192 

193 Base64-encoded audio data.

194 

195 - `format: "mp3" or "wav"`

196 

197 The format of the audio data. Currently supported formats are `mp3` and

198 `wav`.

199 

200 - `"mp3"`

201 

202 - `"wav"`

203 

204 - `type: "input_audio"`

205 

206 The type of the input item. Always `input_audio`.

207 

208 - `"input_audio"`

209 

210 - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`

211 

212 A list of inputs, each of which may be either an input text, output text, input

213 image, or input audio object.

214 

215 - `TextInput = string`

216 

217 A text input to the model.

218 

219 - `ResponseInputText object { text, type }`

220 

221 A text input to the model.

222 

223 - `OutputText object { text, type }`

224 

225 A text output from the model.

226 

227 - `text: string`

228 

229 The text output from the model.

230 

231 - `type: "output_text"`

232 

233 The type of the output text. Always `output_text`.

234 

235 - `"output_text"`

236 

237 - `InputImage object { image_url, type, detail }`

238 

239 An image input block used within EvalItem content arrays.

240 

241 - `image_url: string`

242 

243 The URL of the image input.

244 

245 - `type: "input_image"`

246 

247 The type of the image input. Always `input_image`.

248 

249 - `"input_image"`

250 

251 - `detail: optional string`

252 

253 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

254 

255 - `ResponseInputAudio object { input_audio, type }`

256 

257 An audio input to the model.

258 

259 - `role: "user" or "assistant" or "system" or "developer"`

260 

261 The role of the message input. One of `user`, `assistant`, `system`, or

262 `developer`.

263 

264 - `"user"`

265 

266 - `"assistant"`

267 

268 - `"system"`

269 

270 - `"developer"`

271 

272 - `type: optional "message"`

273 

274 The type of the message input. Always `message`.

275 

276 - `"message"`

277 

278 - `model: string`

279 

280 The model to use for the evaluation.

281 

282 - `name: string`

283 

284 The name of the grader.

285 

286 - `type: "score_model"`

287 

288 The object type, which is always `score_model`.

289 

290 - `"score_model"`

291 

292 - `range: optional array of number`

293 

294 The range of the score. Defaults to `[0, 1]`.

295 

296 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`

297 

298 The sampling parameters for the model.

299 

300 - `max_completions_tokens: optional number`

301 

302 The maximum number of tokens the grader model may generate in its response.

303 

304 - `reasoning_effort: optional ReasoningEffort`

305 

306 Constrains effort on reasoning for

307 [reasoning models](https://platform.openai.com/docs/guides/reasoning).

308 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing

309 reasoning effort can result in faster responses and fewer tokens used

310 on reasoning in a response.

311 

312 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.

313 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.

314 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.

315 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.

316 

317 - `"none"`

318 

319 - `"minimal"`

320 

321 - `"low"`

322 

323 - `"medium"`

324 

325 - `"high"`

326 

327 - `"xhigh"`

328 

329 - `seed: optional number`

330 

331 A seed value to initialize the randomness, during sampling.

332 

333 - `temperature: optional number`

334 

335 A higher temperature increases randomness in the outputs.

336 

337 - `top_p: optional number`

338 

339 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

340 

341 - `MultiGrader object { calculate_output, graders, name, type }`

342 

343 A MultiGrader object combines the output of multiple graders to produce a single score.

344 

345 - `calculate_output: string`

346 

347 A formula to calculate the output based on grader results.

348 

349 - `graders: StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more`

350 

351 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

352 

353 - `StringCheckGrader object { input, name, operation, 2 more }`

354 

355 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

356 

357 - `TextSimilarityGrader object { evaluation_metric, input, name, 2 more }`

358 

359 A TextSimilarityGrader object which grades text based on similarity metrics.

360 

361 - `PythonGrader object { name, source, type, image_tag }`

362 

363 A PythonGrader object that runs a python script on the input.

364 

365 - `ScoreModelGrader object { input, model, name, 3 more }`

366 

367 A ScoreModelGrader object that uses a model to assign a score to the input.

368 

369 - `LabelModelGrader object { input, labels, model, 3 more }`

370 

371 A LabelModelGrader object which uses a model to assign labels to each item

372 in the evaluation.

373 

374 - `input: array of object { content, role, type }`

375 

376 - `content: string or ResponseInputText or object { text, type } or 3 more`

377 

378 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

379 

380 - `TextInput = string`

381 

382 A text input to the model.

383 

384 - `ResponseInputText object { text, type }`

385 

386 A text input to the model.

387 

388 - `OutputText object { text, type }`

389 

390 A text output from the model.

391 

392 - `text: string`

393 

394 The text output from the model.

395 

396 - `type: "output_text"`

397 

398 The type of the output text. Always `output_text`.

399 

400 - `"output_text"`

401 

402 - `InputImage object { image_url, type, detail }`

403 

404 An image input block used within EvalItem content arrays.

405 

406 - `image_url: string`

407 

408 The URL of the image input.

409 

410 - `type: "input_image"`

411 

412 The type of the image input. Always `input_image`.

413 

414 - `"input_image"`

415 

416 - `detail: optional string`

417 

418 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

419 

420 - `ResponseInputAudio object { input_audio, type }`

421 

422 An audio input to the model.

423 

424 - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`

425 

426 A list of inputs, each of which may be either an input text, output text, input

427 image, or input audio object.

428 

429 - `role: "user" or "assistant" or "system" or "developer"`

430 

431 The role of the message input. One of `user`, `assistant`, `system`, or

432 `developer`.

433 

434 - `"user"`

435 

436 - `"assistant"`

437 

438 - `"system"`

439 

440 - `"developer"`

441 

442 - `type: optional "message"`

443 

444 The type of the message input. Always `message`.

445 

446 - `"message"`

447 

448 - `labels: array of string`

449 

450 The labels to assign to each item in the evaluation.

451 

452 - `model: string`

453 

454 The model to use for the evaluation. Must support structured outputs.

455 

456 - `name: string`

457 

458 The name of the grader.

459 

460 - `passing_labels: array of string`

461 

462 The labels that indicate a passing result. Must be a subset of labels.

463 

464 - `type: "label_model"`

465 

466 The object type, which is always `label_model`.

467 

468 - `"label_model"`

469 

470 - `name: string`

471 

472 The name of the grader.

473 

474 - `type: "multi"`

475 

476 The object type, which is always `multi`.

477 

478 - `"multi"`

479 

480- `model_sample: string`

481 

482 The model sample to be evaluated. This value will be used to populate

483 the `sample` namespace. See [the guide](/docs/guides/graders) for more details.

484 The `output_json` variable will be populated if the model sample is a

485 valid JSON string.

486 

487- `item: optional unknown`

488 

489 The dataset item provided to the grader. This will be used to populate

490 the `item` namespace. See [the guide](/docs/guides/graders) for more details.

491 

492### Returns

493 

494- `metadata: object { errors, execution_time, name, 4 more }`

495 

496 - `errors: object { formula_parse_error, invalid_variable_error, model_grader_parse_error, 11 more }`

497 

498 - `formula_parse_error: boolean`

499 

500 - `invalid_variable_error: boolean`

501 

502 - `model_grader_parse_error: boolean`

503 

504 - `model_grader_refusal_error: boolean`

505 

506 - `model_grader_server_error: boolean`

507 

508 - `model_grader_server_error_details: string`

509 

510 - `other_error: boolean`

511 

512 - `python_grader_runtime_error: boolean`

513 

514 - `python_grader_runtime_error_details: string`

515 

516 - `python_grader_server_error: boolean`

517 

518 - `python_grader_server_error_type: string`

519 

520 - `sample_parse_error: boolean`

521 

522 - `truncated_observation_error: boolean`

523 

524 - `unresponsive_reward_error: boolean`

525 

526 - `execution_time: number`

527 

528 - `name: string`

529 

530 - `sampled_model_name: string`

531 

532 - `scores: map[unknown]`

533 

534 - `token_usage: number`

535 

536 - `type: string`

537 

538- `model_grader_token_usage_per_model: map[unknown]`

539 

540- `reward: number`

541 

542- `sub_rewards: map[unknown]`

543 

544### Example

545 

546```http

547curl https://api.openai.com/v1/fine_tuning/alpha/graders/run \

548 -H 'Content-Type: application/json' \

549 -H "Authorization: Bearer $OPENAI_API_KEY" \

550 -d '{

551 "grader": {

552 "input": "input",

553 "name": "name",

554 "operation": "eq",

555 "reference": "reference",

556 "type": "string_check"

557 },

558 "model_sample": "model_sample"

559 }'

560```

561 

562#### Response

563 

564```json

565{

566 "metadata": {

567 "errors": {

568 "formula_parse_error": true,

569 "invalid_variable_error": true,

570 "model_grader_parse_error": true,

571 "model_grader_refusal_error": true,

572 "model_grader_server_error": true,

573 "model_grader_server_error_details": "model_grader_server_error_details",

574 "other_error": true,

575 "python_grader_runtime_error": true,

576 "python_grader_runtime_error_details": "python_grader_runtime_error_details",

577 "python_grader_server_error": true,

578 "python_grader_server_error_type": "python_grader_server_error_type",

579 "sample_parse_error": true,

580 "truncated_observation_error": true,

581 "unresponsive_reward_error": true

582 },

583 "execution_time": 0,

584 "name": "name",

585 "sampled_model_name": "sampled_model_name",

586 "scores": {

587 "foo": "bar"

588 },

589 "token_usage": 0,

590 "type": "type"

591 },

592 "model_grader_token_usage_per_model": {

593 "foo": "bar"

594 },

595 "reward": 0,

596 "sub_rewards": {

597 "foo": "bar"

598 }

599}

600```

601 

602### Score text alignment

603 

604```http

605curl -X POST https://api.openai.com/v1/fine_tuning/alpha/graders/run \

606 -H "Content-Type: application/json" \

607 -H "Authorization: Bearer $OPENAI_API_KEY" \

608 -d '{

609 "grader": {

610 "type": "score_model",

611 "name": "Example score model grader",

612 "input": [

613 {

614 "role": "user",

615 "content": [

616 {

617 "type": "input_text",

618 "text": "Score how close the reference answer is to the model answer on a 0-1 scale. Return only the score.\n\nReference answer: {{item.reference_answer}}\n\nModel answer: {{sample.output_text}}"

619 }

620 ]

621 }

622 ],

623 "model": "gpt-5-mini",

624 "sampling_params": {

625 "temperature": 1,

626 "top_p": 1,

627 "seed": 42

628 }

629 },

630 "item": {

631 "reference_answer": "fuzzy wuzzy was a bear"

632 },

633 "model_sample": "fuzzy wuzzy was a bear"

634 }'

635```

636 

637#### Response

638 

639```json

640{

641 "reward": 1.0,

642 "metadata": {

643 "name": "Example score model grader",

644 "type": "score_model",

645 "errors": {

646 "formula_parse_error": false,

647 "sample_parse_error": false,

648 "truncated_observation_error": false,

649 "unresponsive_reward_error": false,

650 "invalid_variable_error": false,

651 "other_error": false,

652 "python_grader_server_error": false,

653 "python_grader_server_error_type": null,

654 "python_grader_runtime_error": false,

655 "python_grader_runtime_error_details": null,

656 "model_grader_server_error": false,

657 "model_grader_refusal_error": false,

658 "model_grader_parse_error": false,

659 "model_grader_server_error_details": null

660 },

661 "execution_time": 4.365238428115845,

662 "scores": {},

663 "token_usage": {

664 "prompt_tokens": 190,

665 "total_tokens": 324,

666 "completion_tokens": 134,

667 "cached_tokens": 0

668 },

669 "sampled_model_name": "gpt-4o-2024-08-06"

670 },

671 "sub_rewards": {},

672 "model_grader_token_usage_per_model": {

673 "gpt-4o-2024-08-06": {

674 "prompt_tokens": 190,

675 "total_tokens": 324,

676 "completion_tokens": 134,

677 "cached_tokens": 0

678 }

679 }

680}

681```

682 

683### Score an image caption

684 

685```http

686curl -X POST https://api.openai.com/v1/fine_tuning/alpha/graders/run \

687 -H "Content-Type: application/json" \

688 -H "Authorization: Bearer $OPENAI_API_KEY" \

689 -d '{

690 "grader": {

691 "type": "score_model",

692 "name": "Image caption grader",

693 "input": [

694 {

695 "role": "user",

696 "content": [

697 {

698 "type": "input_text",

699 "text": "Score how well the provided caption matches the image on a 0-1 scale. Only return the score.\n\nCaption: {{sample.output_text}}"

700 },

701 {

702 "type": "input_image",

703 "image_url": "https://example.com/dog-catching-ball.png",

704 "file_id": null,

705 "detail": "high"

706 }

707 ]

708 }

709 ],

710 "model": "gpt-5-mini",

711 "sampling_params": {

712 "temperature": 0.2

713 }

714 },

715 "item": {

716 "expected_caption": "A golden retriever jumps to catch a tennis ball"

717 },

718 "model_sample": "A dog leaps to grab a tennis ball mid-air"

719 }'

720```

721 

722### Score an audio response

723 

724```http

725curl -X POST https://api.openai.com/v1/fine_tuning/alpha/graders/run \

726 -H "Content-Type: application/json" \

727 -H "Authorization: Bearer $OPENAI_API_KEY" \

728 -d '{

729 "grader": {

730 "type": "score_model",

731 "name": "Audio clarity grader",

732 "input": [

733 {

734 "role": "user",

735 "content": [

736 {

737 "type": "input_text",

738 "text": "Listen to the clip and return a confidence score from 0 to 1 that the speaker said: {{item.target_phrase}}"

739 },

740 {

741 "type": "input_audio",

742 "input_audio": {

743 "data": "{{item.audio_clip_b64}}",

744 "format": "mp3"

745 }

746 }

747 ]

748 }

749 ],

750 "model": "gpt-audio",

751 "sampling_params": {

752 "temperature": 0.2,

753 "top_p": 1,

754 "seed": 123

755 }

756 },

757 "item": {

758 "target_phrase": "Please deliver the package on Tuesday",

759 "audio_clip_b64": "<base64-encoded mp3>"

760 },

761 "model_sample": "Please deliver the package on Tuesday"

762 }'

763```

764 

765## Validate grader

766 

767**post** `/fine_tuning/alpha/graders/validate`

768 

769Validate a grader.

770 

771### Body Parameters

772 

773- `grader: StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more`

774 

775 The grader used for the fine-tuning job.

776 

777 - `StringCheckGrader object { input, name, operation, 2 more }`

778 

779 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

780 

781 - `input: string`

782 

783 The input text. This may include template strings.

784 

785 - `name: string`

786 

787 The name of the grader.

788 

789 - `operation: "eq" or "ne" or "like" or "ilike"`

790 

791 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

792 

793 - `"eq"`

794 

795 - `"ne"`

796 

797 - `"like"`

798 

799 - `"ilike"`

800 

801 - `reference: string`

802 

803 The reference text. This may include template strings.

804 

805 - `type: "string_check"`

806 

807 The object type, which is always `string_check`.

808 

809 - `"string_check"`

810 

811 - `TextSimilarityGrader object { evaluation_metric, input, name, 2 more }`

812 

813 A TextSimilarityGrader object which grades text based on similarity metrics.

814 

815 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`

816 

817 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

818 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

819 or `rouge_l`.

820 

821 - `"cosine"`

822 

823 - `"fuzzy_match"`

824 

825 - `"bleu"`

826 

827 - `"gleu"`

828 

829 - `"meteor"`

830 

831 - `"rouge_1"`

832 

833 - `"rouge_2"`

834 

835 - `"rouge_3"`

836 

837 - `"rouge_4"`

838 

839 - `"rouge_5"`

840 

841 - `"rouge_l"`

842 

843 - `input: string`

844 

845 The text being graded.

846 

847 - `name: string`

848 

849 The name of the grader.

850 

851 - `reference: string`

852 

853 The text being graded against.

854 

855 - `type: "text_similarity"`

856 

857 The type of grader.

858 

859 - `"text_similarity"`

860 

861 - `PythonGrader object { name, source, type, image_tag }`

862 

863 A PythonGrader object that runs a python script on the input.

864 

865 - `name: string`

866 

867 The name of the grader.

868 

869 - `source: string`

870 

871 The source code of the python script.

872 

873 - `type: "python"`

874 

875 The object type, which is always `python`.

876 

877 - `"python"`

878 

879 - `image_tag: optional string`

880 

881 The image tag to use for the python script.

882 

883 - `ScoreModelGrader object { input, model, name, 3 more }`

884 

885 A ScoreModelGrader object that uses a model to assign a score to the input.

886 

887 - `input: array of object { content, role, type }`

888 

889 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

890 

891 - `content: string or ResponseInputText or object { text, type } or 3 more`

892 

893 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

894 

895 - `TextInput = string`

896 

897 A text input to the model.

898 

899 - `ResponseInputText object { text, type }`

900 

901 A text input to the model.

902 

903 - `text: string`

904 

905 The text input to the model.

906 

907 - `type: "input_text"`

908 

909 The type of the input item. Always `input_text`.

910 

911 - `"input_text"`

912 

913 - `OutputText object { text, type }`

914 

915 A text output from the model.

916 

917 - `text: string`

918 

919 The text output from the model.

920 

921 - `type: "output_text"`

922 

923 The type of the output text. Always `output_text`.

924 

925 - `"output_text"`

926 

927 - `InputImage object { image_url, type, detail }`

928 

929 An image input block used within EvalItem content arrays.

930 

931 - `image_url: string`

932 

933 The URL of the image input.

934 

935 - `type: "input_image"`

936 

937 The type of the image input. Always `input_image`.

938 

939 - `"input_image"`

940 

941 - `detail: optional string`

942 

943 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

944 

945 - `ResponseInputAudio object { input_audio, type }`

946 

947 An audio input to the model.

948 

949 - `input_audio: object { data, format }`

950 

951 - `data: string`

952 

953 Base64-encoded audio data.

954 

955 - `format: "mp3" or "wav"`

956 

957 The format of the audio data. Currently supported formats are `mp3` and

958 `wav`.

959 

960 - `"mp3"`

961 

962 - `"wav"`

963 

964 - `type: "input_audio"`

965 

966 The type of the input item. Always `input_audio`.

967 

968 - `"input_audio"`

969 

970 - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`

971 

972 A list of inputs, each of which may be either an input text, output text, input

973 image, or input audio object.

974 

975 - `TextInput = string`

976 

977 A text input to the model.

978 

979 - `ResponseInputText object { text, type }`

980 

981 A text input to the model.

982 

983 - `OutputText object { text, type }`

984 

985 A text output from the model.

986 

987 - `text: string`

988 

989 The text output from the model.

990 

991 - `type: "output_text"`

992 

993 The type of the output text. Always `output_text`.

994 

995 - `"output_text"`

996 

997 - `InputImage object { image_url, type, detail }`

998 

999 An image input block used within EvalItem content arrays.

1000 

1001 - `image_url: string`

1002 

1003 The URL of the image input.

1004 

1005 - `type: "input_image"`

1006 

1007 The type of the image input. Always `input_image`.

1008 

1009 - `"input_image"`

1010 

1011 - `detail: optional string`

1012 

1013 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1014 

1015 - `ResponseInputAudio object { input_audio, type }`

1016 

1017 An audio input to the model.

1018 

1019 - `role: "user" or "assistant" or "system" or "developer"`

1020 

1021 The role of the message input. One of `user`, `assistant`, `system`, or

1022 `developer`.

1023 

1024 - `"user"`

1025 

1026 - `"assistant"`

1027 

1028 - `"system"`

1029 

1030 - `"developer"`

1031 

1032 - `type: optional "message"`

1033 

1034 The type of the message input. Always `message`.

1035 

1036 - `"message"`

1037 

1038 - `model: string`

1039 

1040 The model to use for the evaluation.

1041 

1042 - `name: string`

1043 

1044 The name of the grader.

1045 

1046 - `type: "score_model"`

1047 

1048 The object type, which is always `score_model`.

1049 

1050 - `"score_model"`

1051 

1052 - `range: optional array of number`

1053 

1054 The range of the score. Defaults to `[0, 1]`.

1055 

1056 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`

1057 

1058 The sampling parameters for the model.

1059 

1060 - `max_completions_tokens: optional number`

1061 

1062 The maximum number of tokens the grader model may generate in its response.

1063 

1064 - `reasoning_effort: optional ReasoningEffort`

1065 

1066 Constrains effort on reasoning for

1067 [reasoning models](https://platform.openai.com/docs/guides/reasoning).

1068 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing

1069 reasoning effort can result in faster responses and fewer tokens used

1070 on reasoning in a response.

1071 

1072 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.

1073 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.

1074 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.

1075 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.

1076 

1077 - `"none"`

1078 

1079 - `"minimal"`

1080 

1081 - `"low"`

1082 

1083 - `"medium"`

1084 

1085 - `"high"`

1086 

1087 - `"xhigh"`

1088 

1089 - `seed: optional number`

1090 

1091 A seed value to initialize the randomness, during sampling.

1092 

1093 - `temperature: optional number`

1094 

1095 A higher temperature increases randomness in the outputs.

1096 

1097 - `top_p: optional number`

1098 

1099 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

1100 

1101 - `MultiGrader object { calculate_output, graders, name, type }`

1102 

1103 A MultiGrader object combines the output of multiple graders to produce a single score.

1104 

1105 - `calculate_output: string`

1106 

1107 A formula to calculate the output based on grader results.

1108 

1109 - `graders: StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more`

1110 

1111 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1112 

1113 - `StringCheckGrader object { input, name, operation, 2 more }`

1114 

1115 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1116 

1117 - `TextSimilarityGrader object { evaluation_metric, input, name, 2 more }`

1118 

1119 A TextSimilarityGrader object which grades text based on similarity metrics.

1120 

1121 - `PythonGrader object { name, source, type, image_tag }`

1122 

1123 A PythonGrader object that runs a python script on the input.

1124 

1125 - `ScoreModelGrader object { input, model, name, 3 more }`

1126 

1127 A ScoreModelGrader object that uses a model to assign a score to the input.

1128 

1129 - `LabelModelGrader object { input, labels, model, 3 more }`

1130 

1131 A LabelModelGrader object which uses a model to assign labels to each item

1132 in the evaluation.

1133 

1134 - `input: array of object { content, role, type }`

1135 

1136 - `content: string or ResponseInputText or object { text, type } or 3 more`

1137 

1138 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1139 

1140 - `TextInput = string`

1141 

1142 A text input to the model.

1143 

1144 - `ResponseInputText object { text, type }`

1145 

1146 A text input to the model.

1147 

1148 - `OutputText object { text, type }`

1149 

1150 A text output from the model.

1151 

1152 - `text: string`

1153 

1154 The text output from the model.

1155 

1156 - `type: "output_text"`

1157 

1158 The type of the output text. Always `output_text`.

1159 

1160 - `"output_text"`

1161 

1162 - `InputImage object { image_url, type, detail }`

1163 

1164 An image input block used within EvalItem content arrays.

1165 

1166 - `image_url: string`

1167 

1168 The URL of the image input.

1169 

1170 - `type: "input_image"`

1171 

1172 The type of the image input. Always `input_image`.

1173 

1174 - `"input_image"`

1175 

1176 - `detail: optional string`

1177 

1178 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1179 

1180 - `ResponseInputAudio object { input_audio, type }`

1181 

1182 An audio input to the model.

1183 

1184 - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`

1185 

1186 A list of inputs, each of which may be either an input text, output text, input

1187 image, or input audio object.

1188 

1189 - `role: "user" or "assistant" or "system" or "developer"`

1190 

1191 The role of the message input. One of `user`, `assistant`, `system`, or

1192 `developer`.

1193 

1194 - `"user"`

1195 

1196 - `"assistant"`

1197 

1198 - `"system"`

1199 

1200 - `"developer"`

1201 

1202 - `type: optional "message"`

1203 

1204 The type of the message input. Always `message`.

1205 

1206 - `"message"`

1207 

1208 - `labels: array of string`

1209 

1210 The labels to assign to each item in the evaluation.

1211 

1212 - `model: string`

1213 

1214 The model to use for the evaluation. Must support structured outputs.

1215 

1216 - `name: string`

1217 

1218 The name of the grader.

1219 

1220 - `passing_labels: array of string`

1221 

1222 The labels that indicate a passing result. Must be a subset of labels.

1223 

1224 - `type: "label_model"`

1225 

1226 The object type, which is always `label_model`.

1227 

1228 - `"label_model"`

1229 

1230 - `name: string`

1231 

1232 The name of the grader.

1233 

1234 - `type: "multi"`

1235 

1236 The object type, which is always `multi`.

1237 

1238 - `"multi"`

1239 

1240### Returns

1241 

1242- `grader: optional StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more`

1243 

1244 The grader used for the fine-tuning job.

1245 

1246 - `StringCheckGrader object { input, name, operation, 2 more }`

1247 

1248 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1249 

1250 - `input: string`

1251 

1252 The input text. This may include template strings.

1253 

1254 - `name: string`

1255 

1256 The name of the grader.

1257 

1258 - `operation: "eq" or "ne" or "like" or "ilike"`

1259 

1260 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

1261 

1262 - `"eq"`

1263 

1264 - `"ne"`

1265 

1266 - `"like"`

1267 

1268 - `"ilike"`

1269 

1270 - `reference: string`

1271 

1272 The reference text. This may include template strings.

1273 

1274 - `type: "string_check"`

1275 

1276 The object type, which is always `string_check`.

1277 

1278 - `"string_check"`

1279 

1280 - `TextSimilarityGrader object { evaluation_metric, input, name, 2 more }`

1281 

1282 A TextSimilarityGrader object which grades text based on similarity metrics.

1283 

1284 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`

1285 

1286 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

1287 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

1288 or `rouge_l`.

1289 

1290 - `"cosine"`

1291 

1292 - `"fuzzy_match"`

1293 

1294 - `"bleu"`

1295 

1296 - `"gleu"`

1297 

1298 - `"meteor"`

1299 

1300 - `"rouge_1"`

1301 

1302 - `"rouge_2"`

1303 

1304 - `"rouge_3"`

1305 

1306 - `"rouge_4"`

1307 

1308 - `"rouge_5"`

1309 

1310 - `"rouge_l"`

1311 

1312 - `input: string`

1313 

1314 The text being graded.

1315 

1316 - `name: string`

1317 

1318 The name of the grader.

1319 

1320 - `reference: string`

1321 

1322 The text being graded against.

1323 

1324 - `type: "text_similarity"`

1325 

1326 The type of grader.

1327 

1328 - `"text_similarity"`

1329 

1330 - `PythonGrader object { name, source, type, image_tag }`

1331 

1332 A PythonGrader object that runs a python script on the input.

1333 

1334 - `name: string`

1335 

1336 The name of the grader.

1337 

1338 - `source: string`

1339 

1340 The source code of the python script.

1341 

1342 - `type: "python"`

1343 

1344 The object type, which is always `python`.

1345 

1346 - `"python"`

1347 

1348 - `image_tag: optional string`

1349 

1350 The image tag to use for the python script.

1351 

1352 - `ScoreModelGrader object { input, model, name, 3 more }`

1353 

1354 A ScoreModelGrader object that uses a model to assign a score to the input.

1355 

1356 - `input: array of object { content, role, type }`

1357 

1358 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

1359 

1360 - `content: string or ResponseInputText or object { text, type } or 3 more`

1361 

1362 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1363 

1364 - `TextInput = string`

1365 

1366 A text input to the model.

1367 

1368 - `ResponseInputText object { text, type }`

1369 

1370 A text input to the model.

1371 

1372 - `text: string`

1373 

1374 The text input to the model.

1375 

1376 - `type: "input_text"`

1377 

1378 The type of the input item. Always `input_text`.

1379 

1380 - `"input_text"`

1381 

1382 - `OutputText object { text, type }`

1383 

1384 A text output from the model.

1385 

1386 - `text: string`

1387 

1388 The text output from the model.

1389 

1390 - `type: "output_text"`

1391 

1392 The type of the output text. Always `output_text`.

1393 

1394 - `"output_text"`

1395 

1396 - `InputImage object { image_url, type, detail }`

1397 

1398 An image input block used within EvalItem content arrays.

1399 

1400 - `image_url: string`

1401 

1402 The URL of the image input.

1403 

1404 - `type: "input_image"`

1405 

1406 The type of the image input. Always `input_image`.

1407 

1408 - `"input_image"`

1409 

1410 - `detail: optional string`

1411 

1412 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1413 

1414 - `ResponseInputAudio object { input_audio, type }`

1415 

1416 An audio input to the model.

1417 

1418 - `input_audio: object { data, format }`

1419 

1420 - `data: string`

1421 

1422 Base64-encoded audio data.

1423 

1424 - `format: "mp3" or "wav"`

1425 

1426 The format of the audio data. Currently supported formats are `mp3` and

1427 `wav`.

1428 

1429 - `"mp3"`

1430 

1431 - `"wav"`

1432 

1433 - `type: "input_audio"`

1434 

1435 The type of the input item. Always `input_audio`.

1436 

1437 - `"input_audio"`

1438 

1439 - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`

1440 

1441 A list of inputs, each of which may be either an input text, output text, input

1442 image, or input audio object.

1443 

1444 - `TextInput = string`

1445 

1446 A text input to the model.

1447 

1448 - `ResponseInputText object { text, type }`

1449 

1450 A text input to the model.

1451 

1452 - `OutputText object { text, type }`

1453 

1454 A text output from the model.

1455 

1456 - `text: string`

1457 

1458 The text output from the model.

1459 

1460 - `type: "output_text"`

1461 

1462 The type of the output text. Always `output_text`.

1463 

1464 - `"output_text"`

1465 

1466 - `InputImage object { image_url, type, detail }`

1467 

1468 An image input block used within EvalItem content arrays.

1469 

1470 - `image_url: string`

1471 

1472 The URL of the image input.

1473 

1474 - `type: "input_image"`

1475 

1476 The type of the image input. Always `input_image`.

1477 

1478 - `"input_image"`

1479 

1480 - `detail: optional string`

1481 

1482 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1483 

1484 - `ResponseInputAudio object { input_audio, type }`

1485 

1486 An audio input to the model.

1487 

1488 - `role: "user" or "assistant" or "system" or "developer"`

1489 

1490 The role of the message input. One of `user`, `assistant`, `system`, or

1491 `developer`.

1492 

1493 - `"user"`

1494 

1495 - `"assistant"`

1496 

1497 - `"system"`

1498 

1499 - `"developer"`

1500 

1501 - `type: optional "message"`

1502 

1503 The type of the message input. Always `message`.

1504 

1505 - `"message"`

1506 

1507 - `model: string`

1508 

1509 The model to use for the evaluation.

1510 

1511 - `name: string`

1512 

1513 The name of the grader.

1514 

1515 - `type: "score_model"`

1516 

1517 The object type, which is always `score_model`.

1518 

1519 - `"score_model"`

1520 

1521 - `range: optional array of number`

1522 

1523 The range of the score. Defaults to `[0, 1]`.

1524 

1525 - `sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more }`

1526 

1527 The sampling parameters for the model.

1528 

1529 - `max_completions_tokens: optional number`

1530 

1531 The maximum number of tokens the grader model may generate in its response.

1532 

1533 - `reasoning_effort: optional ReasoningEffort`

1534 

1535 Constrains effort on reasoning for

1536 [reasoning models](https://platform.openai.com/docs/guides/reasoning).

1537 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing

1538 reasoning effort can result in faster responses and fewer tokens used

1539 on reasoning in a response.

1540 

1541 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.

1542 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.

1543 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.

1544 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.

1545 

1546 - `"none"`

1547 

1548 - `"minimal"`

1549 

1550 - `"low"`

1551 

1552 - `"medium"`

1553 

1554 - `"high"`

1555 

1556 - `"xhigh"`

1557 

1558 - `seed: optional number`

1559 

1560 A seed value to initialize the randomness, during sampling.

1561 

1562 - `temperature: optional number`

1563 

1564 A higher temperature increases randomness in the outputs.

1565 

1566 - `top_p: optional number`

1567 

1568 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

1569 

1570 - `MultiGrader object { calculate_output, graders, name, type }`

1571 

1572 A MultiGrader object combines the output of multiple graders to produce a single score.

1573 

1574 - `calculate_output: string`

1575 

1576 A formula to calculate the output based on grader results.

1577 

1578 - `graders: StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more`

1579 

1580 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1581 

1582 - `StringCheckGrader object { input, name, operation, 2 more }`

1583 

1584 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1585 

1586 - `TextSimilarityGrader object { evaluation_metric, input, name, 2 more }`

1587 

1588 A TextSimilarityGrader object which grades text based on similarity metrics.

1589 

1590 - `PythonGrader object { name, source, type, image_tag }`

1591 

1592 A PythonGrader object that runs a python script on the input.

1593 

1594 - `ScoreModelGrader object { input, model, name, 3 more }`

1595 

1596 A ScoreModelGrader object that uses a model to assign a score to the input.

1597 

1598 - `LabelModelGrader object { input, labels, model, 3 more }`

1599 

1600 A LabelModelGrader object which uses a model to assign labels to each item

1601 in the evaluation.

1602 

1603 - `input: array of object { content, role, type }`

1604 

1605 - `content: string or ResponseInputText or object { text, type } or 3 more`

1606 

1607 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1608 

1609 - `TextInput = string`

1610 

1611 A text input to the model.

1612 

1613 - `ResponseInputText object { text, type }`

1614 

1615 A text input to the model.

1616 

1617 - `OutputText object { text, type }`

1618 

1619 A text output from the model.

1620 

1621 - `text: string`

1622 

1623 The text output from the model.

1624 

1625 - `type: "output_text"`

1626 

1627 The type of the output text. Always `output_text`.

1628 

1629 - `"output_text"`

1630 

1631 - `InputImage object { image_url, type, detail }`

1632 

1633 An image input block used within EvalItem content arrays.

1634 

1635 - `image_url: string`

1636 

1637 The URL of the image input.

1638 

1639 - `type: "input_image"`

1640 

1641 The type of the image input. Always `input_image`.

1642 

1643 - `"input_image"`

1644 

1645 - `detail: optional string`

1646 

1647 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1648 

1649 - `ResponseInputAudio object { input_audio, type }`

1650 

1651 An audio input to the model.

1652 

1653 - `GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more`

1654 

1655 A list of inputs, each of which may be either an input text, output text, input

1656 image, or input audio object.

1657 

1658 - `role: "user" or "assistant" or "system" or "developer"`

1659 

1660 The role of the message input. One of `user`, `assistant`, `system`, or

1661 `developer`.

1662 

1663 - `"user"`

1664 

1665 - `"assistant"`

1666 

1667 - `"system"`

1668 

1669 - `"developer"`

1670 

1671 - `type: optional "message"`

1672 

1673 The type of the message input. Always `message`.

1674 

1675 - `"message"`

1676 

1677 - `labels: array of string`

1678 

1679 The labels to assign to each item in the evaluation.

1680 

1681 - `model: string`

1682 

1683 The model to use for the evaluation. Must support structured outputs.

1684 

1685 - `name: string`

1686 

1687 The name of the grader.

1688 

1689 - `passing_labels: array of string`

1690 

1691 The labels that indicate a passing result. Must be a subset of labels.

1692 

1693 - `type: "label_model"`

1694 

1695 The object type, which is always `label_model`.

1696 

1697 - `"label_model"`

1698 

1699 - `name: string`

1700 

1701 The name of the grader.

1702 

1703 - `type: "multi"`

1704 

1705 The object type, which is always `multi`.

1706 

1707 - `"multi"`

1708 

1709### Example

1710 

1711```http

1712curl https://api.openai.com/v1/fine_tuning/alpha/graders/validate \

1713 -H 'Content-Type: application/json' \

1714 -H "Authorization: Bearer $OPENAI_API_KEY" \

1715 -d '{

1716 "grader": {

1717 "input": "input",

1718 "name": "name",

1719 "operation": "eq",

1720 "reference": "reference",

1721 "type": "string_check"

1722 }

1723 }'

1724```

1725 

1726#### Response

1727 

1728```json

1729{

1730 "grader": {

1731 "input": "input",

1732 "name": "name",

1733 "operation": "eq",

1734 "reference": "reference",

1735 "type": "string_check"

1736 }

1737}

1738```

1739 

1740### Example

1741 

1742```http

1743curl https://api.openai.com/v1/fine_tuning/alpha/graders/validate \

1744 -H "Authorization: Bearer $OPENAI_API_KEY" \

1745 -H "Content-Type: application/json" \

1746 -d '{

1747 "grader": {

1748 "type": "string_check",

1749 "name": "Example string check grader",

1750 "input": "{{sample.output_text}}",

1751 "reference": "{{item.label}}",

1752 "operation": "eq"

1753 }

1754 }'

1755```

1756 

1757#### Response

1758 

1759```json

1760{

1761 "grader": {

1762 "type": "string_check",

1763 "name": "Example string check grader",

1764 "input": "{{sample.output_text}}",

1765 "reference": "{{item.label}}",

1766 "operation": "eq"

1767 }

1768}

1769```

1770 

1771## Domain Types

1772 

1773### Grader Run Response

1774 

1775- `GraderRunResponse object { metadata, model_grader_token_usage_per_model, reward, sub_rewards }`

1776 

1777 - `metadata: object { errors, execution_time, name, 4 more }`

1778 

1779 - `errors: object { formula_parse_error, invalid_variable_error, model_grader_parse_error, 11 more }`

1780 

1781 - `formula_parse_error: boolean`

1782 

1783 - `invalid_variable_error: boolean`

1784 

1785 - `model_grader_parse_error: boolean`

1786 

1787 - `model_grader_refusal_error: boolean`

1788 

1789 - `model_grader_server_error: boolean`

1790 

1791 - `model_grader_server_error_details: string`

1792 

1793 - `other_error: boolean`

1794 

1795 - `python_grader_runtime_error: boolean`

1796 

1797 - `python_grader_runtime_error_details: string`

1798 

1799 - `python_grader_server_error: boolean`

1800 

1801 - `python_grader_server_error_type: string`

1802 

1803 - `sample_parse_error: boolean`

1804 

1805 - `truncated_observation_error: boolean`

1806 

1807 - `unresponsive_reward_error: boolean`

1808 

1809 - `execution_time: number`

1810 

1811 - `name: string`

1812 

1813 - `sampled_model_name: string`

1814 

1815 - `scores: map[unknown]`

1816 

1817 - `token_usage: number`

1818 

1819 - `type: string`

1820 

1821 - `model_grader_token_usage_per_model: map[unknown]`

1822 

1823 - `reward: number`

1824 

1825 - `sub_rewards: map[unknown]`

1826 

1827### Grader Validate Response

1828 

1829- `GraderValidateResponse object { grader }`

1830 

1831 - `grader: optional StringCheckGrader or TextSimilarityGrader or PythonGrader or 2 more`

1832 

1833 The grader used for the fine-tuning job.

1834 

1835 - `StringCheckGrader object { input, name, operation, 2 more }`

1836 

1837 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1838 

1839 - `input: string`

1840 

1841 The input text. This may include template strings.

1842 

1843 - `name: string`

1844 

1845 The name of the grader.

1846 

1847 - `operation: "eq" or "ne" or "like" or "ilike"`

1848 

1849 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

1850 

1851 - `"eq"`

1852 

1853 - `"ne"`

1854 

1855 - `"like"`

1856 

1857 - `"ilike"`

1858 

1859 - `reference: string`

1860 

1861 The reference text. This may include template strings.

1862 

1863 - `type: "string_check"`

1864 

1865 The object type, which is always `string_check`.

1866 

1867 - `"string_check"`

1868 

1869 - `TextSimilarityGrader object { evaluation_metric, input, name, 2 more }`

1870 

1871 A TextSimilarityGrader object which grades text based on similarity metrics.

1872 

1873 - `evaluation_metric: "cosine" or "fuzzy_match" or "bleu" or 8 more`

1874 

1875 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

1876 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

1877 or `rouge_l`.

1878 

1879 - `"cosine"`

1880 

1881 - `"fuzzy_match"`

1882 

1883 - `"bleu"`

1884 

1885 - `"gleu"`

1886 

1887 - `"meteor"`

1888 

1889 - `"rouge_1"`

1890 

1891 - `"rouge_2"`

1892 

1893 - `"rouge_3"`

1894 

1895 - `"rouge_4"`

1896 

1897 - `"rouge_5"`

1898 

1899 - `"rouge_l"`

1900 

1901 - `input: string`

1902 

1903 The text being graded.

1904 

1905 - `name: string`

1906 

1907 The name of the grader.

1908 

1909 - `reference: string`

1910 

1911 The text being graded against.

1912 

1913 - `type: "text_similarity"`

1914 

1915 The type of grader.

1916 

1917 - `"text_similarity"`

1918 

1919 - `PythonGrader object { name, source, type, image_tag }`

1920 

1921 A PythonGrader object that runs a python script on the input.

1922 

1923 - `name: string`

1924 

1925 The name of the grader.

1926 

1927 - `source: string`

1928 

1929 The source code of the python script.

1930 

1931 - `type: "python"`

1932 

1933 The object type, which is always `python`.

1934 

1935 - `"python"`

1936 

1937 - `image_tag: optional string`

1938 

1939 The image tag to use for the python script.

1940 

1941 - `ScoreModelGrader object { input, model, name, 3 more }`

1942 

1943 A ScoreModelGrader object that uses a model to assign a score to the input.

1944 

1945 - `input: array of object { content, role, type }`

1946 

1947 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

1948 

1949 - `content: string or ResponseInputText or object { text, type } or 3 more`

1950 

1951 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1952 

1953 - `TextInput = string`

1954 

1955 A text input to the model.

1956 

1957 - `ResponseInputText object { text, type }`

1958 

1959 A text input to the model.

1960 

1961 - `text: string`

1962 

1963 The text input to the model.

1964 

1965 - `type: "input_text"`

1966 

1967 The type of the input item. Always `input_text`.

1968 

1969 - `"input_text"`

1970 

1971 - `OutputText object { text, type }`

1972 

1973 A text output from the model.

1974 

1975 - `text: string`

1976 

1977 The text output from the model.

1978 

1979 - `type: "output_text"`

1980 

1981 The type of the output text. Always `output_text`.

1982 

1983 - `"output_text"`

1984 

1985 - `InputImage object { image_url, type, detail }`

1986 

1987 An image input block used within EvalItem content arrays.

1988 

1989 - `image_url: string`

1990 

1991 The URL of the image input.

1992 

1993 - `type: "input_image"`

1994 

1995 The type of the image input. Always `input_image`.

1996