ruby/resources/fine_tuning/subresources/alpha/index.md 2026-05-18 22:01 UTC to 2026-05-19 06:34 UTC

1# Alpha

2 

3# Graders

4 

5## Run grader

6 

7`fine_tuning.alpha.graders.run(**kwargs) -> GraderRunResponse`

8 

9**post** `/fine_tuning/alpha/graders/run`

10 

11Run a grader.

12 

13### Parameters

14 

15- `grader: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more`

16 

17 The grader used for the fine-tuning job.

18 

19 - `class StringCheckGrader`

20 

21 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

22 

23 - `input: String`

24 

25 The input text. This may include template strings.

26 

27 - `name: String`

28 

29 The name of the grader.

30 

31 - `operation: :eq | :ne | :like | :ilike`

32 

33 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

34 

35 - `:eq`

36 

37 - `:ne`

38 

39 - `:like`

40 

41 - `:ilike`

42 

43 - `reference: String`

44 

45 The reference text. This may include template strings.

46 

47 - `type: :string_check`

48 

49 The object type, which is always `string_check`.

50 

51 - `:string_check`

52 

53 - `class TextSimilarityGrader`

54 

55 A TextSimilarityGrader object which grades text based on similarity metrics.

56 

57 - `evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more`

58 

59 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

60 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

61 or `rouge_l`.

62 

63 - `:cosine`

64 

65 - `:fuzzy_match`

66 

67 - `:bleu`

68 

69 - `:gleu`

70 

71 - `:meteor`

72 

73 - `:rouge_1`

74 

75 - `:rouge_2`

76 

77 - `:rouge_3`

78 

79 - `:rouge_4`

80 

81 - `:rouge_5`

82 

83 - `:rouge_l`

84 

85 - `input: String`

86 

87 The text being graded.

88 

89 - `name: String`

90 

91 The name of the grader.

92 

93 - `reference: String`

94 

95 The text being graded against.

96 

97 - `type: :text_similarity`

98 

99 The type of grader.

100 

101 - `:text_similarity`

102 

103 - `class PythonGrader`

104 

105 A PythonGrader object that runs a python script on the input.

106 

107 - `name: String`

108 

109 The name of the grader.

110 

111 - `source: String`

112 

113 The source code of the python script.

114 

115 - `type: :python`

116 

117 The object type, which is always `python`.

118 

119 - `:python`

120 

121 - `image_tag: String`

122 

123 The image tag to use for the python script.

124 

125 - `class ScoreModelGrader`

126 

127 A ScoreModelGrader object that uses a model to assign a score to the input.

128 

129 - `input: Array[Input{ content, role, type}]`

130 

131 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

132 

133 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`

134 

135 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

136 

137 - `String = String`

138 

139 A text input to the model.

140 

141 - `class ResponseInputText`

142 

143 A text input to the model.

144 

145 - `text: String`

146 

147 The text input to the model.

148 

149 - `type: :input_text`

150 

151 The type of the input item. Always `input_text`.

152 

153 - `:input_text`

154 

155 - `class OutputText`

156 

157 A text output from the model.

158 

159 - `text: String`

160 

161 The text output from the model.

162 

163 - `type: :output_text`

164 

165 The type of the output text. Always `output_text`.

166 

167 - `:output_text`

168 

169 - `class InputImage`

170 

171 An image input block used within EvalItem content arrays.

172 

173 - `image_url: String`

174 

175 The URL of the image input.

176 

177 - `type: :input_image`

178 

179 The type of the image input. Always `input_image`.

180 

181 - `:input_image`

182 

183 - `detail: String`

184 

185 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

186 

187 - `class ResponseInputAudio`

188 

189 An audio input to the model.

190 

191 - `input_audio: InputAudio{ data, format_}`

192 

193 - `data: String`

194 

195 Base64-encoded audio data.

196 

197 - `format_: :mp3 | :wav`

198 

199 The format of the audio data. Currently supported formats are `mp3` and

200 `wav`.

201 

202 - `:mp3`

203 

204 - `:wav`

205 

206 - `type: :input_audio`

207 

208 The type of the input item. Always `input_audio`.

209 

210 - `:input_audio`

211 

212 - `GraderInputs = Array[GraderInputItem]`

213 

214 A list of inputs, each of which may be either an input text, output text, input

215 image, or input audio object.

216 

217 - `String = String`

218 

219 A text input to the model.

220 

221 - `class ResponseInputText`

222 

223 A text input to the model.

224 

225 - `class OutputText`

226 

227 A text output from the model.

228 

229 - `text: String`

230 

231 The text output from the model.

232 

233 - `type: :output_text`

234 

235 The type of the output text. Always `output_text`.

236 

237 - `:output_text`

238 

239 - `class InputImage`

240 

241 An image input block used within EvalItem content arrays.

242 

243 - `image_url: String`

244 

245 The URL of the image input.

246 

247 - `type: :input_image`

248 

249 The type of the image input. Always `input_image`.

250 

251 - `:input_image`

252 

253 - `detail: String`

254 

255 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

256 

257 - `class ResponseInputAudio`

258 

259 An audio input to the model.

260 

261 - `role: :user | :assistant | :system | :developer`

262 

263 The role of the message input. One of `user`, `assistant`, `system`, or

264 `developer`.

265 

266 - `:user`

267 

268 - `:assistant`

269 

270 - `:system`

271 

272 - `:developer`

273 

274 - `type: :message`

275 

276 The type of the message input. Always `message`.

277 

278 - `:message`

279 

280 - `model: String`

281 

282 The model to use for the evaluation.

283 

284 - `name: String`

285 

286 The name of the grader.

287 

288 - `type: :score_model`

289 

290 The object type, which is always `score_model`.

291 

292 - `:score_model`

293 

294 - `range: Array[Float]`

295 

296 The range of the score. Defaults to `[0, 1]`.

297 

298 - `sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}`

299 

300 The sampling parameters for the model.

301 

302 - `max_completions_tokens: Integer`

303 

304 The maximum number of tokens the grader model may generate in its response.

305 

306 - `reasoning_effort: ReasoningEffort`

307 

308 Constrains effort on reasoning for

309 [reasoning models](https://platform.openai.com/docs/guides/reasoning).

310 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing

311 reasoning effort can result in faster responses and fewer tokens used

312 on reasoning in a response.

313 

314 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.

315 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.

316 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.

317 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.

318 

319 - `:none`

320 

321 - `:minimal`

322 

323 - `:low`

324 

325 - `:medium`

326 

327 - `:high`

328 

329 - `:xhigh`

330 

331 - `seed: Integer`

332 

333 A seed value to initialize the randomness, during sampling.

334 

335 - `temperature: Float`

336 

337 A higher temperature increases randomness in the outputs.

338 

339 - `top_p: Float`

340 

341 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

342 

343 - `class MultiGrader`

344 

345 A MultiGrader object combines the output of multiple graders to produce a single score.

346 

347 - `calculate_output: String`

348 

349 A formula to calculate the output based on grader results.

350 

351 - `graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more`

352 

353 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

354 

355 - `class StringCheckGrader`

356 

357 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

358 

359 - `class TextSimilarityGrader`

360 

361 A TextSimilarityGrader object which grades text based on similarity metrics.

362 

363 - `class PythonGrader`

364 

365 A PythonGrader object that runs a python script on the input.

366 

367 - `class ScoreModelGrader`

368 

369 A ScoreModelGrader object that uses a model to assign a score to the input.

370 

371 - `class LabelModelGrader`

372 

373 A LabelModelGrader object which uses a model to assign labels to each item

374 in the evaluation.

375 

376 - `input: Array[Input{ content, role, type}]`

377 

378 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`

379 

380 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

381 

382 - `String = String`

383 

384 A text input to the model.

385 

386 - `class ResponseInputText`

387 

388 A text input to the model.

389 

390 - `class OutputText`

391 

392 A text output from the model.

393 

394 - `text: String`

395 

396 The text output from the model.

397 

398 - `type: :output_text`

399 

400 The type of the output text. Always `output_text`.

401 

402 - `:output_text`

403 

404 - `class InputImage`

405 

406 An image input block used within EvalItem content arrays.

407 

408 - `image_url: String`

409 

410 The URL of the image input.

411 

412 - `type: :input_image`

413 

414 The type of the image input. Always `input_image`.

415 

416 - `:input_image`

417 

418 - `detail: String`

419 

420 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

421 

422 - `class ResponseInputAudio`

423 

424 An audio input to the model.

425 

426 - `GraderInputs = Array[GraderInputItem]`

427 

428 A list of inputs, each of which may be either an input text, output text, input

429 image, or input audio object.

430 

431 - `role: :user | :assistant | :system | :developer`

432 

433 The role of the message input. One of `user`, `assistant`, `system`, or

434 `developer`.

435 

436 - `:user`

437 

438 - `:assistant`

439 

440 - `:system`

441 

442 - `:developer`

443 

444 - `type: :message`

445 

446 The type of the message input. Always `message`.

447 

448 - `:message`

449 

450 - `labels: Array[String]`

451 

452 The labels to assign to each item in the evaluation.

453 

454 - `model: String`

455 

456 The model to use for the evaluation. Must support structured outputs.

457 

458 - `name: String`

459 

460 The name of the grader.

461 

462 - `passing_labels: Array[String]`

463 

464 The labels that indicate a passing result. Must be a subset of labels.

465 

466 - `type: :label_model`

467 

468 The object type, which is always `label_model`.

469 

470 - `:label_model`

471 

472 - `name: String`

473 

474 The name of the grader.

475 

476 - `type: :multi`

477 

478 The object type, which is always `multi`.

479 

480 - `:multi`

481 

482- `model_sample: String`

483 

484 The model sample to be evaluated. This value will be used to populate

485 the `sample` namespace. See [the guide](https://platform.openai.com/docs/guides/graders) for more details.

486 The `output_json` variable will be populated if the model sample is a

487 valid JSON string.

488 

489- `item: untyped`

490 

491 The dataset item provided to the grader. This will be used to populate

492 the `item` namespace. See [the guide](https://platform.openai.com/docs/guides/graders) for more details.

493 

494### Returns

495 

496- `class GraderRunResponse`

497 

498 - `metadata: Metadata{ errors, execution_time, name, 4 more}`

499 

500 - `errors: Errors{ formula_parse_error, invalid_variable_error, model_grader_parse_error, 11 more}`

501 

502 - `formula_parse_error: bool`

503 

504 - `invalid_variable_error: bool`

505 

506 - `model_grader_parse_error: bool`

507 

508 - `model_grader_refusal_error: bool`

509 

510 - `model_grader_server_error: bool`

511 

512 - `model_grader_server_error_details: String`

513 

514 - `other_error: bool`

515 

516 - `python_grader_runtime_error: bool`

517 

518 - `python_grader_runtime_error_details: String`

519 

520 - `python_grader_server_error: bool`

521 

522 - `python_grader_server_error_type: String`

523 

524 - `sample_parse_error: bool`

525 

526 - `truncated_observation_error: bool`

527 

528 - `unresponsive_reward_error: bool`

529 

530 - `execution_time: Float`

531 

532 - `name: String`

533 

534 - `sampled_model_name: String`

535 

536 - `scores: Hash[Symbol, untyped]`

537 

538 - `token_usage: Integer`

539 

540 - `type: String`

541 

542 - `model_grader_token_usage_per_model: Hash[Symbol, untyped]`

543 

544 - `reward: Float`

545 

546 - `sub_rewards: Hash[Symbol, untyped]`

547 

548### Example

549 

550```ruby

551require "openai"

552 

553openai = OpenAI::Client.new(api_key: "My API Key")

554 

555response = openai.fine_tuning.alpha.graders.run(

556 grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check},

557 model_sample: "model_sample"

558)

559 

560puts(response)

561```

562 

563#### Response

564 

565```json

566{

567 "metadata": {

568 "errors": {

569 "formula_parse_error": true,

570 "invalid_variable_error": true,

571 "model_grader_parse_error": true,

572 "model_grader_refusal_error": true,

573 "model_grader_server_error": true,

574 "model_grader_server_error_details": "model_grader_server_error_details",

575 "other_error": true,

576 "python_grader_runtime_error": true,

577 "python_grader_runtime_error_details": "python_grader_runtime_error_details",

578 "python_grader_server_error": true,

579 "python_grader_server_error_type": "python_grader_server_error_type",

580 "sample_parse_error": true,

581 "truncated_observation_error": true,

582 "unresponsive_reward_error": true

583 },

584 "execution_time": 0,

585 "name": "name",

586 "sampled_model_name": "sampled_model_name",

587 "scores": {

588 "foo": "bar"

589 },

590 "token_usage": 0,

591 "type": "type"

592 },

593 "model_grader_token_usage_per_model": {

594 "foo": "bar"

595 },

596 "reward": 0,

597 "sub_rewards": {

598 "foo": "bar"

599 }

600}

601```

602 

603## Validate grader

604 

605`fine_tuning.alpha.graders.validate(**kwargs) -> GraderValidateResponse`

606 

607**post** `/fine_tuning/alpha/graders/validate`

608 

609Validate a grader.

610 

611### Parameters

612 

613- `grader: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more`

614 

615 The grader used for the fine-tuning job.

616 

617 - `class StringCheckGrader`

618 

619 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

620 

621 - `input: String`

622 

623 The input text. This may include template strings.

624 

625 - `name: String`

626 

627 The name of the grader.

628 

629 - `operation: :eq | :ne | :like | :ilike`

630 

631 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

632 

633 - `:eq`

634 

635 - `:ne`

636 

637 - `:like`

638 

639 - `:ilike`

640 

641 - `reference: String`

642 

643 The reference text. This may include template strings.

644 

645 - `type: :string_check`

646 

647 The object type, which is always `string_check`.

648 

649 - `:string_check`

650 

651 - `class TextSimilarityGrader`

652 

653 A TextSimilarityGrader object which grades text based on similarity metrics.

654 

655 - `evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more`

656 

657 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

658 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

659 or `rouge_l`.

660 

661 - `:cosine`

662 

663 - `:fuzzy_match`

664 

665 - `:bleu`

666 

667 - `:gleu`

668 

669 - `:meteor`

670 

671 - `:rouge_1`

672 

673 - `:rouge_2`

674 

675 - `:rouge_3`

676 

677 - `:rouge_4`

678 

679 - `:rouge_5`

680 

681 - `:rouge_l`

682 

683 - `input: String`

684 

685 The text being graded.

686 

687 - `name: String`

688 

689 The name of the grader.

690 

691 - `reference: String`

692 

693 The text being graded against.

694 

695 - `type: :text_similarity`

696 

697 The type of grader.

698 

699 - `:text_similarity`

700 

701 - `class PythonGrader`

702 

703 A PythonGrader object that runs a python script on the input.

704 

705 - `name: String`

706 

707 The name of the grader.

708 

709 - `source: String`

710 

711 The source code of the python script.

712 

713 - `type: :python`

714 

715 The object type, which is always `python`.

716 

717 - `:python`

718 

719 - `image_tag: String`

720 

721 The image tag to use for the python script.

722 

723 - `class ScoreModelGrader`

724 

725 A ScoreModelGrader object that uses a model to assign a score to the input.

726 

727 - `input: Array[Input{ content, role, type}]`

728 

729 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

730 

731 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`

732 

733 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

734 

735 - `String = String`

736 

737 A text input to the model.

738 

739 - `class ResponseInputText`

740 

741 A text input to the model.

742 

743 - `text: String`

744 

745 The text input to the model.

746 

747 - `type: :input_text`

748 

749 The type of the input item. Always `input_text`.

750 

751 - `:input_text`

752 

753 - `class OutputText`

754 

755 A text output from the model.

756 

757 - `text: String`

758 

759 The text output from the model.

760 

761 - `type: :output_text`

762 

763 The type of the output text. Always `output_text`.

764 

765 - `:output_text`

766 

767 - `class InputImage`

768 

769 An image input block used within EvalItem content arrays.

770 

771 - `image_url: String`

772 

773 The URL of the image input.

774 

775 - `type: :input_image`

776 

777 The type of the image input. Always `input_image`.

778 

779 - `:input_image`

780 

781 - `detail: String`

782 

783 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

784 

785 - `class ResponseInputAudio`

786 

787 An audio input to the model.

788 

789 - `input_audio: InputAudio{ data, format_}`

790 

791 - `data: String`

792 

793 Base64-encoded audio data.

794 

795 - `format_: :mp3 | :wav`

796 

797 The format of the audio data. Currently supported formats are `mp3` and

798 `wav`.

799 

800 - `:mp3`

801 

802 - `:wav`

803 

804 - `type: :input_audio`

805 

806 The type of the input item. Always `input_audio`.

807 

808 - `:input_audio`

809 

810 - `GraderInputs = Array[GraderInputItem]`

811 

812 A list of inputs, each of which may be either an input text, output text, input

813 image, or input audio object.

814 

815 - `String = String`

816 

817 A text input to the model.

818 

819 - `class ResponseInputText`

820 

821 A text input to the model.

822 

823 - `class OutputText`

824 

825 A text output from the model.

826 

827 - `text: String`

828 

829 The text output from the model.

830 

831 - `type: :output_text`

832 

833 The type of the output text. Always `output_text`.

834 

835 - `:output_text`

836 

837 - `class InputImage`

838 

839 An image input block used within EvalItem content arrays.

840 

841 - `image_url: String`

842 

843 The URL of the image input.

844 

845 - `type: :input_image`

846 

847 The type of the image input. Always `input_image`.

848 

849 - `:input_image`

850 

851 - `detail: String`

852 

853 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

854 

855 - `class ResponseInputAudio`

856 

857 An audio input to the model.

858 

859 - `role: :user | :assistant | :system | :developer`

860 

861 The role of the message input. One of `user`, `assistant`, `system`, or

862 `developer`.

863 

864 - `:user`

865 

866 - `:assistant`

867 

868 - `:system`

869 

870 - `:developer`

871 

872 - `type: :message`

873 

874 The type of the message input. Always `message`.

875 

876 - `:message`

877 

878 - `model: String`

879 

880 The model to use for the evaluation.

881 

882 - `name: String`

883 

884 The name of the grader.

885 

886 - `type: :score_model`

887 

888 The object type, which is always `score_model`.

889 

890 - `:score_model`

891 

892 - `range: Array[Float]`

893 

894 The range of the score. Defaults to `[0, 1]`.

895 

896 - `sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}`

897 

898 The sampling parameters for the model.

899 

900 - `max_completions_tokens: Integer`

901 

902 The maximum number of tokens the grader model may generate in its response.

903 

904 - `reasoning_effort: ReasoningEffort`

905 

906 Constrains effort on reasoning for

907 [reasoning models](https://platform.openai.com/docs/guides/reasoning).

908 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing

909 reasoning effort can result in faster responses and fewer tokens used

910 on reasoning in a response.

911 

912 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.

913 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.

914 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.

915 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.

916 

917 - `:none`

918 

919 - `:minimal`

920 

921 - `:low`

922 

923 - `:medium`

924 

925 - `:high`

926 

927 - `:xhigh`

928 

929 - `seed: Integer`

930 

931 A seed value to initialize the randomness, during sampling.

932 

933 - `temperature: Float`

934 

935 A higher temperature increases randomness in the outputs.

936 

937 - `top_p: Float`

938 

939 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

940 

941 - `class MultiGrader`

942 

943 A MultiGrader object combines the output of multiple graders to produce a single score.

944 

945 - `calculate_output: String`

946 

947 A formula to calculate the output based on grader results.

948 

949 - `graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more`

950 

951 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

952 

953 - `class StringCheckGrader`

954 

955 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

956 

957 - `class TextSimilarityGrader`

958 

959 A TextSimilarityGrader object which grades text based on similarity metrics.

960 

961 - `class PythonGrader`

962 

963 A PythonGrader object that runs a python script on the input.

964 

965 - `class ScoreModelGrader`

966 

967 A ScoreModelGrader object that uses a model to assign a score to the input.

968 

969 - `class LabelModelGrader`

970 

971 A LabelModelGrader object which uses a model to assign labels to each item

972 in the evaluation.

973 

974 - `input: Array[Input{ content, role, type}]`

975 

976 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`

977 

978 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

979 

980 - `String = String`

981 

982 A text input to the model.

983 

984 - `class ResponseInputText`

985 

986 A text input to the model.

987 

988 - `class OutputText`

989 

990 A text output from the model.

991 

992 - `text: String`

993 

994 The text output from the model.

995 

996 - `type: :output_text`

997 

998 The type of the output text. Always `output_text`.

999 

1000 - `:output_text`

1001 

1002 - `class InputImage`

1003 

1004 An image input block used within EvalItem content arrays.

1005 

1006 - `image_url: String`

1007 

1008 The URL of the image input.

1009 

1010 - `type: :input_image`

1011 

1012 The type of the image input. Always `input_image`.

1013 

1014 - `:input_image`

1015 

1016 - `detail: String`

1017 

1018 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1019 

1020 - `class ResponseInputAudio`

1021 

1022 An audio input to the model.

1023 

1024 - `GraderInputs = Array[GraderInputItem]`

1025 

1026 A list of inputs, each of which may be either an input text, output text, input

1027 image, or input audio object.

1028 

1029 - `role: :user | :assistant | :system | :developer`

1030 

1031 The role of the message input. One of `user`, `assistant`, `system`, or

1032 `developer`.

1033 

1034 - `:user`

1035 

1036 - `:assistant`

1037 

1038 - `:system`

1039 

1040 - `:developer`

1041 

1042 - `type: :message`

1043 

1044 The type of the message input. Always `message`.

1045 

1046 - `:message`

1047 

1048 - `labels: Array[String]`

1049 

1050 The labels to assign to each item in the evaluation.

1051 

1052 - `model: String`

1053 

1054 The model to use for the evaluation. Must support structured outputs.

1055 

1056 - `name: String`

1057 

1058 The name of the grader.

1059 

1060 - `passing_labels: Array[String]`

1061 

1062 The labels that indicate a passing result. Must be a subset of labels.

1063 

1064 - `type: :label_model`

1065 

1066 The object type, which is always `label_model`.

1067 

1068 - `:label_model`

1069 

1070 - `name: String`

1071 

1072 The name of the grader.

1073 

1074 - `type: :multi`

1075 

1076 The object type, which is always `multi`.

1077 

1078 - `:multi`

1079 

1080### Returns

1081 

1082- `class GraderValidateResponse`

1083 

1084 - `grader: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more`

1085 

1086 The grader used for the fine-tuning job.

1087 

1088 - `class StringCheckGrader`

1089 

1090 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1091 

1092 - `input: String`

1093 

1094 The input text. This may include template strings.

1095 

1096 - `name: String`

1097 

1098 The name of the grader.

1099 

1100 - `operation: :eq | :ne | :like | :ilike`

1101 

1102 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

1103 

1104 - `:eq`

1105 

1106 - `:ne`

1107 

1108 - `:like`

1109 

1110 - `:ilike`

1111 

1112 - `reference: String`

1113 

1114 The reference text. This may include template strings.

1115 

1116 - `type: :string_check`

1117 

1118 The object type, which is always `string_check`.

1119 

1120 - `:string_check`

1121 

1122 - `class TextSimilarityGrader`

1123 

1124 A TextSimilarityGrader object which grades text based on similarity metrics.

1125 

1126 - `evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more`

1127 

1128 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

1129 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

1130 or `rouge_l`.

1131 

1132 - `:cosine`

1133 

1134 - `:fuzzy_match`

1135 

1136 - `:bleu`

1137 

1138 - `:gleu`

1139 

1140 - `:meteor`

1141 

1142 - `:rouge_1`

1143 

1144 - `:rouge_2`

1145 

1146 - `:rouge_3`

1147 

1148 - `:rouge_4`

1149 

1150 - `:rouge_5`

1151 

1152 - `:rouge_l`

1153 

1154 - `input: String`

1155 

1156 The text being graded.

1157 

1158 - `name: String`

1159 

1160 The name of the grader.

1161 

1162 - `reference: String`

1163 

1164 The text being graded against.

1165 

1166 - `type: :text_similarity`

1167 

1168 The type of grader.

1169 

1170 - `:text_similarity`

1171 

1172 - `class PythonGrader`

1173 

1174 A PythonGrader object that runs a python script on the input.

1175 

1176 - `name: String`

1177 

1178 The name of the grader.

1179 

1180 - `source: String`

1181 

1182 The source code of the python script.

1183 

1184 - `type: :python`

1185 

1186 The object type, which is always `python`.

1187 

1188 - `:python`

1189 

1190 - `image_tag: String`

1191 

1192 The image tag to use for the python script.

1193 

1194 - `class ScoreModelGrader`

1195 

1196 A ScoreModelGrader object that uses a model to assign a score to the input.

1197 

1198 - `input: Array[Input{ content, role, type}]`

1199 

1200 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

1201 

1202 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`

1203 

1204 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1205 

1206 - `String = String`

1207 

1208 A text input to the model.

1209 

1210 - `class ResponseInputText`

1211 

1212 A text input to the model.

1213 

1214 - `text: String`

1215 

1216 The text input to the model.

1217 

1218 - `type: :input_text`

1219 

1220 The type of the input item. Always `input_text`.

1221 

1222 - `:input_text`

1223 

1224 - `class OutputText`

1225 

1226 A text output from the model.

1227 

1228 - `text: String`

1229 

1230 The text output from the model.

1231 

1232 - `type: :output_text`

1233 

1234 The type of the output text. Always `output_text`.

1235 

1236 - `:output_text`

1237 

1238 - `class InputImage`

1239 

1240 An image input block used within EvalItem content arrays.

1241 

1242 - `image_url: String`

1243 

1244 The URL of the image input.

1245 

1246 - `type: :input_image`

1247 

1248 The type of the image input. Always `input_image`.

1249 

1250 - `:input_image`

1251 

1252 - `detail: String`

1253 

1254 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1255 

1256 - `class ResponseInputAudio`

1257 

1258 An audio input to the model.

1259 

1260 - `input_audio: InputAudio{ data, format_}`

1261 

1262 - `data: String`

1263 

1264 Base64-encoded audio data.

1265 

1266 - `format_: :mp3 | :wav`

1267 

1268 The format of the audio data. Currently supported formats are `mp3` and

1269 `wav`.

1270 

1271 - `:mp3`

1272 

1273 - `:wav`

1274 

1275 - `type: :input_audio`

1276 

1277 The type of the input item. Always `input_audio`.

1278 

1279 - `:input_audio`

1280 

1281 - `GraderInputs = Array[GraderInputItem]`

1282 

1283 A list of inputs, each of which may be either an input text, output text, input

1284 image, or input audio object.

1285 

1286 - `String = String`

1287 

1288 A text input to the model.

1289 

1290 - `class ResponseInputText`

1291 

1292 A text input to the model.

1293 

1294 - `class OutputText`

1295 

1296 A text output from the model.

1297 

1298 - `text: String`

1299 

1300 The text output from the model.

1301 

1302 - `type: :output_text`

1303 

1304 The type of the output text. Always `output_text`.

1305 

1306 - `:output_text`

1307 

1308 - `class InputImage`

1309 

1310 An image input block used within EvalItem content arrays.

1311 

1312 - `image_url: String`

1313 

1314 The URL of the image input.

1315 

1316 - `type: :input_image`

1317 

1318 The type of the image input. Always `input_image`.

1319 

1320 - `:input_image`

1321 

1322 - `detail: String`

1323 

1324 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1325 

1326 - `class ResponseInputAudio`

1327 

1328 An audio input to the model.

1329 

1330 - `role: :user | :assistant | :system | :developer`

1331 

1332 The role of the message input. One of `user`, `assistant`, `system`, or

1333 `developer`.

1334 

1335 - `:user`

1336 

1337 - `:assistant`

1338 

1339 - `:system`

1340 

1341 - `:developer`

1342 

1343 - `type: :message`

1344 

1345 The type of the message input. Always `message`.

1346 

1347 - `:message`

1348 

1349 - `model: String`

1350 

1351 The model to use for the evaluation.

1352 

1353 - `name: String`

1354 

1355 The name of the grader.

1356 

1357 - `type: :score_model`

1358 

1359 The object type, which is always `score_model`.

1360 

1361 - `:score_model`

1362 

1363 - `range: Array[Float]`

1364 

1365 The range of the score. Defaults to `[0, 1]`.

1366 

1367 - `sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}`

1368 

1369 The sampling parameters for the model.

1370 

1371 - `max_completions_tokens: Integer`

1372 

1373 The maximum number of tokens the grader model may generate in its response.

1374 

1375 - `reasoning_effort: ReasoningEffort`

1376 

1377 Constrains effort on reasoning for

1378 [reasoning models](https://platform.openai.com/docs/guides/reasoning).

1379 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing

1380 reasoning effort can result in faster responses and fewer tokens used

1381 on reasoning in a response.

1382 

1383 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.

1384 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.

1385 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.

1386 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.

1387 

1388 - `:none`

1389 

1390 - `:minimal`

1391 

1392 - `:low`

1393 

1394 - `:medium`

1395 

1396 - `:high`

1397 

1398 - `:xhigh`

1399 

1400 - `seed: Integer`

1401 

1402 A seed value to initialize the randomness, during sampling.

1403 

1404 - `temperature: Float`

1405 

1406 A higher temperature increases randomness in the outputs.

1407 

1408 - `top_p: Float`

1409 

1410 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

1411 

1412 - `class MultiGrader`

1413 

1414 A MultiGrader object combines the output of multiple graders to produce a single score.

1415 

1416 - `calculate_output: String`

1417 

1418 A formula to calculate the output based on grader results.

1419 

1420 - `graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more`

1421 

1422 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1423 

1424 - `class StringCheckGrader`

1425 

1426 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1427 

1428 - `class TextSimilarityGrader`

1429 

1430 A TextSimilarityGrader object which grades text based on similarity metrics.

1431 

1432 - `class PythonGrader`

1433 

1434 A PythonGrader object that runs a python script on the input.

1435 

1436 - `class ScoreModelGrader`

1437 

1438 A ScoreModelGrader object that uses a model to assign a score to the input.

1439 

1440 - `class LabelModelGrader`

1441 

1442 A LabelModelGrader object which uses a model to assign labels to each item

1443 in the evaluation.

1444 

1445 - `input: Array[Input{ content, role, type}]`

1446 

1447 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`

1448 

1449 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1450 

1451 - `String = String`

1452 

1453 A text input to the model.

1454 

1455 - `class ResponseInputText`

1456 

1457 A text input to the model.

1458 

1459 - `class OutputText`

1460 

1461 A text output from the model.

1462 

1463 - `text: String`

1464 

1465 The text output from the model.

1466 

1467 - `type: :output_text`

1468 

1469 The type of the output text. Always `output_text`.

1470 

1471 - `:output_text`

1472 

1473 - `class InputImage`

1474 

1475 An image input block used within EvalItem content arrays.

1476 

1477 - `image_url: String`

1478 

1479 The URL of the image input.

1480 

1481 - `type: :input_image`

1482 

1483 The type of the image input. Always `input_image`.

1484 

1485 - `:input_image`

1486 

1487 - `detail: String`

1488 

1489 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1490 

1491 - `class ResponseInputAudio`

1492 

1493 An audio input to the model.

1494 

1495 - `GraderInputs = Array[GraderInputItem]`

1496 

1497 A list of inputs, each of which may be either an input text, output text, input

1498 image, or input audio object.

1499 

1500 - `role: :user | :assistant | :system | :developer`

1501 

1502 The role of the message input. One of `user`, `assistant`, `system`, or

1503 `developer`.

1504 

1505 - `:user`

1506 

1507 - `:assistant`

1508 

1509 - `:system`

1510 

1511 - `:developer`

1512 

1513 - `type: :message`

1514 

1515 The type of the message input. Always `message`.

1516 

1517 - `:message`

1518 

1519 - `labels: Array[String]`

1520 

1521 The labels to assign to each item in the evaluation.

1522 

1523 - `model: String`

1524 

1525 The model to use for the evaluation. Must support structured outputs.

1526 

1527 - `name: String`

1528 

1529 The name of the grader.

1530 

1531 - `passing_labels: Array[String]`

1532 

1533 The labels that indicate a passing result. Must be a subset of labels.

1534 

1535 - `type: :label_model`

1536 

1537 The object type, which is always `label_model`.

1538 

1539 - `:label_model`

1540 

1541 - `name: String`

1542 

1543 The name of the grader.

1544 

1545 - `type: :multi`

1546 

1547 The object type, which is always `multi`.

1548 

1549 - `:multi`

1550 

1551### Example

1552 

1553```ruby

1554require "openai"

1555 

1556openai = OpenAI::Client.new(api_key: "My API Key")

1557 

1558response = openai.fine_tuning.alpha.graders.validate(

1559 grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check}

1560)

1561 

1562puts(response)

1563```

1564 

1565#### Response

1566 

1567```json

1568{

1569 "grader": {

1570 "input": "input",

1571 "name": "name",

1572 "operation": "eq",

1573 "reference": "reference",

1574 "type": "string_check"

1575 }

1576}

1577```

1578 

1579## Domain Types

1580 

1581### Grader Run Response

1582 

1583- `class GraderRunResponse`

1584 

1585 - `metadata: Metadata{ errors, execution_time, name, 4 more}`

1586 

1587 - `errors: Errors{ formula_parse_error, invalid_variable_error, model_grader_parse_error, 11 more}`

1588 

1589 - `formula_parse_error: bool`

1590 

1591 - `invalid_variable_error: bool`

1592 

1593 - `model_grader_parse_error: bool`

1594 

1595 - `model_grader_refusal_error: bool`

1596 

1597 - `model_grader_server_error: bool`

1598 

1599 - `model_grader_server_error_details: String`

1600 

1601 - `other_error: bool`

1602 

1603 - `python_grader_runtime_error: bool`

1604 

1605 - `python_grader_runtime_error_details: String`

1606 

1607 - `python_grader_server_error: bool`

1608 

1609 - `python_grader_server_error_type: String`

1610 

1611 - `sample_parse_error: bool`

1612 

1613 - `truncated_observation_error: bool`

1614 

1615 - `unresponsive_reward_error: bool`

1616 

1617 - `execution_time: Float`

1618 

1619 - `name: String`

1620 

1621 - `sampled_model_name: String`

1622 

1623 - `scores: Hash[Symbol, untyped]`

1624 

1625 - `token_usage: Integer`

1626 

1627 - `type: String`

1628 

1629 - `model_grader_token_usage_per_model: Hash[Symbol, untyped]`

1630 

1631 - `reward: Float`

1632 

1633 - `sub_rewards: Hash[Symbol, untyped]`

1634 

1635### Grader Validate Response

1636 

1637- `class GraderValidateResponse`

1638 

1639 - `grader: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more`

1640 

1641 The grader used for the fine-tuning job.

1642 

1643 - `class StringCheckGrader`

1644 

1645 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1646 

1647 - `input: String`

1648 

1649 The input text. This may include template strings.

1650 

1651 - `name: String`

1652 

1653 The name of the grader.

1654 

1655 - `operation: :eq | :ne | :like | :ilike`

1656 

1657 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

1658 

1659 - `:eq`

1660 

1661 - `:ne`

1662 

1663 - `:like`

1664 

1665 - `:ilike`

1666 

1667 - `reference: String`

1668 

1669 The reference text. This may include template strings.

1670 

1671 - `type: :string_check`

1672 

1673 The object type, which is always `string_check`.

1674 

1675 - `:string_check`

1676 

1677 - `class TextSimilarityGrader`

1678 

1679 A TextSimilarityGrader object which grades text based on similarity metrics.

1680 

1681 - `evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more`

1682 

1683 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

1684 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

1685 or `rouge_l`.

1686 

1687 - `:cosine`

1688 

1689 - `:fuzzy_match`

1690 

1691 - `:bleu`

1692 

1693 - `:gleu`

1694 

1695 - `:meteor`

1696 

1697 - `:rouge_1`

1698 

1699 - `:rouge_2`

1700 

1701 - `:rouge_3`

1702 

1703 - `:rouge_4`

1704 

1705 - `:rouge_5`

1706 

1707 - `:rouge_l`

1708 

1709 - `input: String`

1710 

1711 The text being graded.

1712 

1713 - `name: String`

1714 

1715 The name of the grader.

1716 

1717 - `reference: String`

1718 

1719 The text being graded against.

1720 

1721 - `type: :text_similarity`

1722 

1723 The type of grader.

1724 

1725 - `:text_similarity`

1726 

1727 - `class PythonGrader`

1728 

1729 A PythonGrader object that runs a python script on the input.

1730 

1731 - `name: String`

1732 

1733 The name of the grader.

1734 

1735 - `source: String`

1736 

1737 The source code of the python script.

1738 

1739 - `type: :python`

1740 

1741 The object type, which is always `python`.

1742 

1743 - `:python`

1744 

1745 - `image_tag: String`

1746 

1747 The image tag to use for the python script.

1748 

1749 - `class ScoreModelGrader`

1750 

1751 A ScoreModelGrader object that uses a model to assign a score to the input.

1752 

1753 - `input: Array[Input{ content, role, type}]`

1754 

1755 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

1756 

1757 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`

1758 

1759 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1760 

1761 - `String = String`

1762 

1763 A text input to the model.

1764 

1765 - `class ResponseInputText`

1766 

1767 A text input to the model.

1768 

1769 - `text: String`

1770 

1771 The text input to the model.

1772 

1773 - `type: :input_text`

1774 

1775 The type of the input item. Always `input_text`.

1776 

1777 - `:input_text`

1778 

1779 - `class OutputText`

1780 

1781 A text output from the model.

1782 

1783 - `text: String`

1784 

1785 The text output from the model.

1786 

1787 - `type: :output_text`

1788 

1789 The type of the output text. Always `output_text`.

1790 

1791 - `:output_text`

1792 

1793 - `class InputImage`

1794 

1795 An image input block used within EvalItem content arrays.

1796 

1797 - `image_url: String`

1798 

1799 The URL of the image input.

1800 

1801 - `type: :input_image`

1802 

1803 The type of the image input. Always `input_image`.

1804 

1805 - `:input_image`

1806 

1807 - `detail: String`

1808 

1809 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1810 

1811 - `class ResponseInputAudio`

1812 

1813 An audio input to the model.

1814 

1815 - `input_audio: InputAudio{ data, format_}`

1816 

1817 - `data: String`

1818 

1819 Base64-encoded audio data.

1820 

1821 - `format_: :mp3 | :wav`

1822 

1823 The format of the audio data. Currently supported formats are `mp3` and

1824 `wav`.

1825 

1826 - `:mp3`

1827 

1828 - `:wav`

1829 

1830 - `type: :input_audio`

1831 

1832 The type of the input item. Always `input_audio`.

1833 

1834 - `:input_audio`

1835 

1836 - `GraderInputs = Array[GraderInputItem]`

1837 

1838 A list of inputs, each of which may be either an input text, output text, input

1839 image, or input audio object.

1840 

1841 - `String = String`

1842 

1843 A text input to the model.

1844 

1845 - `class ResponseInputText`

1846 

1847 A text input to the model.

1848 

1849 - `class OutputText`

1850 

1851 A text output from the model.

1852 

1853 - `text: String`

1854 

1855 The text output from the model.

1856 

1857 - `type: :output_text`

1858 

1859 The type of the output text. Always `output_text`.

1860 

1861 - `:output_text`

1862 

1863 - `class InputImage`

1864 

1865 An image input block used within EvalItem content arrays.

1866 

1867 - `image_url: String`

1868 

1869 The URL of the image input.

1870 

1871 - `type: :input_image`

1872 

1873 The type of the image input. Always `input_image`.

1874 

1875 - `:input_image`

1876 

1877 - `detail: String`

1878 

1879 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1880 

1881 - `class ResponseInputAudio`

1882 

1883 An audio input to the model.

1884 

1885 - `role: :user | :assistant | :system | :developer`

1886 

1887 The role of the message input. One of `user`, `assistant`, `system`, or

1888 `developer`.

1889 

1890 - `:user`

1891 

1892 - `:assistant`

1893 

1894 - `:system`

1895 

1896 - `:developer`

1897 

1898 - `type: :message`

1899 

1900 The type of the message input. Always `message`.

1901 

1902 - `:message`

1903 

1904 - `model: String`

1905 

1906 The model to use for the evaluation.

1907 

1908 - `name: String`

1909 

1910 The name of the grader.

1911 

1912 - `type: :score_model`

1913 

1914 The object type, which is always `score_model`.

1915 

1916 - `:score_model`

1917 

1918 - `range: Array[Float]`

1919 

1920 The range of the score. Defaults to `[0, 1]`.

1921 

1922 - `sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}`

1923 

1924 The sampling parameters for the model.

1925 

1926 - `max_completions_tokens: Integer`

1927 

1928 The maximum number of tokens the grader model may generate in its response.

1929 

1930 - `reasoning_effort: ReasoningEffort`

1931 

1932 Constrains effort on reasoning for

1933 [reasoning models](https://platform.openai.com/docs/guides/reasoning).

1934 Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing

1935 reasoning effort can result in faster responses and fewer tokens used

1936 on reasoning in a response.

1937 

1938 - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1.

1939 - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`.

1940 - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort.

1941 - `xhigh` is supported for all models after `gpt-5.1-codex-max`.

1942 

1943 - `:none`

1944 

1945 - `:minimal`

1946 

1947 - `:low`

1948 

1949 - `:medium`

1950 

1951 - `:high`

1952 

1953 - `:xhigh`

1954 

1955 - `seed: Integer`

1956 

1957 A seed value to initialize the randomness, during sampling.

1958 

1959 - `temperature: Float`

1960 

1961 A higher temperature increases randomness in the outputs.

1962 

1963 - `top_p: Float`

1964 

1965 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

1966 

1967 - `class MultiGrader`

1968 

1969 A MultiGrader object combines the output of multiple graders to produce a single score.

1970 

1971 - `calculate_output: String`

1972 

1973 A formula to calculate the output based on grader results.

1974 

1975 - `graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more`

1976 

1977 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1978 

1979 - `class StringCheckGrader`

1980 

1981 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1982 

1983 - `class TextSimilarityGrader`

1984 

1985 A TextSimilarityGrader object which grades text based on similarity metrics.

1986 

1987 - `class PythonGrader`

1988 

1989 A PythonGrader object that runs a python script on the input.

1990 

1991 - `class ScoreModelGrader`

1992 

1993 A ScoreModelGrader object that uses a model to assign a score to the input.

1994 

1995 - `class LabelModelGrader`

1996