Alpha

Graders

Run grader

fine_tuning.alpha.graders.run(**kwargs) -> GraderRunResponse

post /fine_tuning/alpha/graders/run

Run a grader.

Parameters

grader: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more

The grader used for the fine-tuning job.
- class StringCheckGrader
  
  A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
  - input: String
    
    The input text. This may include template strings.
  - name: String
    
    The name of the grader.
  - operation: :eq | :ne | :like | :ilike
    
    The string check operation to perform. One of eq, ne, like, or ilike.
    - :eq
    - :ne
    - :like
    - :ilike
  - reference: String
    
    The reference text. This may include template strings.
  - type: :string_check
    
    The object type, which is always string_check.
    - :string_check
- class TextSimilarityGrader
  
  A TextSimilarityGrader object which grades text based on similarity metrics.
  - evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more
    
    The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
    - :cosine
    - :fuzzy_match
    - :bleu
    - :gleu
    - :meteor
    - :rouge_1
    - :rouge_2
    - :rouge_3
    - :rouge_4
    - :rouge_5
    - :rouge_l
  - input: String
    
    The text being graded.
  - name: String
    
    The name of the grader.
  - reference: String
    
    The text being graded against.
  - type: :text_similarity
    
    The type of grader.
    - :text_similarity
- class PythonGrader
  
  A PythonGrader object that runs a python script on the input.
  - name: String
    
    The name of the grader.
  - source: String
    
    The source code of the python script.
  - type: :python
    
    The object type, which is always python.
    - :python
  - image_tag: String
    
    The image tag to use for the python script.
- class ScoreModelGrader
  
  A ScoreModelGrader object that uses a model to assign a score to the input.
  - input: Array[Input{ content, role, type}]
    
    The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
    - content: String | ResponseInputText | OutputText{ text, type} | 3 more
      
      Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
      - String = String
        
        A text input to the model.
      - class ResponseInputText
        
        A text input to the model.
        
        text: String
        
        The text input to the model.
        
        type: :input_text
        
        The type of the input item. Always input_text.
        
        :input_text
        
        prompt_cache_breakpoint: PromptCacheBreakpoint{ mode}
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        mode: :explicit
        
        The breakpoint mode. Always explicit.
        
        :explicit
      - class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
      - class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
      - class ResponseInputAudio
        
        An audio input to the model.
        
        input_audio: InputAudio{ data, format_}
        
        data: String
        
        Base64-encoded audio data.
        
        format_: :mp3 | :wav
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        :mp3
        
        :wav
        
        type: :input_audio
        
        The type of the input item. Always input_audio.
        
        :input_audio
      - GraderInputs = Array[GraderInputItem]
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
    - role: :user | :assistant | :system | :developer
      
      The role of the message input. One of user, assistant, system, or developer.
      - :user
      - :assistant
      - :system
      - :developer
    - type: :message
      
      The type of the message input. Always message.
      - :message
  - model: String
    
    The model to use for the evaluation.
  - name: String
    
    The name of the grader.
  - type: :score_model
    
    The object type, which is always score_model.
    - :score_model
  - range: Array[Float]
    
    The range of the score. Defaults to [0, 1].
  - sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}
    
    The sampling parameters for the model.
    - max_completions_tokens: Integer
      
      The maximum number of tokens the grader model may generate in its response.
    - reasoning_effort: ReasoningEffort
      
      Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
      - :none
      - :minimal
      - :low
      - :medium
      - :high
      - :xhigh
      - :max
    - seed: Integer
      
      A seed value to initialize the randomness, during sampling.
    - temperature: Float
      
      A higher temperature increases randomness in the outputs.
    - top_p: Float
      
      An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- class MultiGrader
  
  A MultiGrader object combines the output of multiple graders to produce a single score.
  - calculate_output: String
    
    A formula to calculate the output based on grader results.
  - graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - class StringCheckGrader
      
      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - class TextSimilarityGrader
      
      A TextSimilarityGrader object which grades text based on similarity metrics.
    - class PythonGrader
      
      A PythonGrader object that runs a python script on the input.
    - class ScoreModelGrader
      
      A ScoreModelGrader object that uses a model to assign a score to the input.
    - class LabelModelGrader
      
      A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
      - input: Array[Input{ content, role, type}]
        
        content: String | ResponseInputText | OutputText{ text, type} | 3 more
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
        
        GraderInputs = Array[GraderInputItem]
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        role: :user | :assistant | :system | :developer
        
        The role of the message input. One of user, assistant, system, or developer.
        
        :user
        
        :assistant
        
        :system
        
        :developer
        
        type: :message
        
        The type of the message input. Always message.
        
        :message
      - labels: Array[String]
        
        The labels to assign to each item in the evaluation.
      - model: String
        
        The model to use for the evaluation. Must support structured outputs.
      - name: String
        
        The name of the grader.
      - passing_labels: Array[String]
        
        The labels that indicate a passing result. Must be a subset of labels.
      - type: :label_model
        
        The object type, which is always label_model.
        
        :label_model
  - name: String
    
    The name of the grader.
  - type: :multi
    
    The object type, which is always multi.
    - :multi
model_sample: String

The model sample to be evaluated. This value will be used to populate the sample namespace. See the guide for more details. The output_json variable will be populated if the model sample is a valid JSON string.
item: untyped

The dataset item provided to the grader. This will be used to populate the item namespace. See the guide for more details.

Returns

class GraderRunResponse
- metadata: Metadata{ errors, execution_time, name, 4 more}
  - errors: Errors{ formula_parse_error, invalid_variable_error, model_grader_parse_error, 11 more}
    - formula_parse_error: bool
    - invalid_variable_error: bool
    - model_grader_parse_error: bool
    - model_grader_refusal_error: bool
    - model_grader_server_error: bool
    - model_grader_server_error_details: String
    - other_error: bool
    - python_grader_runtime_error: bool
    - python_grader_runtime_error_details: String
    - python_grader_server_error: bool
    - python_grader_server_error_type: String
    - sample_parse_error: bool
    - truncated_observation_error: bool
    - unresponsive_reward_error: bool
  - execution_time: Float
  - name: String
  - sampled_model_name: String
  - scores: Hash[Symbol, untyped]
  - token_usage: Integer
  - type: String
- model_grader_token_usage_per_model: Hash[Symbol, untyped]
- reward: Float
- sub_rewards: Hash[Symbol, untyped]

Example

require "openai"

openai = OpenAI::Client.new(api_key: "My API Key")

response = openai.fine_tuning.alpha.graders.run(
  grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check},
  model_sample: "model_sample"
)

puts(response)

Response

{
  "metadata": {
    "errors": {
      "formula_parse_error": true,
      "invalid_variable_error": true,
      "model_grader_parse_error": true,
      "model_grader_refusal_error": true,
      "model_grader_server_error": true,
      "model_grader_server_error_details": "model_grader_server_error_details",
      "other_error": true,
      "python_grader_runtime_error": true,
      "python_grader_runtime_error_details": "python_grader_runtime_error_details",
      "python_grader_server_error": true,
      "python_grader_server_error_type": "python_grader_server_error_type",
      "sample_parse_error": true,
      "truncated_observation_error": true,
      "unresponsive_reward_error": true
    },
    "execution_time": 0,
    "name": "name",
    "sampled_model_name": "sampled_model_name",
    "scores": {
      "foo": "bar"
    },
    "token_usage": 0,
    "type": "type"
  },
  "model_grader_token_usage_per_model": {
    "foo": "bar"
  },
  "reward": 0,
  "sub_rewards": {
    "foo": "bar"
  }
}

Validate grader

fine_tuning.alpha.graders.validate(**kwargs) -> GraderValidateResponse

post /fine_tuning/alpha/graders/validate

Validate a grader.

Parameters

grader: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more

The grader used for the fine-tuning job.
- class StringCheckGrader
  
  A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
  - input: String
    
    The input text. This may include template strings.
  - name: String
    
    The name of the grader.
  - operation: :eq | :ne | :like | :ilike
    
    The string check operation to perform. One of eq, ne, like, or ilike.
    - :eq
    - :ne
    - :like
    - :ilike
  - reference: String
    
    The reference text. This may include template strings.
  - type: :string_check
    
    The object type, which is always string_check.
    - :string_check
- class TextSimilarityGrader
  
  A TextSimilarityGrader object which grades text based on similarity metrics.
  - evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more
    
    The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
    - :cosine
    - :fuzzy_match
    - :bleu
    - :gleu
    - :meteor
    - :rouge_1
    - :rouge_2
    - :rouge_3
    - :rouge_4
    - :rouge_5
    - :rouge_l
  - input: String
    
    The text being graded.
  - name: String
    
    The name of the grader.
  - reference: String
    
    The text being graded against.
  - type: :text_similarity
    
    The type of grader.
    - :text_similarity
- class PythonGrader
  
  A PythonGrader object that runs a python script on the input.
  - name: String
    
    The name of the grader.
  - source: String
    
    The source code of the python script.
  - type: :python
    
    The object type, which is always python.
    - :python
  - image_tag: String
    
    The image tag to use for the python script.
- class ScoreModelGrader
  
  A ScoreModelGrader object that uses a model to assign a score to the input.
  - input: Array[Input{ content, role, type}]
    
    The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
    - content: String | ResponseInputText | OutputText{ text, type} | 3 more
      
      Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
      - String = String
        
        A text input to the model.
      - class ResponseInputText
        
        A text input to the model.
        
        text: String
        
        The text input to the model.
        
        type: :input_text
        
        The type of the input item. Always input_text.
        
        :input_text
        
        prompt_cache_breakpoint: PromptCacheBreakpoint{ mode}
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        mode: :explicit
        
        The breakpoint mode. Always explicit.
        
        :explicit
      - class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
      - class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
      - class ResponseInputAudio
        
        An audio input to the model.
        
        input_audio: InputAudio{ data, format_}
        
        data: String
        
        Base64-encoded audio data.
        
        format_: :mp3 | :wav
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        :mp3
        
        :wav
        
        type: :input_audio
        
        The type of the input item. Always input_audio.
        
        :input_audio
      - GraderInputs = Array[GraderInputItem]
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
    - role: :user | :assistant | :system | :developer
      
      The role of the message input. One of user, assistant, system, or developer.
      - :user
      - :assistant
      - :system
      - :developer
    - type: :message
      
      The type of the message input. Always message.
      - :message
  - model: String
    
    The model to use for the evaluation.
  - name: String
    
    The name of the grader.
  - type: :score_model
    
    The object type, which is always score_model.
    - :score_model
  - range: Array[Float]
    
    The range of the score. Defaults to [0, 1].
  - sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}
    
    The sampling parameters for the model.
    - max_completions_tokens: Integer
      
      The maximum number of tokens the grader model may generate in its response.
    - reasoning_effort: ReasoningEffort
      
      Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
      - :none
      - :minimal
      - :low
      - :medium
      - :high
      - :xhigh
      - :max
    - seed: Integer
      
      A seed value to initialize the randomness, during sampling.
    - temperature: Float
      
      A higher temperature increases randomness in the outputs.
    - top_p: Float
      
      An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- class MultiGrader
  
  A MultiGrader object combines the output of multiple graders to produce a single score.
  - calculate_output: String
    
    A formula to calculate the output based on grader results.
  - graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - class StringCheckGrader
      
      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - class TextSimilarityGrader
      
      A TextSimilarityGrader object which grades text based on similarity metrics.
    - class PythonGrader
      
      A PythonGrader object that runs a python script on the input.
    - class ScoreModelGrader
      
      A ScoreModelGrader object that uses a model to assign a score to the input.
    - class LabelModelGrader
      
      A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
      - input: Array[Input{ content, role, type}]
        
        content: String | ResponseInputText | OutputText{ text, type} | 3 more
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
        
        GraderInputs = Array[GraderInputItem]
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        role: :user | :assistant | :system | :developer
        
        The role of the message input. One of user, assistant, system, or developer.
        
        :user
        
        :assistant
        
        :system
        
        :developer
        
        type: :message
        
        The type of the message input. Always message.
        
        :message
      - labels: Array[String]
        
        The labels to assign to each item in the evaluation.
      - model: String
        
        The model to use for the evaluation. Must support structured outputs.
      - name: String
        
        The name of the grader.
      - passing_labels: Array[String]
        
        The labels that indicate a passing result. Must be a subset of labels.
      - type: :label_model
        
        The object type, which is always label_model.
        
        :label_model
  - name: String
    
    The name of the grader.
  - type: :multi
    
    The object type, which is always multi.
    - :multi

Returns

class GraderValidateResponse
- grader: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more
  
  The grader used for the fine-tuning job.
  - class StringCheckGrader
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - input: String
      
      The input text. This may include template strings.
    - name: String
      
      The name of the grader.
    - operation: :eq | :ne | :like | :ilike
      
      The string check operation to perform. One of eq, ne, like, or ilike.
      - :eq
      - :ne
      - :like
      - :ilike
    - reference: String
      
      The reference text. This may include template strings.
    - type: :string_check
      
      The object type, which is always string_check.
      - :string_check
  - class TextSimilarityGrader
    
    A TextSimilarityGrader object which grades text based on similarity metrics.
    - evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more
      
      The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
      - :cosine
      - :fuzzy_match
      - :bleu
      - :gleu
      - :meteor
      - :rouge_1
      - :rouge_2
      - :rouge_3
      - :rouge_4
      - :rouge_5
      - :rouge_l
    - input: String
      
      The text being graded.
    - name: String
      
      The name of the grader.
    - reference: String
      
      The text being graded against.
    - type: :text_similarity
      
      The type of grader.
      - :text_similarity
  - class PythonGrader
    
    A PythonGrader object that runs a python script on the input.
    - name: String
      
      The name of the grader.
    - source: String
      
      The source code of the python script.
    - type: :python
      
      The object type, which is always python.
      - :python
    - image_tag: String
      
      The image tag to use for the python script.
  - class ScoreModelGrader
    
    A ScoreModelGrader object that uses a model to assign a score to the input.
    - input: Array[Input{ content, role, type}]
      
      The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
      - content: String | ResponseInputText | OutputText{ text, type} | 3 more
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        text: String
        
        The text input to the model.
        
        type: :input_text
        
        The type of the input item. Always input_text.
        
        :input_text
        
        prompt_cache_breakpoint: PromptCacheBreakpoint{ mode}
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        mode: :explicit
        
        The breakpoint mode. Always explicit.
        
        :explicit
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
        
        input_audio: InputAudio{ data, format_}
        
        data: String
        
        Base64-encoded audio data.
        
        format_: :mp3 | :wav
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        :mp3
        
        :wav
        
        type: :input_audio
        
        The type of the input item. Always input_audio.
        
        :input_audio
        
        GraderInputs = Array[GraderInputItem]
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
      - role: :user | :assistant | :system | :developer
        
        The role of the message input. One of user, assistant, system, or developer.
        
        :user
        
        :assistant
        
        :system
        
        :developer
      - type: :message
        
        The type of the message input. Always message.
        
        :message
    - model: String
      
      The model to use for the evaluation.
    - name: String
      
      The name of the grader.
    - type: :score_model
      
      The object type, which is always score_model.
      - :score_model
    - range: Array[Float]
      
      The range of the score. Defaults to [0, 1].
    - sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}
      
      The sampling parameters for the model.
      - max_completions_tokens: Integer
        
        The maximum number of tokens the grader model may generate in its response.
      - reasoning_effort: ReasoningEffort
        
        Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
        
        :none
        
        :minimal
        
        :low
        
        :medium
        
        :high
        
        :xhigh
        
        :max
      - seed: Integer
        
        A seed value to initialize the randomness, during sampling.
      - temperature: Float
        
        A higher temperature increases randomness in the outputs.
      - top_p: Float
        
        An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
  - class MultiGrader
    
    A MultiGrader object combines the output of multiple graders to produce a single score.
    - calculate_output: String
      
      A formula to calculate the output based on grader results.
    - graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more
      
      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class StringCheckGrader
        
        A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class TextSimilarityGrader
        
        A TextSimilarityGrader object which grades text based on similarity metrics.
      - class PythonGrader
        
        A PythonGrader object that runs a python script on the input.
      - class ScoreModelGrader
        
        A ScoreModelGrader object that uses a model to assign a score to the input.
      - class LabelModelGrader
        
        A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
        
        input: Array[Input{ content, role, type}]
        
        content: String | ResponseInputText | OutputText{ text, type} | 3 more
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
        
        GraderInputs = Array[GraderInputItem]
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        role: :user | :assistant | :system | :developer
        
        The role of the message input. One of user, assistant, system, or developer.
        
        :user
        
        :assistant
        
        :system
        
        :developer
        
        type: :message
        
        The type of the message input. Always message.
        
        :message
        
        labels: Array[String]
        
        The labels to assign to each item in the evaluation.
        
        model: String
        
        The model to use for the evaluation. Must support structured outputs.
        
        name: String
        
        The name of the grader.
        
        passing_labels: Array[String]
        
        The labels that indicate a passing result. Must be a subset of labels.
        
        type: :label_model
        
        The object type, which is always label_model.
        
        :label_model
    - name: String
      
      The name of the grader.
    - type: :multi
      
      The object type, which is always multi.
      - :multi

Example

require "openai"

openai = OpenAI::Client.new(api_key: "My API Key")

response = openai.fine_tuning.alpha.graders.validate(
  grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check}
)

puts(response)

Response

{
  "grader": {
    "input": "input",
    "name": "name",
    "operation": "eq",
    "reference": "reference",
    "type": "string_check"
  }
}

Domain Types

Grader Run Response

class GraderRunResponse
- metadata: Metadata{ errors, execution_time, name, 4 more}
  - errors: Errors{ formula_parse_error, invalid_variable_error, model_grader_parse_error, 11 more}
    - formula_parse_error: bool
    - invalid_variable_error: bool
    - model_grader_parse_error: bool
    - model_grader_refusal_error: bool
    - model_grader_server_error: bool
    - model_grader_server_error_details: String
    - other_error: bool
    - python_grader_runtime_error: bool
    - python_grader_runtime_error_details: String
    - python_grader_server_error: bool
    - python_grader_server_error_type: String
    - sample_parse_error: bool
    - truncated_observation_error: bool
    - unresponsive_reward_error: bool
  - execution_time: Float
  - name: String
  - sampled_model_name: String
  - scores: Hash[Symbol, untyped]
  - token_usage: Integer
  - type: String
- model_grader_token_usage_per_model: Hash[Symbol, untyped]
- reward: Float
- sub_rewards: Hash[Symbol, untyped]

Grader Validate Response

class GraderValidateResponse
- grader: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more
  
  The grader used for the fine-tuning job.
  - class StringCheckGrader
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - input: String
      
      The input text. This may include template strings.
    - name: String
      
      The name of the grader.
    - operation: :eq | :ne | :like | :ilike
      
      The string check operation to perform. One of eq, ne, like, or ilike.
      - :eq
      - :ne
      - :like
      - :ilike
    - reference: String
      
      The reference text. This may include template strings.
    - type: :string_check
      
      The object type, which is always string_check.
      - :string_check
  - class TextSimilarityGrader
    
    A TextSimilarityGrader object which grades text based on similarity metrics.
    - evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more
      
      The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
      - :cosine
      - :fuzzy_match
      - :bleu
      - :gleu
      - :meteor
      - :rouge_1
      - :rouge_2
      - :rouge_3
      - :rouge_4
      - :rouge_5
      - :rouge_l
    - input: String
      
      The text being graded.
    - name: String
      
      The name of the grader.
    - reference: String
      
      The text being graded against.
    - type: :text_similarity
      
      The type of grader.
      - :text_similarity
  - class PythonGrader
    
    A PythonGrader object that runs a python script on the input.
    - name: String
      
      The name of the grader.
    - source: String
      
      The source code of the python script.
    - type: :python
      
      The object type, which is always python.
      - :python
    - image_tag: String
      
      The image tag to use for the python script.
  - class ScoreModelGrader
    
    A ScoreModelGrader object that uses a model to assign a score to the input.
    - input: Array[Input{ content, role, type}]
      
      The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
      - content: String | ResponseInputText | OutputText{ text, type} | 3 more
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        text: String
        
        The text input to the model.
        
        type: :input_text
        
        The type of the input item. Always input_text.
        
        :input_text
        
        prompt_cache_breakpoint: PromptCacheBreakpoint{ mode}
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        mode: :explicit
        
        The breakpoint mode. Always explicit.
        
        :explicit
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
        
        input_audio: InputAudio{ data, format_}
        
        data: String
        
        Base64-encoded audio data.
        
        format_: :mp3 | :wav
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        :mp3
        
        :wav
        
        type: :input_audio
        
        The type of the input item. Always input_audio.
        
        :input_audio
        
        GraderInputs = Array[GraderInputItem]
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
      - role: :user | :assistant | :system | :developer
        
        The role of the message input. One of user, assistant, system, or developer.
        
        :user
        
        :assistant
        
        :system
        
        :developer
      - type: :message
        
        The type of the message input. Always message.
        
        :message
    - model: String
      
      The model to use for the evaluation.
    - name: String
      
      The name of the grader.
    - type: :score_model
      
      The object type, which is always score_model.
      - :score_model
    - range: Array[Float]
      
      The range of the score. Defaults to [0, 1].
    - sampling_params: SamplingParams{ max_completions_tokens, reasoning_effort, seed, 2 more}
      
      The sampling parameters for the model.
      - max_completions_tokens: Integer
        
        The maximum number of tokens the grader model may generate in its response.
      - reasoning_effort: ReasoningEffort
        
        Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
        
        :none
        
        :minimal
        
        :low
        
        :medium
        
        :high
        
        :xhigh
        
        :max
      - seed: Integer
        
        A seed value to initialize the randomness, during sampling.
      - temperature: Float
        
        A higher temperature increases randomness in the outputs.
      - top_p: Float
        
        An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
  - class MultiGrader
    
    A MultiGrader object combines the output of multiple graders to produce a single score.
    - calculate_output: String
      
      A formula to calculate the output based on grader results.
    - graders: StringCheckGrader | TextSimilarityGrader | PythonGrader | 2 more
      
      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class StringCheckGrader
        
        A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class TextSimilarityGrader
        
        A TextSimilarityGrader object which grades text based on similarity metrics.
      - class PythonGrader
        
        A PythonGrader object that runs a python script on the input.
      - class ScoreModelGrader
        
        A ScoreModelGrader object that uses a model to assign a score to the input.
      - class LabelModelGrader
        
        A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
        
        input: Array[Input{ content, role, type}]
        
        content: String | ResponseInputText | OutputText{ text, type} | 3 more
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String = String
        
        A text input to the model.
        
        class ResponseInputText
        
        A text input to the model.
        
        class OutputText
        
        A text output from the model.
        
        text: String
        
        The text output from the model.
        
        type: :output_text
        
        The type of the output text. Always output_text.
        
        :output_text
        
        class InputImage
        
        An image input block used within EvalItem content arrays.
        
        image_url: String
        
        The URL of the image input.
        
        type: :input_image
        
        The type of the image input. Always input_image.
        
        :input_image
        
        detail: String
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio
        
        An audio input to the model.
        
        GraderInputs = Array[GraderInputItem]
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        role: :user | :assistant | :system | :developer
        
        The role of the message input. One of user, assistant, system, or developer.
        
        :user
        
        :assistant
        
        :system
        
        :developer
        
        type: :message
        
        The type of the message input. Always message.
        
        :message
        
        labels: Array[String]
        
        The labels to assign to each item in the evaluation.
        
        model: String
        
        The model to use for the evaluation. Must support structured outputs.
        
        name: String
        
        The name of the grader.
        
        passing_labels: Array[String]
        
        The labels that indicate a passing result. Must be a subset of labels.
        
        type: :label_model
        
        The object type, which is always label_model.
        
        :label_model
    - name: String
      
      The name of the grader.
    - type: :multi
      
      The object type, which is always multi.
      - :multi

ruby/resources/fine_tuning/subresources/alpha/index.md 2026-07-10 23:02 UTC to 2026-07-12 06:58 UTC

Alpha

Graders

Run grader

Parameters

Returns

Example

Response

Validate grader

Parameters

Returns

Example

Response

Domain Types

Grader Run Response

Grader Validate Response

ruby/resources/fine_tuning/subresources/alpha/index.md +2 −2