Alpha

Graders

Run grader

fine_tuning.alpha.graders.run(GraderRunParams**kwargs) -> GraderRunResponse

post /fine_tuning/alpha/graders/run

Run grader

Parameters

grader: Grader

The grader used for the fine-tuning job.
- class StringCheckGrader: …
  
  A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
  - input: str
    
    The input text. This may include template strings.
  - name: str
    
    The name of the grader.
  - operation: Literal["eq", "ne", "like", "ilike"]
    
    The string check operation to perform. One of eq, ne, like, or ilike.
    - "eq"
    - "ne"
    - "like"
    - "ilike"
  - reference: str
    
    The reference text. This may include template strings.
  - type: Literal["string_check"]
    
    The object type, which is always string_check.
    - "string_check"
- class TextSimilarityGrader: …
  
  A TextSimilarityGrader object which grades text based on similarity metrics.
  - evaluation_metric: Literal["cosine", "fuzzy_match", "bleu", 8 more]
    
    The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
    - "cosine"
    - "fuzzy_match"
    - "bleu"
    - "gleu"
    - "meteor"
    - "rouge_1"
    - "rouge_2"
    - "rouge_3"
    - "rouge_4"
    - "rouge_5"
    - "rouge_l"
  - input: str
    
    The text being graded.
  - name: str
    
    The name of the grader.
  - reference: str
    
    The text being graded against.
  - type: Literal["text_similarity"]
    
    The type of grader.
    - "text_similarity"
- class PythonGrader: …
  
  A PythonGrader object that runs a python script on the input.
  - name: str
    
    The name of the grader.
  - source: str
    
    The source code of the python script.
  - type: Literal["python"]
    
    The object type, which is always python.
    - "python"
  - image_tag: Optional[str]
    
    The image tag to use for the python script.
- class ScoreModelGrader: …
  
  A ScoreModelGrader object that uses a model to assign a score to the input.
  - input: List[Input]
    
    The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
    - content: InputContent
      
      Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
      - str
        
        A text input to the model.
      - class ResponseInputText: …
        
        A text input to the model.
        
        text: str
        
        The text input to the model.
        
        type: Literal["input_text"]
        
        The type of the input item. Always input_text.
        
        "input_text"
        
        prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        mode: Literal["explicit"]
        
        The breakpoint mode. Always explicit.
        
        "explicit"
      - class InputContentOutputText: …
        
        A text output from the model.
        
        text: str
        
        The text output from the model.
        
        type: Literal["output_text"]
        
        The type of the output text. Always output_text.
        
        "output_text"
      - class InputContentInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        image_url: str
        
        The URL of the image input.
        
        type: Literal["input_image"]
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: Optional[str]
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
      - class ResponseInputAudio: …
        
        An audio input to the model.
        
        input_audio: InputAudio
        
        data: str
        
        Base64-encoded audio data.
        
        format: Literal["mp3", "wav"]
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        "mp3"
        
        "wav"
        
        type: Literal["input_audio"]
        
        The type of the input item. Always input_audio.
        
        "input_audio"
      - List[GraderInputItem]
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        class GraderInputItemOutputText: …
        
        A text output from the model.
        
        text: str
        
        The text output from the model.
        
        type: Literal["output_text"]
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        class GraderInputItemInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        image_url: str
        
        The URL of the image input.
        
        type: Literal["input_image"]
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: Optional[str]
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
    - role: Literal["user", "assistant", "system", "developer"]
      
      The role of the message input. One of user, assistant, system, or developer.
      - "user"
      - "assistant"
      - "system"
      - "developer"
    - type: Optional[Literal["message"]]
      
      The type of the message input. Always message.
      - "message"
  - model: str
    
    The model to use for the evaluation.
  - name: str
    
    The name of the grader.
  - type: Literal["score_model"]
    
    The object type, which is always score_model.
    - "score_model"
  - range: Optional[List[float]]
    
    The range of the score. Defaults to [0, 1].
  - sampling_params: Optional[SamplingParams]
    
    The sampling parameters for the model.
    - max_completions_tokens: Optional[int]
      
      The maximum number of tokens the grader model may generate in its response.
    - reasoning_effort: Optional[ReasoningEffort]
      
      Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
      - "none"
      - "minimal"
      - "low"
      - "medium"
      - "high"
      - "xhigh"
      - "max"
    - seed: Optional[int]
      
      A seed value to initialize the randomness, during sampling.
    - temperature: Optional[float]
      
      A higher temperature increases randomness in the outputs.
    - top_p: Optional[float]
      
      An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- class MultiGrader: …
  
  A MultiGrader object combines the output of multiple graders to produce a single score.
  - calculate_output: str
    
    A formula to calculate the output based on grader results.
  - graders: Graders
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - class StringCheckGrader: …
      
      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - class TextSimilarityGrader: …
      
      A TextSimilarityGrader object which grades text based on similarity metrics.
    - class PythonGrader: …
      
      A PythonGrader object that runs a python script on the input.
    - class ScoreModelGrader: …
      
      A ScoreModelGrader object that uses a model to assign a score to the input.
    - class LabelModelGrader: …
      
      A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
      - input: List[Input]
        
        content: InputContent
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        class InputContentOutputText: …
        
        A text output from the model.
        
        text: str
        
        The text output from the model.
        
        type: Literal["output_text"]
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        class InputContentInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        image_url: str
        
        The URL of the image input.
        
        type: Literal["input_image"]
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: Optional[str]
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
        
        List[GraderInputItem]
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        class GraderInputItemOutputText: …
        
        A text output from the model.
        
        class GraderInputItemInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
        
        role: Literal["user", "assistant", "system", "developer"]
        
        The role of the message input. One of user, assistant, system, or developer.
        
        "user"
        
        "assistant"
        
        "system"
        
        "developer"
        
        type: Optional[Literal["message"]]
        
        The type of the message input. Always message.
        
        "message"
      - labels: List[str]
        
        The labels to assign to each item in the evaluation.
      - model: str
        
        The model to use for the evaluation. Must support structured outputs.
      - name: str
        
        The name of the grader.
      - passing_labels: List[str]
        
        The labels that indicate a passing result. Must be a subset of labels.
      - type: Literal["label_model"]
        
        The object type, which is always label_model.
        
        "label_model"
  - name: str
    
    The name of the grader.
  - type: Literal["multi"]
    
    The object type, which is always multi.
    - "multi"
model_sample: str

The model sample to be evaluated. This value will be used to populate the sample namespace. See the guide for more details. The output_json variable will be populated if the model sample is a valid JSON string.
item: Optional[object]

The dataset item provided to the grader. This will be used to populate the item namespace. See the guide for more details.

Returns

class GraderRunResponse: …
- metadata: Metadata
  - errors: MetadataErrors
    - formula_parse_error: bool
    - invalid_variable_error: bool
    - model_grader_parse_error: bool
    - model_grader_refusal_error: bool
    - model_grader_server_error: bool
    - model_grader_server_error_details: Optional[str]
    - other_error: bool
    - python_grader_runtime_error: bool
    - python_grader_runtime_error_details: Optional[str]
    - python_grader_server_error: bool
    - python_grader_server_error_type: Optional[str]
    - sample_parse_error: bool
    - truncated_observation_error: bool
    - unresponsive_reward_error: bool
  - execution_time: float
  - name: str
  - sampled_model_name: Optional[str]
  - scores: Dict[str, object]
  - token_usage: Optional[int]
  - type: str
- model_grader_token_usage_per_model: Dict[str, object]
- reward: float
- sub_rewards: Dict[str, object]

Example

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY"),  # This is the default and can be omitted
)
response = client.fine_tuning.alpha.graders.run(
    grader={
        "input": "input",
        "name": "name",
        "operation": "eq",
        "reference": "reference",
        "type": "string_check",
    },
    model_sample="model_sample",
)
print(response.metadata)

Response

{
  "metadata": {
    "errors": {
      "formula_parse_error": true,
      "invalid_variable_error": true,
      "model_grader_parse_error": true,
      "model_grader_refusal_error": true,
      "model_grader_server_error": true,
      "model_grader_server_error_details": "model_grader_server_error_details",
      "other_error": true,
      "python_grader_runtime_error": true,
      "python_grader_runtime_error_details": "python_grader_runtime_error_details",
      "python_grader_server_error": true,
      "python_grader_server_error_type": "python_grader_server_error_type",
      "sample_parse_error": true,
      "truncated_observation_error": true,
      "unresponsive_reward_error": true
    },
    "execution_time": 0,
    "name": "name",
    "sampled_model_name": "sampled_model_name",
    "scores": {
      "foo": "bar"
    },
    "token_usage": 0,
    "type": "type"
  },
  "model_grader_token_usage_per_model": {
    "foo": "bar"
  },
  "reward": 0,
  "sub_rewards": {
    "foo": "bar"
  }
}

Score text alignment

from openai import OpenAI

client = OpenAI()
result = client.fine_tuning.alpha.graders.run(
  grader={
    "type": "score_model",
    "name": "Example score model grader",
    "input": [
      {
        "role": "user",
        "content": [
          {
            "type": "input_text",
            "text": "Score how close the reference answer is to the model answer on a 0-1 scale. Return only the score.\n\nReference answer: {{item.reference_answer}}\n\nModel answer: {{sample.output_text}}",
          }
        ],
      }
    ],
    "model": "gpt-5-mini",
    "sampling_params": {"temperature": 1, "top_p": 1, "seed": 42},
  },
  item={"reference_answer": "fuzzy wuzzy was a bear"},
  model_sample="fuzzy wuzzy was a bear",
)
print(result)

Response

{
  "reward": 1.0,
  "metadata": {
    "name": "Example score model grader",
    "type": "score_model",
    "errors": {
      "formula_parse_error": false,
      "sample_parse_error": false,
      "truncated_observation_error": false,
      "unresponsive_reward_error": false,
      "invalid_variable_error": false,
      "other_error": false,
      "python_grader_server_error": false,
      "python_grader_server_error_type": null,
      "python_grader_runtime_error": false,
      "python_grader_runtime_error_details": null,
      "model_grader_server_error": false,
      "model_grader_refusal_error": false,
      "model_grader_parse_error": false,
      "model_grader_server_error_details": null
    },
    "execution_time": 4.365238428115845,
    "scores": {},
    "token_usage": {
      "prompt_tokens": 190,
      "total_tokens": 324,
      "completion_tokens": 134,
      "cached_tokens": 0
    },
    "sampled_model_name": "gpt-4o-2024-08-06"
  },
  "sub_rewards": {},
  "model_grader_token_usage_per_model": {
    "gpt-4o-2024-08-06": {
      "prompt_tokens": 190,
      "total_tokens": 324,
      "completion_tokens": 134,
      "cached_tokens": 0
    }
  }
}

Validate grader

fine_tuning.alpha.graders.validate(GraderValidateParams**kwargs) -> GraderValidateResponse

post /fine_tuning/alpha/graders/validate

Validate grader

Parameters

grader: Grader

The grader used for the fine-tuning job.
- class StringCheckGrader: …
  
  A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
  - input: str
    
    The input text. This may include template strings.
  - name: str
    
    The name of the grader.
  - operation: Literal["eq", "ne", "like", "ilike"]
    
    The string check operation to perform. One of eq, ne, like, or ilike.
    - "eq"
    - "ne"
    - "like"
    - "ilike"
  - reference: str
    
    The reference text. This may include template strings.
  - type: Literal["string_check"]
    
    The object type, which is always string_check.
    - "string_check"
- class TextSimilarityGrader: …
  
  A TextSimilarityGrader object which grades text based on similarity metrics.
  - evaluation_metric: Literal["cosine", "fuzzy_match", "bleu", 8 more]
    
    The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
    - "cosine"
    - "fuzzy_match"
    - "bleu"
    - "gleu"
    - "meteor"
    - "rouge_1"
    - "rouge_2"
    - "rouge_3"
    - "rouge_4"
    - "rouge_5"
    - "rouge_l"
  - input: str
    
    The text being graded.
  - name: str
    
    The name of the grader.
  - reference: str
    
    The text being graded against.
  - type: Literal["text_similarity"]
    
    The type of grader.
    - "text_similarity"
- class PythonGrader: …
  
  A PythonGrader object that runs a python script on the input.
  - name: str
    
    The name of the grader.
  - source: str
    
    The source code of the python script.
  - type: Literal["python"]
    
    The object type, which is always python.
    - "python"
  - image_tag: Optional[str]
    
    The image tag to use for the python script.
- class ScoreModelGrader: …
  
  A ScoreModelGrader object that uses a model to assign a score to the input.
  - input: List[Input]
    
    The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
    - content: InputContent
      
      Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
      - str
        
        A text input to the model.
      - class ResponseInputText: …
        
        A text input to the model.
        
        text: str
        
        The text input to the model.
        
        type: Literal["input_text"]
        
        The type of the input item. Always input_text.
        
        "input_text"
        
        prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        mode: Literal["explicit"]
        
        The breakpoint mode. Always explicit.
        
        "explicit"
      - class InputContentOutputText: …
        
        A text output from the model.
        
        text: str
        
        The text output from the model.
        
        type: Literal["output_text"]
        
        The type of the output text. Always output_text.
        
        "output_text"
      - class InputContentInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        image_url: str
        
        The URL of the image input.
        
        type: Literal["input_image"]
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: Optional[str]
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
      - class ResponseInputAudio: …
        
        An audio input to the model.
        
        input_audio: InputAudio
        
        data: str
        
        Base64-encoded audio data.
        
        format: Literal["mp3", "wav"]
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        "mp3"
        
        "wav"
        
        type: Literal["input_audio"]
        
        The type of the input item. Always input_audio.
        
        "input_audio"
      - List[GraderInputItem]
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        class GraderInputItemOutputText: …
        
        A text output from the model.
        
        text: str
        
        The text output from the model.
        
        type: Literal["output_text"]
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        class GraderInputItemInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        image_url: str
        
        The URL of the image input.
        
        type: Literal["input_image"]
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: Optional[str]
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
    - role: Literal["user", "assistant", "system", "developer"]
      
      The role of the message input. One of user, assistant, system, or developer.
      - "user"
      - "assistant"
      - "system"
      - "developer"
    - type: Optional[Literal["message"]]
      
      The type of the message input. Always message.
      - "message"
  - model: str
    
    The model to use for the evaluation.
  - name: str
    
    The name of the grader.
  - type: Literal["score_model"]
    
    The object type, which is always score_model.
    - "score_model"
  - range: Optional[List[float]]
    
    The range of the score. Defaults to [0, 1].
  - sampling_params: Optional[SamplingParams]
    
    The sampling parameters for the model.
    - max_completions_tokens: Optional[int]
      
      The maximum number of tokens the grader model may generate in its response.
    - reasoning_effort: Optional[ReasoningEffort]
      
      Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
      - "none"
      - "minimal"
      - "low"
      - "medium"
      - "high"
      - "xhigh"
      - "max"
    - seed: Optional[int]
      
      A seed value to initialize the randomness, during sampling.
    - temperature: Optional[float]
      
      A higher temperature increases randomness in the outputs.
    - top_p: Optional[float]
      
      An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
- class MultiGrader: …
  
  A MultiGrader object combines the output of multiple graders to produce a single score.
  - calculate_output: str
    
    A formula to calculate the output based on grader results.
  - graders: Graders
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - class StringCheckGrader: …
      
      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - class TextSimilarityGrader: …
      
      A TextSimilarityGrader object which grades text based on similarity metrics.
    - class PythonGrader: …
      
      A PythonGrader object that runs a python script on the input.
    - class ScoreModelGrader: …
      
      A ScoreModelGrader object that uses a model to assign a score to the input.
    - class LabelModelGrader: …
      
      A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
      - input: List[Input]
        
        content: InputContent
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        class InputContentOutputText: …
        
        A text output from the model.
        
        text: str
        
        The text output from the model.
        
        type: Literal["output_text"]
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        class InputContentInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        image_url: str
        
        The URL of the image input.
        
        type: Literal["input_image"]
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: Optional[str]
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
        
        List[GraderInputItem]
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        class GraderInputItemOutputText: …
        
        A text output from the model.
        
        class GraderInputItemInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
        
        role: Literal["user", "assistant", "system", "developer"]
        
        The role of the message input. One of user, assistant, system, or developer.
        
        "user"
        
        "assistant"
        
        "system"
        
        "developer"
        
        type: Optional[Literal["message"]]
        
        The type of the message input. Always message.
        
        "message"
      - labels: List[str]
        
        The labels to assign to each item in the evaluation.
      - model: str
        
        The model to use for the evaluation. Must support structured outputs.
      - name: str
        
        The name of the grader.
      - passing_labels: List[str]
        
        The labels that indicate a passing result. Must be a subset of labels.
      - type: Literal["label_model"]
        
        The object type, which is always label_model.
        
        "label_model"
  - name: str
    
    The name of the grader.
  - type: Literal["multi"]
    
    The object type, which is always multi.
    - "multi"

Returns

class GraderValidateResponse: …
- grader: Optional[Grader]
  
  The grader used for the fine-tuning job.
  - class StringCheckGrader: …
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - input: str
      
      The input text. This may include template strings.
    - name: str
      
      The name of the grader.
    - operation: Literal["eq", "ne", "like", "ilike"]
      
      The string check operation to perform. One of eq, ne, like, or ilike.
      - "eq"
      - "ne"
      - "like"
      - "ilike"
    - reference: str
      
      The reference text. This may include template strings.
    - type: Literal["string_check"]
      
      The object type, which is always string_check.
      - "string_check"
  - class TextSimilarityGrader: …
    
    A TextSimilarityGrader object which grades text based on similarity metrics.
    - evaluation_metric: Literal["cosine", "fuzzy_match", "bleu", 8 more]
      
      The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
      - "cosine"
      - "fuzzy_match"
      - "bleu"
      - "gleu"
      - "meteor"
      - "rouge_1"
      - "rouge_2"
      - "rouge_3"
      - "rouge_4"
      - "rouge_5"
      - "rouge_l"
    - input: str
      
      The text being graded.
    - name: str
      
      The name of the grader.
    - reference: str
      
      The text being graded against.
    - type: Literal["text_similarity"]
      
      The type of grader.
      - "text_similarity"
  - class PythonGrader: …
    
    A PythonGrader object that runs a python script on the input.
    - name: str
      
      The name of the grader.
    - source: str
      
      The source code of the python script.
    - type: Literal["python"]
      
      The object type, which is always python.
      - "python"
    - image_tag: Optional[str]
      
      The image tag to use for the python script.
  - class ScoreModelGrader: …
    
    A ScoreModelGrader object that uses a model to assign a score to the input.
    - input: List[Input]
      
      The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
      - content: InputContent
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        text: str
        
        The text input to the model.
        
        type: Literal["input_text"]
        
        The type of the input item. Always input_text.
        
        "input_text"
        
        prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        mode: Literal["explicit"]
        
        The breakpoint mode. Always explicit.
        
        "explicit"
        
        class InputContentOutputText: …
        
        A text output from the model.
        
        text: str
        
        The text output from the model.
        
        type: Literal["output_text"]
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        class InputContentInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        image_url: str
        
        The URL of the image input.
        
        type: Literal["input_image"]
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: Optional[str]
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
        
        input_audio: InputAudio
        
        data: str
        
        Base64-encoded audio data.
        
        format: Literal["mp3", "wav"]
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        "mp3"
        
        "wav"
        
        type: Literal["input_audio"]
        
        The type of the input item. Always input_audio.
        
        "input_audio"
        
        List[GraderInputItem]
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        class GraderInputItemOutputText: …
        
        A text output from the model.
        
        text: str
        
        The text output from the model.
        
        type: Literal["output_text"]
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        class GraderInputItemInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        image_url: str
        
        The URL of the image input.
        
        type: Literal["input_image"]
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: Optional[str]
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
      - role: Literal["user", "assistant", "system", "developer"]
        
        The role of the message input. One of user, assistant, system, or developer.
        
        "user"
        
        "assistant"
        
        "system"
        
        "developer"
      - type: Optional[Literal["message"]]
        
        The type of the message input. Always message.
        
        "message"
    - model: str
      
      The model to use for the evaluation.
    - name: str
      
      The name of the grader.
    - type: Literal["score_model"]
      
      The object type, which is always score_model.
      - "score_model"
    - range: Optional[List[float]]
      
      The range of the score. Defaults to [0, 1].
    - sampling_params: Optional[SamplingParams]
      
      The sampling parameters for the model.
      - max_completions_tokens: Optional[int]
        
        The maximum number of tokens the grader model may generate in its response.
      - reasoning_effort: Optional[ReasoningEffort]
        
        Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
        
        "none"
        
        "minimal"
        
        "low"
        
        "medium"
        
        "high"
        
        "xhigh"
        
        "max"
      - seed: Optional[int]
        
        A seed value to initialize the randomness, during sampling.
      - temperature: Optional[float]
        
        A higher temperature increases randomness in the outputs.
      - top_p: Optional[float]
        
        An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
  - class MultiGrader: …
    
    A MultiGrader object combines the output of multiple graders to produce a single score.
    - calculate_output: str
      
      A formula to calculate the output based on grader results.
    - graders: Graders
      
      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class StringCheckGrader: …
        
        A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class TextSimilarityGrader: …
        
        A TextSimilarityGrader object which grades text based on similarity metrics.
      - class PythonGrader: …
        
        A PythonGrader object that runs a python script on the input.
      - class ScoreModelGrader: …
        
        A ScoreModelGrader object that uses a model to assign a score to the input.
      - class LabelModelGrader: …
        
        A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
        
        input: List[Input]
        
        content: InputContent
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        class InputContentOutputText: …
        
        A text output from the model.
        
        text: str
        
        The text output from the model.
        
        type: Literal["output_text"]
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        class InputContentInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        image_url: str
        
        The URL of the image input.
        
        type: Literal["input_image"]
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: Optional[str]
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
        
        List[GraderInputItem]
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        class GraderInputItemOutputText: …
        
        A text output from the model.
        
        class GraderInputItemInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
        
        role: Literal["user", "assistant", "system", "developer"]
        
        The role of the message input. One of user, assistant, system, or developer.
        
        "user"
        
        "assistant"
        
        "system"
        
        "developer"
        
        type: Optional[Literal["message"]]
        
        The type of the message input. Always message.
        
        "message"
        
        labels: List[str]
        
        The labels to assign to each item in the evaluation.
        
        model: str
        
        The model to use for the evaluation. Must support structured outputs.
        
        name: str
        
        The name of the grader.
        
        passing_labels: List[str]
        
        The labels that indicate a passing result. Must be a subset of labels.
        
        type: Literal["label_model"]
        
        The object type, which is always label_model.
        
        "label_model"
    - name: str
      
      The name of the grader.
    - type: Literal["multi"]
      
      The object type, which is always multi.
      - "multi"

Example

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY"),  # This is the default and can be omitted
)
response = client.fine_tuning.alpha.graders.validate(
    grader={
        "input": "input",
        "name": "name",
        "operation": "eq",
        "reference": "reference",
        "type": "string_check",
    },
)
print(response.grader)

Response

{
  "grader": {
    "input": "input",
    "name": "name",
    "operation": "eq",
    "reference": "reference",
    "type": "string_check"
  }
}

Domain Types

Grader Run Response

class GraderRunResponse: …
- metadata: Metadata
  - errors: MetadataErrors
    - formula_parse_error: bool
    - invalid_variable_error: bool
    - model_grader_parse_error: bool
    - model_grader_refusal_error: bool
    - model_grader_server_error: bool
    - model_grader_server_error_details: Optional[str]
    - other_error: bool
    - python_grader_runtime_error: bool
    - python_grader_runtime_error_details: Optional[str]
    - python_grader_server_error: bool
    - python_grader_server_error_type: Optional[str]
    - sample_parse_error: bool
    - truncated_observation_error: bool
    - unresponsive_reward_error: bool
  - execution_time: float
  - name: str
  - sampled_model_name: Optional[str]
  - scores: Dict[str, object]
  - token_usage: Optional[int]
  - type: str
- model_grader_token_usage_per_model: Dict[str, object]
- reward: float
- sub_rewards: Dict[str, object]

Grader Validate Response

class GraderValidateResponse: …
- grader: Optional[Grader]
  
  The grader used for the fine-tuning job.
  - class StringCheckGrader: …
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - input: str
      
      The input text. This may include template strings.
    - name: str
      
      The name of the grader.
    - operation: Literal["eq", "ne", "like", "ilike"]
      
      The string check operation to perform. One of eq, ne, like, or ilike.
      - "eq"
      - "ne"
      - "like"
      - "ilike"
    - reference: str
      
      The reference text. This may include template strings.
    - type: Literal["string_check"]
      
      The object type, which is always string_check.
      - "string_check"
  - class TextSimilarityGrader: …
    
    A TextSimilarityGrader object which grades text based on similarity metrics.
    - evaluation_metric: Literal["cosine", "fuzzy_match", "bleu", 8 more]
      
      The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
      - "cosine"
      - "fuzzy_match"
      - "bleu"
      - "gleu"
      - "meteor"
      - "rouge_1"
      - "rouge_2"
      - "rouge_3"
      - "rouge_4"
      - "rouge_5"
      - "rouge_l"
    - input: str
      
      The text being graded.
    - name: str
      
      The name of the grader.
    - reference: str
      
      The text being graded against.
    - type: Literal["text_similarity"]
      
      The type of grader.
      - "text_similarity"
  - class PythonGrader: …
    
    A PythonGrader object that runs a python script on the input.
    - name: str
      
      The name of the grader.
    - source: str
      
      The source code of the python script.
    - type: Literal["python"]
      
      The object type, which is always python.
      - "python"
    - image_tag: Optional[str]
      
      The image tag to use for the python script.
  - class ScoreModelGrader: …
    
    A ScoreModelGrader object that uses a model to assign a score to the input.
    - input: List[Input]
      
      The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
      - content: InputContent
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        text: str
        
        The text input to the model.
        
        type: Literal["input_text"]
        
        The type of the input item. Always input_text.
        
        "input_text"
        
        prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        mode: Literal["explicit"]
        
        The breakpoint mode. Always explicit.
        
        "explicit"
        
        class InputContentOutputText: …
        
        A text output from the model.
        
        text: str
        
        The text output from the model.
        
        type: Literal["output_text"]
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        class InputContentInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        image_url: str
        
        The URL of the image input.
        
        type: Literal["input_image"]
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: Optional[str]
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
        
        input_audio: InputAudio
        
        data: str
        
        Base64-encoded audio data.
        
        format: Literal["mp3", "wav"]
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        "mp3"
        
        "wav"
        
        type: Literal["input_audio"]
        
        The type of the input item. Always input_audio.
        
        "input_audio"
        
        List[GraderInputItem]
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        class GraderInputItemOutputText: …
        
        A text output from the model.
        
        text: str
        
        The text output from the model.
        
        type: Literal["output_text"]
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        class GraderInputItemInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        image_url: str
        
        The URL of the image input.
        
        type: Literal["input_image"]
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: Optional[str]
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
      - role: Literal["user", "assistant", "system", "developer"]
        
        The role of the message input. One of user, assistant, system, or developer.
        
        "user"
        
        "assistant"
        
        "system"
        
        "developer"
      - type: Optional[Literal["message"]]
        
        The type of the message input. Always message.
        
        "message"
    - model: str
      
      The model to use for the evaluation.
    - name: str
      
      The name of the grader.
    - type: Literal["score_model"]
      
      The object type, which is always score_model.
      - "score_model"
    - range: Optional[List[float]]
      
      The range of the score. Defaults to [0, 1].
    - sampling_params: Optional[SamplingParams]
      
      The sampling parameters for the model.
      - max_completions_tokens: Optional[int]
        
        The maximum number of tokens the grader model may generate in its response.
      - reasoning_effort: Optional[ReasoningEffort]
        
        Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
        
        "none"
        
        "minimal"
        
        "low"
        
        "medium"
        
        "high"
        
        "xhigh"
        
        "max"
      - seed: Optional[int]
        
        A seed value to initialize the randomness, during sampling.
      - temperature: Optional[float]
        
        A higher temperature increases randomness in the outputs.
      - top_p: Optional[float]
        
        An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
  - class MultiGrader: …
    
    A MultiGrader object combines the output of multiple graders to produce a single score.
    - calculate_output: str
      
      A formula to calculate the output based on grader results.
    - graders: Graders
      
      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class StringCheckGrader: …
        
        A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class TextSimilarityGrader: …
        
        A TextSimilarityGrader object which grades text based on similarity metrics.
      - class PythonGrader: …
        
        A PythonGrader object that runs a python script on the input.
      - class ScoreModelGrader: …
        
        A ScoreModelGrader object that uses a model to assign a score to the input.
      - class LabelModelGrader: …
        
        A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
        
        input: List[Input]
        
        content: InputContent
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        class InputContentOutputText: …
        
        A text output from the model.
        
        text: str
        
        The text output from the model.
        
        type: Literal["output_text"]
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        class InputContentInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        image_url: str
        
        The URL of the image input.
        
        type: Literal["input_image"]
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: Optional[str]
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
        
        List[GraderInputItem]
        
        str
        
        A text input to the model.
        
        class ResponseInputText: …
        
        A text input to the model.
        
        class GraderInputItemOutputText: …
        
        A text output from the model.
        
        class GraderInputItemInputImage: …
        
        An image input block used within EvalItem content arrays.
        
        class ResponseInputAudio: …
        
        An audio input to the model.
        
        role: Literal["user", "assistant", "system", "developer"]
        
        The role of the message input. One of user, assistant, system, or developer.
        
        "user"
        
        "assistant"
        
        "system"
        
        "developer"
        
        type: Optional[Literal["message"]]
        
        The type of the message input. Always message.
        
        "message"
        
        labels: List[str]
        
        The labels to assign to each item in the evaluation.
        
        model: str
        
        The model to use for the evaluation. Must support structured outputs.
        
        name: str
        
        The name of the grader.
        
        passing_labels: List[str]
        
        The labels that indicate a passing result. Must be a subset of labels.
        
        type: Literal["label_model"]
        
        The object type, which is always label_model.
        
        "label_model"
    - name: str
      
      The name of the grader.
    - type: Literal["multi"]
      
      The object type, which is always multi.
      - "multi"

python/resources/fine_tuning/subresources/alpha/index.md +2286 −0 created

1# Alpha

3# Graders

5## Run grader

7`fine_tuning.alpha.graders.run(GraderRunParams**kwargs) -> GraderRunResponse`

9**post** `/fine_tuning/alpha/graders/run`

11Run grader

13### Parameters

15- `grader: Grader`

17 The grader used for the fine-tuning job.

19 - `class StringCheckGrader: …`

21 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

23 - `input: str`

25 The input text. This may include template strings.

27 - `name: str`

29 The name of the grader.

31 - `operation: Literal["eq", "ne", "like", "ilike"]`

33 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

35 - `"eq"`

37 - `"ne"`

39 - `"like"`

41 - `"ilike"`

43 - `reference: str`

45 The reference text. This may include template strings.

47 - `type: Literal["string_check"]`

49 The object type, which is always `string_check`.

51 - `"string_check"`

53 - `class TextSimilarityGrader: …`

55 A TextSimilarityGrader object which grades text based on similarity metrics.

57 - `evaluation_metric: Literal["cosine", "fuzzy_match", "bleu", 8 more]`

59 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

60 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

61 or `rouge_l`.

63 - `"cosine"`

65 - `"fuzzy_match"`

67 - `"bleu"`

69 - `"gleu"`

71 - `"meteor"`

73 - `"rouge_1"`

75 - `"rouge_2"`

77 - `"rouge_3"`

79 - `"rouge_4"`

81 - `"rouge_5"`

83 - `"rouge_l"`

85 - `input: str`

87 The text being graded.

89 - `name: str`

91 The name of the grader.

93 - `reference: str`

95 The text being graded against.

97 - `type: Literal["text_similarity"]`

99 The type of grader.

100

101 - `"text_similarity"`

102

103 - `class PythonGrader: …`

104

105 A PythonGrader object that runs a python script on the input.

106

107 - `name: str`

108

109 The name of the grader.

110

111 - `source: str`

112

113 The source code of the python script.

114

115 - `type: Literal["python"]`

116

117 The object type, which is always `python`.

118

119 - `"python"`

120

121 - `image_tag: Optional[str]`

122

123 The image tag to use for the python script.

124

125 - `class ScoreModelGrader: …`

126

127 A ScoreModelGrader object that uses a model to assign a score to the input.

128

129 - `input: List[Input]`

130

131 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

132

133 - `content: InputContent`

134

135 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

136

137 - `str`

138

139 A text input to the model.

140

141 - `class ResponseInputText: …`

142

143 A text input to the model.

144

145 - `text: str`

146

147 The text input to the model.

148

149 - `type: Literal["input_text"]`

150

151 The type of the input item. Always `input_text`.

152

153 - `"input_text"`

154

155 - `prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]`

156

157 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.

158

159 - `mode: Literal["explicit"]`

160

161 The breakpoint mode. Always `explicit`.

162

163 - `"explicit"`

164

165 - `class InputContentOutputText: …`

166

167 A text output from the model.

168

169 - `text: str`

170

171 The text output from the model.

172

173 - `type: Literal["output_text"]`

174

175 The type of the output text. Always `output_text`.

176

177 - `"output_text"`

178

179 - `class InputContentInputImage: …`

180

181 An image input block used within EvalItem content arrays.

182

183 - `image_url: str`

184

185 The URL of the image input.

186

187 - `type: Literal["input_image"]`

188

189 The type of the image input. Always `input_image`.

190

191 - `"input_image"`

192

193 - `detail: Optional[str]`

194

195 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

196

197 - `class ResponseInputAudio: …`

198

199 An audio input to the model.

200

201 - `input_audio: InputAudio`

202

203 - `data: str`

204

205 Base64-encoded audio data.

206

207 - `format: Literal["mp3", "wav"]`

208

209 The format of the audio data. Currently supported formats are `mp3` and

210 `wav`.

211

212 - `"mp3"`

213

214 - `"wav"`

215

216 - `type: Literal["input_audio"]`

217

218 The type of the input item. Always `input_audio`.

219

220 - `"input_audio"`

221

222 - `List[GraderInputItem]`

223

224 - `str`

225

226 A text input to the model.

227

228 - `class ResponseInputText: …`

229

230 A text input to the model.

231

232 - `class GraderInputItemOutputText: …`

233

234 A text output from the model.

235

236 - `text: str`

237

238 The text output from the model.

239

240 - `type: Literal["output_text"]`

241

242 The type of the output text. Always `output_text`.

243

244 - `"output_text"`

245

246 - `class GraderInputItemInputImage: …`

247

248 An image input block used within EvalItem content arrays.

249

250 - `image_url: str`

251

252 The URL of the image input.

253

254 - `type: Literal["input_image"]`

255

256 The type of the image input. Always `input_image`.

257

258 - `"input_image"`

259

260 - `detail: Optional[str]`

261

262 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

263

264 - `class ResponseInputAudio: …`

265

266 An audio input to the model.

267

268 - `role: Literal["user", "assistant", "system", "developer"]`

269

270 The role of the message input. One of `user`, `assistant`, `system`, or

271 `developer`.

272

273 - `"user"`

274

275 - `"assistant"`

276

277 - `"system"`

278

279 - `"developer"`

280

281 - `type: Optional[Literal["message"]]`

282

283 The type of the message input. Always `message`.

284

285 - `"message"`

286

287 - `model: str`

288

289 The model to use for the evaluation.

290

291 - `name: str`

292

293 The name of the grader.

294

295 - `type: Literal["score_model"]`

296

297 The object type, which is always `score_model`.

298

299 - `"score_model"`

300

301 - `range: Optional[List[float]]`

302

303 The range of the score. Defaults to `[0, 1]`.

304

305 - `sampling_params: Optional[SamplingParams]`

306

307 The sampling parameters for the model.

308

309 - `max_completions_tokens: Optional[int]`

310

311 The maximum number of tokens the grader model may generate in its response.

312

313 - `reasoning_effort: Optional[ReasoningEffort]`

314

315 Constrains effort on reasoning for reasoning models. Currently supported

316 values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`.

317 Reducing reasoning effort can result in faster responses and fewer tokens

318 used on reasoning in a response. Not all reasoning models support every

319 value. See the

320 [reasoning guide](https://platform.openai.com/docs/guides/reasoning)

321 for model-specific support.

322

323 - `"none"`

324

325 - `"minimal"`

326

327 - `"low"`

328

329 - `"medium"`

330

331 - `"high"`

332

333 - `"xhigh"`

334

335 - `"max"`

336

337 - `seed: Optional[int]`

338

339 A seed value to initialize the randomness, during sampling.

340

341 - `temperature: Optional[float]`

342

343 A higher temperature increases randomness in the outputs.

344

345 - `top_p: Optional[float]`

346

347 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

348

349 - `class MultiGrader: …`

350

351 A MultiGrader object combines the output of multiple graders to produce a single score.

352

353 - `calculate_output: str`

354

355 A formula to calculate the output based on grader results.

356

357 - `graders: Graders`

358

359 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

360

361 - `class StringCheckGrader: …`

362

363 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

364

365 - `class TextSimilarityGrader: …`

366

367 A TextSimilarityGrader object which grades text based on similarity metrics.

368

369 - `class PythonGrader: …`

370

371 A PythonGrader object that runs a python script on the input.

372

373 - `class ScoreModelGrader: …`

374

375 A ScoreModelGrader object that uses a model to assign a score to the input.

376

377 - `class LabelModelGrader: …`

378

379 A LabelModelGrader object which uses a model to assign labels to each item

380 in the evaluation.

381

382 - `input: List[Input]`

383

384 - `content: InputContent`

385

386 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

387

388 - `str`

389

390 A text input to the model.

391

392 - `class ResponseInputText: …`

393

394 A text input to the model.

395

396 - `class InputContentOutputText: …`

397

398 A text output from the model.

399

400 - `text: str`

401

402 The text output from the model.

403

404 - `type: Literal["output_text"]`

405

406 The type of the output text. Always `output_text`.

407

408 - `"output_text"`

409

410 - `class InputContentInputImage: …`

411

412 An image input block used within EvalItem content arrays.

413

414 - `image_url: str`

415

416 The URL of the image input.

417

418 - `type: Literal["input_image"]`

419

420 The type of the image input. Always `input_image`.

421

422 - `"input_image"`

423

424 - `detail: Optional[str]`

425

426 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

427

428 - `class ResponseInputAudio: …`

429

430 An audio input to the model.

431

432 - `List[GraderInputItem]`

433

434 - `str`

435

436 A text input to the model.

437

438 - `class ResponseInputText: …`

439

440 A text input to the model.

441

442 - `class GraderInputItemOutputText: …`

443

444 A text output from the model.

445

446 - `class GraderInputItemInputImage: …`

447

448 An image input block used within EvalItem content arrays.

449

450 - `class ResponseInputAudio: …`

451

452 An audio input to the model.

453

454 - `role: Literal["user", "assistant", "system", "developer"]`

455

456 The role of the message input. One of `user`, `assistant`, `system`, or

457 `developer`.

458

459 - `"user"`

460

461 - `"assistant"`

462

463 - `"system"`

464

465 - `"developer"`

466

467 - `type: Optional[Literal["message"]]`

468

469 The type of the message input. Always `message`.

470

471 - `"message"`

472

473 - `labels: List[str]`

474

475 The labels to assign to each item in the evaluation.

476

477 - `model: str`

478

479 The model to use for the evaluation. Must support structured outputs.

480

481 - `name: str`

482

483 The name of the grader.

484

485 - `passing_labels: List[str]`

486

487 The labels that indicate a passing result. Must be a subset of labels.

488

489 - `type: Literal["label_model"]`

490

491 The object type, which is always `label_model`.

492

493 - `"label_model"`

494

495 - `name: str`

496

497 The name of the grader.

498

499 - `type: Literal["multi"]`

500

501 The object type, which is always `multi`.

502

503 - `"multi"`

504

505- `model_sample: str`

506

507 The model sample to be evaluated. This value will be used to populate

508 the `sample` namespace. See [the guide](https://platform.openai.com/docs/guides/graders) for more details.

509 The `output_json` variable will be populated if the model sample is a

510 valid JSON string.

511

512- `item: Optional[object]`

513

514 The dataset item provided to the grader. This will be used to populate

515 the `item` namespace. See [the guide](https://platform.openai.com/docs/guides/graders) for more details.

516

517### Returns

518

519- `class GraderRunResponse: …`

520

521 - `metadata: Metadata`

522

523 - `errors: MetadataErrors`

524

525 - `formula_parse_error: bool`

526

527 - `invalid_variable_error: bool`

528

529 - `model_grader_parse_error: bool`

530

531 - `model_grader_refusal_error: bool`

532

533 - `model_grader_server_error: bool`

534

535 - `model_grader_server_error_details: Optional[str]`

536

537 - `other_error: bool`

538

539 - `python_grader_runtime_error: bool`

540

541 - `python_grader_runtime_error_details: Optional[str]`

542

543 - `python_grader_server_error: bool`

544

545 - `python_grader_server_error_type: Optional[str]`

546

547 - `sample_parse_error: bool`

548

549 - `truncated_observation_error: bool`

550

551 - `unresponsive_reward_error: bool`

552

553 - `execution_time: float`

554

555 - `name: str`

556

557 - `sampled_model_name: Optional[str]`

558

559 - `scores: Dict[str, object]`

560

561 - `token_usage: Optional[int]`

562

563 - `type: str`

564

565 - `model_grader_token_usage_per_model: Dict[str, object]`

566

567 - `reward: float`

568

569 - `sub_rewards: Dict[str, object]`

570

571### Example

572

573```python

574import os

575from openai import OpenAI

576

577client = OpenAI(

578 api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted

579)

580response = client.fine_tuning.alpha.graders.run(

581 grader={

582 "input": "input",

583 "name": "name",

584 "operation": "eq",

585 "reference": "reference",

586 "type": "string_check",

587 },

588 model_sample="model_sample",

589)

590print(response.metadata)

591```

592

593#### Response

594

595```json

596{

597 "metadata": {

598 "errors": {

599 "formula_parse_error": true,

600 "invalid_variable_error": true,

601 "model_grader_parse_error": true,

602 "model_grader_refusal_error": true,

603 "model_grader_server_error": true,

604 "model_grader_server_error_details": "model_grader_server_error_details",

605 "other_error": true,

606 "python_grader_runtime_error": true,

607 "python_grader_runtime_error_details": "python_grader_runtime_error_details",

608 "python_grader_server_error": true,

609 "python_grader_server_error_type": "python_grader_server_error_type",

610 "sample_parse_error": true,

611 "truncated_observation_error": true,

612 "unresponsive_reward_error": true

613 },

614 "execution_time": 0,

615 "name": "name",

616 "sampled_model_name": "sampled_model_name",

617 "scores": {

618 "foo": "bar"

619 },

620 "token_usage": 0,

621 "type": "type"

622 },

623 "model_grader_token_usage_per_model": {

624 "foo": "bar"

625 },

626 "reward": 0,

627 "sub_rewards": {

628 "foo": "bar"

629 }

630}

631```

632

633### Score text alignment

634

635```python

636from openai import OpenAI

637

638client = OpenAI()

639result = client.fine_tuning.alpha.graders.run(

640 grader={

641 "type": "score_model",

642 "name": "Example score model grader",

643 "input": [

644 {

645 "role": "user",

646 "content": [

647 {

648 "type": "input_text",

649 "text": "Score how close the reference answer is to the model answer on a 0-1 scale. Return only the score.\n\nReference answer: {{item.reference_answer}}\n\nModel answer: {{sample.output_text}}",

650 }

651 ],

652 }

653 ],

654 "model": "gpt-5-mini",

655 "sampling_params": {"temperature": 1, "top_p": 1, "seed": 42},

656 },

657 item={"reference_answer": "fuzzy wuzzy was a bear"},

658 model_sample="fuzzy wuzzy was a bear",

659)

660print(result)

661```

662

663#### Response

664

665```json

666{

667 "reward": 1.0,

668 "metadata": {

669 "name": "Example score model grader",

670 "type": "score_model",

671 "errors": {

672 "formula_parse_error": false,

673 "sample_parse_error": false,

674 "truncated_observation_error": false,

675 "unresponsive_reward_error": false,

676 "invalid_variable_error": false,

677 "other_error": false,

678 "python_grader_server_error": false,

679 "python_grader_server_error_type": null,

680 "python_grader_runtime_error": false,

681 "python_grader_runtime_error_details": null,

682 "model_grader_server_error": false,

683 "model_grader_refusal_error": false,

684 "model_grader_parse_error": false,

685 "model_grader_server_error_details": null

686 },

687 "execution_time": 4.365238428115845,

688 "scores": {},

689 "token_usage": {

690 "prompt_tokens": 190,

691 "total_tokens": 324,

692 "completion_tokens": 134,

693 "cached_tokens": 0

694 },

695 "sampled_model_name": "gpt-4o-2024-08-06"

696 },

697 "sub_rewards": {},

698 "model_grader_token_usage_per_model": {

699 "gpt-4o-2024-08-06": {

700 "prompt_tokens": 190,

701 "total_tokens": 324,

702 "completion_tokens": 134,

703 "cached_tokens": 0

704 }

705 }

706}

707```

708

709## Validate grader

710

711`fine_tuning.alpha.graders.validate(GraderValidateParams**kwargs) -> GraderValidateResponse`

712

713**post** `/fine_tuning/alpha/graders/validate`

714

715Validate grader

716

717### Parameters

718

719- `grader: Grader`

720

721 The grader used for the fine-tuning job.

722

723 - `class StringCheckGrader: …`

724

725 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

726

727 - `input: str`

728

729 The input text. This may include template strings.

730

731 - `name: str`

732

733 The name of the grader.

734

735 - `operation: Literal["eq", "ne", "like", "ilike"]`

736

737 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

738

739 - `"eq"`

740

741 - `"ne"`

742

743 - `"like"`

744

745 - `"ilike"`

746

747 - `reference: str`

748

749 The reference text. This may include template strings.

750

751 - `type: Literal["string_check"]`

752

753 The object type, which is always `string_check`.

754

755 - `"string_check"`

756

757 - `class TextSimilarityGrader: …`

758

759 A TextSimilarityGrader object which grades text based on similarity metrics.

760

761 - `evaluation_metric: Literal["cosine", "fuzzy_match", "bleu", 8 more]`

762

763 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

764 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

765 or `rouge_l`.

766

767 - `"cosine"`

768

769 - `"fuzzy_match"`

770

771 - `"bleu"`

772

773 - `"gleu"`

774

775 - `"meteor"`

776

777 - `"rouge_1"`

778

779 - `"rouge_2"`

780

781 - `"rouge_3"`

782

783 - `"rouge_4"`

784

785 - `"rouge_5"`

786

787 - `"rouge_l"`

788

789 - `input: str`

790

791 The text being graded.

792

793 - `name: str`

794

795 The name of the grader.

796

797 - `reference: str`

798

799 The text being graded against.

800

801 - `type: Literal["text_similarity"]`

802

803 The type of grader.

804

805 - `"text_similarity"`

806

807 - `class PythonGrader: …`

808

809 A PythonGrader object that runs a python script on the input.

810

811 - `name: str`

812

813 The name of the grader.

814

815 - `source: str`

816

817 The source code of the python script.

818

819 - `type: Literal["python"]`

820

821 The object type, which is always `python`.

822

823 - `"python"`

824

825 - `image_tag: Optional[str]`

826

827 The image tag to use for the python script.

828

829 - `class ScoreModelGrader: …`

830

831 A ScoreModelGrader object that uses a model to assign a score to the input.

832

833 - `input: List[Input]`

834

835 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

836

837 - `content: InputContent`

838

839 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

840

841 - `str`

842

843 A text input to the model.

844

845 - `class ResponseInputText: …`

846

847 A text input to the model.

848

849 - `text: str`

850

851 The text input to the model.

852

853 - `type: Literal["input_text"]`

854

855 The type of the input item. Always `input_text`.

856

857 - `"input_text"`

858

859 - `prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]`

860

861 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.

862

863 - `mode: Literal["explicit"]`

864

865 The breakpoint mode. Always `explicit`.

866

867 - `"explicit"`

868

869 - `class InputContentOutputText: …`

870

871 A text output from the model.

872

873 - `text: str`

874

875 The text output from the model.

876

877 - `type: Literal["output_text"]`

878

879 The type of the output text. Always `output_text`.

880

881 - `"output_text"`

882

883 - `class InputContentInputImage: …`

884

885 An image input block used within EvalItem content arrays.

886

887 - `image_url: str`

888

889 The URL of the image input.

890

891 - `type: Literal["input_image"]`

892

893 The type of the image input. Always `input_image`.

894

895 - `"input_image"`

896

897 - `detail: Optional[str]`

898

899 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

900

901 - `class ResponseInputAudio: …`

902

903 An audio input to the model.

904

905 - `input_audio: InputAudio`

906

907 - `data: str`

908

909 Base64-encoded audio data.

910

911 - `format: Literal["mp3", "wav"]`

912

913 The format of the audio data. Currently supported formats are `mp3` and

914 `wav`.

915

916 - `"mp3"`

917

918 - `"wav"`

919

920 - `type: Literal["input_audio"]`

921

922 The type of the input item. Always `input_audio`.

923

924 - `"input_audio"`

925

926 - `List[GraderInputItem]`

927

928 - `str`

929

930 A text input to the model.

931

932 - `class ResponseInputText: …`

933

934 A text input to the model.

935

936 - `class GraderInputItemOutputText: …`

937

938 A text output from the model.

939

940 - `text: str`

941

942 The text output from the model.

943

944 - `type: Literal["output_text"]`

945

946 The type of the output text. Always `output_text`.

947

948 - `"output_text"`

949

950 - `class GraderInputItemInputImage: …`

951

952 An image input block used within EvalItem content arrays.

953

954 - `image_url: str`

955

956 The URL of the image input.

957

958 - `type: Literal["input_image"]`

959

960 The type of the image input. Always `input_image`.

961

962 - `"input_image"`

963

964 - `detail: Optional[str]`

965

966 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

967

968 - `class ResponseInputAudio: …`

969

970 An audio input to the model.

971

972 - `role: Literal["user", "assistant", "system", "developer"]`

973

974 The role of the message input. One of `user`, `assistant`, `system`, or

975 `developer`.

976

977 - `"user"`

978

979 - `"assistant"`

980

981 - `"system"`

982

983 - `"developer"`

984

985 - `type: Optional[Literal["message"]]`

986

987 The type of the message input. Always `message`.

988

989 - `"message"`

990

991 - `model: str`

992

993 The model to use for the evaluation.

994

995 - `name: str`

996

997 The name of the grader.

998

999 - `type: Literal["score_model"]`

1000

1001 The object type, which is always `score_model`.

1002

1003 - `"score_model"`

1004

1005 - `range: Optional[List[float]]`

1006

1007 The range of the score. Defaults to `[0, 1]`.

1008

1009 - `sampling_params: Optional[SamplingParams]`

1010

1011 The sampling parameters for the model.

1012

1013 - `max_completions_tokens: Optional[int]`

1014

1015 The maximum number of tokens the grader model may generate in its response.

1016

1017 - `reasoning_effort: Optional[ReasoningEffort]`

1018

1019 Constrains effort on reasoning for reasoning models. Currently supported

1020 values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`.

1021 Reducing reasoning effort can result in faster responses and fewer tokens

1022 used on reasoning in a response. Not all reasoning models support every

1023 value. See the

1024 [reasoning guide](https://platform.openai.com/docs/guides/reasoning)

1025 for model-specific support.

1026

1027 - `"none"`

1028

1029 - `"minimal"`

1030

1031 - `"low"`

1032

1033 - `"medium"`

1034

1035 - `"high"`

1036

1037 - `"xhigh"`

1038

1039 - `"max"`

1040

1041 - `seed: Optional[int]`

1042

1043 A seed value to initialize the randomness, during sampling.

1044

1045 - `temperature: Optional[float]`

1046

1047 A higher temperature increases randomness in the outputs.

1048

1049 - `top_p: Optional[float]`

1050

1051 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

1052

1053 - `class MultiGrader: …`

1054

1055 A MultiGrader object combines the output of multiple graders to produce a single score.

1056

1057 - `calculate_output: str`

1058

1059 A formula to calculate the output based on grader results.

1060

1061 - `graders: Graders`

1062

1063 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1064

1065 - `class StringCheckGrader: …`

1066

1067 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1068

1069 - `class TextSimilarityGrader: …`

1070

1071 A TextSimilarityGrader object which grades text based on similarity metrics.

1072

1073 - `class PythonGrader: …`

1074

1075 A PythonGrader object that runs a python script on the input.

1076

1077 - `class ScoreModelGrader: …`

1078

1079 A ScoreModelGrader object that uses a model to assign a score to the input.

1080

1081 - `class LabelModelGrader: …`

1082

1083 A LabelModelGrader object which uses a model to assign labels to each item

1084 in the evaluation.

1085

1086 - `input: List[Input]`

1087

1088 - `content: InputContent`

1089

1090 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1091

1092 - `str`

1093

1094 A text input to the model.

1095

1096 - `class ResponseInputText: …`

1097

1098 A text input to the model.

1099

1100 - `class InputContentOutputText: …`

1101

1102 A text output from the model.

1103

1104 - `text: str`

1105

1106 The text output from the model.

1107

1108 - `type: Literal["output_text"]`

1109

1110 The type of the output text. Always `output_text`.

1111

1112 - `"output_text"`

1113

1114 - `class InputContentInputImage: …`

1115

1116 An image input block used within EvalItem content arrays.

1117

1118 - `image_url: str`

1119

1120 The URL of the image input.

1121

1122 - `type: Literal["input_image"]`

1123

1124 The type of the image input. Always `input_image`.

1125

1126 - `"input_image"`

1127

1128 - `detail: Optional[str]`

1129

1130 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1131

1132 - `class ResponseInputAudio: …`

1133

1134 An audio input to the model.

1135

1136 - `List[GraderInputItem]`

1137

1138 - `str`

1139

1140 A text input to the model.

1141

1142 - `class ResponseInputText: …`

1143

1144 A text input to the model.

1145

1146 - `class GraderInputItemOutputText: …`

1147

1148 A text output from the model.

1149

1150 - `class GraderInputItemInputImage: …`

1151

1152 An image input block used within EvalItem content arrays.

1153

1154 - `class ResponseInputAudio: …`

1155

1156 An audio input to the model.

1157

1158 - `role: Literal["user", "assistant", "system", "developer"]`

1159

1160 The role of the message input. One of `user`, `assistant`, `system`, or

1161 `developer`.

1162

1163 - `"user"`

1164

1165 - `"assistant"`

1166

1167 - `"system"`

1168

1169 - `"developer"`

1170

1171 - `type: Optional[Literal["message"]]`

1172

1173 The type of the message input. Always `message`.

1174

1175 - `"message"`

1176

1177 - `labels: List[str]`

1178

1179 The labels to assign to each item in the evaluation.

1180

1181 - `model: str`

1182

1183 The model to use for the evaluation. Must support structured outputs.

1184

1185 - `name: str`

1186

1187 The name of the grader.

1188

1189 - `passing_labels: List[str]`

1190

1191 The labels that indicate a passing result. Must be a subset of labels.

1192

1193 - `type: Literal["label_model"]`

1194

1195 The object type, which is always `label_model`.

1196

1197 - `"label_model"`

1198

1199 - `name: str`

1200

1201 The name of the grader.

1202

1203 - `type: Literal["multi"]`

1204

1205 The object type, which is always `multi`.

1206

1207 - `"multi"`

1208

1209### Returns

1210

1211- `class GraderValidateResponse: …`

1212

1213 - `grader: Optional[Grader]`

1214

1215 The grader used for the fine-tuning job.

1216

1217 - `class StringCheckGrader: …`

1218

1219 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1220

1221 - `input: str`

1222

1223 The input text. This may include template strings.

1224

1225 - `name: str`

1226

1227 The name of the grader.

1228

1229 - `operation: Literal["eq", "ne", "like", "ilike"]`

1230

1231 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

1232

1233 - `"eq"`

1234

1235 - `"ne"`

1236

1237 - `"like"`

1238

1239 - `"ilike"`

1240

1241 - `reference: str`

1242

1243 The reference text. This may include template strings.

1244

1245 - `type: Literal["string_check"]`

1246

1247 The object type, which is always `string_check`.

1248

1249 - `"string_check"`

1250

1251 - `class TextSimilarityGrader: …`

1252

1253 A TextSimilarityGrader object which grades text based on similarity metrics.

1254

1255 - `evaluation_metric: Literal["cosine", "fuzzy_match", "bleu", 8 more]`

1256

1257 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

1258 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

1259 or `rouge_l`.

1260

1261 - `"cosine"`

1262

1263 - `"fuzzy_match"`

1264

1265 - `"bleu"`

1266

1267 - `"gleu"`

1268

1269 - `"meteor"`

1270

1271 - `"rouge_1"`

1272

1273 - `"rouge_2"`

1274

1275 - `"rouge_3"`

1276

1277 - `"rouge_4"`

1278

1279 - `"rouge_5"`

1280

1281 - `"rouge_l"`

1282

1283 - `input: str`

1284

1285 The text being graded.

1286

1287 - `name: str`

1288

1289 The name of the grader.

1290

1291 - `reference: str`

1292

1293 The text being graded against.

1294

1295 - `type: Literal["text_similarity"]`

1296

1297 The type of grader.

1298

1299 - `"text_similarity"`

1300

1301 - `class PythonGrader: …`

1302

1303 A PythonGrader object that runs a python script on the input.

1304

1305 - `name: str`

1306

1307 The name of the grader.

1308

1309 - `source: str`

1310

1311 The source code of the python script.

1312

1313 - `type: Literal["python"]`

1314

1315 The object type, which is always `python`.

1316

1317 - `"python"`

1318

1319 - `image_tag: Optional[str]`

1320

1321 The image tag to use for the python script.

1322

1323 - `class ScoreModelGrader: …`

1324

1325 A ScoreModelGrader object that uses a model to assign a score to the input.

1326

1327 - `input: List[Input]`

1328

1329 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

1330

1331 - `content: InputContent`

1332

1333 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1334

1335 - `str`

1336

1337 A text input to the model.

1338

1339 - `class ResponseInputText: …`

1340

1341 A text input to the model.

1342

1343 - `text: str`

1344

1345 The text input to the model.

1346

1347 - `type: Literal["input_text"]`

1348

1349 The type of the input item. Always `input_text`.

1350

1351 - `"input_text"`

1352

1353 - `prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]`

1354

1355 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.

1356

1357 - `mode: Literal["explicit"]`

1358

1359 The breakpoint mode. Always `explicit`.

1360

1361 - `"explicit"`

1362

1363 - `class InputContentOutputText: …`

1364

1365 A text output from the model.

1366

1367 - `text: str`

1368

1369 The text output from the model.

1370

1371 - `type: Literal["output_text"]`

1372

1373 The type of the output text. Always `output_text`.

1374

1375 - `"output_text"`

1376

1377 - `class InputContentInputImage: …`

1378

1379 An image input block used within EvalItem content arrays.

1380

1381 - `image_url: str`

1382

1383 The URL of the image input.

1384

1385 - `type: Literal["input_image"]`

1386

1387 The type of the image input. Always `input_image`.

1388

1389 - `"input_image"`

1390

1391 - `detail: Optional[str]`

1392

1393 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1394

1395 - `class ResponseInputAudio: …`

1396

1397 An audio input to the model.

1398

1399 - `input_audio: InputAudio`

1400

1401 - `data: str`

1402

1403 Base64-encoded audio data.

1404

1405 - `format: Literal["mp3", "wav"]`

1406

1407 The format of the audio data. Currently supported formats are `mp3` and

1408 `wav`.

1409

1410 - `"mp3"`

1411

1412 - `"wav"`

1413

1414 - `type: Literal["input_audio"]`

1415

1416 The type of the input item. Always `input_audio`.

1417

1418 - `"input_audio"`

1419

1420 - `List[GraderInputItem]`

1421

1422 - `str`

1423

1424 A text input to the model.

1425

1426 - `class ResponseInputText: …`

1427

1428 A text input to the model.

1429

1430 - `class GraderInputItemOutputText: …`

1431

1432 A text output from the model.

1433

1434 - `text: str`

1435

1436 The text output from the model.

1437

1438 - `type: Literal["output_text"]`

1439

1440 The type of the output text. Always `output_text`.

1441

1442 - `"output_text"`

1443

1444 - `class GraderInputItemInputImage: …`

1445

1446 An image input block used within EvalItem content arrays.

1447

1448 - `image_url: str`

1449

1450 The URL of the image input.

1451

1452 - `type: Literal["input_image"]`

1453

1454 The type of the image input. Always `input_image`.

1455

1456 - `"input_image"`

1457

1458 - `detail: Optional[str]`

1459

1460 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1461

1462 - `class ResponseInputAudio: …`

1463

1464 An audio input to the model.

1465

1466 - `role: Literal["user", "assistant", "system", "developer"]`

1467

1468 The role of the message input. One of `user`, `assistant`, `system`, or

1469 `developer`.

1470

1471 - `"user"`

1472

1473 - `"assistant"`

1474

1475 - `"system"`

1476

1477 - `"developer"`

1478

1479 - `type: Optional[Literal["message"]]`

1480

1481 The type of the message input. Always `message`.

1482

1483 - `"message"`

1484

1485 - `model: str`

1486

1487 The model to use for the evaluation.

1488

1489 - `name: str`

1490

1491 The name of the grader.

1492

1493 - `type: Literal["score_model"]`

1494

1495 The object type, which is always `score_model`.

1496

1497 - `"score_model"`

1498

1499 - `range: Optional[List[float]]`

1500

1501 The range of the score. Defaults to `[0, 1]`.

1502

1503 - `sampling_params: Optional[SamplingParams]`

1504

1505 The sampling parameters for the model.

1506

1507 - `max_completions_tokens: Optional[int]`

1508

1509 The maximum number of tokens the grader model may generate in its response.

1510

1511 - `reasoning_effort: Optional[ReasoningEffort]`

1512

1513 Constrains effort on reasoning for reasoning models. Currently supported

1514 values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`.

1515 Reducing reasoning effort can result in faster responses and fewer tokens

1516 used on reasoning in a response. Not all reasoning models support every

1517 value. See the

1518 [reasoning guide](https://platform.openai.com/docs/guides/reasoning)

1519 for model-specific support.

1520

1521 - `"none"`

1522

1523 - `"minimal"`

1524

1525 - `"low"`

1526

1527 - `"medium"`

1528

1529 - `"high"`

1530

1531 - `"xhigh"`

1532

1533 - `"max"`

1534

1535 - `seed: Optional[int]`

1536

1537 A seed value to initialize the randomness, during sampling.

1538

1539 - `temperature: Optional[float]`

1540

1541 A higher temperature increases randomness in the outputs.

1542

1543 - `top_p: Optional[float]`

1544

1545 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

1546

1547 - `class MultiGrader: …`

1548

1549 A MultiGrader object combines the output of multiple graders to produce a single score.

1550

1551 - `calculate_output: str`

1552

1553 A formula to calculate the output based on grader results.

1554

1555 - `graders: Graders`

1556

1557 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1558

1559 - `class StringCheckGrader: …`

1560

1561 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1562

1563 - `class TextSimilarityGrader: …`

1564

1565 A TextSimilarityGrader object which grades text based on similarity metrics.

1566

1567 - `class PythonGrader: …`

1568

1569 A PythonGrader object that runs a python script on the input.

1570

1571 - `class ScoreModelGrader: …`

1572

1573 A ScoreModelGrader object that uses a model to assign a score to the input.

1574

1575 - `class LabelModelGrader: …`

1576

1577 A LabelModelGrader object which uses a model to assign labels to each item

1578 in the evaluation.

1579

1580 - `input: List[Input]`

1581

1582 - `content: InputContent`

1583

1584 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1585

1586 - `str`

1587

1588 A text input to the model.

1589

1590 - `class ResponseInputText: …`

1591

1592 A text input to the model.

1593

1594 - `class InputContentOutputText: …`

1595

1596 A text output from the model.

1597

1598 - `text: str`

1599

1600 The text output from the model.

1601

1602 - `type: Literal["output_text"]`

1603

1604 The type of the output text. Always `output_text`.

1605

1606 - `"output_text"`

1607

1608 - `class InputContentInputImage: …`

1609

1610 An image input block used within EvalItem content arrays.

1611

1612 - `image_url: str`

1613

1614 The URL of the image input.

1615

1616 - `type: Literal["input_image"]`

1617

1618 The type of the image input. Always `input_image`.

1619

1620 - `"input_image"`

1621

1622 - `detail: Optional[str]`

1623

1624 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1625

1626 - `class ResponseInputAudio: …`

1627

1628 An audio input to the model.

1629

1630 - `List[GraderInputItem]`

1631

1632 - `str`

1633

1634 A text input to the model.

1635

1636 - `class ResponseInputText: …`

1637

1638 A text input to the model.

1639

1640 - `class GraderInputItemOutputText: …`

1641

1642 A text output from the model.

1643

1644 - `class GraderInputItemInputImage: …`

1645

1646 An image input block used within EvalItem content arrays.

1647

1648 - `class ResponseInputAudio: …`

1649

1650 An audio input to the model.

1651

1652 - `role: Literal["user", "assistant", "system", "developer"]`

1653

1654 The role of the message input. One of `user`, `assistant`, `system`, or

1655 `developer`.

1656

1657 - `"user"`

1658

1659 - `"assistant"`

1660

1661 - `"system"`

1662

1663 - `"developer"`

1664

1665 - `type: Optional[Literal["message"]]`

1666

1667 The type of the message input. Always `message`.

1668

1669 - `"message"`

1670

1671 - `labels: List[str]`

1672

1673 The labels to assign to each item in the evaluation.

1674

1675 - `model: str`

1676

1677 The model to use for the evaluation. Must support structured outputs.

1678

1679 - `name: str`

1680

1681 The name of the grader.

1682

1683 - `passing_labels: List[str]`

1684

1685 The labels that indicate a passing result. Must be a subset of labels.

1686

1687 - `type: Literal["label_model"]`

1688

1689 The object type, which is always `label_model`.

1690

1691 - `"label_model"`

1692

1693 - `name: str`

1694

1695 The name of the grader.

1696

1697 - `type: Literal["multi"]`

1698

1699 The object type, which is always `multi`.

1700

1701 - `"multi"`

1702

1703### Example

1704

1705```python

1706import os

1707from openai import OpenAI

1708

1709client = OpenAI(

1710 api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted

1711)

1712response = client.fine_tuning.alpha.graders.validate(

1713 grader={

1714 "input": "input",

1715 "name": "name",

1716 "operation": "eq",

1717 "reference": "reference",

1718 "type": "string_check",

1719 },

1720)

1721print(response.grader)

1722```

1723

1724#### Response

1725

1726```json

1727{

1728 "grader": {

1729 "input": "input",

1730 "name": "name",

1731 "operation": "eq",

1732 "reference": "reference",

1733 "type": "string_check"

1734 }

1735}

1736```

1737

1738## Domain Types

1739

1740### Grader Run Response

1741

1742- `class GraderRunResponse: …`

1743

1744 - `metadata: Metadata`

1745

1746 - `errors: MetadataErrors`

1747

1748 - `formula_parse_error: bool`

1749

1750 - `invalid_variable_error: bool`

1751

1752 - `model_grader_parse_error: bool`

1753

1754 - `model_grader_refusal_error: bool`

1755

1756 - `model_grader_server_error: bool`

1757

1758 - `model_grader_server_error_details: Optional[str]`

1759

1760 - `other_error: bool`

1761

1762 - `python_grader_runtime_error: bool`

1763

1764 - `python_grader_runtime_error_details: Optional[str]`

1765

1766 - `python_grader_server_error: bool`

1767

1768 - `python_grader_server_error_type: Optional[str]`

1769

1770 - `sample_parse_error: bool`

1771

1772 - `truncated_observation_error: bool`

1773

1774 - `unresponsive_reward_error: bool`

1775

1776 - `execution_time: float`

1777

1778 - `name: str`

1779

1780 - `sampled_model_name: Optional[str]`

1781

1782 - `scores: Dict[str, object]`

1783

1784 - `token_usage: Optional[int]`

1785

1786 - `type: str`

1787

1788 - `model_grader_token_usage_per_model: Dict[str, object]`

1789

1790 - `reward: float`

1791

1792 - `sub_rewards: Dict[str, object]`

1793

1794### Grader Validate Response

1795

1796- `class GraderValidateResponse: …`

1797

1798 - `grader: Optional[Grader]`

1799

1800 The grader used for the fine-tuning job.

1801

1802 - `class StringCheckGrader: …`

1803

1804 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1805

1806 - `input: str`

1807

1808 The input text. This may include template strings.

1809

1810 - `name: str`

1811

1812 The name of the grader.

1813

1814 - `operation: Literal["eq", "ne", "like", "ilike"]`

1815

1816 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

1817

1818 - `"eq"`

1819

1820 - `"ne"`

1821

1822 - `"like"`

1823

1824 - `"ilike"`

1825

1826 - `reference: str`

1827

1828 The reference text. This may include template strings.

1829

1830 - `type: Literal["string_check"]`

1831

1832 The object type, which is always `string_check`.

1833

1834 - `"string_check"`

1835

1836 - `class TextSimilarityGrader: …`

1837

1838 A TextSimilarityGrader object which grades text based on similarity metrics.

1839

1840 - `evaluation_metric: Literal["cosine", "fuzzy_match", "bleu", 8 more]`

1841

1842 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

1843 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

1844 or `rouge_l`.

1845

1846 - `"cosine"`

1847

1848 - `"fuzzy_match"`

1849

1850 - `"bleu"`

1851

1852 - `"gleu"`

1853

1854 - `"meteor"`

1855

1856 - `"rouge_1"`

1857

1858 - `"rouge_2"`

1859

1860 - `"rouge_3"`

1861

1862 - `"rouge_4"`

1863

1864 - `"rouge_5"`

1865

1866 - `"rouge_l"`

1867

1868 - `input: str`

1869

1870 The text being graded.

1871

1872 - `name: str`

1873

1874 The name of the grader.

1875

1876 - `reference: str`

1877

1878 The text being graded against.

1879

1880 - `type: Literal["text_similarity"]`

1881

1882 The type of grader.

1883

1884 - `"text_similarity"`

1885

1886 - `class PythonGrader: …`

1887

1888 A PythonGrader object that runs a python script on the input.

1889

1890 - `name: str`

1891

1892 The name of the grader.

1893

1894 - `source: str`

1895

1896 The source code of the python script.

1897

1898 - `type: Literal["python"]`

1899

1900 The object type, which is always `python`.

1901

1902 - `"python"`

1903

1904 - `image_tag: Optional[str]`

1905

1906 The image tag to use for the python script.

1907

1908 - `class ScoreModelGrader: …`

1909

1910 A ScoreModelGrader object that uses a model to assign a score to the input.

1911

1912 - `input: List[Input]`

1913

1914 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

1915

1916 - `content: InputContent`

1917

1918 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1919

1920 - `str`

1921

1922 A text input to the model.

1923

1924 - `class ResponseInputText: …`

1925

1926 A text input to the model.

1927

1928 - `text: str`

1929

1930 The text input to the model.

1931

1932 - `type: Literal["input_text"]`

1933

1934 The type of the input item. Always `input_text`.

1935

1936 - `"input_text"`

1937

1938 - `prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]`

1939

1940 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.

1941

1942 - `mode: Literal["explicit"]`

1943

1944 The breakpoint mode. Always `explicit`.

1945

1946 - `"explicit"`

1947

1948 - `class InputContentOutputText: …`

1949

1950 A text output from the model.

1951

1952 - `text: str`

1953

1954 The text output from the model.

1955

1956 - `type: Literal["output_text"]`

1957

1958 The type of the output text. Always `output_text`.

1959

1960 - `"output_text"`

1961

1962 - `class InputContentInputImage: …`

1963

1964 An image input block used within EvalItem content arrays.

1965

1966 - `image_url: str`

1967

1968 The URL of the image input.

1969

1970 - `type: Literal["input_image"]`

1971

1972 The type of the image input. Always `input_image`.

1973

1974 - `"input_image"`

1975

1976 - `detail: Optional[str]`

1977

1978 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1979

1980 - `class ResponseInputAudio: …`

1981

1982 An audio input to the model.

1983

1984 - `input_audio: InputAudio`

1985

1986 - `data: str`

1987

1988 Base64-encoded audio data.

1989

1990 - `format: Literal["mp3", "wav"]`

1991

1992 The format of the audio data. Currently supported formats are `mp3` and

1993 `wav`.

1994

1995 - `"mp3"`

1996

1997 - `"wav"`

1998

1999 - `type: Literal["input_audio"]`

2000

2001 The type of the input item. Always `input_audio`.

2002

2003 - `"input_audio"`

2004

2005 - `List[GraderInputItem]`

2006

2007 - `str`

2008

2009 A text input to the model.

2010

2011 - `class ResponseInputText: …`

2012

2013 A text input to the model.

2014

2015 - `class GraderInputItemOutputText: …`

2016

2017 A text output from the model.

2018

2019 - `text: str`

2020

2021 The text output from the model.

2022

2023 - `type: Literal["output_text"]`

2024

2025 The type of the output text. Always `output_text`.

2026

2027 - `"output_text"`

2028

2029 - `class GraderInputItemInputImage: …`

2030

2031 An image input block used within EvalItem content arrays.

2032

2033 - `image_url: str`

2034

2035 The URL of the image input.

2036

2037 - `type: Literal["input_image"]`

2038

2039 The type of the image input. Always `input_image`.

2040

2041 - `"input_image"`

2042

2043 - `detail: Optional[str]`

2044

2045 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

2046

2047 - `class ResponseInputAudio: …`

2048

2049 An audio input to the model.

2050

2051 - `role: Literal["user", "assistant", "system", "developer"]`

2052

2053 The role of the message input. One of `user`, `assistant`, `system`, or

2054 `developer`.

2055

2056 - `"user"`

2057

2058 - `"assistant"`

2059

2060 - `"system"`

2061

2062 - `"developer"`

2063

2064 - `type: Optional[Literal["message"]]`

2065

2066 The type of the message input. Always `message`.

2067

2068 - `"message"`

2069

2070 - `model: str`

2071

2072 The model to use for the evaluation.

2073

2074 - `name: str`

2075

2076 The name of the grader.

2077

2078 - `type: Literal["score_model"]`

2079

2080 The object type, which is always `score_model`.

2081

2082 - `"score_model"`

2083

2084 - `range: Optional[List[float]]`

2085

2086 The range of the score. Defaults to `[0, 1]`.

2087

2088 - `sampling_params: Optional[SamplingParams]`

2089

2090 The sampling parameters for the model.

2091

2092 - `max_completions_tokens: Optional[int]`

2093

2094 The maximum number of tokens the grader model may generate in its response.

2095

2096 - `reasoning_effort: Optional[ReasoningEffort]`

2097

2098 Constrains effort on reasoning for reasoning models. Currently supported

2099 values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`.

2100 Reducing reasoning effort can result in faster responses and fewer tokens

2101 used on reasoning in a response. Not all reasoning models support every

2102 value. See the

2103 [reasoning guide](https://platform.openai.com/docs/guides/reasoning)

2104 for model-specific support.

2105

2106 - `"none"`

2107

2108 - `"minimal"`

2109

2110 - `"low"`

2111

2112 - `"medium"`

2113

2114 - `"high"`

2115

2116 - `"xhigh"`

2117

2118 - `"max"`

2119

2120 - `seed: Optional[int]`

2121

2122 A seed value to initialize the randomness, during sampling.

2123

2124 - `temperature: Optional[float]`

2125

2126 A higher temperature increases randomness in the outputs.

2127

2128 - `top_p: Optional[float]`

2129

2130 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

2131

2132 - `class MultiGrader: …`

2133

2134 A MultiGrader object combines the output of multiple graders to produce a single score.

2135

2136 - `calculate_output: str`

2137

2138 A formula to calculate the output based on grader results.

2139

2140 - `graders: Graders`

2141

2142 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

2143

2144 - `class StringCheckGrader: …`

2145

2146 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

2147

2148 - `class TextSimilarityGrader: …`

2149

2150 A TextSimilarityGrader object which grades text based on similarity metrics.

2151

2152 - `class PythonGrader: …`

2153

2154 A PythonGrader object that runs a python script on the input.

2155

2156 - `class ScoreModelGrader: …`

2157

2158 A ScoreModelGrader object that uses a model to assign a score to the input.

2159

2160 - `class LabelModelGrader: …`

2161

2162 A LabelModelGrader object which uses a model to assign labels to each item

2163 in the evaluation.

2164

2165 - `input: List[Input]`

2166

2167 - `content: InputContent`

2168

2169 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

2170

2171 - `str`

2172

2173 A text input to the model.

2174

2175 - `class ResponseInputText: …`

2176

2177 A text input to the model.

2178

2179 - `class InputContentOutputText: …`

2180

2181 A text output from the model.

2182

2183 - `text: str`

2184

2185 The text output from the model.

2186

2187 - `type: Literal["output_text"]`

2188

2189 The type of the output text. Always `output_text`.

2190

2191 - `"output_text"`

2192

2193 - `class InputContentInputImage: …`

2194

2195 An image input block used within EvalItem content arrays.

2196

2197 - `image_url: str`

2198

2199 The URL of the image input.

2200

2201 - `type: Literal["input_image"]`

2202

2203 The type of the image input. Always `input_image`.

2204

2205 - `"input_image"`

2206

2207 - `detail: Optional[str]`

2208

2209 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

2210

2211 - `class ResponseInputAudio: …`

2212

2213 An audio input to the model.

2214

2215 - `List[GraderInputItem]`

2216

2217 - `str`

2218

2219 A text input to the model.

2220

2221 - `class ResponseInputText: …`

2222

2223 A text input to the model.

2224

2225 - `class GraderInputItemOutputText: …`

2226

2227 A text output from the model.

2228

2229 - `class GraderInputItemInputImage: …`

2230

2231 An image input block used within EvalItem content arrays.

2232

2233 - `class ResponseInputAudio: …`

2234

2235 An audio input to the model.

2236

2237 - `role: Literal["user", "assistant", "system", "developer"]`

2238

2239 The role of the message input. One of `user`, `assistant`, `system`, or

2240 `developer`.

2241

2242 - `"user"`

2243

2244 - `"assistant"`

2245

2246 - `"system"`

2247

2248 - `"developer"`

2249

2250 - `type: Optional[Literal["message"]]`

2251

2252 The type of the message input. Always `message`.

2253

2254 - `"message"`

2255

2256 - `labels: List[str]`

2257

2258 The labels to assign to each item in the evaluation.

2259

2260 - `model: str`

2261

2262 The model to use for the evaluation. Must support structured outputs.

2263

2264 - `name: str`

2265

2266 The name of the grader.

2267

2268 - `passing_labels: List[str]`

2269

2270 The labels that indicate a passing result. Must be a subset of labels.

2271

2272 - `type: Literal["label_model"]`

2273

2274 The object type, which is always `label_model`.

2275

2276 - `"label_model"`

2277

2278 - `name: str`

2279

2280 The name of the grader.

2281

2282 - `type: Literal["multi"]`

2283

2284 The object type, which is always `multi`.

2285

2286 - `"multi"`

python/resources/fine_tuning/subresources/alpha/index.md 2026-07-07 08:02 UTC to 2026-07-09 20:58 UTC

Alpha

Graders

Run grader

Parameters

Returns

Example

Response

Score text alignment

Response

Validate grader

Parameters

Returns

Example

Response

Domain Types

Grader Run Response

Grader Validate Response