Alpha

Graders

Run grader

GraderRunResponse fineTuning().alpha().graders().run(GraderRunParamsparams, RequestOptionsrequestOptions = RequestOptions.none())

post /fine_tuning/alpha/graders/run

Run grader

Parameters

GraderRunParams params
- Grader grader
  
  The grader used for the fine-tuning job.
  - class StringCheckGrader:
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - String input
      
      The input text. This may include template strings.
    - String name
      
      The name of the grader.
    - Operation operation
      
      The string check operation to perform. One of eq, ne, like, or ilike.
      - EQ("eq")
      - NE("ne")
      - LIKE("like")
      - ILIKE("ilike")
    - String reference
      
      The reference text. This may include template strings.
    - JsonValue; type "string_check"constant
      
      The object type, which is always string_check.
      - STRING_CHECK("string_check")
  - class TextSimilarityGrader:
    
    A TextSimilarityGrader object which grades text based on similarity metrics.
    - EvaluationMetric evaluationMetric
      
      The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
      - COSINE("cosine")
      - FUZZY_MATCH("fuzzy_match")
      - BLEU("bleu")
      - GLEU("gleu")
      - METEOR("meteor")
      - ROUGE_1("rouge_1")
      - ROUGE_2("rouge_2")
      - ROUGE_3("rouge_3")
      - ROUGE_4("rouge_4")
      - ROUGE_5("rouge_5")
      - ROUGE_L("rouge_l")
    - String input
      
      The text being graded.
    - String name
      
      The name of the grader.
    - String reference
      
      The text being graded against.
    - JsonValue; type "text_similarity"constant
      
      The type of grader.
      - TEXT_SIMILARITY("text_similarity")
  - class PythonGrader:
    
    A PythonGrader object that runs a python script on the input.
    - String name
      
      The name of the grader.
    - String source
      
      The source code of the python script.
    - JsonValue; type "python"constant
      
      The object type, which is always python.
      - PYTHON("python")
    - Optional<String> imageTag
      
      The image tag to use for the python script.
  - class ScoreModelGrader:
    
    A ScoreModelGrader object that uses a model to assign a score to the input.
    - List<Input> input
      
      The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
      - Content content
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String
        
        class ResponseInputText:
        
        A text input to the model.
        
        String text
        
        The text input to the model.
        
        JsonValue; type "input_text"constant
        
        The type of the input item. Always input_text.
        
        INPUT_TEXT("input_text")
        
        Optional<PromptCacheBreakpoint> promptCacheBreakpoint
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        JsonValue; mode "explicit"constant
        
        The breakpoint mode. Always explicit.
        
        EXPLICIT("explicit")
        
        class OutputText:
        
        A text output from the model.
        
        String text
        
        The text output from the model.
        
        JsonValue; type "output_text"constant
        
        The type of the output text. Always output_text.
        
        OUTPUT_TEXT("output_text")
        
        class InputImage:
        
        An image input block used within EvalItem content arrays.
        
        String imageUrl
        
        The URL of the image input.
        
        JsonValue; type "input_image"constant
        
        The type of the image input. Always input_image.
        
        INPUT_IMAGE("input_image")
        
        Optional<String> detail
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio:
        
        An audio input to the model.
        
        InputAudio inputAudio
        
        String data
        
        Base64-encoded audio data.
        
        Format format
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        MP3("mp3")
        
        WAV("wav")
        
        JsonValue; type "input_audio"constant
        
        The type of the input item. Always input_audio.
        
        INPUT_AUDIO("input_audio")
        
        List<EvalContentItem>
        
        String
        
        class ResponseInputText:
        
        A text input to the model.
        
        OutputText
        
        String text
        
        The text output from the model.
        
        JsonValue; type "output_text"constant
        
        The type of the output text. Always output_text.
        
        OUTPUT_TEXT("output_text")
        
        InputImage
        
        String imageUrl
        
        The URL of the image input.
        
        JsonValue; type "input_image"constant
        
        The type of the image input. Always input_image.
        
        INPUT_IMAGE("input_image")
        
        Optional<String> detail
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio:
        
        An audio input to the model.
      - Role role
        
        The role of the message input. One of user, assistant, system, or developer.
        
        USER("user")
        
        ASSISTANT("assistant")
        
        SYSTEM("system")
        
        DEVELOPER("developer")
      - Optional<Type> type
        
        The type of the message input. Always message.
        
        MESSAGE("message")
    - String model
      
      The model to use for the evaluation.
    - String name
      
      The name of the grader.
    - JsonValue; type "score_model"constant
      
      The object type, which is always score_model.
      - SCORE_MODEL("score_model")
    - Optional<List<Double>> range
      
      The range of the score. Defaults to [0, 1].
    - Optional<SamplingParams> samplingParams
      
      The sampling parameters for the model.
      - Optional<Long> maxCompletionsTokens
        
        The maximum number of tokens the grader model may generate in its response.
      - Optional<ReasoningEffort> reasoningEffort
        
        Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
        
        NONE("none")
        
        MINIMAL("minimal")
        
        LOW("low")
        
        MEDIUM("medium")
        
        HIGH("high")
        
        XHIGH("xhigh")
        
        MAX("max")
      - Optional<Long> seed
        
        A seed value to initialize the randomness, during sampling.
      - Optional<Double> temperature
        
        A higher temperature increases randomness in the outputs.
      - Optional<Double> topP
        
        An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
  - class MultiGrader:
    
    A MultiGrader object combines the output of multiple graders to produce a single score.
    - String calculateOutput
      
      A formula to calculate the output based on grader results.
    - Graders graders
      
      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class StringCheckGrader:
        
        A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class TextSimilarityGrader:
        
        A TextSimilarityGrader object which grades text based on similarity metrics.
      - class PythonGrader:
        
        A PythonGrader object that runs a python script on the input.
      - class ScoreModelGrader:
        
        A ScoreModelGrader object that uses a model to assign a score to the input.
      - class LabelModelGrader:
        
        A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
        
        List<Input> input
        
        Content content
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String
        
        class ResponseInputText:
        
        A text input to the model.
        
        class OutputText:
        
        A text output from the model.
        
        String text
        
        The text output from the model.
        
        JsonValue; type "output_text"constant
        
        The type of the output text. Always output_text.
        
        OUTPUT_TEXT("output_text")
        
        class InputImage:
        
        An image input block used within EvalItem content arrays.
        
        String imageUrl
        
        The URL of the image input.
        
        JsonValue; type "input_image"constant
        
        The type of the image input. Always input_image.
        
        INPUT_IMAGE("input_image")
        
        Optional<String> detail
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio:
        
        An audio input to the model.
        
        List<EvalContentItem>
        
        String
        
        class ResponseInputText:
        
        A text input to the model.
        
        OutputText
        
        InputImage
        
        class ResponseInputAudio:
        
        An audio input to the model.
        
        Role role
        
        The role of the message input. One of user, assistant, system, or developer.
        
        USER("user")
        
        ASSISTANT("assistant")
        
        SYSTEM("system")
        
        DEVELOPER("developer")
        
        Optional<Type> type
        
        The type of the message input. Always message.
        
        MESSAGE("message")
        
        List<String> labels
        
        The labels to assign to each item in the evaluation.
        
        String model
        
        The model to use for the evaluation. Must support structured outputs.
        
        String name
        
        The name of the grader.
        
        List<String> passingLabels
        
        The labels that indicate a passing result. Must be a subset of labels.
        
        JsonValue; type "label_model"constant
        
        The object type, which is always label_model.
        
        LABEL_MODEL("label_model")
    - String name
      
      The name of the grader.
    - JsonValue; type "multi"constant
      
      The object type, which is always multi.
      - MULTI("multi")
- String modelSample
  
  The model sample to be evaluated. This value will be used to populate the sample namespace. See the guide for more details. The output_json variable will be populated if the model sample is a valid JSON string.
- Optional<JsonValue> item
  
  The dataset item provided to the grader. This will be used to populate the item namespace. See the guide for more details.

Returns

class GraderRunResponse:
- Metadata metadata
  - Errors errors
    - boolean formulaParseError
    - boolean invalidVariableError
    - boolean modelGraderParseError
    - boolean modelGraderRefusalError
    - boolean modelGraderServerError
    - Optional<String> modelGraderServerErrorDetails
    - boolean otherError
    - boolean pythonGraderRuntimeError
    - Optional<String> pythonGraderRuntimeErrorDetails
    - boolean pythonGraderServerError
    - Optional<String> pythonGraderServerErrorType
    - boolean sampleParseError
    - boolean truncatedObservationError
    - boolean unresponsiveRewardError
  - double executionTime
  - String name
  - Optional<String> sampledModelName
  - Scores scores
  - Optional<Long> tokenUsage
  - String type
- ModelGraderTokenUsagePerModel modelGraderTokenUsagePerModel
- double reward
- SubRewards subRewards

Example

package com.openai.example;

import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.finetuning.alpha.graders.GraderRunParams;
import com.openai.models.finetuning.alpha.graders.GraderRunResponse;
import com.openai.models.graders.gradermodels.StringCheckGrader;

public final class Main {
    private Main() {}

    public static void main(String[] args) {
        OpenAIClient client = OpenAIOkHttpClient.fromEnv();

        GraderRunParams params = GraderRunParams.builder()
            .grader(StringCheckGrader.builder()
                .input("input")
                .name("name")
                .operation(StringCheckGrader.Operation.EQ)
                .reference("reference")
                .build())
            .modelSample("model_sample")
            .build();
        GraderRunResponse response = client.fineTuning().alpha().graders().run(params);
    }
}

Response

{
  "metadata": {
    "errors": {
      "formula_parse_error": true,
      "invalid_variable_error": true,
      "model_grader_parse_error": true,
      "model_grader_refusal_error": true,
      "model_grader_server_error": true,
      "model_grader_server_error_details": "model_grader_server_error_details",
      "other_error": true,
      "python_grader_runtime_error": true,
      "python_grader_runtime_error_details": "python_grader_runtime_error_details",
      "python_grader_server_error": true,
      "python_grader_server_error_type": "python_grader_server_error_type",
      "sample_parse_error": true,
      "truncated_observation_error": true,
      "unresponsive_reward_error": true
    },
    "execution_time": 0,
    "name": "name",
    "sampled_model_name": "sampled_model_name",
    "scores": {
      "foo": "bar"
    },
    "token_usage": 0,
    "type": "type"
  },
  "model_grader_token_usage_per_model": {
    "foo": "bar"
  },
  "reward": 0,
  "sub_rewards": {
    "foo": "bar"
  }
}

Validate grader

GraderValidateResponse fineTuning().alpha().graders().validate(GraderValidateParamsparams, RequestOptionsrequestOptions = RequestOptions.none())

post /fine_tuning/alpha/graders/validate

Validate grader

Parameters

GraderValidateParams params
- Grader grader
  
  The grader used for the fine-tuning job.
  - class StringCheckGrader:
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - String input
      
      The input text. This may include template strings.
    - String name
      
      The name of the grader.
    - Operation operation
      
      The string check operation to perform. One of eq, ne, like, or ilike.
      - EQ("eq")
      - NE("ne")
      - LIKE("like")
      - ILIKE("ilike")
    - String reference
      
      The reference text. This may include template strings.
    - JsonValue; type "string_check"constant
      
      The object type, which is always string_check.
      - STRING_CHECK("string_check")
  - class TextSimilarityGrader:
    
    A TextSimilarityGrader object which grades text based on similarity metrics.
    - EvaluationMetric evaluationMetric
      
      The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
      - COSINE("cosine")
      - FUZZY_MATCH("fuzzy_match")
      - BLEU("bleu")
      - GLEU("gleu")
      - METEOR("meteor")
      - ROUGE_1("rouge_1")
      - ROUGE_2("rouge_2")
      - ROUGE_3("rouge_3")
      - ROUGE_4("rouge_4")
      - ROUGE_5("rouge_5")
      - ROUGE_L("rouge_l")
    - String input
      
      The text being graded.
    - String name
      
      The name of the grader.
    - String reference
      
      The text being graded against.
    - JsonValue; type "text_similarity"constant
      
      The type of grader.
      - TEXT_SIMILARITY("text_similarity")
  - class PythonGrader:
    
    A PythonGrader object that runs a python script on the input.
    - String name
      
      The name of the grader.
    - String source
      
      The source code of the python script.
    - JsonValue; type "python"constant
      
      The object type, which is always python.
      - PYTHON("python")
    - Optional<String> imageTag
      
      The image tag to use for the python script.
  - class ScoreModelGrader:
    
    A ScoreModelGrader object that uses a model to assign a score to the input.
    - List<Input> input
      
      The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
      - Content content
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String
        
        class ResponseInputText:
        
        A text input to the model.
        
        String text
        
        The text input to the model.
        
        JsonValue; type "input_text"constant
        
        The type of the input item. Always input_text.
        
        INPUT_TEXT("input_text")
        
        Optional<PromptCacheBreakpoint> promptCacheBreakpoint
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        JsonValue; mode "explicit"constant
        
        The breakpoint mode. Always explicit.
        
        EXPLICIT("explicit")
        
        class OutputText:
        
        A text output from the model.
        
        String text
        
        The text output from the model.
        
        JsonValue; type "output_text"constant
        
        The type of the output text. Always output_text.
        
        OUTPUT_TEXT("output_text")
        
        class InputImage:
        
        An image input block used within EvalItem content arrays.
        
        String imageUrl
        
        The URL of the image input.
        
        JsonValue; type "input_image"constant
        
        The type of the image input. Always input_image.
        
        INPUT_IMAGE("input_image")
        
        Optional<String> detail
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio:
        
        An audio input to the model.
        
        InputAudio inputAudio
        
        String data
        
        Base64-encoded audio data.
        
        Format format
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        MP3("mp3")
        
        WAV("wav")
        
        JsonValue; type "input_audio"constant
        
        The type of the input item. Always input_audio.
        
        INPUT_AUDIO("input_audio")
        
        List<EvalContentItem>
        
        String
        
        class ResponseInputText:
        
        A text input to the model.
        
        OutputText
        
        String text
        
        The text output from the model.
        
        JsonValue; type "output_text"constant
        
        The type of the output text. Always output_text.
        
        OUTPUT_TEXT("output_text")
        
        InputImage
        
        String imageUrl
        
        The URL of the image input.
        
        JsonValue; type "input_image"constant
        
        The type of the image input. Always input_image.
        
        INPUT_IMAGE("input_image")
        
        Optional<String> detail
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio:
        
        An audio input to the model.
      - Role role
        
        The role of the message input. One of user, assistant, system, or developer.
        
        USER("user")
        
        ASSISTANT("assistant")
        
        SYSTEM("system")
        
        DEVELOPER("developer")
      - Optional<Type> type
        
        The type of the message input. Always message.
        
        MESSAGE("message")
    - String model
      
      The model to use for the evaluation.
    - String name
      
      The name of the grader.
    - JsonValue; type "score_model"constant
      
      The object type, which is always score_model.
      - SCORE_MODEL("score_model")
    - Optional<List<Double>> range
      
      The range of the score. Defaults to [0, 1].
    - Optional<SamplingParams> samplingParams
      
      The sampling parameters for the model.
      - Optional<Long> maxCompletionsTokens
        
        The maximum number of tokens the grader model may generate in its response.
      - Optional<ReasoningEffort> reasoningEffort
        
        Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
        
        NONE("none")
        
        MINIMAL("minimal")
        
        LOW("low")
        
        MEDIUM("medium")
        
        HIGH("high")
        
        XHIGH("xhigh")
        
        MAX("max")
      - Optional<Long> seed
        
        A seed value to initialize the randomness, during sampling.
      - Optional<Double> temperature
        
        A higher temperature increases randomness in the outputs.
      - Optional<Double> topP
        
        An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
  - class MultiGrader:
    
    A MultiGrader object combines the output of multiple graders to produce a single score.
    - String calculateOutput
      
      A formula to calculate the output based on grader results.
    - Graders graders
      
      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class StringCheckGrader:
        
        A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class TextSimilarityGrader:
        
        A TextSimilarityGrader object which grades text based on similarity metrics.
      - class PythonGrader:
        
        A PythonGrader object that runs a python script on the input.
      - class ScoreModelGrader:
        
        A ScoreModelGrader object that uses a model to assign a score to the input.
      - class LabelModelGrader:
        
        A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
        
        List<Input> input
        
        Content content
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String
        
        class ResponseInputText:
        
        A text input to the model.
        
        class OutputText:
        
        A text output from the model.
        
        String text
        
        The text output from the model.
        
        JsonValue; type "output_text"constant
        
        The type of the output text. Always output_text.
        
        OUTPUT_TEXT("output_text")
        
        class InputImage:
        
        An image input block used within EvalItem content arrays.
        
        String imageUrl
        
        The URL of the image input.
        
        JsonValue; type "input_image"constant
        
        The type of the image input. Always input_image.
        
        INPUT_IMAGE("input_image")
        
        Optional<String> detail
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio:
        
        An audio input to the model.
        
        List<EvalContentItem>
        
        String
        
        class ResponseInputText:
        
        A text input to the model.
        
        OutputText
        
        InputImage
        
        class ResponseInputAudio:
        
        An audio input to the model.
        
        Role role
        
        The role of the message input. One of user, assistant, system, or developer.
        
        USER("user")
        
        ASSISTANT("assistant")
        
        SYSTEM("system")
        
        DEVELOPER("developer")
        
        Optional<Type> type
        
        The type of the message input. Always message.
        
        MESSAGE("message")
        
        List<String> labels
        
        The labels to assign to each item in the evaluation.
        
        String model
        
        The model to use for the evaluation. Must support structured outputs.
        
        String name
        
        The name of the grader.
        
        List<String> passingLabels
        
        The labels that indicate a passing result. Must be a subset of labels.
        
        JsonValue; type "label_model"constant
        
        The object type, which is always label_model.
        
        LABEL_MODEL("label_model")
    - String name
      
      The name of the grader.
    - JsonValue; type "multi"constant
      
      The object type, which is always multi.
      - MULTI("multi")

Returns

class GraderValidateResponse:
- Optional<Grader> grader
  
  The grader used for the fine-tuning job.
  - class StringCheckGrader:
    
    A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
    - String input
      
      The input text. This may include template strings.
    - String name
      
      The name of the grader.
    - Operation operation
      
      The string check operation to perform. One of eq, ne, like, or ilike.
      - EQ("eq")
      - NE("ne")
      - LIKE("like")
      - ILIKE("ilike")
    - String reference
      
      The reference text. This may include template strings.
    - JsonValue; type "string_check"constant
      
      The object type, which is always string_check.
      - STRING_CHECK("string_check")
  - class TextSimilarityGrader:
    
    A TextSimilarityGrader object which grades text based on similarity metrics.
    - EvaluationMetric evaluationMetric
      
      The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.
      - COSINE("cosine")
      - FUZZY_MATCH("fuzzy_match")
      - BLEU("bleu")
      - GLEU("gleu")
      - METEOR("meteor")
      - ROUGE_1("rouge_1")
      - ROUGE_2("rouge_2")
      - ROUGE_3("rouge_3")
      - ROUGE_4("rouge_4")
      - ROUGE_5("rouge_5")
      - ROUGE_L("rouge_l")
    - String input
      
      The text being graded.
    - String name
      
      The name of the grader.
    - String reference
      
      The text being graded against.
    - JsonValue; type "text_similarity"constant
      
      The type of grader.
      - TEXT_SIMILARITY("text_similarity")
  - class PythonGrader:
    
    A PythonGrader object that runs a python script on the input.
    - String name
      
      The name of the grader.
    - String source
      
      The source code of the python script.
    - JsonValue; type "python"constant
      
      The object type, which is always python.
      - PYTHON("python")
    - Optional<String> imageTag
      
      The image tag to use for the python script.
  - class ScoreModelGrader:
    
    A ScoreModelGrader object that uses a model to assign a score to the input.
    - List<Input> input
      
      The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
      - Content content
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String
        
        class ResponseInputText:
        
        A text input to the model.
        
        String text
        
        The text input to the model.
        
        JsonValue; type "input_text"constant
        
        The type of the input item. Always input_text.
        
        INPUT_TEXT("input_text")
        
        Optional<PromptCacheBreakpoint> promptCacheBreakpoint
        
        Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.
        
        JsonValue; mode "explicit"constant
        
        The breakpoint mode. Always explicit.
        
        EXPLICIT("explicit")
        
        class OutputText:
        
        A text output from the model.
        
        String text
        
        The text output from the model.
        
        JsonValue; type "output_text"constant
        
        The type of the output text. Always output_text.
        
        OUTPUT_TEXT("output_text")
        
        class InputImage:
        
        An image input block used within EvalItem content arrays.
        
        String imageUrl
        
        The URL of the image input.
        
        JsonValue; type "input_image"constant
        
        The type of the image input. Always input_image.
        
        INPUT_IMAGE("input_image")
        
        Optional<String> detail
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio:
        
        An audio input to the model.
        
        InputAudio inputAudio
        
        String data
        
        Base64-encoded audio data.
        
        Format format
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        MP3("mp3")
        
        WAV("wav")
        
        JsonValue; type "input_audio"constant
        
        The type of the input item. Always input_audio.
        
        INPUT_AUDIO("input_audio")
        
        List<EvalContentItem>
        
        String
        
        class ResponseInputText:
        
        A text input to the model.
        
        OutputText
        
        String text
        
        The text output from the model.
        
        JsonValue; type "output_text"constant
        
        The type of the output text. Always output_text.
        
        OUTPUT_TEXT("output_text")
        
        InputImage
        
        String imageUrl
        
        The URL of the image input.
        
        JsonValue; type "input_image"constant
        
        The type of the image input. Always input_image.
        
        INPUT_IMAGE("input_image")
        
        Optional<String> detail
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio:
        
        An audio input to the model.
      - Role role
        
        The role of the message input. One of user, assistant, system, or developer.
        
        USER("user")
        
        ASSISTANT("assistant")
        
        SYSTEM("system")
        
        DEVELOPER("developer")
      - Optional<Type> type
        
        The type of the message input. Always message.
        
        MESSAGE("message")
    - String model
      
      The model to use for the evaluation.
    - String name
      
      The name of the grader.
    - JsonValue; type "score_model"constant
      
      The object type, which is always score_model.
      - SCORE_MODEL("score_model")
    - Optional<List<Double>> range
      
      The range of the score. Defaults to [0, 1].
    - Optional<SamplingParams> samplingParams
      
      The sampling parameters for the model.
      - Optional<Long> maxCompletionsTokens
        
        The maximum number of tokens the grader model may generate in its response.
      - Optional<ReasoningEffort> reasoningEffort
        
        Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
        
        NONE("none")
        
        MINIMAL("minimal")
        
        LOW("low")
        
        MEDIUM("medium")
        
        HIGH("high")
        
        XHIGH("xhigh")
        
        MAX("max")
      - Optional<Long> seed
        
        A seed value to initialize the randomness, during sampling.
      - Optional<Double> temperature
        
        A higher temperature increases randomness in the outputs.
      - Optional<Double> topP
        
        An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
  - class MultiGrader:
    
    A MultiGrader object combines the output of multiple graders to produce a single score.
    - String calculateOutput
      
      A formula to calculate the output based on grader results.
    - Graders graders
      
      A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class StringCheckGrader:
        
        A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
      - class TextSimilarityGrader:
        
        A TextSimilarityGrader object which grades text based on similarity metrics.
      - class PythonGrader:
        
        A PythonGrader object that runs a python script on the input.
      - class ScoreModelGrader:
        
        A ScoreModelGrader object that uses a model to assign a score to the input.
      - class LabelModelGrader:
        
        A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
        
        List<Input> input
        
        Content content
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        String
        
        class ResponseInputText:
        
        A text input to the model.
        
        class OutputText:
        
        A text output from the model.
        
        String text
        
        The text output from the model.
        
        JsonValue; type "output_text"constant
        
        The type of the output text. Always output_text.
        
        OUTPUT_TEXT("output_text")
        
        class InputImage:
        
        An image input block used within EvalItem content arrays.
        
        String imageUrl
        
        The URL of the image input.
        
        JsonValue; type "input_image"constant
        
        The type of the image input. Always input_image.
        
        INPUT_IMAGE("input_image")
        
        Optional<String> detail
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        class ResponseInputAudio:
        
        An audio input to the model.
        
        List<EvalContentItem>
        
        String
        
        class ResponseInputText:
        
        A text input to the model.
        
        OutputText
        
        InputImage
        
        class ResponseInputAudio:
        
        An audio input to the model.
        
        Role role
        
        The role of the message input. One of user, assistant, system, or developer.
        
        USER("user")
        
        ASSISTANT("assistant")
        
        SYSTEM("system")
        
        DEVELOPER("developer")
        
        Optional<Type> type
        
        The type of the message input. Always message.
        
        MESSAGE("message")
        
        List<String> labels
        
        The labels to assign to each item in the evaluation.
        
        String model
        
        The model to use for the evaluation. Must support structured outputs.
        
        String name
        
        The name of the grader.
        
        List<String> passingLabels
        
        The labels that indicate a passing result. Must be a subset of labels.
        
        JsonValue; type "label_model"constant
        
        The object type, which is always label_model.
        
        LABEL_MODEL("label_model")
    - String name
      
      The name of the grader.
    - JsonValue; type "multi"constant
      
      The object type, which is always multi.
      - MULTI("multi")

Example

package com.openai.example;

import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.finetuning.alpha.graders.GraderValidateParams;
import com.openai.models.finetuning.alpha.graders.GraderValidateResponse;
import com.openai.models.graders.gradermodels.StringCheckGrader;

public final class Main {
    private Main() {}

    public static void main(String[] args) {
        OpenAIClient client = OpenAIOkHttpClient.fromEnv();

        GraderValidateParams params = GraderValidateParams.builder()
            .grader(StringCheckGrader.builder()
                .input("input")
                .name("name")
                .operation(StringCheckGrader.Operation.EQ)
                .reference("reference")
                .build())
            .build();
        GraderValidateResponse response = client.fineTuning().alpha().graders().validate(params);
    }
}

Response

{
  "grader": {
    "input": "input",
    "name": "name",
    "operation": "eq",
    "reference": "reference",
    "type": "string_check"
  }
}

java/resources/fine_tuning/subresources/alpha/index.md +1634 −0 created

1# Alpha

3# Graders

5## Run grader

7`GraderRunResponse fineTuning().alpha().graders().run(GraderRunParamsparams, RequestOptionsrequestOptions = RequestOptions.none())`

9**post** `/fine_tuning/alpha/graders/run`

11Run grader

13### Parameters

15- `GraderRunParams params`

17 - `Grader grader`

19 The grader used for the fine-tuning job.

21 - `class StringCheckGrader:`

23 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

25 - `String input`

27 The input text. This may include template strings.

29 - `String name`

31 The name of the grader.

33 - `Operation operation`

35 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

37 - `EQ("eq")`

39 - `NE("ne")`

41 - `LIKE("like")`

43 - `ILIKE("ilike")`

45 - `String reference`

47 The reference text. This may include template strings.

49 - `JsonValue; type "string_check"constant`

51 The object type, which is always `string_check`.

53 - `STRING_CHECK("string_check")`

55 - `class TextSimilarityGrader:`

57 A TextSimilarityGrader object which grades text based on similarity metrics.

59 - `EvaluationMetric evaluationMetric`

61 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

62 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

63 or `rouge_l`.

65 - `COSINE("cosine")`

67 - `FUZZY_MATCH("fuzzy_match")`

69 - `BLEU("bleu")`

71 - `GLEU("gleu")`

73 - `METEOR("meteor")`

75 - `ROUGE_1("rouge_1")`

77 - `ROUGE_2("rouge_2")`

79 - `ROUGE_3("rouge_3")`

81 - `ROUGE_4("rouge_4")`

83 - `ROUGE_5("rouge_5")`

85 - `ROUGE_L("rouge_l")`

87 - `String input`

89 The text being graded.

91 - `String name`

93 The name of the grader.

95 - `String reference`

97 The text being graded against.

99 - `JsonValue; type "text_similarity"constant`

100

101 The type of grader.

102

103 - `TEXT_SIMILARITY("text_similarity")`

104

105 - `class PythonGrader:`

106

107 A PythonGrader object that runs a python script on the input.

108

109 - `String name`

110

111 The name of the grader.

112

113 - `String source`

114

115 The source code of the python script.

116

117 - `JsonValue; type "python"constant`

118

119 The object type, which is always `python`.

120

121 - `PYTHON("python")`

122

123 - `Optional<String> imageTag`

124

125 The image tag to use for the python script.

126

127 - `class ScoreModelGrader:`

128

129 A ScoreModelGrader object that uses a model to assign a score to the input.

130

131 - `List<Input> input`

132

133 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

134

135 - `Content content`

136

137 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

138

139 - `String`

140

141 - `class ResponseInputText:`

142

143 A text input to the model.

144

145 - `String text`

146

147 The text input to the model.

148

149 - `JsonValue; type "input_text"constant`

150

151 The type of the input item. Always `input_text`.

152

153 - `INPUT_TEXT("input_text")`

154

155 - `Optional<PromptCacheBreakpoint> promptCacheBreakpoint`

156

157 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.

158

159 - `JsonValue; mode "explicit"constant`

160

161 The breakpoint mode. Always `explicit`.

162

163 - `EXPLICIT("explicit")`

164

165 - `class OutputText:`

166

167 A text output from the model.

168

169 - `String text`

170

171 The text output from the model.

172

173 - `JsonValue; type "output_text"constant`

174

175 The type of the output text. Always `output_text`.

176

177 - `OUTPUT_TEXT("output_text")`

178

179 - `class InputImage:`

180

181 An image input block used within EvalItem content arrays.

182

183 - `String imageUrl`

184

185 The URL of the image input.

186

187 - `JsonValue; type "input_image"constant`

188

189 The type of the image input. Always `input_image`.

190

191 - `INPUT_IMAGE("input_image")`

192

193 - `Optional<String> detail`

194

195 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

196

197 - `class ResponseInputAudio:`

198

199 An audio input to the model.

200

201 - `InputAudio inputAudio`

202

203 - `String data`

204

205 Base64-encoded audio data.

206

207 - `Format format`

208

209 The format of the audio data. Currently supported formats are `mp3` and

210 `wav`.

211

212 - `MP3("mp3")`

213

214 - `WAV("wav")`

215

216 - `JsonValue; type "input_audio"constant`

217

218 The type of the input item. Always `input_audio`.

219

220 - `INPUT_AUDIO("input_audio")`

221

222 - `List<EvalContentItem>`

223

224 - `String`

225

226 - `class ResponseInputText:`

227

228 A text input to the model.

229

230 - `OutputText`

231

232 - `String text`

233

234 The text output from the model.

235

236 - `JsonValue; type "output_text"constant`

237

238 The type of the output text. Always `output_text`.

239

240 - `OUTPUT_TEXT("output_text")`

241

242 - `InputImage`

243

244 - `String imageUrl`

245

246 The URL of the image input.

247

248 - `JsonValue; type "input_image"constant`

249

250 The type of the image input. Always `input_image`.

251

252 - `INPUT_IMAGE("input_image")`

253

254 - `Optional<String> detail`

255

256 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

257

258 - `class ResponseInputAudio:`

259

260 An audio input to the model.

261

262 - `Role role`

263

264 The role of the message input. One of `user`, `assistant`, `system`, or

265 `developer`.

266

267 - `USER("user")`

268

269 - `ASSISTANT("assistant")`

270

271 - `SYSTEM("system")`

272

273 - `DEVELOPER("developer")`

274

275 - `Optional<Type> type`

276

277 The type of the message input. Always `message`.

278

279 - `MESSAGE("message")`

280

281 - `String model`

282

283 The model to use for the evaluation.

284

285 - `String name`

286

287 The name of the grader.

288

289 - `JsonValue; type "score_model"constant`

290

291 The object type, which is always `score_model`.

292

293 - `SCORE_MODEL("score_model")`

294

295 - `Optional<List<Double>> range`

296

297 The range of the score. Defaults to `[0, 1]`.

298

299 - `Optional<SamplingParams> samplingParams`

300

301 The sampling parameters for the model.

302

303 - `Optional<Long> maxCompletionsTokens`

304

305 The maximum number of tokens the grader model may generate in its response.

306

307 - `Optional<ReasoningEffort> reasoningEffort`

308

309 Constrains effort on reasoning for reasoning models. Currently supported

310 values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`.

311 Reducing reasoning effort can result in faster responses and fewer tokens

312 used on reasoning in a response. Not all reasoning models support every

313 value. See the

314 [reasoning guide](https://platform.openai.com/docs/guides/reasoning)

315 for model-specific support.

316

317 - `NONE("none")`

318

319 - `MINIMAL("minimal")`

320

321 - `LOW("low")`

322

323 - `MEDIUM("medium")`

324

325 - `HIGH("high")`

326

327 - `XHIGH("xhigh")`

328

329 - `MAX("max")`

330

331 - `Optional<Long> seed`

332

333 A seed value to initialize the randomness, during sampling.

334

335 - `Optional<Double> temperature`

336

337 A higher temperature increases randomness in the outputs.

338

339 - `Optional<Double> topP`

340

341 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

342

343 - `class MultiGrader:`

344

345 A MultiGrader object combines the output of multiple graders to produce a single score.

346

347 - `String calculateOutput`

348

349 A formula to calculate the output based on grader results.

350

351 - `Graders graders`

352

353 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

354

355 - `class StringCheckGrader:`

356

357 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

358

359 - `class TextSimilarityGrader:`

360

361 A TextSimilarityGrader object which grades text based on similarity metrics.

362

363 - `class PythonGrader:`

364

365 A PythonGrader object that runs a python script on the input.

366

367 - `class ScoreModelGrader:`

368

369 A ScoreModelGrader object that uses a model to assign a score to the input.

370

371 - `class LabelModelGrader:`

372

373 A LabelModelGrader object which uses a model to assign labels to each item

374 in the evaluation.

375

376 - `List<Input> input`

377

378 - `Content content`

379

380 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

381

382 - `String`

383

384 - `class ResponseInputText:`

385

386 A text input to the model.

387

388 - `class OutputText:`

389

390 A text output from the model.

391

392 - `String text`

393

394 The text output from the model.

395

396 - `JsonValue; type "output_text"constant`

397

398 The type of the output text. Always `output_text`.

399

400 - `OUTPUT_TEXT("output_text")`

401

402 - `class InputImage:`

403

404 An image input block used within EvalItem content arrays.

405

406 - `String imageUrl`

407

408 The URL of the image input.

409

410 - `JsonValue; type "input_image"constant`

411

412 The type of the image input. Always `input_image`.

413

414 - `INPUT_IMAGE("input_image")`

415

416 - `Optional<String> detail`

417

418 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

419

420 - `class ResponseInputAudio:`

421

422 An audio input to the model.

423

424 - `List<EvalContentItem>`

425

426 - `String`

427

428 - `class ResponseInputText:`

429

430 A text input to the model.

431

432 - `OutputText`

433

434 - `InputImage`

435

436 - `class ResponseInputAudio:`

437

438 An audio input to the model.

439

440 - `Role role`

441

442 The role of the message input. One of `user`, `assistant`, `system`, or

443 `developer`.

444

445 - `USER("user")`

446

447 - `ASSISTANT("assistant")`

448

449 - `SYSTEM("system")`

450

451 - `DEVELOPER("developer")`

452

453 - `Optional<Type> type`

454

455 The type of the message input. Always `message`.

456

457 - `MESSAGE("message")`

458

459 - `List<String> labels`

460

461 The labels to assign to each item in the evaluation.

462

463 - `String model`

464

465 The model to use for the evaluation. Must support structured outputs.

466

467 - `String name`

468

469 The name of the grader.

470

471 - `List<String> passingLabels`

472

473 The labels that indicate a passing result. Must be a subset of labels.

474

475 - `JsonValue; type "label_model"constant`

476

477 The object type, which is always `label_model`.

478

479 - `LABEL_MODEL("label_model")`

480

481 - `String name`

482

483 The name of the grader.

484

485 - `JsonValue; type "multi"constant`

486

487 The object type, which is always `multi`.

488

489 - `MULTI("multi")`

490

491 - `String modelSample`

492

493 The model sample to be evaluated. This value will be used to populate

494 the `sample` namespace. See [the guide](https://platform.openai.com/docs/guides/graders) for more details.

495 The `output_json` variable will be populated if the model sample is a

496 valid JSON string.

497

498 - `Optional<JsonValue> item`

499

500 The dataset item provided to the grader. This will be used to populate

501 the `item` namespace. See [the guide](https://platform.openai.com/docs/guides/graders) for more details.

502

503### Returns

504

505- `class GraderRunResponse:`

506

507 - `Metadata metadata`

508

509 - `Errors errors`

510

511 - `boolean formulaParseError`

512

513 - `boolean invalidVariableError`

514

515 - `boolean modelGraderParseError`

516

517 - `boolean modelGraderRefusalError`

518

519 - `boolean modelGraderServerError`

520

521 - `Optional<String> modelGraderServerErrorDetails`

522

523 - `boolean otherError`

524

525 - `boolean pythonGraderRuntimeError`

526

527 - `Optional<String> pythonGraderRuntimeErrorDetails`

528

529 - `boolean pythonGraderServerError`

530

531 - `Optional<String> pythonGraderServerErrorType`

532

533 - `boolean sampleParseError`

534

535 - `boolean truncatedObservationError`

536

537 - `boolean unresponsiveRewardError`

538

539 - `double executionTime`

540

541 - `String name`

542

543 - `Optional<String> sampledModelName`

544

545 - `Scores scores`

546

547 - `Optional<Long> tokenUsage`

548

549 - `String type`

550

551 - `ModelGraderTokenUsagePerModel modelGraderTokenUsagePerModel`

552

553 - `double reward`

554

555 - `SubRewards subRewards`

556

557### Example

558

559```java

560package com.openai.example;

561

562import com.openai.client.OpenAIClient;

563import com.openai.client.okhttp.OpenAIOkHttpClient;

564import com.openai.models.finetuning.alpha.graders.GraderRunParams;

565import com.openai.models.finetuning.alpha.graders.GraderRunResponse;

566import com.openai.models.graders.gradermodels.StringCheckGrader;

567

568public final class Main {

569 private Main() {}

570

571 public static void main(String[] args) {

572 OpenAIClient client = OpenAIOkHttpClient.fromEnv();

573

574 GraderRunParams params = GraderRunParams.builder()

575 .grader(StringCheckGrader.builder()

576 .input("input")

577 .name("name")

578 .operation(StringCheckGrader.Operation.EQ)

579 .reference("reference")

580 .build())

581 .modelSample("model_sample")

582 .build();

583 GraderRunResponse response = client.fineTuning().alpha().graders().run(params);

584 }

585}

586```

587

588#### Response

589

590```json

591{

592 "metadata": {

593 "errors": {

594 "formula_parse_error": true,

595 "invalid_variable_error": true,

596 "model_grader_parse_error": true,

597 "model_grader_refusal_error": true,

598 "model_grader_server_error": true,

599 "model_grader_server_error_details": "model_grader_server_error_details",

600 "other_error": true,

601 "python_grader_runtime_error": true,

602 "python_grader_runtime_error_details": "python_grader_runtime_error_details",

603 "python_grader_server_error": true,

604 "python_grader_server_error_type": "python_grader_server_error_type",

605 "sample_parse_error": true,

606 "truncated_observation_error": true,

607 "unresponsive_reward_error": true

608 },

609 "execution_time": 0,

610 "name": "name",

611 "sampled_model_name": "sampled_model_name",

612 "scores": {

613 "foo": "bar"

614 },

615 "token_usage": 0,

616 "type": "type"

617 },

618 "model_grader_token_usage_per_model": {

619 "foo": "bar"

620 },

621 "reward": 0,

622 "sub_rewards": {

623 "foo": "bar"

624 }

625}

626```

627

628## Validate grader

629

630`GraderValidateResponse fineTuning().alpha().graders().validate(GraderValidateParamsparams, RequestOptionsrequestOptions = RequestOptions.none())`

631

632**post** `/fine_tuning/alpha/graders/validate`

633

634Validate grader

635

636### Parameters

637

638- `GraderValidateParams params`

639

640 - `Grader grader`

641

642 The grader used for the fine-tuning job.

643

644 - `class StringCheckGrader:`

645

646 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

647

648 - `String input`

649

650 The input text. This may include template strings.

651

652 - `String name`

653

654 The name of the grader.

655

656 - `Operation operation`

657

658 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

659

660 - `EQ("eq")`

661

662 - `NE("ne")`

663

664 - `LIKE("like")`

665

666 - `ILIKE("ilike")`

667

668 - `String reference`

669

670 The reference text. This may include template strings.

671

672 - `JsonValue; type "string_check"constant`

673

674 The object type, which is always `string_check`.

675

676 - `STRING_CHECK("string_check")`

677

678 - `class TextSimilarityGrader:`

679

680 A TextSimilarityGrader object which grades text based on similarity metrics.

681

682 - `EvaluationMetric evaluationMetric`

683

684 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

685 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

686 or `rouge_l`.

687

688 - `COSINE("cosine")`

689

690 - `FUZZY_MATCH("fuzzy_match")`

691

692 - `BLEU("bleu")`

693

694 - `GLEU("gleu")`

695

696 - `METEOR("meteor")`

697

698 - `ROUGE_1("rouge_1")`

699

700 - `ROUGE_2("rouge_2")`

701

702 - `ROUGE_3("rouge_3")`

703

704 - `ROUGE_4("rouge_4")`

705

706 - `ROUGE_5("rouge_5")`

707

708 - `ROUGE_L("rouge_l")`

709

710 - `String input`

711

712 The text being graded.

713

714 - `String name`

715

716 The name of the grader.

717

718 - `String reference`

719

720 The text being graded against.

721

722 - `JsonValue; type "text_similarity"constant`

723

724 The type of grader.

725

726 - `TEXT_SIMILARITY("text_similarity")`

727

728 - `class PythonGrader:`

729

730 A PythonGrader object that runs a python script on the input.

731

732 - `String name`

733

734 The name of the grader.

735

736 - `String source`

737

738 The source code of the python script.

739

740 - `JsonValue; type "python"constant`

741

742 The object type, which is always `python`.

743

744 - `PYTHON("python")`

745

746 - `Optional<String> imageTag`

747

748 The image tag to use for the python script.

749

750 - `class ScoreModelGrader:`

751

752 A ScoreModelGrader object that uses a model to assign a score to the input.

753

754 - `List<Input> input`

755

756 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

757

758 - `Content content`

759

760 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

761

762 - `String`

763

764 - `class ResponseInputText:`

765

766 A text input to the model.

767

768 - `String text`

769

770 The text input to the model.

771

772 - `JsonValue; type "input_text"constant`

773

774 The type of the input item. Always `input_text`.

775

776 - `INPUT_TEXT("input_text")`

777

778 - `Optional<PromptCacheBreakpoint> promptCacheBreakpoint`

779

780 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.

781

782 - `JsonValue; mode "explicit"constant`

783

784 The breakpoint mode. Always `explicit`.

785

786 - `EXPLICIT("explicit")`

787

788 - `class OutputText:`

789

790 A text output from the model.

791

792 - `String text`

793

794 The text output from the model.

795

796 - `JsonValue; type "output_text"constant`

797

798 The type of the output text. Always `output_text`.

799

800 - `OUTPUT_TEXT("output_text")`

801

802 - `class InputImage:`

803

804 An image input block used within EvalItem content arrays.

805

806 - `String imageUrl`

807

808 The URL of the image input.

809

810 - `JsonValue; type "input_image"constant`

811

812 The type of the image input. Always `input_image`.

813

814 - `INPUT_IMAGE("input_image")`

815

816 - `Optional<String> detail`

817

818 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

819

820 - `class ResponseInputAudio:`

821

822 An audio input to the model.

823

824 - `InputAudio inputAudio`

825

826 - `String data`

827

828 Base64-encoded audio data.

829

830 - `Format format`

831

832 The format of the audio data. Currently supported formats are `mp3` and

833 `wav`.

834

835 - `MP3("mp3")`

836

837 - `WAV("wav")`

838

839 - `JsonValue; type "input_audio"constant`

840

841 The type of the input item. Always `input_audio`.

842

843 - `INPUT_AUDIO("input_audio")`

844

845 - `List<EvalContentItem>`

846

847 - `String`

848

849 - `class ResponseInputText:`

850

851 A text input to the model.

852

853 - `OutputText`

854

855 - `String text`

856

857 The text output from the model.

858

859 - `JsonValue; type "output_text"constant`

860

861 The type of the output text. Always `output_text`.

862

863 - `OUTPUT_TEXT("output_text")`

864

865 - `InputImage`

866

867 - `String imageUrl`

868

869 The URL of the image input.

870

871 - `JsonValue; type "input_image"constant`

872

873 The type of the image input. Always `input_image`.

874

875 - `INPUT_IMAGE("input_image")`

876

877 - `Optional<String> detail`

878

879 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

880

881 - `class ResponseInputAudio:`

882

883 An audio input to the model.

884

885 - `Role role`

886

887 The role of the message input. One of `user`, `assistant`, `system`, or

888 `developer`.

889

890 - `USER("user")`

891

892 - `ASSISTANT("assistant")`

893

894 - `SYSTEM("system")`

895

896 - `DEVELOPER("developer")`

897

898 - `Optional<Type> type`

899

900 The type of the message input. Always `message`.

901

902 - `MESSAGE("message")`

903

904 - `String model`

905

906 The model to use for the evaluation.

907

908 - `String name`

909

910 The name of the grader.

911

912 - `JsonValue; type "score_model"constant`

913

914 The object type, which is always `score_model`.

915

916 - `SCORE_MODEL("score_model")`

917

918 - `Optional<List<Double>> range`

919

920 The range of the score. Defaults to `[0, 1]`.

921

922 - `Optional<SamplingParams> samplingParams`

923

924 The sampling parameters for the model.

925

926 - `Optional<Long> maxCompletionsTokens`

927

928 The maximum number of tokens the grader model may generate in its response.

929

930 - `Optional<ReasoningEffort> reasoningEffort`

931

932 Constrains effort on reasoning for reasoning models. Currently supported

933 values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`.

934 Reducing reasoning effort can result in faster responses and fewer tokens

935 used on reasoning in a response. Not all reasoning models support every

936 value. See the

937 [reasoning guide](https://platform.openai.com/docs/guides/reasoning)

938 for model-specific support.

939

940 - `NONE("none")`

941

942 - `MINIMAL("minimal")`

943

944 - `LOW("low")`

945

946 - `MEDIUM("medium")`

947

948 - `HIGH("high")`

949

950 - `XHIGH("xhigh")`

951

952 - `MAX("max")`

953

954 - `Optional<Long> seed`

955

956 A seed value to initialize the randomness, during sampling.

957

958 - `Optional<Double> temperature`

959

960 A higher temperature increases randomness in the outputs.

961

962 - `Optional<Double> topP`

963

964 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

965

966 - `class MultiGrader:`

967

968 A MultiGrader object combines the output of multiple graders to produce a single score.

969

970 - `String calculateOutput`

971

972 A formula to calculate the output based on grader results.

973

974 - `Graders graders`

975

976 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

977

978 - `class StringCheckGrader:`

979

980 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

981

982 - `class TextSimilarityGrader:`

983

984 A TextSimilarityGrader object which grades text based on similarity metrics.

985

986 - `class PythonGrader:`

987

988 A PythonGrader object that runs a python script on the input.

989

990 - `class ScoreModelGrader:`

991

992 A ScoreModelGrader object that uses a model to assign a score to the input.

993

994 - `class LabelModelGrader:`

995

996 A LabelModelGrader object which uses a model to assign labels to each item

997 in the evaluation.

998

999 - `List<Input> input`

1000

1001 - `Content content`

1002

1003 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1004

1005 - `String`

1006

1007 - `class ResponseInputText:`

1008

1009 A text input to the model.

1010

1011 - `class OutputText:`

1012

1013 A text output from the model.

1014

1015 - `String text`

1016

1017 The text output from the model.

1018

1019 - `JsonValue; type "output_text"constant`

1020

1021 The type of the output text. Always `output_text`.

1022

1023 - `OUTPUT_TEXT("output_text")`

1024

1025 - `class InputImage:`

1026

1027 An image input block used within EvalItem content arrays.

1028

1029 - `String imageUrl`

1030

1031 The URL of the image input.

1032

1033 - `JsonValue; type "input_image"constant`

1034

1035 The type of the image input. Always `input_image`.

1036

1037 - `INPUT_IMAGE("input_image")`

1038

1039 - `Optional<String> detail`

1040

1041 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1042

1043 - `class ResponseInputAudio:`

1044

1045 An audio input to the model.

1046

1047 - `List<EvalContentItem>`

1048

1049 - `String`

1050

1051 - `class ResponseInputText:`

1052

1053 A text input to the model.

1054

1055 - `OutputText`

1056

1057 - `InputImage`

1058

1059 - `class ResponseInputAudio:`

1060

1061 An audio input to the model.

1062

1063 - `Role role`

1064

1065 The role of the message input. One of `user`, `assistant`, `system`, or

1066 `developer`.

1067

1068 - `USER("user")`

1069

1070 - `ASSISTANT("assistant")`

1071

1072 - `SYSTEM("system")`

1073

1074 - `DEVELOPER("developer")`

1075

1076 - `Optional<Type> type`

1077

1078 The type of the message input. Always `message`.

1079

1080 - `MESSAGE("message")`

1081

1082 - `List<String> labels`

1083

1084 The labels to assign to each item in the evaluation.

1085

1086 - `String model`

1087

1088 The model to use for the evaluation. Must support structured outputs.

1089

1090 - `String name`

1091

1092 The name of the grader.

1093

1094 - `List<String> passingLabels`

1095

1096 The labels that indicate a passing result. Must be a subset of labels.

1097

1098 - `JsonValue; type "label_model"constant`

1099

1100 The object type, which is always `label_model`.

1101

1102 - `LABEL_MODEL("label_model")`

1103

1104 - `String name`

1105

1106 The name of the grader.

1107

1108 - `JsonValue; type "multi"constant`

1109

1110 The object type, which is always `multi`.

1111

1112 - `MULTI("multi")`

1113

1114### Returns

1115

1116- `class GraderValidateResponse:`

1117

1118 - `Optional<Grader> grader`

1119

1120 The grader used for the fine-tuning job.

1121

1122 - `class StringCheckGrader:`

1123

1124 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1125

1126 - `String input`

1127

1128 The input text. This may include template strings.

1129

1130 - `String name`

1131

1132 The name of the grader.

1133

1134 - `Operation operation`

1135

1136 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.

1137

1138 - `EQ("eq")`

1139

1140 - `NE("ne")`

1141

1142 - `LIKE("like")`

1143

1144 - `ILIKE("ilike")`

1145

1146 - `String reference`

1147

1148 The reference text. This may include template strings.

1149

1150 - `JsonValue; type "string_check"constant`

1151

1152 The object type, which is always `string_check`.

1153

1154 - `STRING_CHECK("string_check")`

1155

1156 - `class TextSimilarityGrader:`

1157

1158 A TextSimilarityGrader object which grades text based on similarity metrics.

1159

1160 - `EvaluationMetric evaluationMetric`

1161

1162 The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`,

1163 `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`,

1164 or `rouge_l`.

1165

1166 - `COSINE("cosine")`

1167

1168 - `FUZZY_MATCH("fuzzy_match")`

1169

1170 - `BLEU("bleu")`

1171

1172 - `GLEU("gleu")`

1173

1174 - `METEOR("meteor")`

1175

1176 - `ROUGE_1("rouge_1")`

1177

1178 - `ROUGE_2("rouge_2")`

1179

1180 - `ROUGE_3("rouge_3")`

1181

1182 - `ROUGE_4("rouge_4")`

1183

1184 - `ROUGE_5("rouge_5")`

1185

1186 - `ROUGE_L("rouge_l")`

1187

1188 - `String input`

1189

1190 The text being graded.

1191

1192 - `String name`

1193

1194 The name of the grader.

1195

1196 - `String reference`

1197

1198 The text being graded against.

1199

1200 - `JsonValue; type "text_similarity"constant`

1201

1202 The type of grader.

1203

1204 - `TEXT_SIMILARITY("text_similarity")`

1205

1206 - `class PythonGrader:`

1207

1208 A PythonGrader object that runs a python script on the input.

1209

1210 - `String name`

1211

1212 The name of the grader.

1213

1214 - `String source`

1215

1216 The source code of the python script.

1217

1218 - `JsonValue; type "python"constant`

1219

1220 The object type, which is always `python`.

1221

1222 - `PYTHON("python")`

1223

1224 - `Optional<String> imageTag`

1225

1226 The image tag to use for the python script.

1227

1228 - `class ScoreModelGrader:`

1229

1230 A ScoreModelGrader object that uses a model to assign a score to the input.

1231

1232 - `List<Input> input`

1233

1234 The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

1235

1236 - `Content content`

1237

1238 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1239

1240 - `String`

1241

1242 - `class ResponseInputText:`

1243

1244 A text input to the model.

1245

1246 - `String text`

1247

1248 The text input to the model.

1249

1250 - `JsonValue; type "input_text"constant`

1251

1252 The type of the input item. Always `input_text`.

1253

1254 - `INPUT_TEXT("input_text")`

1255

1256 - `Optional<PromptCacheBreakpoint> promptCacheBreakpoint`

1257

1258 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.

1259

1260 - `JsonValue; mode "explicit"constant`

1261

1262 The breakpoint mode. Always `explicit`.

1263

1264 - `EXPLICIT("explicit")`

1265

1266 - `class OutputText:`

1267

1268 A text output from the model.

1269

1270 - `String text`

1271

1272 The text output from the model.

1273

1274 - `JsonValue; type "output_text"constant`

1275

1276 The type of the output text. Always `output_text`.

1277

1278 - `OUTPUT_TEXT("output_text")`

1279

1280 - `class InputImage:`

1281

1282 An image input block used within EvalItem content arrays.

1283

1284 - `String imageUrl`

1285

1286 The URL of the image input.

1287

1288 - `JsonValue; type "input_image"constant`

1289

1290 The type of the image input. Always `input_image`.

1291

1292 - `INPUT_IMAGE("input_image")`

1293

1294 - `Optional<String> detail`

1295

1296 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1297

1298 - `class ResponseInputAudio:`

1299

1300 An audio input to the model.

1301

1302 - `InputAudio inputAudio`

1303

1304 - `String data`

1305

1306 Base64-encoded audio data.

1307

1308 - `Format format`

1309

1310 The format of the audio data. Currently supported formats are `mp3` and

1311 `wav`.

1312

1313 - `MP3("mp3")`

1314

1315 - `WAV("wav")`

1316

1317 - `JsonValue; type "input_audio"constant`

1318

1319 The type of the input item. Always `input_audio`.

1320

1321 - `INPUT_AUDIO("input_audio")`

1322

1323 - `List<EvalContentItem>`

1324

1325 - `String`

1326

1327 - `class ResponseInputText:`

1328

1329 A text input to the model.

1330

1331 - `OutputText`

1332

1333 - `String text`

1334

1335 The text output from the model.

1336

1337 - `JsonValue; type "output_text"constant`

1338

1339 The type of the output text. Always `output_text`.

1340

1341 - `OUTPUT_TEXT("output_text")`

1342

1343 - `InputImage`

1344

1345 - `String imageUrl`

1346

1347 The URL of the image input.

1348

1349 - `JsonValue; type "input_image"constant`

1350

1351 The type of the image input. Always `input_image`.

1352

1353 - `INPUT_IMAGE("input_image")`

1354

1355 - `Optional<String> detail`

1356

1357 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1358

1359 - `class ResponseInputAudio:`

1360

1361 An audio input to the model.

1362

1363 - `Role role`

1364

1365 The role of the message input. One of `user`, `assistant`, `system`, or

1366 `developer`.

1367

1368 - `USER("user")`

1369

1370 - `ASSISTANT("assistant")`

1371

1372 - `SYSTEM("system")`

1373

1374 - `DEVELOPER("developer")`

1375

1376 - `Optional<Type> type`

1377

1378 The type of the message input. Always `message`.

1379

1380 - `MESSAGE("message")`

1381

1382 - `String model`

1383

1384 The model to use for the evaluation.

1385

1386 - `String name`

1387

1388 The name of the grader.

1389

1390 - `JsonValue; type "score_model"constant`

1391

1392 The object type, which is always `score_model`.

1393

1394 - `SCORE_MODEL("score_model")`

1395

1396 - `Optional<List<Double>> range`

1397

1398 The range of the score. Defaults to `[0, 1]`.

1399

1400 - `Optional<SamplingParams> samplingParams`

1401

1402 The sampling parameters for the model.

1403

1404 - `Optional<Long> maxCompletionsTokens`

1405

1406 The maximum number of tokens the grader model may generate in its response.

1407

1408 - `Optional<ReasoningEffort> reasoningEffort`

1409

1410 Constrains effort on reasoning for reasoning models. Currently supported

1411 values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`.

1412 Reducing reasoning effort can result in faster responses and fewer tokens

1413 used on reasoning in a response. Not all reasoning models support every

1414 value. See the

1415 [reasoning guide](https://platform.openai.com/docs/guides/reasoning)

1416 for model-specific support.

1417

1418 - `NONE("none")`

1419

1420 - `MINIMAL("minimal")`

1421

1422 - `LOW("low")`

1423

1424 - `MEDIUM("medium")`

1425

1426 - `HIGH("high")`

1427

1428 - `XHIGH("xhigh")`

1429

1430 - `MAX("max")`

1431

1432 - `Optional<Long> seed`

1433

1434 A seed value to initialize the randomness, during sampling.

1435

1436 - `Optional<Double> temperature`

1437

1438 A higher temperature increases randomness in the outputs.

1439

1440 - `Optional<Double> topP`

1441

1442 An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

1443

1444 - `class MultiGrader:`

1445

1446 A MultiGrader object combines the output of multiple graders to produce a single score.

1447

1448 - `String calculateOutput`

1449

1450 A formula to calculate the output based on grader results.

1451

1452 - `Graders graders`

1453

1454 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1455

1456 - `class StringCheckGrader:`

1457

1458 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

1459

1460 - `class TextSimilarityGrader:`

1461

1462 A TextSimilarityGrader object which grades text based on similarity metrics.

1463

1464 - `class PythonGrader:`

1465

1466 A PythonGrader object that runs a python script on the input.

1467

1468 - `class ScoreModelGrader:`

1469

1470 A ScoreModelGrader object that uses a model to assign a score to the input.

1471

1472 - `class LabelModelGrader:`

1473

1474 A LabelModelGrader object which uses a model to assign labels to each item

1475 in the evaluation.

1476

1477 - `List<Input> input`

1478

1479 - `Content content`

1480

1481 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

1482

1483 - `String`

1484

1485 - `class ResponseInputText:`

1486

1487 A text input to the model.

1488

1489 - `class OutputText:`

1490

1491 A text output from the model.

1492

1493 - `String text`

1494

1495 The text output from the model.

1496

1497 - `JsonValue; type "output_text"constant`

1498

1499 The type of the output text. Always `output_text`.

1500

1501 - `OUTPUT_TEXT("output_text")`

1502

1503 - `class InputImage:`

1504

1505 An image input block used within EvalItem content arrays.

1506

1507 - `String imageUrl`

1508

1509 The URL of the image input.

1510

1511 - `JsonValue; type "input_image"constant`

1512

1513 The type of the image input. Always `input_image`.

1514

1515 - `INPUT_IMAGE("input_image")`

1516

1517 - `Optional<String> detail`

1518

1519 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.

1520

1521 - `class ResponseInputAudio:`

1522

1523 An audio input to the model.

1524

1525 - `List<EvalContentItem>`

1526

1527 - `String`

1528

1529 - `class ResponseInputText:`

1530

1531 A text input to the model.

1532

1533 - `OutputText`

1534

1535 - `InputImage`

1536

1537 - `class ResponseInputAudio:`

1538

1539 An audio input to the model.

1540

1541 - `Role role`

1542

1543 The role of the message input. One of `user`, `assistant`, `system`, or

1544 `developer`.

1545

1546 - `USER("user")`

1547

1548 - `ASSISTANT("assistant")`

1549

1550 - `SYSTEM("system")`

1551

1552 - `DEVELOPER("developer")`

1553

1554 - `Optional<Type> type`

1555

1556 The type of the message input. Always `message`.

1557

1558 - `MESSAGE("message")`

1559

1560 - `List<String> labels`

1561

1562 The labels to assign to each item in the evaluation.

1563

1564 - `String model`

1565

1566 The model to use for the evaluation. Must support structured outputs.

1567

1568 - `String name`

1569

1570 The name of the grader.

1571

1572 - `List<String> passingLabels`

1573

1574 The labels that indicate a passing result. Must be a subset of labels.

1575

1576 - `JsonValue; type "label_model"constant`

1577

1578 The object type, which is always `label_model`.

1579

1580 - `LABEL_MODEL("label_model")`

1581

1582 - `String name`

1583

1584 The name of the grader.

1585

1586 - `JsonValue; type "multi"constant`

1587

1588 The object type, which is always `multi`.

1589

1590 - `MULTI("multi")`

1591

1592### Example

1593

1594```java

1595package com.openai.example;

1596

1597import com.openai.client.OpenAIClient;

1598import com.openai.client.okhttp.OpenAIOkHttpClient;

1599import com.openai.models.finetuning.alpha.graders.GraderValidateParams;

1600import com.openai.models.finetuning.alpha.graders.GraderValidateResponse;

1601import com.openai.models.graders.gradermodels.StringCheckGrader;

1602

1603public final class Main {

1604 private Main() {}

1605

1606 public static void main(String[] args) {

1607 OpenAIClient client = OpenAIOkHttpClient.fromEnv();

1608

1609 GraderValidateParams params = GraderValidateParams.builder()

1610 .grader(StringCheckGrader.builder()

1611 .input("input")

1612 .name("name")

1613 .operation(StringCheckGrader.Operation.EQ)

1614 .reference("reference")

1615 .build())

1616 .build();

1617 GraderValidateResponse response = client.fineTuning().alpha().graders().validate(params);

1618 }

1619}

1620```

1621

1622#### Response

1623

1624```json

1625{

1626 "grader": {

1627 "input": "input",

1628 "name": "name",

1629 "operation": "eq",

1630 "reference": "reference",

1631 "type": "string_check"

1632 }

1633}

1634```

java/resources/fine_tuning/subresources/alpha/index.md 2026-07-07 08:02 UTC to 2026-07-09 20:58 UTC

Alpha

Graders

Run grader

Parameters

Returns

Example

Response

Validate grader

Parameters

Returns

Example

Response