Get an eval
evals.retrieve(streval_id) -> EvalRetrieveResponse
get /evals/{eval_id}
Get an evaluation by ID.
Parameters
eval_id: str
Returns
-
class EvalRetrieveResponse: …An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:
-
Improve the quality of my chatbot
-
See how well my chatbot handles customer support
-
Check if o4-mini is better at my usecase than gpt-4o
-
id: strUnique identifier for the evaluation.
-
created_at: intThe Unix timestamp (in seconds) for when the eval was created.
-
data_source_config: DataSourceConfigConfiguration of data sources used in runs of the evaluation.
-
class EvalCustomDataSourceConfig: …A CustomDataSourceConfig which specifies the schema of your
itemand optionallysamplenamespaces. The response schema defines the shape of the data that will be:-
Used to define your testing criteria and
-
What data is required when creating a run
-
schema: Dict[str, object]The json schema for the run data source items. Learn how to build JSON schemas here.
-
type: Literal["custom"]The type of data source. Always
custom."custom"
-
-
class DataSourceConfigLogs: …A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like
usecase=chatbotorprompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals.itemandsampleare both defined when using this data source config.-
schema: Dict[str, object]The json schema for the run data source items. Learn how to build JSON schemas here.
-
type: Literal["logs"]The type of data source. Always
logs."logs"
-
metadata: Optional[Metadata]Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
-
-
class EvalStoredCompletionsDataSourceConfig: …Deprecated in favor of LogsDataSourceConfig.
-
schema: Dict[str, object]The json schema for the run data source items. Learn how to build JSON schemas here.
-
type: Literal["stored_completions"]The type of data source. Always
stored_completions."stored_completions"
-
metadata: Optional[Metadata]Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
-
-
-
metadata: Optional[Metadata]Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
-
name: strThe name of the evaluation.
-
object: Literal["eval"]The object type.
"eval"
-
testing_criteria: List[TestingCriterion]A list of testing criteria.
-
class LabelModelGrader: …A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
-
input: List[Input]-
content: InputContentInputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
-
strA text input to the model.
-
class ResponseInputText: …A text input to the model.
-
text: strThe text input to the model.
-
type: Literal["input_text"]The type of the input item. Always
input_text."input_text"
-
prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's
prompt_cache_options.ttl; the boundary is not rounded to a token block.-
mode: Literal["explicit"]The breakpoint mode. Always
explicit."explicit"
-
-
-
class InputContentOutputText: …A text output from the model.
-
text: strThe text output from the model.
-
type: Literal["output_text"]The type of the output text. Always
output_text."output_text"
-
-
class InputContentInputImage: …An image input block used within EvalItem content arrays.
-
image_url: strThe URL of the image input.
-
type: Literal["input_image"]The type of the image input. Always
input_image."input_image"
-
detail: Optional[str]The detail level of the image to be sent to the model. One of
high,low, orauto. Defaults toauto.
-
-
class ResponseInputAudio: …An audio input to the model.
-
input_audio: InputAudio-
data: strBase64-encoded audio data.
-
format: Literal["mp3", "wav"]The format of the audio data. Currently supported formats are
mp3andwav.-
"mp3" -
"wav"
-
-
-
type: Literal["input_audio"]The type of the input item. Always
input_audio."input_audio"
-
-
List[GraderInputItem]-
strA text input to the model.
-
class ResponseInputText: …A text input to the model.
-
class GraderInputItemOutputText: …A text output from the model.
-
text: strThe text output from the model.
-
type: Literal["output_text"]The type of the output text. Always
output_text."output_text"
-
-
class GraderInputItemInputImage: …An image input block used within EvalItem content arrays.
-
image_url: strThe URL of the image input.
-
type: Literal["input_image"]The type of the image input. Always
input_image."input_image"
-
detail: Optional[str]The detail level of the image to be sent to the model. One of
high,low, orauto. Defaults toauto.
-
-
class ResponseInputAudio: …An audio input to the model.
-
-
-
role: Literal["user", "assistant", "system", "developer"]The role of the message input. One of
user,assistant,system, ordeveloper.-
"user" -
"assistant" -
"system" -
"developer"
-
-
type: Optional[Literal["message"]]The type of the message input. Always
message."message"
-
-
labels: List[str]The labels to assign to each item in the evaluation.
-
model: strThe model to use for the evaluation. Must support structured outputs.
-
name: strThe name of the grader.
-
passing_labels: List[str]The labels that indicate a passing result. Must be a subset of labels.
-
type: Literal["label_model"]The object type, which is always
label_model."label_model"
-
-
class StringCheckGrader: …A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
-
input: strThe input text. This may include template strings.
-
name: strThe name of the grader.
-
operation: Literal["eq", "ne", "like", "ilike"]The string check operation to perform. One of
eq,ne,like, orilike.-
"eq" -
"ne" -
"like" -
"ilike"
-
-
reference: strThe reference text. This may include template strings.
-
type: Literal["string_check"]The object type, which is always
string_check."string_check"
-
-
class TestingCriterionEvalGraderTextSimilarity: …A TextSimilarityGrader object which grades text based on similarity metrics.
-
pass_threshold: floatThe threshold for the score.
-
-
class TestingCriterionEvalGraderPython: …A PythonGrader object that runs a python script on the input.
-
pass_threshold: Optional[float]The threshold for the score.
-
-
class TestingCriterionEvalGraderScoreModel: …A ScoreModelGrader object that uses a model to assign a score to the input.
-
pass_threshold: Optional[float]The threshold for the score.
-
-
-
Example
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted
)
eval = client.evals.retrieve(
"eval_id",
)
print(eval.id)
Response
{
"id": "id",
"created_at": 0,
"data_source_config": {
"schema": {
"foo": "bar"
},
"type": "custom"
},
"metadata": {
"foo": "string"
},
"name": "Chatbot effectiveness Evaluation",
"object": "eval",
"testing_criteria": [
{
"input": [
{
"content": "string",
"role": "user",
"type": "message"
}
],
"labels": [
"string"
],
"model": "model",
"name": "name",
"passing_labels": [
"string"
],
"type": "label_model"
}
]
}
Example
from openai import OpenAI
client = OpenAI()
eval_obj = client.evals.retrieve("eval_67abd54d9b0081909a86353f6fb9317a")
print(eval_obj)
Response
{
"object": "eval",
"id": "eval_67abd54d9b0081909a86353f6fb9317a",
"data_source_config": {
"type": "custom",
"schema": {
"type": "object",
"properties": {
"item": {
"type": "object",
"properties": {
"input": {
"type": "string"
},
"ground_truth": {
"type": "string"
}
},
"required": [
"input",
"ground_truth"
]
}
},
"required": [
"item"
]
}
},
"testing_criteria": [
{
"name": "String check",
"id": "String check-2eaf2d8d-d649-4335-8148-9535a7ca73c2",
"type": "string_check",
"input": "{{item.input}}",
"reference": "{{item.ground_truth}}",
"operation": "eq"
}
],
"name": "External Data Eval",
"created_at": 1739314509,
"metadata": {},
}