Create eval
evals.create(EvalCreateParams**kwargs) -> EvalCreateResponse
post /evals
Create the structure of an evaluation that can be used to test a model's performance. An evaluation is a set of testing criteria and the config for a data source, which dictates the schema of the data used in the evaluation. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources. For more information, see the Evals guide.
Parameters
-
data_source_config: DataSourceConfigThe configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
-
class DataSourceConfigCustom: …A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs. This schema is used to define the shape of the data that will be:
-
Used to define your testing criteria and
-
What data is required when creating a run
-
item_schema: Dict[str, object]The json schema for each row in the data source.
-
type: Literal["custom"]The type of data source. Always
custom."custom"
-
include_sample_schema: Optional[bool]Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)
-
-
class DataSourceConfigLogs: …A data source config which specifies the metadata property of your logs query. This is usually metadata like
usecase=chatbotorprompt-version=v2, etc.-
type: Literal["logs"]The type of data source. Always
logs."logs"
-
metadata: Optional[Dict[str, object]]Metadata filters for the logs data source.
-
-
class DataSourceConfigStoredCompletions: …Deprecated in favor of LogsDataSourceConfig.
-
type: Literal["stored_completions"]The type of data source. Always
stored_completions."stored_completions"
-
metadata: Optional[Dict[str, object]]Metadata filters for the stored completions data source.
-
-
-
testing_criteria: Iterable[TestingCriterion]A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like
{{item.variable_name}}. To reference the model's output, use thesamplenamespace (ie,{{sample.output_text}}).-
class TestingCriterionLabelModel: …A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
-
input: Iterable[TestingCriterionLabelModelInput]A list of chat messages forming the prompt or context. May include variable references to the
itemnamespace, ie {{item.name}}.-
class TestingCriterionLabelModelInputSimpleInputMessage: …-
content: strThe content of the message.
-
role: strThe role of the message (e.g. "system", "assistant", "user").
-
-
class TestingCriterionLabelModelInputEvalItem: …A message input to the model with a role indicating instruction following hierarchy. Instructions given with the
developerorsystemrole take precedence over instructions given with theuserrole. Messages with theassistantrole are presumed to have been generated by the model in previous interactions.-
content: TestingCriterionLabelModelInputEvalItemContentInputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
-
strA text input to the model.
-
class ResponseInputText: …A text input to the model.
-
text: strThe text input to the model.
-
type: Literal["input_text"]The type of the input item. Always
input_text."input_text"
-
prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's
prompt_cache_options.ttl; the boundary is not rounded to a token block.-
mode: Literal["explicit"]The breakpoint mode. Always
explicit."explicit"
-
-
-
class TestingCriterionLabelModelInputEvalItemContentOutputText: …A text output from the model.
-
text: strThe text output from the model.
-
type: Literal["output_text"]The type of the output text. Always
output_text."output_text"
-
-
class TestingCriterionLabelModelInputEvalItemContentInputImage: …An image input block used within EvalItem content arrays.
-
image_url: strThe URL of the image input.
-
type: Literal["input_image"]The type of the image input. Always
input_image."input_image"
-
detail: Optional[str]The detail level of the image to be sent to the model. One of
high,low, orauto. Defaults toauto.
-
-
class ResponseInputAudio: …An audio input to the model.
-
input_audio: InputAudio-
data: strBase64-encoded audio data.
-
format: Literal["mp3", "wav"]The format of the audio data. Currently supported formats are
mp3andwav.-
"mp3" -
"wav"
-
-
-
type: Literal["input_audio"]The type of the input item. Always
input_audio."input_audio"
-
-
Sequence[GraderInputsParamItem]-
strA text input to the model.
-
class ResponseInputText: …A text input to the model.
-
class GraderInputItemOutputText: …A text output from the model.
-
text: strThe text output from the model.
-
type: Literal["output_text"]The type of the output text. Always
output_text."output_text"
-
-
class GraderInputItemInputImage: …An image input block used within EvalItem content arrays.
-
image_url: strThe URL of the image input.
-
type: Literal["input_image"]The type of the image input. Always
input_image."input_image"
-
detail: Optional[str]The detail level of the image to be sent to the model. One of
high,low, orauto. Defaults toauto.
-
-
class ResponseInputAudio: …An audio input to the model.
-
-
-
role: Literal["user", "assistant", "system", "developer"]The role of the message input. One of
user,assistant,system, ordeveloper.-
"user" -
"assistant" -
"system" -
"developer"
-
-
type: Optional[Literal["message"]]The type of the message input. Always
message."message"
-
-
-
labels: Sequence[str]The labels to classify to each item in the evaluation.
-
model: strThe model to use for the evaluation. Must support structured outputs.
-
name: strThe name of the grader.
-
passing_labels: Sequence[str]The labels that indicate a passing result. Must be a subset of labels.
-
type: Literal["label_model"]The object type, which is always
label_model."label_model"
-
-
class StringCheckGrader: …A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
-
input: strThe input text. This may include template strings.
-
name: strThe name of the grader.
-
operation: Literal["eq", "ne", "like", "ilike"]The string check operation to perform. One of
eq,ne,like, orilike.-
"eq" -
"ne" -
"like" -
"ilike"
-
-
reference: strThe reference text. This may include template strings.
-
type: Literal["string_check"]The object type, which is always
string_check."string_check"
-
-
class TestingCriterionTextSimilarity: …A TextSimilarityGrader object which grades text based on similarity metrics.
-
pass_threshold: floatThe threshold for the score.
-
-
class TestingCriterionPython: …A PythonGrader object that runs a python script on the input.
-
pass_threshold: Optional[float]The threshold for the score.
-
-
class TestingCriterionScoreModel: …A ScoreModelGrader object that uses a model to assign a score to the input.
-
pass_threshold: Optional[float]The threshold for the score.
-
-
-
metadata: Optional[Metadata]Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
-
name: Optional[str]The name of the evaluation.
Returns
-
class EvalCreateResponse: …An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:
-
Improve the quality of my chatbot
-
See how well my chatbot handles customer support
-
Check if o4-mini is better at my usecase than gpt-4o
-
id: strUnique identifier for the evaluation.
-
created_at: intThe Unix timestamp (in seconds) for when the eval was created.
-
data_source_config: DataSourceConfigConfiguration of data sources used in runs of the evaluation.
-
class EvalCustomDataSourceConfig: …A CustomDataSourceConfig which specifies the schema of your
itemand optionallysamplenamespaces. The response schema defines the shape of the data that will be:-
Used to define your testing criteria and
-
What data is required when creating a run
-
schema: Dict[str, object]The json schema for the run data source items. Learn how to build JSON schemas here.
-
type: Literal["custom"]The type of data source. Always
custom."custom"
-
-
class DataSourceConfigLogs: …A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like
usecase=chatbotorprompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals.itemandsampleare both defined when using this data source config.-
schema: Dict[str, object]The json schema for the run data source items. Learn how to build JSON schemas here.
-
type: Literal["logs"]The type of data source. Always
logs."logs"
-
metadata: Optional[Metadata]Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
-
-
class EvalStoredCompletionsDataSourceConfig: …Deprecated in favor of LogsDataSourceConfig.
-
schema: Dict[str, object]The json schema for the run data source items. Learn how to build JSON schemas here.
-
type: Literal["stored_completions"]The type of data source. Always
stored_completions."stored_completions"
-
metadata: Optional[Metadata]Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
-
-
-
metadata: Optional[Metadata]Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
-
name: strThe name of the evaluation.
-
object: Literal["eval"]The object type.
"eval"
-
testing_criteria: List[TestingCriterion]A list of testing criteria.
-
class LabelModelGrader: …A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
-
input: List[Input]-
content: InputContentInputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
-
strA text input to the model.
-
class ResponseInputText: …A text input to the model.
-
text: strThe text input to the model.
-
type: Literal["input_text"]The type of the input item. Always
input_text."input_text"
-
prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's
prompt_cache_options.ttl; the boundary is not rounded to a token block.-
mode: Literal["explicit"]The breakpoint mode. Always
explicit."explicit"
-
-
-
class InputContentOutputText: …A text output from the model.
-
text: strThe text output from the model.
-
type: Literal["output_text"]The type of the output text. Always
output_text."output_text"
-
-
class InputContentInputImage: …An image input block used within EvalItem content arrays.
-
image_url: strThe URL of the image input.
-
type: Literal["input_image"]The type of the image input. Always
input_image."input_image"
-
detail: Optional[str]The detail level of the image to be sent to the model. One of
high,low, orauto. Defaults toauto.
-
-
class ResponseInputAudio: …An audio input to the model.
-
input_audio: InputAudio-
data: strBase64-encoded audio data.
-
format: Literal["mp3", "wav"]The format of the audio data. Currently supported formats are
mp3andwav.-
"mp3" -
"wav"
-
-
-
type: Literal["input_audio"]The type of the input item. Always
input_audio."input_audio"
-
-
List[GraderInputItem]-
strA text input to the model.
-
class ResponseInputText: …A text input to the model.
-
class GraderInputItemOutputText: …A text output from the model.
-
text: strThe text output from the model.
-
type: Literal["output_text"]The type of the output text. Always
output_text."output_text"
-
-
class GraderInputItemInputImage: …An image input block used within EvalItem content arrays.
-
image_url: strThe URL of the image input.
-
type: Literal["input_image"]The type of the image input. Always
input_image."input_image"
-
detail: Optional[str]The detail level of the image to be sent to the model. One of
high,low, orauto. Defaults toauto.
-
-
class ResponseInputAudio: …An audio input to the model.
-
-
-
role: Literal["user", "assistant", "system", "developer"]The role of the message input. One of
user,assistant,system, ordeveloper.-
"user" -
"assistant" -
"system" -
"developer"
-
-
type: Optional[Literal["message"]]The type of the message input. Always
message."message"
-
-
labels: List[str]The labels to assign to each item in the evaluation.
-
model: strThe model to use for the evaluation. Must support structured outputs.
-
name: strThe name of the grader.
-
passing_labels: List[str]The labels that indicate a passing result. Must be a subset of labels.
-
type: Literal["label_model"]The object type, which is always
label_model."label_model"
-
-
class StringCheckGrader: …A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
-
input: strThe input text. This may include template strings.
-
name: strThe name of the grader.
-
operation: Literal["eq", "ne", "like", "ilike"]The string check operation to perform. One of
eq,ne,like, orilike.-
"eq" -
"ne" -
"like" -
"ilike"
-
-
reference: strThe reference text. This may include template strings.
-
type: Literal["string_check"]The object type, which is always
string_check."string_check"
-
-
class TestingCriterionEvalGraderTextSimilarity: …A TextSimilarityGrader object which grades text based on similarity metrics.
-
pass_threshold: floatThe threshold for the score.
-
-
class TestingCriterionEvalGraderPython: …A PythonGrader object that runs a python script on the input.
-
pass_threshold: Optional[float]The threshold for the score.
-
-
class TestingCriterionEvalGraderScoreModel: …A ScoreModelGrader object that uses a model to assign a score to the input.
-
pass_threshold: Optional[float]The threshold for the score.
-
-
-
Example
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted
)
eval = client.evals.create(
data_source_config={
"item_schema": {
"foo": "bar"
},
"type": "custom",
},
testing_criteria=[{
"input": [{
"content": "content",
"role": "role",
}],
"labels": ["string"],
"model": "model",
"name": "name",
"passing_labels": ["string"],
"type": "label_model",
}],
)
print(eval.id)
Response
{
"id": "id",
"created_at": 0,
"data_source_config": {
"schema": {
"foo": "bar"
},
"type": "custom"
},
"metadata": {
"foo": "string"
},
"name": "Chatbot effectiveness Evaluation",
"object": "eval",
"testing_criteria": [
{
"input": [
{
"content": "string",
"role": "user",
"type": "message"
}
],
"labels": [
"string"
],
"model": "model",
"name": "name",
"passing_labels": [
"string"
],
"type": "label_model"
}
]
}
Example
from openai import OpenAI
client = OpenAI()
eval_obj = client.evals.create(
name="Sentiment",
data_source_config={
"type": "stored_completions",
"metadata": {"usecase": "chatbot"}
},
testing_criteria=[
{
"type": "label_model",
"model": "o3-mini",
"input": [
{"role": "developer", "content": "Classify the sentiment of the following statement as one of 'positive', 'neutral', or 'negative'"},
{"role": "user", "content": "Statement: {{item.input}}"}
],
"passing_labels": ["positive"],
"labels": ["positive", "neutral", "negative"],
"name": "Example label grader"
}
]
)
print(eval_obj)
Response
{
"object": "eval",
"id": "eval_67b7fa9a81a88190ab4aa417e397ea21",
"data_source_config": {
"type": "stored_completions",
"metadata": {
"usecase": "chatbot"
},
"schema": {
"type": "object",
"properties": {
"item": {
"type": "object"
},
"sample": {
"type": "object"
}
},
"required": [
"item",
"sample"
]
},
"testing_criteria": [
{
"name": "Example label grader",
"type": "label_model",
"model": "o3-mini",
"input": [
{
"type": "message",
"role": "developer",
"content": {
"type": "input_text",
"text": "Classify the sentiment of the following statement as one of positive, neutral, or negative"
}
},
{
"type": "message",
"role": "user",
"content": {
"type": "input_text",
"text": "Statement: {{item.input}}"
}
}
],
"passing_labels": [
"positive"
],
"labels": [
"positive",
"neutral",
"negative"
]
}
],
"name": "Sentiment",
"created_at": 1740110490,
"metadata": {
"description": "An eval for sentiment analysis"
}
}