python/resources/evals/methods/retrieve/index.md +0 −482 deleted
File Deleted View Diff
1## Get an eval
2
3`evals.retrieve(streval_id) -> EvalRetrieveResponse`
4
5**get** `/evals/{eval_id}`
6
7Get an evaluation by ID.
8
9### Parameters
10
11- `eval_id: str`
12
13### Returns
14
15- `class EvalRetrieveResponse: …`
16
17 An Eval object with a data source config and testing criteria.
18 An Eval represents a task to be done for your LLM integration.
19 Like:
20
21 - Improve the quality of my chatbot
22 - See how well my chatbot handles customer support
23 - Check if o4-mini is better at my usecase than gpt-4o
24
25 - `id: str`
26
27 Unique identifier for the evaluation.
28
29 - `created_at: int`
30
31 The Unix timestamp (in seconds) for when the eval was created.
32
33 - `data_source_config: DataSourceConfig`
34
35 Configuration of data sources used in runs of the evaluation.
36
37 - `class EvalCustomDataSourceConfig: …`
38
39 A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
40 The response schema defines the shape of the data that will be:
41
42 - Used to define your testing criteria and
43 - What data is required when creating a run
44
45 - `schema: Dict[str, object]`
46
47 The json schema for the run data source items.
48 Learn how to build JSON schemas [here](https://json-schema.org/).
49
50 - `type: Literal["custom"]`
51
52 The type of data source. Always `custom`.
53
54 - `"custom"`
55
56 - `class DataSourceConfigLogs: …`
57
58 A LogsDataSourceConfig which specifies the metadata property of your logs query.
59 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
60 The schema returned by this data source config is used to defined what variables are available in your evals.
61 `item` and `sample` are both defined when using this data source config.
62
63 - `schema: Dict[str, object]`
64
65 The json schema for the run data source items.
66 Learn how to build JSON schemas [here](https://json-schema.org/).
67
68 - `type: Literal["logs"]`
69
70 The type of data source. Always `logs`.
71
72 - `"logs"`
73
74 - `metadata: Optional[Metadata]`
75
76 Set of 16 key-value pairs that can be attached to an object. This can be
77 useful for storing additional information about the object in a structured
78 format, and querying for objects via API or the dashboard.
79
80 Keys are strings with a maximum length of 64 characters. Values are strings
81 with a maximum length of 512 characters.
82
83 - `class EvalStoredCompletionsDataSourceConfig: …`
84
85 Deprecated in favor of LogsDataSourceConfig.
86
87 - `schema: Dict[str, object]`
88
89 The json schema for the run data source items.
90 Learn how to build JSON schemas [here](https://json-schema.org/).
91
92 - `type: Literal["stored_completions"]`
93
94 The type of data source. Always `stored_completions`.
95
96 - `"stored_completions"`
97
98 - `metadata: Optional[Metadata]`
99
100 Set of 16 key-value pairs that can be attached to an object. This can be
101 useful for storing additional information about the object in a structured
102 format, and querying for objects via API or the dashboard.
103
104 Keys are strings with a maximum length of 64 characters. Values are strings
105 with a maximum length of 512 characters.
106
107 - `metadata: Optional[Metadata]`
108
109 Set of 16 key-value pairs that can be attached to an object. This can be
110 useful for storing additional information about the object in a structured
111 format, and querying for objects via API or the dashboard.
112
113 Keys are strings with a maximum length of 64 characters. Values are strings
114 with a maximum length of 512 characters.
115
116 - `name: str`
117
118 The name of the evaluation.
119
120 - `object: Literal["eval"]`
121
122 The object type.
123
124 - `"eval"`
125
126 - `testing_criteria: List[TestingCriterion]`
127
128 A list of testing criteria.
129
130 - `class LabelModelGrader: …`
131
132 A LabelModelGrader object which uses a model to assign labels to each item
133 in the evaluation.
134
135 - `input: List[Input]`
136
137 - `content: InputContent`
138
139 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
140
141 - `str`
142
143 A text input to the model.
144
145 - `class ResponseInputText: …`
146
147 A text input to the model.
148
149 - `text: str`
150
151 The text input to the model.
152
153 - `type: Literal["input_text"]`
154
155 The type of the input item. Always `input_text`.
156
157 - `"input_text"`
158
159 - `prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]`
160
161 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
162
163 - `mode: Literal["explicit"]`
164
165 The breakpoint mode. Always `explicit`.
166
167 - `"explicit"`
168
169 - `class InputContentOutputText: …`
170
171 A text output from the model.
172
173 - `text: str`
174
175 The text output from the model.
176
177 - `type: Literal["output_text"]`
178
179 The type of the output text. Always `output_text`.
180
181 - `"output_text"`
182
183 - `class InputContentInputImage: …`
184
185 An image input block used within EvalItem content arrays.
186
187 - `image_url: str`
188
189 The URL of the image input.
190
191 - `type: Literal["input_image"]`
192
193 The type of the image input. Always `input_image`.
194
195 - `"input_image"`
196
197 - `detail: Optional[str]`
198
199 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
200
201 - `class ResponseInputAudio: …`
202
203 An audio input to the model.
204
205 - `input_audio: InputAudio`
206
207 - `data: str`
208
209 Base64-encoded audio data.
210
211 - `format: Literal["mp3", "wav"]`
212
213 The format of the audio data. Currently supported formats are `mp3` and
214 `wav`.
215
216 - `"mp3"`
217
218 - `"wav"`
219
220 - `type: Literal["input_audio"]`
221
222 The type of the input item. Always `input_audio`.
223
224 - `"input_audio"`
225
226 - `List[GraderInputItem]`
227
228 - `str`
229
230 A text input to the model.
231
232 - `class ResponseInputText: …`
233
234 A text input to the model.
235
236 - `class GraderInputItemOutputText: …`
237
238 A text output from the model.
239
240 - `text: str`
241
242 The text output from the model.
243
244 - `type: Literal["output_text"]`
245
246 The type of the output text. Always `output_text`.
247
248 - `"output_text"`
249
250 - `class GraderInputItemInputImage: …`
251
252 An image input block used within EvalItem content arrays.
253
254 - `image_url: str`
255
256 The URL of the image input.
257
258 - `type: Literal["input_image"]`
259
260 The type of the image input. Always `input_image`.
261
262 - `"input_image"`
263
264 - `detail: Optional[str]`
265
266 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
267
268 - `class ResponseInputAudio: …`
269
270 An audio input to the model.
271
272 - `role: Literal["user", "assistant", "system", "developer"]`
273
274 The role of the message input. One of `user`, `assistant`, `system`, or
275 `developer`.
276
277 - `"user"`
278
279 - `"assistant"`
280
281 - `"system"`
282
283 - `"developer"`
284
285 - `type: Optional[Literal["message"]]`
286
287 The type of the message input. Always `message`.
288
289 - `"message"`
290
291 - `labels: List[str]`
292
293 The labels to assign to each item in the evaluation.
294
295 - `model: str`
296
297 The model to use for the evaluation. Must support structured outputs.
298
299 - `name: str`
300
301 The name of the grader.
302
303 - `passing_labels: List[str]`
304
305 The labels that indicate a passing result. Must be a subset of labels.
306
307 - `type: Literal["label_model"]`
308
309 The object type, which is always `label_model`.
310
311 - `"label_model"`
312
313 - `class StringCheckGrader: …`
314
315 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
316
317 - `input: str`
318
319 The input text. This may include template strings.
320
321 - `name: str`
322
323 The name of the grader.
324
325 - `operation: Literal["eq", "ne", "like", "ilike"]`
326
327 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
328
329 - `"eq"`
330
331 - `"ne"`
332
333 - `"like"`
334
335 - `"ilike"`
336
337 - `reference: str`
338
339 The reference text. This may include template strings.
340
341 - `type: Literal["string_check"]`
342
343 The object type, which is always `string_check`.
344
345 - `"string_check"`
346
347 - `class TestingCriterionEvalGraderTextSimilarity: …`
348
349 A TextSimilarityGrader object which grades text based on similarity metrics.
350
351 - `pass_threshold: float`
352
353 The threshold for the score.
354
355 - `class TestingCriterionEvalGraderPython: …`
356
357 A PythonGrader object that runs a python script on the input.
358
359 - `pass_threshold: Optional[float]`
360
361 The threshold for the score.
362
363 - `class TestingCriterionEvalGraderScoreModel: …`
364
365 A ScoreModelGrader object that uses a model to assign a score to the input.
366
367 - `pass_threshold: Optional[float]`
368
369 The threshold for the score.
370
371### Example
372
373```python
374import os
375from openai import OpenAI
376
377client = OpenAI(
378 api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted
379)
380eval = client.evals.retrieve(
381 "eval_id",
382)
383print(eval.id)
384```
385
386#### Response
387
388```json
389{
390 "id": "id",
391 "created_at": 0,
392 "data_source_config": {
393 "schema": {
394 "foo": "bar"
395 },
396 "type": "custom"
397 },
398 "metadata": {
399 "foo": "string"
400 },
401 "name": "Chatbot effectiveness Evaluation",
402 "object": "eval",
403 "testing_criteria": [
404 {
405 "input": [
406 {
407 "content": "string",
408 "role": "user",
409 "type": "message"
410 }
411 ],
412 "labels": [
413 "string"
414 ],
415 "model": "model",
416 "name": "name",
417 "passing_labels": [
418 "string"
419 ],
420 "type": "label_model"
421 }
422 ]
423}
424```
425
426### Example
427
428```python
429from openai import OpenAI
430client = OpenAI()
431
432eval_obj = client.evals.retrieve("eval_67abd54d9b0081909a86353f6fb9317a")
433print(eval_obj)
434```
435
436#### Response
437
438```json
439{
440 "object": "eval",
441 "id": "eval_67abd54d9b0081909a86353f6fb9317a",
442 "data_source_config": {
443 "type": "custom",
444 "schema": {
445 "type": "object",
446 "properties": {
447 "item": {
448 "type": "object",
449 "properties": {
450 "input": {
451 "type": "string"
452 },
453 "ground_truth": {
454 "type": "string"
455 }
456 },
457 "required": [
458 "input",
459 "ground_truth"
460 ]
461 }
462 },
463 "required": [
464 "item"
465 ]
466 }
467 },
468 "testing_criteria": [
469 {
470 "name": "String check",
471 "id": "String check-2eaf2d8d-d649-4335-8148-9535a7ca73c2",
472 "type": "string_check",
473 "input": "{{item.input}}",
474 "reference": "{{item.ground_truth}}",
475 "operation": "eq"
476 }
477 ],
478 "name": "External Data Eval",
479 "created_at": 1739314509,
480 "metadata": {},
481}
482```