python/resources/evals/methods/update/index.md +0 −499 deleted
File Deleted View Diff
1## Update an eval
2
3`evals.update(streval_id, EvalUpdateParams**kwargs) -> EvalUpdateResponse`
4
5**post** `/evals/{eval_id}`
6
7Update certain properties of an evaluation.
8
9### Parameters
10
11- `eval_id: str`
12
13- `metadata: Optional[Metadata]`
14
15 Set of 16 key-value pairs that can be attached to an object. This can be
16 useful for storing additional information about the object in a structured
17 format, and querying for objects via API or the dashboard.
18
19 Keys are strings with a maximum length of 64 characters. Values are strings
20 with a maximum length of 512 characters.
21
22- `name: Optional[str]`
23
24 Rename the evaluation.
25
26### Returns
27
28- `class EvalUpdateResponse: …`
29
30 An Eval object with a data source config and testing criteria.
31 An Eval represents a task to be done for your LLM integration.
32 Like:
33
34 - Improve the quality of my chatbot
35 - See how well my chatbot handles customer support
36 - Check if o4-mini is better at my usecase than gpt-4o
37
38 - `id: str`
39
40 Unique identifier for the evaluation.
41
42 - `created_at: int`
43
44 The Unix timestamp (in seconds) for when the eval was created.
45
46 - `data_source_config: DataSourceConfig`
47
48 Configuration of data sources used in runs of the evaluation.
49
50 - `class EvalCustomDataSourceConfig: …`
51
52 A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
53 The response schema defines the shape of the data that will be:
54
55 - Used to define your testing criteria and
56 - What data is required when creating a run
57
58 - `schema: Dict[str, object]`
59
60 The json schema for the run data source items.
61 Learn how to build JSON schemas [here](https://json-schema.org/).
62
63 - `type: Literal["custom"]`
64
65 The type of data source. Always `custom`.
66
67 - `"custom"`
68
69 - `class DataSourceConfigLogs: …`
70
71 A LogsDataSourceConfig which specifies the metadata property of your logs query.
72 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
73 The schema returned by this data source config is used to defined what variables are available in your evals.
74 `item` and `sample` are both defined when using this data source config.
75
76 - `schema: Dict[str, object]`
77
78 The json schema for the run data source items.
79 Learn how to build JSON schemas [here](https://json-schema.org/).
80
81 - `type: Literal["logs"]`
82
83 The type of data source. Always `logs`.
84
85 - `"logs"`
86
87 - `metadata: Optional[Metadata]`
88
89 Set of 16 key-value pairs that can be attached to an object. This can be
90 useful for storing additional information about the object in a structured
91 format, and querying for objects via API or the dashboard.
92
93 Keys are strings with a maximum length of 64 characters. Values are strings
94 with a maximum length of 512 characters.
95
96 - `class EvalStoredCompletionsDataSourceConfig: …`
97
98 Deprecated in favor of LogsDataSourceConfig.
99
100 - `schema: Dict[str, object]`
101
102 The json schema for the run data source items.
103 Learn how to build JSON schemas [here](https://json-schema.org/).
104
105 - `type: Literal["stored_completions"]`
106
107 The type of data source. Always `stored_completions`.
108
109 - `"stored_completions"`
110
111 - `metadata: Optional[Metadata]`
112
113 Set of 16 key-value pairs that can be attached to an object. This can be
114 useful for storing additional information about the object in a structured
115 format, and querying for objects via API or the dashboard.
116
117 Keys are strings with a maximum length of 64 characters. Values are strings
118 with a maximum length of 512 characters.
119
120 - `metadata: Optional[Metadata]`
121
122 Set of 16 key-value pairs that can be attached to an object. This can be
123 useful for storing additional information about the object in a structured
124 format, and querying for objects via API or the dashboard.
125
126 Keys are strings with a maximum length of 64 characters. Values are strings
127 with a maximum length of 512 characters.
128
129 - `name: str`
130
131 The name of the evaluation.
132
133 - `object: Literal["eval"]`
134
135 The object type.
136
137 - `"eval"`
138
139 - `testing_criteria: List[TestingCriterion]`
140
141 A list of testing criteria.
142
143 - `class LabelModelGrader: …`
144
145 A LabelModelGrader object which uses a model to assign labels to each item
146 in the evaluation.
147
148 - `input: List[Input]`
149
150 - `content: InputContent`
151
152 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
153
154 - `str`
155
156 A text input to the model.
157
158 - `class ResponseInputText: …`
159
160 A text input to the model.
161
162 - `text: str`
163
164 The text input to the model.
165
166 - `type: Literal["input_text"]`
167
168 The type of the input item. Always `input_text`.
169
170 - `"input_text"`
171
172 - `prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]`
173
174 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
175
176 - `mode: Literal["explicit"]`
177
178 The breakpoint mode. Always `explicit`.
179
180 - `"explicit"`
181
182 - `class InputContentOutputText: …`
183
184 A text output from the model.
185
186 - `text: str`
187
188 The text output from the model.
189
190 - `type: Literal["output_text"]`
191
192 The type of the output text. Always `output_text`.
193
194 - `"output_text"`
195
196 - `class InputContentInputImage: …`
197
198 An image input block used within EvalItem content arrays.
199
200 - `image_url: str`
201
202 The URL of the image input.
203
204 - `type: Literal["input_image"]`
205
206 The type of the image input. Always `input_image`.
207
208 - `"input_image"`
209
210 - `detail: Optional[str]`
211
212 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
213
214 - `class ResponseInputAudio: …`
215
216 An audio input to the model.
217
218 - `input_audio: InputAudio`
219
220 - `data: str`
221
222 Base64-encoded audio data.
223
224 - `format: Literal["mp3", "wav"]`
225
226 The format of the audio data. Currently supported formats are `mp3` and
227 `wav`.
228
229 - `"mp3"`
230
231 - `"wav"`
232
233 - `type: Literal["input_audio"]`
234
235 The type of the input item. Always `input_audio`.
236
237 - `"input_audio"`
238
239 - `List[GraderInputItem]`
240
241 - `str`
242
243 A text input to the model.
244
245 - `class ResponseInputText: …`
246
247 A text input to the model.
248
249 - `class GraderInputItemOutputText: …`
250
251 A text output from the model.
252
253 - `text: str`
254
255 The text output from the model.
256
257 - `type: Literal["output_text"]`
258
259 The type of the output text. Always `output_text`.
260
261 - `"output_text"`
262
263 - `class GraderInputItemInputImage: …`
264
265 An image input block used within EvalItem content arrays.
266
267 - `image_url: str`
268
269 The URL of the image input.
270
271 - `type: Literal["input_image"]`
272
273 The type of the image input. Always `input_image`.
274
275 - `"input_image"`
276
277 - `detail: Optional[str]`
278
279 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
280
281 - `class ResponseInputAudio: …`
282
283 An audio input to the model.
284
285 - `role: Literal["user", "assistant", "system", "developer"]`
286
287 The role of the message input. One of `user`, `assistant`, `system`, or
288 `developer`.
289
290 - `"user"`
291
292 - `"assistant"`
293
294 - `"system"`
295
296 - `"developer"`
297
298 - `type: Optional[Literal["message"]]`
299
300 The type of the message input. Always `message`.
301
302 - `"message"`
303
304 - `labels: List[str]`
305
306 The labels to assign to each item in the evaluation.
307
308 - `model: str`
309
310 The model to use for the evaluation. Must support structured outputs.
311
312 - `name: str`
313
314 The name of the grader.
315
316 - `passing_labels: List[str]`
317
318 The labels that indicate a passing result. Must be a subset of labels.
319
320 - `type: Literal["label_model"]`
321
322 The object type, which is always `label_model`.
323
324 - `"label_model"`
325
326 - `class StringCheckGrader: …`
327
328 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
329
330 - `input: str`
331
332 The input text. This may include template strings.
333
334 - `name: str`
335
336 The name of the grader.
337
338 - `operation: Literal["eq", "ne", "like", "ilike"]`
339
340 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
341
342 - `"eq"`
343
344 - `"ne"`
345
346 - `"like"`
347
348 - `"ilike"`
349
350 - `reference: str`
351
352 The reference text. This may include template strings.
353
354 - `type: Literal["string_check"]`
355
356 The object type, which is always `string_check`.
357
358 - `"string_check"`
359
360 - `class TestingCriterionEvalGraderTextSimilarity: …`
361
362 A TextSimilarityGrader object which grades text based on similarity metrics.
363
364 - `pass_threshold: float`
365
366 The threshold for the score.
367
368 - `class TestingCriterionEvalGraderPython: …`
369
370 A PythonGrader object that runs a python script on the input.
371
372 - `pass_threshold: Optional[float]`
373
374 The threshold for the score.
375
376 - `class TestingCriterionEvalGraderScoreModel: …`
377
378 A ScoreModelGrader object that uses a model to assign a score to the input.
379
380 - `pass_threshold: Optional[float]`
381
382 The threshold for the score.
383
384### Example
385
386```python
387import os
388from openai import OpenAI
389
390client = OpenAI(
391 api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted
392)
393eval = client.evals.update(
394 eval_id="eval_id",
395)
396print(eval.id)
397```
398
399#### Response
400
401```json
402{
403 "id": "id",
404 "created_at": 0,
405 "data_source_config": {
406 "schema": {
407 "foo": "bar"
408 },
409 "type": "custom"
410 },
411 "metadata": {
412 "foo": "string"
413 },
414 "name": "Chatbot effectiveness Evaluation",
415 "object": "eval",
416 "testing_criteria": [
417 {
418 "input": [
419 {
420 "content": "string",
421 "role": "user",
422 "type": "message"
423 }
424 ],
425 "labels": [
426 "string"
427 ],
428 "model": "model",
429 "name": "name",
430 "passing_labels": [
431 "string"
432 ],
433 "type": "label_model"
434 }
435 ]
436}
437```
438
439### Example
440
441```python
442from openai import OpenAI
443client = OpenAI()
444
445updated_eval = client.evals.update(
446 "eval_67abd54d9b0081909a86353f6fb9317a",
447 name="Updated Eval",
448 metadata={"description": "Updated description"}
449)
450print(updated_eval)
451```
452
453#### Response
454
455```json
456{
457 "object": "eval",
458 "id": "eval_67abd54d9b0081909a86353f6fb9317a",
459 "data_source_config": {
460 "type": "custom",
461 "schema": {
462 "type": "object",
463 "properties": {
464 "item": {
465 "type": "object",
466 "properties": {
467 "input": {
468 "type": "string"
469 },
470 "ground_truth": {
471 "type": "string"
472 }
473 },
474 "required": [
475 "input",
476 "ground_truth"
477 ]
478 }
479 },
480 "required": [
481 "item"
482 ]
483 }
484 },
485 "testing_criteria": [
486 {
487 "name": "String check",
488 "id": "String check-2eaf2d8d-d649-4335-8148-9535a7ca73c2",
489 "type": "string_check",
490 "input": "{{item.input}}",
491 "reference": "{{item.ground_truth}}",
492 "operation": "eq"
493 }
494 ],
495 "name": "Updated Eval",
496 "created_at": 1739314509,
497 "metadata": {"description": "Updated description"},
498}
499```