python/resources/evals/methods/update/index.md +0 −489 deleted
File Deleted View Diff
1## Update an eval
2
3`evals.update(streval_id, EvalUpdateParams**kwargs) -> EvalUpdateResponse`
4
5**post** `/evals/{eval_id}`
6
7Update certain properties of an evaluation.
8
9### Parameters
10
11- `eval_id: str`
12
13- `metadata: Optional[Metadata]`
14
15 Set of 16 key-value pairs that can be attached to an object. This can be
16 useful for storing additional information about the object in a structured
17 format, and querying for objects via API or the dashboard.
18
19 Keys are strings with a maximum length of 64 characters. Values are strings
20 with a maximum length of 512 characters.
21
22- `name: Optional[str]`
23
24 Rename the evaluation.
25
26### Returns
27
28- `class EvalUpdateResponse: …`
29
30 An Eval object with a data source config and testing criteria.
31 An Eval represents a task to be done for your LLM integration.
32 Like:
33
34 - Improve the quality of my chatbot
35 - See how well my chatbot handles customer support
36 - Check if o4-mini is better at my usecase than gpt-4o
37
38 - `id: str`
39
40 Unique identifier for the evaluation.
41
42 - `created_at: int`
43
44 The Unix timestamp (in seconds) for when the eval was created.
45
46 - `data_source_config: DataSourceConfig`
47
48 Configuration of data sources used in runs of the evaluation.
49
50 - `class EvalCustomDataSourceConfig: …`
51
52 A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
53 The response schema defines the shape of the data that will be:
54
55 - Used to define your testing criteria and
56 - What data is required when creating a run
57
58 - `schema: Dict[str, object]`
59
60 The json schema for the run data source items.
61 Learn how to build JSON schemas [here](https://json-schema.org/).
62
63 - `type: Literal["custom"]`
64
65 The type of data source. Always `custom`.
66
67 - `"custom"`
68
69 - `class DataSourceConfigLogs: …`
70
71 A LogsDataSourceConfig which specifies the metadata property of your logs query.
72 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
73 The schema returned by this data source config is used to defined what variables are available in your evals.
74 `item` and `sample` are both defined when using this data source config.
75
76 - `schema: Dict[str, object]`
77
78 The json schema for the run data source items.
79 Learn how to build JSON schemas [here](https://json-schema.org/).
80
81 - `type: Literal["logs"]`
82
83 The type of data source. Always `logs`.
84
85 - `"logs"`
86
87 - `metadata: Optional[Metadata]`
88
89 Set of 16 key-value pairs that can be attached to an object. This can be
90 useful for storing additional information about the object in a structured
91 format, and querying for objects via API or the dashboard.
92
93 Keys are strings with a maximum length of 64 characters. Values are strings
94 with a maximum length of 512 characters.
95
96 - `class EvalStoredCompletionsDataSourceConfig: …`
97
98 Deprecated in favor of LogsDataSourceConfig.
99
100 - `schema: Dict[str, object]`
101
102 The json schema for the run data source items.
103 Learn how to build JSON schemas [here](https://json-schema.org/).
104
105 - `type: Literal["stored_completions"]`
106
107 The type of data source. Always `stored_completions`.
108
109 - `"stored_completions"`
110
111 - `metadata: Optional[Metadata]`
112
113 Set of 16 key-value pairs that can be attached to an object. This can be
114 useful for storing additional information about the object in a structured
115 format, and querying for objects via API or the dashboard.
116
117 Keys are strings with a maximum length of 64 characters. Values are strings
118 with a maximum length of 512 characters.
119
120 - `metadata: Optional[Metadata]`
121
122 Set of 16 key-value pairs that can be attached to an object. This can be
123 useful for storing additional information about the object in a structured
124 format, and querying for objects via API or the dashboard.
125
126 Keys are strings with a maximum length of 64 characters. Values are strings
127 with a maximum length of 512 characters.
128
129 - `name: str`
130
131 The name of the evaluation.
132
133 - `object: Literal["eval"]`
134
135 The object type.
136
137 - `"eval"`
138
139 - `testing_criteria: List[TestingCriterion]`
140
141 A list of testing criteria.
142
143 - `class LabelModelGrader: …`
144
145 A LabelModelGrader object which uses a model to assign labels to each item
146 in the evaluation.
147
148 - `input: List[Input]`
149
150 - `content: InputContent`
151
152 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
153
154 - `str`
155
156 A text input to the model.
157
158 - `class ResponseInputText: …`
159
160 A text input to the model.
161
162 - `text: str`
163
164 The text input to the model.
165
166 - `type: Literal["input_text"]`
167
168 The type of the input item. Always `input_text`.
169
170 - `"input_text"`
171
172 - `class InputContentOutputText: …`
173
174 A text output from the model.
175
176 - `text: str`
177
178 The text output from the model.
179
180 - `type: Literal["output_text"]`
181
182 The type of the output text. Always `output_text`.
183
184 - `"output_text"`
185
186 - `class InputContentInputImage: …`
187
188 An image input block used within EvalItem content arrays.
189
190 - `image_url: str`
191
192 The URL of the image input.
193
194 - `type: Literal["input_image"]`
195
196 The type of the image input. Always `input_image`.
197
198 - `"input_image"`
199
200 - `detail: Optional[str]`
201
202 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
203
204 - `class ResponseInputAudio: …`
205
206 An audio input to the model.
207
208 - `input_audio: InputAudio`
209
210 - `data: str`
211
212 Base64-encoded audio data.
213
214 - `format: Literal["mp3", "wav"]`
215
216 The format of the audio data. Currently supported formats are `mp3` and
217 `wav`.
218
219 - `"mp3"`
220
221 - `"wav"`
222
223 - `type: Literal["input_audio"]`
224
225 The type of the input item. Always `input_audio`.
226
227 - `"input_audio"`
228
229 - `List[GraderInputItem]`
230
231 - `str`
232
233 A text input to the model.
234
235 - `class ResponseInputText: …`
236
237 A text input to the model.
238
239 - `class GraderInputItemOutputText: …`
240
241 A text output from the model.
242
243 - `text: str`
244
245 The text output from the model.
246
247 - `type: Literal["output_text"]`
248
249 The type of the output text. Always `output_text`.
250
251 - `"output_text"`
252
253 - `class GraderInputItemInputImage: …`
254
255 An image input block used within EvalItem content arrays.
256
257 - `image_url: str`
258
259 The URL of the image input.
260
261 - `type: Literal["input_image"]`
262
263 The type of the image input. Always `input_image`.
264
265 - `"input_image"`
266
267 - `detail: Optional[str]`
268
269 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
270
271 - `class ResponseInputAudio: …`
272
273 An audio input to the model.
274
275 - `role: Literal["user", "assistant", "system", "developer"]`
276
277 The role of the message input. One of `user`, `assistant`, `system`, or
278 `developer`.
279
280 - `"user"`
281
282 - `"assistant"`
283
284 - `"system"`
285
286 - `"developer"`
287
288 - `type: Optional[Literal["message"]]`
289
290 The type of the message input. Always `message`.
291
292 - `"message"`
293
294 - `labels: List[str]`
295
296 The labels to assign to each item in the evaluation.
297
298 - `model: str`
299
300 The model to use for the evaluation. Must support structured outputs.
301
302 - `name: str`
303
304 The name of the grader.
305
306 - `passing_labels: List[str]`
307
308 The labels that indicate a passing result. Must be a subset of labels.
309
310 - `type: Literal["label_model"]`
311
312 The object type, which is always `label_model`.
313
314 - `"label_model"`
315
316 - `class StringCheckGrader: …`
317
318 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
319
320 - `input: str`
321
322 The input text. This may include template strings.
323
324 - `name: str`
325
326 The name of the grader.
327
328 - `operation: Literal["eq", "ne", "like", "ilike"]`
329
330 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
331
332 - `"eq"`
333
334 - `"ne"`
335
336 - `"like"`
337
338 - `"ilike"`
339
340 - `reference: str`
341
342 The reference text. This may include template strings.
343
344 - `type: Literal["string_check"]`
345
346 The object type, which is always `string_check`.
347
348 - `"string_check"`
349
350 - `class TestingCriterionEvalGraderTextSimilarity: …`
351
352 A TextSimilarityGrader object which grades text based on similarity metrics.
353
354 - `pass_threshold: float`
355
356 The threshold for the score.
357
358 - `class TestingCriterionEvalGraderPython: …`
359
360 A PythonGrader object that runs a python script on the input.
361
362 - `pass_threshold: Optional[float]`
363
364 The threshold for the score.
365
366 - `class TestingCriterionEvalGraderScoreModel: …`
367
368 A ScoreModelGrader object that uses a model to assign a score to the input.
369
370 - `pass_threshold: Optional[float]`
371
372 The threshold for the score.
373
374### Example
375
376```python
377import os
378from openai import OpenAI
379
380client = OpenAI(
381 api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted
382)
383eval = client.evals.update(
384 eval_id="eval_id",
385)
386print(eval.id)
387```
388
389#### Response
390
391```json
392{
393 "id": "id",
394 "created_at": 0,
395 "data_source_config": {
396 "schema": {
397 "foo": "bar"
398 },
399 "type": "custom"
400 },
401 "metadata": {
402 "foo": "string"
403 },
404 "name": "Chatbot effectiveness Evaluation",
405 "object": "eval",
406 "testing_criteria": [
407 {
408 "input": [
409 {
410 "content": "string",
411 "role": "user",
412 "type": "message"
413 }
414 ],
415 "labels": [
416 "string"
417 ],
418 "model": "model",
419 "name": "name",
420 "passing_labels": [
421 "string"
422 ],
423 "type": "label_model"
424 }
425 ]
426}
427```
428
429### Example
430
431```python
432from openai import OpenAI
433client = OpenAI()
434
435updated_eval = client.evals.update(
436 "eval_67abd54d9b0081909a86353f6fb9317a",
437 name="Updated Eval",
438 metadata={"description": "Updated description"}
439)
440print(updated_eval)
441```
442
443#### Response
444
445```json
446{
447 "object": "eval",
448 "id": "eval_67abd54d9b0081909a86353f6fb9317a",
449 "data_source_config": {
450 "type": "custom",
451 "schema": {
452 "type": "object",
453 "properties": {
454 "item": {
455 "type": "object",
456 "properties": {
457 "input": {
458 "type": "string"
459 },
460 "ground_truth": {
461 "type": "string"
462 }
463 },
464 "required": [
465 "input",
466 "ground_truth"
467 ]
468 }
469 },
470 "required": [
471 "item"
472 ]
473 }
474 },
475 "testing_criteria": [
476 {
477 "name": "String check",
478 "id": "String check-2eaf2d8d-d649-4335-8148-9535a7ca73c2",
479 "type": "string_check",
480 "input": "{{item.input}}",
481 "reference": "{{item.ground_truth}}",
482 "operation": "eq"
483 }
484 ],
485 "name": "Updated Eval",
486 "created_at": 1739314509,
487 "metadata": {"description": "Updated description"},
488}
489```