ruby/resources/evals/methods/update/index.md +0 −427 deleted
File Deleted View Diff
1## Update an eval
2
3`evals.update(eval_id, **kwargs) -> EvalUpdateResponse`
4
5**post** `/evals/{eval_id}`
6
7Update certain properties of an evaluation.
8
9### Parameters
10
11- `eval_id: String`
12
13- `metadata: Metadata`
14
15 Set of 16 key-value pairs that can be attached to an object. This can be
16 useful for storing additional information about the object in a structured
17 format, and querying for objects via API or the dashboard.
18
19 Keys are strings with a maximum length of 64 characters. Values are strings
20 with a maximum length of 512 characters.
21
22- `name: String`
23
24 Rename the evaluation.
25
26### Returns
27
28- `class EvalUpdateResponse`
29
30 An Eval object with a data source config and testing criteria.
31 An Eval represents a task to be done for your LLM integration.
32 Like:
33
34 - Improve the quality of my chatbot
35 - See how well my chatbot handles customer support
36 - Check if o4-mini is better at my usecase than gpt-4o
37
38 - `id: String`
39
40 Unique identifier for the evaluation.
41
42 - `created_at: Integer`
43
44 The Unix timestamp (in seconds) for when the eval was created.
45
46 - `data_source_config: EvalCustomDataSourceConfig | Logs{ schema, type, metadata} | EvalStoredCompletionsDataSourceConfig`
47
48 Configuration of data sources used in runs of the evaluation.
49
50 - `class EvalCustomDataSourceConfig`
51
52 A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
53 The response schema defines the shape of the data that will be:
54
55 - Used to define your testing criteria and
56 - What data is required when creating a run
57
58 - `schema: Hash[Symbol, untyped]`
59
60 The json schema for the run data source items.
61 Learn how to build JSON schemas [here](https://json-schema.org/).
62
63 - `type: :custom`
64
65 The type of data source. Always `custom`.
66
67 - `:custom`
68
69 - `class Logs`
70
71 A LogsDataSourceConfig which specifies the metadata property of your logs query.
72 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
73 The schema returned by this data source config is used to defined what variables are available in your evals.
74 `item` and `sample` are both defined when using this data source config.
75
76 - `schema: Hash[Symbol, untyped]`
77
78 The json schema for the run data source items.
79 Learn how to build JSON schemas [here](https://json-schema.org/).
80
81 - `type: :logs`
82
83 The type of data source. Always `logs`.
84
85 - `:logs`
86
87 - `metadata: Metadata`
88
89 Set of 16 key-value pairs that can be attached to an object. This can be
90 useful for storing additional information about the object in a structured
91 format, and querying for objects via API or the dashboard.
92
93 Keys are strings with a maximum length of 64 characters. Values are strings
94 with a maximum length of 512 characters.
95
96 - `class EvalStoredCompletionsDataSourceConfig`
97
98 Deprecated in favor of LogsDataSourceConfig.
99
100 - `schema: Hash[Symbol, untyped]`
101
102 The json schema for the run data source items.
103 Learn how to build JSON schemas [here](https://json-schema.org/).
104
105 - `type: :stored_completions`
106
107 The type of data source. Always `stored_completions`.
108
109 - `:stored_completions`
110
111 - `metadata: Metadata`
112
113 Set of 16 key-value pairs that can be attached to an object. This can be
114 useful for storing additional information about the object in a structured
115 format, and querying for objects via API or the dashboard.
116
117 Keys are strings with a maximum length of 64 characters. Values are strings
118 with a maximum length of 512 characters.
119
120 - `metadata: Metadata`
121
122 Set of 16 key-value pairs that can be attached to an object. This can be
123 useful for storing additional information about the object in a structured
124 format, and querying for objects via API or the dashboard.
125
126 Keys are strings with a maximum length of 64 characters. Values are strings
127 with a maximum length of 512 characters.
128
129 - `name: String`
130
131 The name of the evaluation.
132
133 - `object: :eval`
134
135 The object type.
136
137 - `:eval`
138
139 - `testing_criteria: Array[LabelModelGrader | StringCheckGrader | TextSimilarityGrader & { pass_threshold} | 2 more]`
140
141 A list of testing criteria.
142
143 - `class LabelModelGrader`
144
145 A LabelModelGrader object which uses a model to assign labels to each item
146 in the evaluation.
147
148 - `input: Array[Input{ content, role, type}]`
149
150 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`
151
152 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
153
154 - `String = String`
155
156 A text input to the model.
157
158 - `class ResponseInputText`
159
160 A text input to the model.
161
162 - `text: String`
163
164 The text input to the model.
165
166 - `type: :input_text`
167
168 The type of the input item. Always `input_text`.
169
170 - `:input_text`
171
172 - `class OutputText`
173
174 A text output from the model.
175
176 - `text: String`
177
178 The text output from the model.
179
180 - `type: :output_text`
181
182 The type of the output text. Always `output_text`.
183
184 - `:output_text`
185
186 - `class InputImage`
187
188 An image input block used within EvalItem content arrays.
189
190 - `image_url: String`
191
192 The URL of the image input.
193
194 - `type: :input_image`
195
196 The type of the image input. Always `input_image`.
197
198 - `:input_image`
199
200 - `detail: String`
201
202 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
203
204 - `class ResponseInputAudio`
205
206 An audio input to the model.
207
208 - `input_audio: InputAudio{ data, format_}`
209
210 - `data: String`
211
212 Base64-encoded audio data.
213
214 - `format_: :mp3 | :wav`
215
216 The format of the audio data. Currently supported formats are `mp3` and
217 `wav`.
218
219 - `:mp3`
220
221 - `:wav`
222
223 - `type: :input_audio`
224
225 The type of the input item. Always `input_audio`.
226
227 - `:input_audio`
228
229 - `GraderInputs = Array[GraderInputItem]`
230
231 A list of inputs, each of which may be either an input text, output text, input
232 image, or input audio object.
233
234 - `String = String`
235
236 A text input to the model.
237
238 - `class ResponseInputText`
239
240 A text input to the model.
241
242 - `class OutputText`
243
244 A text output from the model.
245
246 - `text: String`
247
248 The text output from the model.
249
250 - `type: :output_text`
251
252 The type of the output text. Always `output_text`.
253
254 - `:output_text`
255
256 - `class InputImage`
257
258 An image input block used within EvalItem content arrays.
259
260 - `image_url: String`
261
262 The URL of the image input.
263
264 - `type: :input_image`
265
266 The type of the image input. Always `input_image`.
267
268 - `:input_image`
269
270 - `detail: String`
271
272 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
273
274 - `class ResponseInputAudio`
275
276 An audio input to the model.
277
278 - `role: :user | :assistant | :system | :developer`
279
280 The role of the message input. One of `user`, `assistant`, `system`, or
281 `developer`.
282
283 - `:user`
284
285 - `:assistant`
286
287 - `:system`
288
289 - `:developer`
290
291 - `type: :message`
292
293 The type of the message input. Always `message`.
294
295 - `:message`
296
297 - `labels: Array[String]`
298
299 The labels to assign to each item in the evaluation.
300
301 - `model: String`
302
303 The model to use for the evaluation. Must support structured outputs.
304
305 - `name: String`
306
307 The name of the grader.
308
309 - `passing_labels: Array[String]`
310
311 The labels that indicate a passing result. Must be a subset of labels.
312
313 - `type: :label_model`
314
315 The object type, which is always `label_model`.
316
317 - `:label_model`
318
319 - `class StringCheckGrader`
320
321 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
322
323 - `input: String`
324
325 The input text. This may include template strings.
326
327 - `name: String`
328
329 The name of the grader.
330
331 - `operation: :eq | :ne | :like | :ilike`
332
333 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
334
335 - `:eq`
336
337 - `:ne`
338
339 - `:like`
340
341 - `:ilike`
342
343 - `reference: String`
344
345 The reference text. This may include template strings.
346
347 - `type: :string_check`
348
349 The object type, which is always `string_check`.
350
351 - `:string_check`
352
353 - `class EvalGraderTextSimilarity`
354
355 A TextSimilarityGrader object which grades text based on similarity metrics.
356
357 - `pass_threshold: Float`
358
359 The threshold for the score.
360
361 - `class EvalGraderPython`
362
363 A PythonGrader object that runs a python script on the input.
364
365 - `pass_threshold: Float`
366
367 The threshold for the score.
368
369 - `class EvalGraderScoreModel`
370
371 A ScoreModelGrader object that uses a model to assign a score to the input.
372
373 - `pass_threshold: Float`
374
375 The threshold for the score.
376
377### Example
378
379```ruby
380require "openai"
381
382openai = OpenAI::Client.new(api_key: "My API Key")
383
384eval_ = openai.evals.update("eval_id")
385
386puts(eval_)
387```
388
389#### Response
390
391```json
392{
393 "id": "id",
394 "created_at": 0,
395 "data_source_config": {
396 "schema": {
397 "foo": "bar"
398 },
399 "type": "custom"
400 },
401 "metadata": {
402 "foo": "string"
403 },
404 "name": "Chatbot effectiveness Evaluation",
405 "object": "eval",
406 "testing_criteria": [
407 {
408 "input": [
409 {
410 "content": "string",
411 "role": "user",
412 "type": "message"
413 }
414 ],
415 "labels": [
416 "string"
417 ],
418 "model": "model",
419 "name": "name",
420 "passing_labels": [
421 "string"
422 ],
423 "type": "label_model"
424 }
425 ]
426}
427```