ruby/resources/evals/methods/retrieve/index.md +0 −424 deleted
File Deleted View Diff
1## Get an eval
2
3`evals.retrieve(eval_id) -> EvalRetrieveResponse`
4
5**get** `/evals/{eval_id}`
6
7Get an evaluation by ID.
8
9### Parameters
10
11- `eval_id: String`
12
13### Returns
14
15- `class EvalRetrieveResponse`
16
17 An Eval object with a data source config and testing criteria.
18 An Eval represents a task to be done for your LLM integration.
19 Like:
20
21 - Improve the quality of my chatbot
22 - See how well my chatbot handles customer support
23 - Check if o4-mini is better at my usecase than gpt-4o
24
25 - `id: String`
26
27 Unique identifier for the evaluation.
28
29 - `created_at: Integer`
30
31 The Unix timestamp (in seconds) for when the eval was created.
32
33 - `data_source_config: EvalCustomDataSourceConfig | Logs{ schema, type, metadata} | EvalStoredCompletionsDataSourceConfig`
34
35 Configuration of data sources used in runs of the evaluation.
36
37 - `class EvalCustomDataSourceConfig`
38
39 A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
40 The response schema defines the shape of the data that will be:
41
42 - Used to define your testing criteria and
43 - What data is required when creating a run
44
45 - `schema: Hash[Symbol, untyped]`
46
47 The json schema for the run data source items.
48 Learn how to build JSON schemas [here](https://json-schema.org/).
49
50 - `type: :custom`
51
52 The type of data source. Always `custom`.
53
54 - `:custom`
55
56 - `class Logs`
57
58 A LogsDataSourceConfig which specifies the metadata property of your logs query.
59 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
60 The schema returned by this data source config is used to defined what variables are available in your evals.
61 `item` and `sample` are both defined when using this data source config.
62
63 - `schema: Hash[Symbol, untyped]`
64
65 The json schema for the run data source items.
66 Learn how to build JSON schemas [here](https://json-schema.org/).
67
68 - `type: :logs`
69
70 The type of data source. Always `logs`.
71
72 - `:logs`
73
74 - `metadata: Metadata`
75
76 Set of 16 key-value pairs that can be attached to an object. This can be
77 useful for storing additional information about the object in a structured
78 format, and querying for objects via API or the dashboard.
79
80 Keys are strings with a maximum length of 64 characters. Values are strings
81 with a maximum length of 512 characters.
82
83 - `class EvalStoredCompletionsDataSourceConfig`
84
85 Deprecated in favor of LogsDataSourceConfig.
86
87 - `schema: Hash[Symbol, untyped]`
88
89 The json schema for the run data source items.
90 Learn how to build JSON schemas [here](https://json-schema.org/).
91
92 - `type: :stored_completions`
93
94 The type of data source. Always `stored_completions`.
95
96 - `:stored_completions`
97
98 - `metadata: Metadata`
99
100 Set of 16 key-value pairs that can be attached to an object. This can be
101 useful for storing additional information about the object in a structured
102 format, and querying for objects via API or the dashboard.
103
104 Keys are strings with a maximum length of 64 characters. Values are strings
105 with a maximum length of 512 characters.
106
107 - `metadata: Metadata`
108
109 Set of 16 key-value pairs that can be attached to an object. This can be
110 useful for storing additional information about the object in a structured
111 format, and querying for objects via API or the dashboard.
112
113 Keys are strings with a maximum length of 64 characters. Values are strings
114 with a maximum length of 512 characters.
115
116 - `name: String`
117
118 The name of the evaluation.
119
120 - `object: :eval`
121
122 The object type.
123
124 - `:eval`
125
126 - `testing_criteria: Array[LabelModelGrader | StringCheckGrader | TextSimilarityGrader & { pass_threshold} | 2 more]`
127
128 A list of testing criteria.
129
130 - `class LabelModelGrader`
131
132 A LabelModelGrader object which uses a model to assign labels to each item
133 in the evaluation.
134
135 - `input: Array[Input{ content, role, type}]`
136
137 - `content: String | ResponseInputText | OutputText{ text, type} | 3 more`
138
139 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
140
141 - `String = String`
142
143 A text input to the model.
144
145 - `class ResponseInputText`
146
147 A text input to the model.
148
149 - `text: String`
150
151 The text input to the model.
152
153 - `type: :input_text`
154
155 The type of the input item. Always `input_text`.
156
157 - `:input_text`
158
159 - `prompt_cache_breakpoint: PromptCacheBreakpoint{ mode}`
160
161 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
162
163 - `mode: :explicit`
164
165 The breakpoint mode. Always `explicit`.
166
167 - `:explicit`
168
169 - `class OutputText`
170
171 A text output from the model.
172
173 - `text: String`
174
175 The text output from the model.
176
177 - `type: :output_text`
178
179 The type of the output text. Always `output_text`.
180
181 - `:output_text`
182
183 - `class InputImage`
184
185 An image input block used within EvalItem content arrays.
186
187 - `image_url: String`
188
189 The URL of the image input.
190
191 - `type: :input_image`
192
193 The type of the image input. Always `input_image`.
194
195 - `:input_image`
196
197 - `detail: String`
198
199 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
200
201 - `class ResponseInputAudio`
202
203 An audio input to the model.
204
205 - `input_audio: InputAudio{ data, format_}`
206
207 - `data: String`
208
209 Base64-encoded audio data.
210
211 - `format_: :mp3 | :wav`
212
213 The format of the audio data. Currently supported formats are `mp3` and
214 `wav`.
215
216 - `:mp3`
217
218 - `:wav`
219
220 - `type: :input_audio`
221
222 The type of the input item. Always `input_audio`.
223
224 - `:input_audio`
225
226 - `GraderInputs = Array[GraderInputItem]`
227
228 A list of inputs, each of which may be either an input text, output text, input
229 image, or input audio object.
230
231 - `String = String`
232
233 A text input to the model.
234
235 - `class ResponseInputText`
236
237 A text input to the model.
238
239 - `class OutputText`
240
241 A text output from the model.
242
243 - `text: String`
244
245 The text output from the model.
246
247 - `type: :output_text`
248
249 The type of the output text. Always `output_text`.
250
251 - `:output_text`
252
253 - `class InputImage`
254
255 An image input block used within EvalItem content arrays.
256
257 - `image_url: String`
258
259 The URL of the image input.
260
261 - `type: :input_image`
262
263 The type of the image input. Always `input_image`.
264
265 - `:input_image`
266
267 - `detail: String`
268
269 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
270
271 - `class ResponseInputAudio`
272
273 An audio input to the model.
274
275 - `role: :user | :assistant | :system | :developer`
276
277 The role of the message input. One of `user`, `assistant`, `system`, or
278 `developer`.
279
280 - `:user`
281
282 - `:assistant`
283
284 - `:system`
285
286 - `:developer`
287
288 - `type: :message`
289
290 The type of the message input. Always `message`.
291
292 - `:message`
293
294 - `labels: Array[String]`
295
296 The labels to assign to each item in the evaluation.
297
298 - `model: String`
299
300 The model to use for the evaluation. Must support structured outputs.
301
302 - `name: String`
303
304 The name of the grader.
305
306 - `passing_labels: Array[String]`
307
308 The labels that indicate a passing result. Must be a subset of labels.
309
310 - `type: :label_model`
311
312 The object type, which is always `label_model`.
313
314 - `:label_model`
315
316 - `class StringCheckGrader`
317
318 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
319
320 - `input: String`
321
322 The input text. This may include template strings.
323
324 - `name: String`
325
326 The name of the grader.
327
328 - `operation: :eq | :ne | :like | :ilike`
329
330 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
331
332 - `:eq`
333
334 - `:ne`
335
336 - `:like`
337
338 - `:ilike`
339
340 - `reference: String`
341
342 The reference text. This may include template strings.
343
344 - `type: :string_check`
345
346 The object type, which is always `string_check`.
347
348 - `:string_check`
349
350 - `class EvalGraderTextSimilarity`
351
352 A TextSimilarityGrader object which grades text based on similarity metrics.
353
354 - `pass_threshold: Float`
355
356 The threshold for the score.
357
358 - `class EvalGraderPython`
359
360 A PythonGrader object that runs a python script on the input.
361
362 - `pass_threshold: Float`
363
364 The threshold for the score.
365
366 - `class EvalGraderScoreModel`
367
368 A ScoreModelGrader object that uses a model to assign a score to the input.
369
370 - `pass_threshold: Float`
371
372 The threshold for the score.
373
374### Example
375
376```ruby
377require "openai"
378
379openai = OpenAI::Client.new(api_key: "My API Key")
380
381eval_ = openai.evals.retrieve("eval_id")
382
383puts(eval_)
384```
385
386#### Response
387
388```json
389{
390 "id": "id",
391 "created_at": 0,
392 "data_source_config": {
393 "schema": {
394 "foo": "bar"
395 },
396 "type": "custom"
397 },
398 "metadata": {
399 "foo": "string"
400 },
401 "name": "Chatbot effectiveness Evaluation",
402 "object": "eval",
403 "testing_criteria": [
404 {
405 "input": [
406 {
407 "content": "string",
408 "role": "user",
409 "type": "message"
410 }
411 ],
412 "labels": [
413 "string"
414 ],
415 "model": "model",
416 "name": "name",
417 "passing_labels": [
418 "string"
419 ],
420 "type": "label_model"
421 }
422 ]
423}
424```