java/resources/evals/methods/create/index.md +0 −765 deleted
File Deleted View Diff
1## Create eval
2
3`EvalCreateResponse evals().create(EvalCreateParamsparams, RequestOptionsrequestOptions = RequestOptions.none())`
4
5**post** `/evals`
6
7Create the structure of an evaluation that can be used to test a model's performance.
8An evaluation is a set of testing criteria and the config for a data source, which dictates the schema of the data used in the evaluation. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources.
9For more information, see the [Evals guide](https://platform.openai.com/docs/guides/evals).
10
11### Parameters
12
13- `EvalCreateParams params`
14
15 - `DataSourceConfig dataSourceConfig`
16
17 The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
18
19 - `class Custom:`
20
21 A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs.
22 This schema is used to define the shape of the data that will be:
23
24 - Used to define your testing criteria and
25 - What data is required when creating a run
26
27 - `ItemSchema itemSchema`
28
29 The json schema for each row in the data source.
30
31 - `JsonValue; type "custom"constant`
32
33 The type of data source. Always `custom`.
34
35 - `CUSTOM("custom")`
36
37 - `Optional<Boolean> includeSampleSchema`
38
39 Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)
40
41 - `class Logs:`
42
43 A data source config which specifies the metadata property of your logs query.
44 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
45
46 - `JsonValue; type "logs"constant`
47
48 The type of data source. Always `logs`.
49
50 - `LOGS("logs")`
51
52 - `Optional<Metadata> metadata`
53
54 Metadata filters for the logs data source.
55
56 - `class StoredCompletions:`
57
58 Deprecated in favor of LogsDataSourceConfig.
59
60 - `JsonValue; type "stored_completions"constant`
61
62 The type of data source. Always `stored_completions`.
63
64 - `STORED_COMPLETIONS("stored_completions")`
65
66 - `Optional<Metadata> metadata`
67
68 Metadata filters for the stored completions data source.
69
70 - `List<TestingCriterion> testingCriteria`
71
72 A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).
73
74 - `class LabelModel:`
75
76 A LabelModelGrader object which uses a model to assign labels to each item
77 in the evaluation.
78
79 - `List<Input> input`
80
81 A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
82
83 - `class SimpleInputMessage:`
84
85 - `String content`
86
87 The content of the message.
88
89 - `String role`
90
91 The role of the message (e.g. "system", "assistant", "user").
92
93 - `class EvalItem:`
94
95 A message input to the model with a role indicating instruction following
96 hierarchy. Instructions given with the `developer` or `system` role take
97 precedence over instructions given with the `user` role. Messages with the
98 `assistant` role are presumed to have been generated by the model in previous
99 interactions.
100
101 - `Content content`
102
103 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
104
105 - `String`
106
107 - `class ResponseInputText:`
108
109 A text input to the model.
110
111 - `String text`
112
113 The text input to the model.
114
115 - `JsonValue; type "input_text"constant`
116
117 The type of the input item. Always `input_text`.
118
119 - `INPUT_TEXT("input_text")`
120
121 - `Optional<PromptCacheBreakpoint> promptCacheBreakpoint`
122
123 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
124
125 - `JsonValue; mode "explicit"constant`
126
127 The breakpoint mode. Always `explicit`.
128
129 - `EXPLICIT("explicit")`
130
131 - `class OutputText:`
132
133 A text output from the model.
134
135 - `String text`
136
137 The text output from the model.
138
139 - `JsonValue; type "output_text"constant`
140
141 The type of the output text. Always `output_text`.
142
143 - `OUTPUT_TEXT("output_text")`
144
145 - `class InputImage:`
146
147 An image input block used within EvalItem content arrays.
148
149 - `String imageUrl`
150
151 The URL of the image input.
152
153 - `JsonValue; type "input_image"constant`
154
155 The type of the image input. Always `input_image`.
156
157 - `INPUT_IMAGE("input_image")`
158
159 - `Optional<String> detail`
160
161 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
162
163 - `class ResponseInputAudio:`
164
165 An audio input to the model.
166
167 - `InputAudio inputAudio`
168
169 - `String data`
170
171 Base64-encoded audio data.
172
173 - `Format format`
174
175 The format of the audio data. Currently supported formats are `mp3` and
176 `wav`.
177
178 - `MP3("mp3")`
179
180 - `WAV("wav")`
181
182 - `JsonValue; type "input_audio"constant`
183
184 The type of the input item. Always `input_audio`.
185
186 - `INPUT_AUDIO("input_audio")`
187
188 - `List<EvalContentItem>`
189
190 - `String`
191
192 - `class ResponseInputText:`
193
194 A text input to the model.
195
196 - `OutputText`
197
198 - `String text`
199
200 The text output from the model.
201
202 - `JsonValue; type "output_text"constant`
203
204 The type of the output text. Always `output_text`.
205
206 - `OUTPUT_TEXT("output_text")`
207
208 - `InputImage`
209
210 - `String imageUrl`
211
212 The URL of the image input.
213
214 - `JsonValue; type "input_image"constant`
215
216 The type of the image input. Always `input_image`.
217
218 - `INPUT_IMAGE("input_image")`
219
220 - `Optional<String> detail`
221
222 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
223
224 - `class ResponseInputAudio:`
225
226 An audio input to the model.
227
228 - `Role role`
229
230 The role of the message input. One of `user`, `assistant`, `system`, or
231 `developer`.
232
233 - `USER("user")`
234
235 - `ASSISTANT("assistant")`
236
237 - `SYSTEM("system")`
238
239 - `DEVELOPER("developer")`
240
241 - `Optional<Type> type`
242
243 The type of the message input. Always `message`.
244
245 - `MESSAGE("message")`
246
247 - `List<String> labels`
248
249 The labels to classify to each item in the evaluation.
250
251 - `String model`
252
253 The model to use for the evaluation. Must support structured outputs.
254
255 - `String name`
256
257 The name of the grader.
258
259 - `List<String> passingLabels`
260
261 The labels that indicate a passing result. Must be a subset of labels.
262
263 - `JsonValue; type "label_model"constant`
264
265 The object type, which is always `label_model`.
266
267 - `LABEL_MODEL("label_model")`
268
269 - `class StringCheckGrader:`
270
271 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
272
273 - `String input`
274
275 The input text. This may include template strings.
276
277 - `String name`
278
279 The name of the grader.
280
281 - `Operation operation`
282
283 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
284
285 - `EQ("eq")`
286
287 - `NE("ne")`
288
289 - `LIKE("like")`
290
291 - `ILIKE("ilike")`
292
293 - `String reference`
294
295 The reference text. This may include template strings.
296
297 - `JsonValue; type "string_check"constant`
298
299 The object type, which is always `string_check`.
300
301 - `STRING_CHECK("string_check")`
302
303 - `class TextSimilarity:`
304
305 A TextSimilarityGrader object which grades text based on similarity metrics.
306
307 - `double passThreshold`
308
309 The threshold for the score.
310
311 - `class Python:`
312
313 A PythonGrader object that runs a python script on the input.
314
315 - `Optional<Double> passThreshold`
316
317 The threshold for the score.
318
319 - `class ScoreModel:`
320
321 A ScoreModelGrader object that uses a model to assign a score to the input.
322
323 - `Optional<Double> passThreshold`
324
325 The threshold for the score.
326
327 - `Optional<Metadata> metadata`
328
329 Set of 16 key-value pairs that can be attached to an object. This can be
330 useful for storing additional information about the object in a structured
331 format, and querying for objects via API or the dashboard.
332
333 Keys are strings with a maximum length of 64 characters. Values are strings
334 with a maximum length of 512 characters.
335
336 - `Optional<String> name`
337
338 The name of the evaluation.
339
340### Returns
341
342- `class EvalCreateResponse:`
343
344 An Eval object with a data source config and testing criteria.
345 An Eval represents a task to be done for your LLM integration.
346 Like:
347
348 - Improve the quality of my chatbot
349 - See how well my chatbot handles customer support
350 - Check if o4-mini is better at my usecase than gpt-4o
351
352 - `String id`
353
354 Unique identifier for the evaluation.
355
356 - `long createdAt`
357
358 The Unix timestamp (in seconds) for when the eval was created.
359
360 - `DataSourceConfig dataSourceConfig`
361
362 Configuration of data sources used in runs of the evaluation.
363
364 - `class EvalCustomDataSourceConfig:`
365
366 A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
367 The response schema defines the shape of the data that will be:
368
369 - Used to define your testing criteria and
370 - What data is required when creating a run
371
372 - `Schema schema`
373
374 The json schema for the run data source items.
375 Learn how to build JSON schemas [here](https://json-schema.org/).
376
377 - `JsonValue; type "custom"constant`
378
379 The type of data source. Always `custom`.
380
381 - `CUSTOM("custom")`
382
383 - `class Logs:`
384
385 A LogsDataSourceConfig which specifies the metadata property of your logs query.
386 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
387 The schema returned by this data source config is used to defined what variables are available in your evals.
388 `item` and `sample` are both defined when using this data source config.
389
390 - `Schema schema`
391
392 The json schema for the run data source items.
393 Learn how to build JSON schemas [here](https://json-schema.org/).
394
395 - `JsonValue; type "logs"constant`
396
397 The type of data source. Always `logs`.
398
399 - `LOGS("logs")`
400
401 - `Optional<Metadata> metadata`
402
403 Set of 16 key-value pairs that can be attached to an object. This can be
404 useful for storing additional information about the object in a structured
405 format, and querying for objects via API or the dashboard.
406
407 Keys are strings with a maximum length of 64 characters. Values are strings
408 with a maximum length of 512 characters.
409
410 - `class EvalStoredCompletionsDataSourceConfig:`
411
412 Deprecated in favor of LogsDataSourceConfig.
413
414 - `Schema schema`
415
416 The json schema for the run data source items.
417 Learn how to build JSON schemas [here](https://json-schema.org/).
418
419 - `JsonValue; type "stored_completions"constant`
420
421 The type of data source. Always `stored_completions`.
422
423 - `STORED_COMPLETIONS("stored_completions")`
424
425 - `Optional<Metadata> metadata`
426
427 Set of 16 key-value pairs that can be attached to an object. This can be
428 useful for storing additional information about the object in a structured
429 format, and querying for objects via API or the dashboard.
430
431 Keys are strings with a maximum length of 64 characters. Values are strings
432 with a maximum length of 512 characters.
433
434 - `Optional<Metadata> metadata`
435
436 Set of 16 key-value pairs that can be attached to an object. This can be
437 useful for storing additional information about the object in a structured
438 format, and querying for objects via API or the dashboard.
439
440 Keys are strings with a maximum length of 64 characters. Values are strings
441 with a maximum length of 512 characters.
442
443 - `String name`
444
445 The name of the evaluation.
446
447 - `JsonValue; object_ "eval"constant`
448
449 The object type.
450
451 - `EVAL("eval")`
452
453 - `List<TestingCriterion> testingCriteria`
454
455 A list of testing criteria.
456
457 - `class LabelModelGrader:`
458
459 A LabelModelGrader object which uses a model to assign labels to each item
460 in the evaluation.
461
462 - `List<Input> input`
463
464 - `Content content`
465
466 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
467
468 - `String`
469
470 - `class ResponseInputText:`
471
472 A text input to the model.
473
474 - `String text`
475
476 The text input to the model.
477
478 - `JsonValue; type "input_text"constant`
479
480 The type of the input item. Always `input_text`.
481
482 - `INPUT_TEXT("input_text")`
483
484 - `Optional<PromptCacheBreakpoint> promptCacheBreakpoint`
485
486 Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's `prompt_cache_options.ttl`; the boundary is not rounded to a token block.
487
488 - `JsonValue; mode "explicit"constant`
489
490 The breakpoint mode. Always `explicit`.
491
492 - `EXPLICIT("explicit")`
493
494 - `class OutputText:`
495
496 A text output from the model.
497
498 - `String text`
499
500 The text output from the model.
501
502 - `JsonValue; type "output_text"constant`
503
504 The type of the output text. Always `output_text`.
505
506 - `OUTPUT_TEXT("output_text")`
507
508 - `class InputImage:`
509
510 An image input block used within EvalItem content arrays.
511
512 - `String imageUrl`
513
514 The URL of the image input.
515
516 - `JsonValue; type "input_image"constant`
517
518 The type of the image input. Always `input_image`.
519
520 - `INPUT_IMAGE("input_image")`
521
522 - `Optional<String> detail`
523
524 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
525
526 - `class ResponseInputAudio:`
527
528 An audio input to the model.
529
530 - `InputAudio inputAudio`
531
532 - `String data`
533
534 Base64-encoded audio data.
535
536 - `Format format`
537
538 The format of the audio data. Currently supported formats are `mp3` and
539 `wav`.
540
541 - `MP3("mp3")`
542
543 - `WAV("wav")`
544
545 - `JsonValue; type "input_audio"constant`
546
547 The type of the input item. Always `input_audio`.
548
549 - `INPUT_AUDIO("input_audio")`
550
551 - `List<EvalContentItem>`
552
553 - `String`
554
555 - `class ResponseInputText:`
556
557 A text input to the model.
558
559 - `OutputText`
560
561 - `String text`
562
563 The text output from the model.
564
565 - `JsonValue; type "output_text"constant`
566
567 The type of the output text. Always `output_text`.
568
569 - `OUTPUT_TEXT("output_text")`
570
571 - `InputImage`
572
573 - `String imageUrl`
574
575 The URL of the image input.
576
577 - `JsonValue; type "input_image"constant`
578
579 The type of the image input. Always `input_image`.
580
581 - `INPUT_IMAGE("input_image")`
582
583 - `Optional<String> detail`
584
585 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
586
587 - `class ResponseInputAudio:`
588
589 An audio input to the model.
590
591 - `Role role`
592
593 The role of the message input. One of `user`, `assistant`, `system`, or
594 `developer`.
595
596 - `USER("user")`
597
598 - `ASSISTANT("assistant")`
599
600 - `SYSTEM("system")`
601
602 - `DEVELOPER("developer")`
603
604 - `Optional<Type> type`
605
606 The type of the message input. Always `message`.
607
608 - `MESSAGE("message")`
609
610 - `List<String> labels`
611
612 The labels to assign to each item in the evaluation.
613
614 - `String model`
615
616 The model to use for the evaluation. Must support structured outputs.
617
618 - `String name`
619
620 The name of the grader.
621
622 - `List<String> passingLabels`
623
624 The labels that indicate a passing result. Must be a subset of labels.
625
626 - `JsonValue; type "label_model"constant`
627
628 The object type, which is always `label_model`.
629
630 - `LABEL_MODEL("label_model")`
631
632 - `class StringCheckGrader:`
633
634 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
635
636 - `String input`
637
638 The input text. This may include template strings.
639
640 - `String name`
641
642 The name of the grader.
643
644 - `Operation operation`
645
646 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
647
648 - `EQ("eq")`
649
650 - `NE("ne")`
651
652 - `LIKE("like")`
653
654 - `ILIKE("ilike")`
655
656 - `String reference`
657
658 The reference text. This may include template strings.
659
660 - `JsonValue; type "string_check"constant`
661
662 The object type, which is always `string_check`.
663
664 - `STRING_CHECK("string_check")`
665
666 - `class EvalGraderTextSimilarity:`
667
668 A TextSimilarityGrader object which grades text based on similarity metrics.
669
670 - `double passThreshold`
671
672 The threshold for the score.
673
674 - `class EvalGraderPython:`
675
676 A PythonGrader object that runs a python script on the input.
677
678 - `Optional<Double> passThreshold`
679
680 The threshold for the score.
681
682 - `class EvalGraderScoreModel:`
683
684 A ScoreModelGrader object that uses a model to assign a score to the input.
685
686 - `Optional<Double> passThreshold`
687
688 The threshold for the score.
689
690### Example
691
692```java
693package com.openai.example;
694
695import com.openai.client.OpenAIClient;
696import com.openai.client.okhttp.OpenAIOkHttpClient;
697import com.openai.core.JsonValue;
698import com.openai.models.evals.EvalCreateParams;
699import com.openai.models.evals.EvalCreateResponse;
700
701public final class Main {
702 private Main() {}
703
704 public static void main(String[] args) {
705 OpenAIClient client = OpenAIOkHttpClient.fromEnv();
706
707 EvalCreateParams params = EvalCreateParams.builder()
708 .customDataSourceConfig(EvalCreateParams.DataSourceConfig.Custom.ItemSchema.builder()
709 .putAdditionalProperty("foo", JsonValue.from("bar"))
710 .build())
711 .addTestingCriterion(EvalCreateParams.TestingCriterion.LabelModel.builder()
712 .addInput(EvalCreateParams.TestingCriterion.LabelModel.Input.SimpleInputMessage.builder()
713 .content("content")
714 .role("role")
715 .build())
716 .addLabel("string")
717 .model("model")
718 .name("name")
719 .addPassingLabel("string")
720 .build())
721 .build();
722 EvalCreateResponse eval = client.evals().create(params);
723 }
724}
725```
726
727#### Response
728
729```json
730{
731 "id": "id",
732 "created_at": 0,
733 "data_source_config": {
734 "schema": {
735 "foo": "bar"
736 },
737 "type": "custom"
738 },
739 "metadata": {
740 "foo": "string"
741 },
742 "name": "Chatbot effectiveness Evaluation",
743 "object": "eval",
744 "testing_criteria": [
745 {
746 "input": [
747 {
748 "content": "string",
749 "role": "user",
750 "type": "message"
751 }
752 ],
753 "labels": [
754 "string"
755 ],
756 "model": "model",
757 "name": "name",
758 "passing_labels": [
759 "string"
760 ],
761 "type": "label_model"
762 }
763 ]
764}
765```