java/resources/evals/methods/create/index.md +0 −745 deleted
File Deleted View Diff
1## Create eval
2
3`EvalCreateResponse evals().create(EvalCreateParamsparams, RequestOptionsrequestOptions = RequestOptions.none())`
4
5**post** `/evals`
6
7Create the structure of an evaluation that can be used to test a model's performance.
8An evaluation is a set of testing criteria and the config for a data source, which dictates the schema of the data used in the evaluation. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources.
9For more information, see the [Evals guide](https://platform.openai.com/docs/guides/evals).
10
11### Parameters
12
13- `EvalCreateParams params`
14
15 - `DataSourceConfig dataSourceConfig`
16
17 The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
18
19 - `class Custom:`
20
21 A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs.
22 This schema is used to define the shape of the data that will be:
23
24 - Used to define your testing criteria and
25 - What data is required when creating a run
26
27 - `ItemSchema itemSchema`
28
29 The json schema for each row in the data source.
30
31 - `JsonValue; type "custom"constant`
32
33 The type of data source. Always `custom`.
34
35 - `CUSTOM("custom")`
36
37 - `Optional<Boolean> includeSampleSchema`
38
39 Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)
40
41 - `class Logs:`
42
43 A data source config which specifies the metadata property of your logs query.
44 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
45
46 - `JsonValue; type "logs"constant`
47
48 The type of data source. Always `logs`.
49
50 - `LOGS("logs")`
51
52 - `Optional<Metadata> metadata`
53
54 Metadata filters for the logs data source.
55
56 - `class StoredCompletions:`
57
58 Deprecated in favor of LogsDataSourceConfig.
59
60 - `JsonValue; type "stored_completions"constant`
61
62 The type of data source. Always `stored_completions`.
63
64 - `STORED_COMPLETIONS("stored_completions")`
65
66 - `Optional<Metadata> metadata`
67
68 Metadata filters for the stored completions data source.
69
70 - `List<TestingCriterion> testingCriteria`
71
72 A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).
73
74 - `class LabelModel:`
75
76 A LabelModelGrader object which uses a model to assign labels to each item
77 in the evaluation.
78
79 - `List<Input> input`
80
81 A list of chat messages forming the prompt or context. May include variable references to the `item` namespace, ie {{item.name}}.
82
83 - `class SimpleInputMessage:`
84
85 - `String content`
86
87 The content of the message.
88
89 - `String role`
90
91 The role of the message (e.g. "system", "assistant", "user").
92
93 - `class EvalItem:`
94
95 A message input to the model with a role indicating instruction following
96 hierarchy. Instructions given with the `developer` or `system` role take
97 precedence over instructions given with the `user` role. Messages with the
98 `assistant` role are presumed to have been generated by the model in previous
99 interactions.
100
101 - `Content content`
102
103 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
104
105 - `String`
106
107 - `class ResponseInputText:`
108
109 A text input to the model.
110
111 - `String text`
112
113 The text input to the model.
114
115 - `JsonValue; type "input_text"constant`
116
117 The type of the input item. Always `input_text`.
118
119 - `INPUT_TEXT("input_text")`
120
121 - `class OutputText:`
122
123 A text output from the model.
124
125 - `String text`
126
127 The text output from the model.
128
129 - `JsonValue; type "output_text"constant`
130
131 The type of the output text. Always `output_text`.
132
133 - `OUTPUT_TEXT("output_text")`
134
135 - `class InputImage:`
136
137 An image input block used within EvalItem content arrays.
138
139 - `String imageUrl`
140
141 The URL of the image input.
142
143 - `JsonValue; type "input_image"constant`
144
145 The type of the image input. Always `input_image`.
146
147 - `INPUT_IMAGE("input_image")`
148
149 - `Optional<String> detail`
150
151 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
152
153 - `class ResponseInputAudio:`
154
155 An audio input to the model.
156
157 - `InputAudio inputAudio`
158
159 - `String data`
160
161 Base64-encoded audio data.
162
163 - `Format format`
164
165 The format of the audio data. Currently supported formats are `mp3` and
166 `wav`.
167
168 - `MP3("mp3")`
169
170 - `WAV("wav")`
171
172 - `JsonValue; type "input_audio"constant`
173
174 The type of the input item. Always `input_audio`.
175
176 - `INPUT_AUDIO("input_audio")`
177
178 - `List<EvalContentItem>`
179
180 - `String`
181
182 - `class ResponseInputText:`
183
184 A text input to the model.
185
186 - `OutputText`
187
188 - `String text`
189
190 The text output from the model.
191
192 - `JsonValue; type "output_text"constant`
193
194 The type of the output text. Always `output_text`.
195
196 - `OUTPUT_TEXT("output_text")`
197
198 - `InputImage`
199
200 - `String imageUrl`
201
202 The URL of the image input.
203
204 - `JsonValue; type "input_image"constant`
205
206 The type of the image input. Always `input_image`.
207
208 - `INPUT_IMAGE("input_image")`
209
210 - `Optional<String> detail`
211
212 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
213
214 - `class ResponseInputAudio:`
215
216 An audio input to the model.
217
218 - `Role role`
219
220 The role of the message input. One of `user`, `assistant`, `system`, or
221 `developer`.
222
223 - `USER("user")`
224
225 - `ASSISTANT("assistant")`
226
227 - `SYSTEM("system")`
228
229 - `DEVELOPER("developer")`
230
231 - `Optional<Type> type`
232
233 The type of the message input. Always `message`.
234
235 - `MESSAGE("message")`
236
237 - `List<String> labels`
238
239 The labels to classify to each item in the evaluation.
240
241 - `String model`
242
243 The model to use for the evaluation. Must support structured outputs.
244
245 - `String name`
246
247 The name of the grader.
248
249 - `List<String> passingLabels`
250
251 The labels that indicate a passing result. Must be a subset of labels.
252
253 - `JsonValue; type "label_model"constant`
254
255 The object type, which is always `label_model`.
256
257 - `LABEL_MODEL("label_model")`
258
259 - `class StringCheckGrader:`
260
261 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
262
263 - `String input`
264
265 The input text. This may include template strings.
266
267 - `String name`
268
269 The name of the grader.
270
271 - `Operation operation`
272
273 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
274
275 - `EQ("eq")`
276
277 - `NE("ne")`
278
279 - `LIKE("like")`
280
281 - `ILIKE("ilike")`
282
283 - `String reference`
284
285 The reference text. This may include template strings.
286
287 - `JsonValue; type "string_check"constant`
288
289 The object type, which is always `string_check`.
290
291 - `STRING_CHECK("string_check")`
292
293 - `class TextSimilarity:`
294
295 A TextSimilarityGrader object which grades text based on similarity metrics.
296
297 - `double passThreshold`
298
299 The threshold for the score.
300
301 - `class Python:`
302
303 A PythonGrader object that runs a python script on the input.
304
305 - `Optional<Double> passThreshold`
306
307 The threshold for the score.
308
309 - `class ScoreModel:`
310
311 A ScoreModelGrader object that uses a model to assign a score to the input.
312
313 - `Optional<Double> passThreshold`
314
315 The threshold for the score.
316
317 - `Optional<Metadata> metadata`
318
319 Set of 16 key-value pairs that can be attached to an object. This can be
320 useful for storing additional information about the object in a structured
321 format, and querying for objects via API or the dashboard.
322
323 Keys are strings with a maximum length of 64 characters. Values are strings
324 with a maximum length of 512 characters.
325
326 - `Optional<String> name`
327
328 The name of the evaluation.
329
330### Returns
331
332- `class EvalCreateResponse:`
333
334 An Eval object with a data source config and testing criteria.
335 An Eval represents a task to be done for your LLM integration.
336 Like:
337
338 - Improve the quality of my chatbot
339 - See how well my chatbot handles customer support
340 - Check if o4-mini is better at my usecase than gpt-4o
341
342 - `String id`
343
344 Unique identifier for the evaluation.
345
346 - `long createdAt`
347
348 The Unix timestamp (in seconds) for when the eval was created.
349
350 - `DataSourceConfig dataSourceConfig`
351
352 Configuration of data sources used in runs of the evaluation.
353
354 - `class EvalCustomDataSourceConfig:`
355
356 A CustomDataSourceConfig which specifies the schema of your `item` and optionally `sample` namespaces.
357 The response schema defines the shape of the data that will be:
358
359 - Used to define your testing criteria and
360 - What data is required when creating a run
361
362 - `Schema schema`
363
364 The json schema for the run data source items.
365 Learn how to build JSON schemas [here](https://json-schema.org/).
366
367 - `JsonValue; type "custom"constant`
368
369 The type of data source. Always `custom`.
370
371 - `CUSTOM("custom")`
372
373 - `class Logs:`
374
375 A LogsDataSourceConfig which specifies the metadata property of your logs query.
376 This is usually metadata like `usecase=chatbot` or `prompt-version=v2`, etc.
377 The schema returned by this data source config is used to defined what variables are available in your evals.
378 `item` and `sample` are both defined when using this data source config.
379
380 - `Schema schema`
381
382 The json schema for the run data source items.
383 Learn how to build JSON schemas [here](https://json-schema.org/).
384
385 - `JsonValue; type "logs"constant`
386
387 The type of data source. Always `logs`.
388
389 - `LOGS("logs")`
390
391 - `Optional<Metadata> metadata`
392
393 Set of 16 key-value pairs that can be attached to an object. This can be
394 useful for storing additional information about the object in a structured
395 format, and querying for objects via API or the dashboard.
396
397 Keys are strings with a maximum length of 64 characters. Values are strings
398 with a maximum length of 512 characters.
399
400 - `class EvalStoredCompletionsDataSourceConfig:`
401
402 Deprecated in favor of LogsDataSourceConfig.
403
404 - `Schema schema`
405
406 The json schema for the run data source items.
407 Learn how to build JSON schemas [here](https://json-schema.org/).
408
409 - `JsonValue; type "stored_completions"constant`
410
411 The type of data source. Always `stored_completions`.
412
413 - `STORED_COMPLETIONS("stored_completions")`
414
415 - `Optional<Metadata> metadata`
416
417 Set of 16 key-value pairs that can be attached to an object. This can be
418 useful for storing additional information about the object in a structured
419 format, and querying for objects via API or the dashboard.
420
421 Keys are strings with a maximum length of 64 characters. Values are strings
422 with a maximum length of 512 characters.
423
424 - `Optional<Metadata> metadata`
425
426 Set of 16 key-value pairs that can be attached to an object. This can be
427 useful for storing additional information about the object in a structured
428 format, and querying for objects via API or the dashboard.
429
430 Keys are strings with a maximum length of 64 characters. Values are strings
431 with a maximum length of 512 characters.
432
433 - `String name`
434
435 The name of the evaluation.
436
437 - `JsonValue; object_ "eval"constant`
438
439 The object type.
440
441 - `EVAL("eval")`
442
443 - `List<TestingCriterion> testingCriteria`
444
445 A list of testing criteria.
446
447 - `class LabelModelGrader:`
448
449 A LabelModelGrader object which uses a model to assign labels to each item
450 in the evaluation.
451
452 - `List<Input> input`
453
454 - `Content content`
455
456 Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
457
458 - `String`
459
460 - `class ResponseInputText:`
461
462 A text input to the model.
463
464 - `String text`
465
466 The text input to the model.
467
468 - `JsonValue; type "input_text"constant`
469
470 The type of the input item. Always `input_text`.
471
472 - `INPUT_TEXT("input_text")`
473
474 - `class OutputText:`
475
476 A text output from the model.
477
478 - `String text`
479
480 The text output from the model.
481
482 - `JsonValue; type "output_text"constant`
483
484 The type of the output text. Always `output_text`.
485
486 - `OUTPUT_TEXT("output_text")`
487
488 - `class InputImage:`
489
490 An image input block used within EvalItem content arrays.
491
492 - `String imageUrl`
493
494 The URL of the image input.
495
496 - `JsonValue; type "input_image"constant`
497
498 The type of the image input. Always `input_image`.
499
500 - `INPUT_IMAGE("input_image")`
501
502 - `Optional<String> detail`
503
504 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
505
506 - `class ResponseInputAudio:`
507
508 An audio input to the model.
509
510 - `InputAudio inputAudio`
511
512 - `String data`
513
514 Base64-encoded audio data.
515
516 - `Format format`
517
518 The format of the audio data. Currently supported formats are `mp3` and
519 `wav`.
520
521 - `MP3("mp3")`
522
523 - `WAV("wav")`
524
525 - `JsonValue; type "input_audio"constant`
526
527 The type of the input item. Always `input_audio`.
528
529 - `INPUT_AUDIO("input_audio")`
530
531 - `List<EvalContentItem>`
532
533 - `String`
534
535 - `class ResponseInputText:`
536
537 A text input to the model.
538
539 - `OutputText`
540
541 - `String text`
542
543 The text output from the model.
544
545 - `JsonValue; type "output_text"constant`
546
547 The type of the output text. Always `output_text`.
548
549 - `OUTPUT_TEXT("output_text")`
550
551 - `InputImage`
552
553 - `String imageUrl`
554
555 The URL of the image input.
556
557 - `JsonValue; type "input_image"constant`
558
559 The type of the image input. Always `input_image`.
560
561 - `INPUT_IMAGE("input_image")`
562
563 - `Optional<String> detail`
564
565 The detail level of the image to be sent to the model. One of `high`, `low`, or `auto`. Defaults to `auto`.
566
567 - `class ResponseInputAudio:`
568
569 An audio input to the model.
570
571 - `Role role`
572
573 The role of the message input. One of `user`, `assistant`, `system`, or
574 `developer`.
575
576 - `USER("user")`
577
578 - `ASSISTANT("assistant")`
579
580 - `SYSTEM("system")`
581
582 - `DEVELOPER("developer")`
583
584 - `Optional<Type> type`
585
586 The type of the message input. Always `message`.
587
588 - `MESSAGE("message")`
589
590 - `List<String> labels`
591
592 The labels to assign to each item in the evaluation.
593
594 - `String model`
595
596 The model to use for the evaluation. Must support structured outputs.
597
598 - `String name`
599
600 The name of the grader.
601
602 - `List<String> passingLabels`
603
604 The labels that indicate a passing result. Must be a subset of labels.
605
606 - `JsonValue; type "label_model"constant`
607
608 The object type, which is always `label_model`.
609
610 - `LABEL_MODEL("label_model")`
611
612 - `class StringCheckGrader:`
613
614 A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
615
616 - `String input`
617
618 The input text. This may include template strings.
619
620 - `String name`
621
622 The name of the grader.
623
624 - `Operation operation`
625
626 The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`.
627
628 - `EQ("eq")`
629
630 - `NE("ne")`
631
632 - `LIKE("like")`
633
634 - `ILIKE("ilike")`
635
636 - `String reference`
637
638 The reference text. This may include template strings.
639
640 - `JsonValue; type "string_check"constant`
641
642 The object type, which is always `string_check`.
643
644 - `STRING_CHECK("string_check")`
645
646 - `class EvalGraderTextSimilarity:`
647
648 A TextSimilarityGrader object which grades text based on similarity metrics.
649
650 - `double passThreshold`
651
652 The threshold for the score.
653
654 - `class EvalGraderPython:`
655
656 A PythonGrader object that runs a python script on the input.
657
658 - `Optional<Double> passThreshold`
659
660 The threshold for the score.
661
662 - `class EvalGraderScoreModel:`
663
664 A ScoreModelGrader object that uses a model to assign a score to the input.
665
666 - `Optional<Double> passThreshold`
667
668 The threshold for the score.
669
670### Example
671
672```java
673package com.openai.example;
674
675import com.openai.client.OpenAIClient;
676import com.openai.client.okhttp.OpenAIOkHttpClient;
677import com.openai.core.JsonValue;
678import com.openai.models.evals.EvalCreateParams;
679import com.openai.models.evals.EvalCreateResponse;
680
681public final class Main {
682 private Main() {}
683
684 public static void main(String[] args) {
685 OpenAIClient client = OpenAIOkHttpClient.fromEnv();
686
687 EvalCreateParams params = EvalCreateParams.builder()
688 .customDataSourceConfig(EvalCreateParams.DataSourceConfig.Custom.ItemSchema.builder()
689 .putAdditionalProperty("foo", JsonValue.from("bar"))
690 .build())
691 .addTestingCriterion(EvalCreateParams.TestingCriterion.LabelModel.builder()
692 .addInput(EvalCreateParams.TestingCriterion.LabelModel.Input.SimpleInputMessage.builder()
693 .content("content")
694 .role("role")
695 .build())
696 .addLabel("string")
697 .model("model")
698 .name("name")
699 .addPassingLabel("string")
700 .build())
701 .build();
702 EvalCreateResponse eval = client.evals().create(params);
703 }
704}
705```
706
707#### Response
708
709```json
710{
711 "id": "id",
712 "created_at": 0,
713 "data_source_config": {
714 "schema": {
715 "foo": "bar"
716 },
717 "type": "custom"
718 },
719 "metadata": {
720 "foo": "string"
721 },
722 "name": "Chatbot effectiveness Evaluation",
723 "object": "eval",
724 "testing_criteria": [
725 {
726 "input": [
727 {
728 "content": "string",
729 "role": "user",
730 "type": "message"
731 }
732 ],
733 "labels": [
734 "string"
735 ],
736 "model": "model",
737 "name": "name",
738 "passing_labels": [
739 "string"
740 ],
741 "type": "label_model"
742 }
743 ]
744}
745```