File Deleted
View Diff
1# Reinforcement fine-tuning
2
3> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.
4
5Reinforcement fine-tuning (RFT) adapts an OpenAI reasoning model with a feedback signal you define. Like [supervised fine-tuning](https://developers.openai.com/api/docs/guides/supervised-fine-tuning), it tailors the model to your task. The difference is that instead of training on fixed “correct” answers, it relies on a programmable grader that scores every candidate response. The training algorithm then shifts the model’s weights, so high-scoring outputs become more likely and low-scoring ones fade.
6
7OpenAI is winding down the fine-tuning platform. The platform is no longer
8 accessible to new users. Eligible existing customers can create training jobs
9 until January 6, 2027, unless the base model shuts down earlier.
10
11
12 Fine-tuned models remain available for inference until their base model shuts
13 down. The `o4-mini-2025-04-16` base model and its fine-tuned versions shut
14 down on [October 23,
15 2026](https://developers.openai.com/api/docs/deprecations#2026-04-22-legacy-gpt-model-snapshots). See the
16 [fine-tuning availability
17 timeline](https://developers.openai.com/api/docs/deprecations#update-to-openais-self-serve-fine-tuning).
18
19
20
21
22<table>
23<tbody>
24<tr>
25<th>How it works</th>
26<th>Best for</th>
27<th>Use with</th>
28</tr>
29
30<tr>
31<td>
32Generate a response for a prompt, provide an expert grade for the result, and reinforce the model's chain-of-thought for higher-scored responses.
33
34Requires expert graders to agree on the ideal output from the model.
35
36</td>
37<td>
38- Complex domain-specific tasks that require advanced reasoning
39- Medical diagnoses based on history and diagnostic guidelines
40- Determining relevant passages from legal case law
41</td>
42<td>
43`o4-mini-2025-04-16`
44
45**Reasoning models only**.
46
47</td>
48</tr>
49</tbody>
50</table>
51
52This optimization lets you align the model with nuanced objectives like style, safety, or domain accuracy—with many [practical use cases](https://developers.openai.com/api/docs/guides/rft-use-cases) emerging. Run RFT in five steps:
53
541. Implement a [grader](https://developers.openai.com/api/docs/guides/graders) that assigns a numeric reward to each model response.
551. Upload your prompt dataset and designate a validation split.
561. Start the fine-tune job.
571. Monitor and [evaluate](https://developers.openai.com/api/docs/guides/evals) checkpoints; revise data or grader if needed.
581. Deploy the resulting model through the standard API.
59
60During training, the platform cycles through the dataset, samples several responses per prompt, scores them with the grader, and applies policy-gradient updates based on those rewards. The loop continues until we hit the end of your training data or you stop the job at a chosen checkpoint, producing a model optimized for the metric that matters to you.
61
62
63
64## When should I use reinforcement fine-tuning?
65
66
67
68It's useful to understand the strengths and weaknesses of reinforcement fine-tuning to identify opportunities and to avoid wasted effort.
69
70- **RFT works best with unambiguous tasks**. Check whether qualified human experts agree on the answers. If conscientious experts working independently (with access only to the same instructions and information as the model) do not converge on the same answers, the task may be too ambiguous and may benefit from revision or reframing.
71- **Your task must be compatible with the grading options**. Review [grading options in the API](https://developers.openai.com/api/reference/resources/graders) first and verify it's possible to grade your task with them.
72- **Your eval results must be variable enough to improve**. Run [evals](https://developers.openai.com/api/docs/guides/evals) before using RFT. If your eval scores between minimum and maximum possible scores, you'll have enough data to work with to reinforce positive answers. If the model you want to fine-tune scores at either the absolute minimum or absolute maximum score, RFT won't be useful to you.
73- **Your model must have some success at the desired task**. Reinforcement fine-tuning makes gradual changes, sampling many answers and choosing the best ones. If a model has a 0% success rate at a given task, you cannot bootstrap to higher performance levels through RFT.
74- **Your task should be guess-proof**. If the model can get a higher reward from a lucky guess, the training signal is too noisy, as the model can get the right answer with an incorrect reasoning process. Reframe your task to make guessing more difficult—for example, by expanding classes into subclasses or revising a multiple choice problem to take open-ended answers.
75
76See common use cases, specific implementations, and grader examples in the [reinforcement fine-tuning use case guide](https://developers.openai.com/api/docs/guides/rft-use-cases).
77
78
79
80
81
82
83
84## What is reinforcement learning?
85
86
87
88Reinforcement learning is a branch of machine learning in which a model learns by acting, receiving feedback, and readjusting itself to maximise future feedback. Instead of memorising one “right” answer per example, the model explores many possible answers, observes a numeric reward for each, and gradually shifts its behaviour so the high-reward answers become more likely and the low-reward ones disappear. Over repeated rounds, the model converges on a policy—a rule for choosing outputs—that best satisfies the reward signal you define.
89
90In reinforcement fine-tuning (RFT), that reward signal comes from a custom grader that you define for your task. For every prompt in your dataset, the platform samples multiple candidate answers, runs your grader to score them, and applies a policy-gradient update that nudges the model toward answers with higher scores. This cycle—sample, grade, update—continues across the dataset (and successive epochs) until the model reliably optimizes for your grader’s understanding of quality. The grader encodes whatever you care about—accuracy, style, safety, or any metric—so the resulting fine-tuned model reflects those priorities and you don't have to manage reinforcement learning infrastructure.
91
92
93
94
95
96Reinforcement fine-tuning is supported on o-series reasoning models only, and
97 currently only for [o4-mini](https://developers.openai.com/api/docs/models/o4-mini).
98
99## Example: LLM-powered security review
100
101To demonstrate reinforcement fine-tuning below, we'll fine-tune an [o4-mini](https://developers.openai.com/api/docs/models/o4-mini) model to provide expert answers about a fictional company's security posture, based on an internal company policy document. We want the model to return a JSON object that conforms to a specific schema with [Structured Outputs](https://developers.openai.com/api/docs/guides/structured-outputs).
102
103Example input question:
104
105```text
106Do you have a dedicated security team?
107```
108
109Using the internal policy document, we want the model to respond with JSON that has two keys:
110
111- `compliant`: A string `yes`, `no`, or `needs review`, indicating whether the company's policy covers the question.
112- `explanation`: A string of text that briefly explains, based on the policy document, why the question is covered in the policy or why it's not covered.
113
114Example desired output from the model:
115
116```json
117{
118 "compliant": "yes",
119 "explanation": "A dedicated security team follows strict protocols for handling incidents."
120}
121```
122
123Let's fine-tune a model with RFT to perform well at this task.
124
125## Define a grader
126
127To perform RFT, define a [grader](https://developers.openai.com/api/docs/guides/graders) to score the model's output during training, indicating the quality of its response. RFT uses the same set of graders as [evals](https://developers.openai.com/api/docs/guides/evals), which you may already be familiar with.
128
129In this example, we define [multiple graders](https://developers.openai.com/api/reference/resources/graders) to examine the properties of the JSON returned by our fine-tuned model:
130
131- The [`string_check`](https://developers.openai.com/api/reference/resources/graders) grader to ensure the proper `compliant` property has been set
132- The [`score_model`](https://developers.openai.com/api/reference/resources/graders) grader to provide a score between zero and one for the explanation text, using another evaluator model
133
134We weight the output of each property equally in the `calculate_output` expression.
135
136Below is the JSON payload data we'll use for this grader in API requests. In both graders, we use `{{ }}` template syntax to refer to the relevant properties of both the `item` (the row of test data being used for evaluation) and `sample` (the model output generated during the training run).
137
138
139
140Grader configuration
141
142 Multi-grader configuration object
143
144```json
145{
146 "type": "multi",
147 "graders": {
148 "explanation": {
149 "name": "Explanation text grader",
150 "type": "score_model",
151 "input": [
152 {
153 "role": "user",
154 "type": "message",
155 "content": "...see other tab for the full prompt..."
156 }
157 ],
158 "model": "gpt-4o-2024-08-06"
159 },
160 "compliant": {
161 "name": "compliant",
162 "type": "string_check",
163 "reference": "{{item.compliant}}",
164 "operation": "eq",
165 "input": "{{sample.output_json.compliant}}"
166 }
167 },
168 "calculate_output": "0.5 * compliant + 0.5 * explanation"
169}
170```
171
172
173
174
175
176
177Grading prompt
178
179 Grading prompt in the grader config
180
181```markdown
182# Overview
183
184Evaluate the accuracy of the model-generated answer based on the
185Copernicus Product Security Policy and an example answer. The response
186should align with the policy, cover key details, and avoid speculative
187or fabricated claims.
188
189Always respond with a single floating point number 0 through 1,
190using the grading criteria below.
191
192## Grading Criteria:
193- **1.0**: The model answer is fully aligned with the policy and factually correct.
194- **0.75**: The model answer is mostly correct but has minor omissions or slight rewording that does not change meaning.
195- **0.5**: The model answer is partially correct but lacks key details or contains speculative statements.
196- **0.25**: The model answer is significantly inaccurate or missing important information.
197- **0.0**: The model answer is completely incorrect, hallucinates policy details, or is irrelevant.
198
199## Copernicus Product Security Policy
200
201### Introduction
202Protecting customer data is a top priority for Copernicus. Our platform is designed with industry-standard security and compliance measures to ensure data integrity, privacy, and reliability.
203
204### Data Classification
205Copernicus safeguards customer data, which includes prompts, responses, file uploads, user preferences, and authentication configurations. Metadata, such as user IDs, organization IDs, IP addresses, and device details, is collected for security purposes and stored securely for monitoring and analytics.
206
207### Data Management
208Copernicus utilizes cloud-based storage with strong encryption (AES-256) and strict access controls. Data is logically segregated to ensure confidentiality and access is restricted to authorized personnel only. Conversations and other customer data are never used for model training.
209
210### Data Retention
211Customer data is retained only for providing core functionalities like conversation history and team collaboration. Customers can configure data retention periods, and deleted content is removed from our system within 30 days.
212
213### User Authentication & Access Control
214Users authenticate via Single Sign-On (SSO) using an Identity Provider (IdP). Roles include Account Owner, Admin, and Standard Member, each with defined permissions. User provisioning can be automated through SCIM integration.
215
216### Compliance & Security Monitoring
217- **Compliance API**: Logs interactions, enabling data export and deletion.
218- **Audit Logging**: Ensures transparency for security audits.
219- **HIPAA Support**: Business Associate Agreements (BAAs) available for customers needing healthcare compliance.
220- **Security Monitoring**: 24/7 monitoring for threats and suspicious activity.
221- **Incident Response**: A dedicated security team follows strict protocols for handling incidents.
222
223### Infrastructure Security
224- **Access Controls**: Role-based authentication with multi-factor security.
225- **Source Code Security**: Controlled code access with mandatory reviews before deployment.
226- **Network Security**: Web application firewalls and strict ingress/egress controls to prevent unauthorized access.
227- **Physical Security**: Data centers have controlled access, surveillance, and environmental risk management.
228
229### Bug Bounty Program
230Security researchers are encouraged to report vulnerabilities through our Bug Bounty Program for responsible disclosure and rewards.
231
232### Compliance & Certifications
233Copernicus maintains compliance with industry standards, including SOC 2 and GDPR. Customers can access security reports and documentation via our Security Portal.
234
235### Conclusion
236Copernicus prioritizes security, privacy, and compliance. For inquiries, contact your account representative or visit our Security Portal.
237
238## Examples
239
240### Example 1: GDPR Compliance
241**Reference Answer**: 'Copernicus maintains compliance with industry standards, including SOC 2 and GDPR. Customers can access security reports and documentation via our Security Portal.'
242
243**Model Answer 1**: 'Yes, Copernicus is GDPR compliant and provides compliance documentation via the Security Portal.'
244**Score: 1.0** (fully correct)
245
246**Model Answer 2**: 'Yes, Copernicus follows GDPR standards.'
247**Score: 0.75** (mostly correct but lacks detail about compliance reports)
248
249**Model Answer 3**: 'Copernicus may comply with GDPR but does not provide documentation.'
250**Score: 0.5** (partially correct, speculative about compliance reports)
251
252**Model Answer 4**: 'Copernicus does not follow GDPR standards.'
253**Score: 0.0** (factually incorrect)
254
255### Example 2: Encryption in Transit
256**Reference Answer**: 'The Copernicus Product Security Policy states that data is stored with strong encryption (AES-256) and that network security measures include web application firewalls and strict ingress/egress controls. However, the policy does not explicitly mention encryption of data in transit (e.g., TLS encryption). A review is needed to confirm whether data transmission is encrypted.'
257
258**Model Answer 1**: 'Data is encrypted at rest using AES-256, but a review is needed to confirm encryption in transit.'
259**Score: 1.0** (fully correct)
260
261**Model Answer 2**: 'Yes, Copernicus encrypts data in transit and at rest.'
262**Score: 0.5** (partially correct, assumes transit encryption without confirmation)
263
264**Model Answer 3**: 'All data is protected with encryption.'
265**Score: 0.25** (vague and lacks clarity on encryption specifics)
266
267**Model Answer 4**: 'Data is not encrypted in transit.'
268**Score: 0.0** (factually incorrect)
269
270Reference Answer: {{item.explanation}}
271Model Answer: {{sample.output_json.explanation}}
272```
273
274
275
276## Prepare your dataset
277
278To create an RFT fine-tune, you'll need both a training and test dataset. Both the training and test datasets will share the same [JSONL format](https://jsonlines.org/). Each line in the JSONL data file will contain a `messages` array, along with any additional fields required to grade the output from the model. The full specification for RFT dataset [can be found here](https://developers.openai.com/api/reference/resources/fine_tuning).
279
280In our case, in addition to the `messages` array, each line in our JSONL file also needs `compliant` and `explanation` properties, which we can use as reference values to test the fine-tuned model's Structured Output.
281
282A single line in our training and test datasets looks like this as indented JSON:
283
284```json
285{
286 "messages": [
287 {
288 "role": "user",
289 "content": "Do you have a dedicated security team?"
290 }
291 ],
292 "compliant": "yes",
293 "explanation": "A dedicated security team follows strict protocols for handling incidents."
294}
295```
296
297Below, find some JSONL data you can use for both training and testing when you create your fine-tune job. Note that these datasets are for illustration purposes only—in your real test data, strive for diverse and representative inputs for your application.
298
299**Training set**
300
301```text
302{"messages":[{"role":"user","content":"Do you have a dedicated security team?"}],"compliant":"yes","explanation":"A dedicated security team follows strict protocols for handling incidents."}
303{"messages":[{"role":"user","content":"Have you undergone third-party security audits or penetration testing in the last 12 months?"}],"compliant":"needs review","explanation":"The policy does not explicitly mention undergoing third-party security audits or penetration testing. It only mentions SOC 2 and GDPR compliance."}
304{"messages":[{"role":"user","content":"Is your software SOC 2, ISO 27001, or similarly certified?"}],"compliant":"yes","explanation":"The policy explicitly mentions SOC 2 compliance."}
305```
306
307**Test set**
308
309```text
310{"messages":[{"role":"user","content":"Will our data be encrypted at rest?"}],"compliant":"yes","explanation":"Copernicus utilizes cloud-based storage with strong encryption (AES-256) and strict access controls."}
311{"messages":[{"role":"user","content":"Will data transmitted to/from your services be encrypted in transit?"}],"compliant":"needs review","explanation":"The policy does not explicitly mention encryption of data in transit. It focuses on encryption in cloud storage."}
312{"messages":[{"role":"user","content":"Do you enforce multi-factor authentication (MFA) internally?"}],"compliant":"yes","explanation":"The policy explicitly mentions role-based authentication with multi-factor security."}
313```
314
315
316
317### How much training data is needed?
318
319
320
321Start small—between several dozen and a few hundred examples—to determine the usefulness of RFT before investing in a large dataset. For product safety reasons, the training set must first pass through an automated screening process. Large datasets take longer to process. This screening process begins when you start a fine-tuning job with a file, not upon initial file upload. Once a file has successfully completed screening, you can use it repeatedly without delay.
322
323Dozens of examples can be meaningful as long as they're high quality. After screening, more data is better, as long as it remains high quality. With larger datasets, you can use a higher batch size, which tends to improve training stability.
324
325Your training file can contain a maximum of 50,000 examples. Test datasets can contain a maximum of 1,000 examples. Test datasets also go through automated screening.
326
327
328
329
330
331### Upload your files
332
333The process for uploading RFT training and test data files is the same as [supervised fine-tuning](https://developers.openai.com/api/docs/guides/supervised-fine-tuning). Upload your training data to OpenAI either through the [API](https://developers.openai.com/api/reference/resources/files/methods/create) or [using our UI](https://platform.openai.com/storage). Files must be uploaded with a purpose of `fine-tune` in order to be used with fine-tuning.
334
335**You need file IDs for both your test and training data files** to create a fine-tune job.
336
337## Create a fine-tune job
338
339Create a fine-tune job using either the [API](https://developers.openai.com/api/reference/resources/fine_tuning) or [fine-tuning dashboard](https://platform.openai.com/finetune). To do this, you need:
340
341- File IDs for both your training and test datasets
342- The grader configuration we created earlier
343- The model ID you want to use as a base for fine-tuning (we'll use `o4-mini-2025-04-16`)
344- If you're fine-tuning a model that will return JSON data as a structured output, you need the JSON schema for the returned object as well (see below)
345- Optionally, any hyperparameters you want to configure for the fine-tune
346- To qualify for [data sharing inference pricing](https://developers.openai.com/api/docs/pricing#fine-tuning), you need to first [share evaluation and fine-tuning data](https://help.openai.com/en/articles/10306912-sharing-feedback-evaluation-and-fine-tuning-data-and-api-inputs-and-outputs-with-openai#h_c93188c569) with OpenAI before creating the job
347
348### Structured Outputs JSON schema
349
350If you're fine-tuning a model to return [Structured Outputs](https://developers.openai.com/api/docs/guides/structured-outputs), provide the JSON schema being used to format the output. See a valid JSON schema for our security interview use case:
351
352```json
353{
354 "type": "json_schema",
355 "json_schema": {
356 "name": "security_assistant",
357 "strict": true,
358 "schema": {
359 "type": "object",
360 "properties": {
361 "compliant": { "type": "string" },
362 "explanation": { "type": "string" }
363 },
364 "required": ["compliant", "explanation"],
365 "additionalProperties": false
366 }
367 }
368}
369```
370
371
372
373#### Generating a JSON schema from a Pydantic model
374
375
376
377To simplify JSON schema generation, start from a [Pydantic BaseModel](https://docs.pydantic.dev/latest/api/base_model/) class:
378
3791. Define your class
3801. Use `to_strict_json_schema` from the OpenAI library to generate a valid schema
3811. Wrap the schema in a dictionary with `type` and `name` keys, and set `strict` to true
3821. Take the resulting object and supply it as the `response_format` in your RFT job
383
384```python
385from openai.lib._pydantic import to_strict_json_schema
386from pydantic import BaseModel
387
388
389class MyCustomClass(BaseModel):
390 name: str
391 age: int
392
393
394# Note: Do not use MyCustomClass.model_json_schema() in place of
395# to_strict_json_schema as it is not equivalent
396schema = to_strict_json_schema(MyCustomClass)
397
398response_format = dict(
399 type="json_schema",
400 json_schema=dict(name=MyCustomClass.__name__, strict=True, schema=schema),
401)
402```
403
404
405
406
407
408
409### Create a job with the API
410
411Configuring a job with the API has a lot of moving parts, so many users prefer to configure them in the [fine-tuning dashboard UI](https://platform.openai.com/finetune). However, here's a complete API request to kick off a fine-tune job with all the configuration we've set up in this guide so far:
412
413```bash
414curl https://api.openai.com/v1/fine_tuning/jobs \
415 -H "Content-Type: application/json" \
416 -H "Authorization: Bearer $OPENAI_API_KEY" \
417 -d '{
418 "training_file": "file-2STiufDaGXWCnT6XUBUEHW",
419 "validation_file": "file-4TcgH85ej7dFCjZ1kThCYb",
420 "model": "o4-mini-2025-04-16",
421 "method": {
422 "type": "reinforcement",
423 "reinforcement": {
424 "grader": {
425 "type": "multi",
426 "graders": {
427 "explanation": {
428 "name": "Explanation text grader",
429 "type": "score_model",
430 "input": [
431 {
432 "role": "user",
433 "type": "message",
434 "content": "# Overview\n\nEvaluate the accuracy of the model-generated answer based on the \nCopernicus Product Security Policy and an example answer. The response \nshould align with the policy, cover key details, and avoid speculative \nor fabricated claims.\n\nAlways respond with a single floating point number 0 through 1,\nusing the grading criteria below.\n\n## Grading Criteria:\n- **1.0**: The model answer is fully aligned with the policy and factually correct.\n- **0.75**: The model answer is mostly correct but has minor omissions or slight rewording that does not change meaning.\n- **0.5**: The model answer is partially correct but lacks key details or contains speculative statements.\n- **0.25**: The model answer is significantly inaccurate or missing important information.\n- **0.0**: The model answer is completely incorrect, hallucinates policy details, or is irrelevant.\n\n## Copernicus Product Security Policy\n\n### Introduction\nProtecting customer data is a top priority for Copernicus. Our platform is designed with industry-standard security and compliance measures to ensure data integrity, privacy, and reliability.\n\n### Data Classification\nCopernicus safeguards customer data, which includes prompts, responses, file uploads, user preferences, and authentication configurations. Metadata, such as user IDs, organization IDs, IP addresses, and device details, is collected for security purposes and stored securely for monitoring and analytics.\n\n### Data Management\nCopernicus utilizes cloud-based storage with strong encryption (AES-256) and strict access controls. Data is logically segregated to ensure confidentiality and access is restricted to authorized personnel only. Conversations and other customer data are never used for model training.\n\n### Data Retention\nCustomer data is retained only for providing core functionalities like conversation history and team collaboration. Customers can configure data retention periods, and deleted content is removed from our system within 30 days.\n\n### User Authentication & Access Control\nUsers authenticate via Single Sign-On (SSO) using an Identity Provider (IdP). Roles include Account Owner, Admin, and Standard Member, each with defined permissions. User provisioning can be automated through SCIM integration.\n\n### Compliance & Security Monitoring\n- **Compliance API**: Logs interactions, enabling data export and deletion.\n- **Audit Logging**: Ensures transparency for security audits.\n- **HIPAA Support**: Business Associate Agreements (BAAs) available for customers needing healthcare compliance.\n- **Security Monitoring**: 24/7 monitoring for threats and suspicious activity.\n- **Incident Response**: A dedicated security team follows strict protocols for handling incidents.\n\n### Infrastructure Security\n- **Access Controls**: Role-based authentication with multi-factor security.\n- **Source Code Security**: Controlled code access with mandatory reviews before deployment.\n- **Network Security**: Web application firewalls and strict ingress/egress controls to prevent unauthorized access.\n- **Physical Security**: Data centers have controlled access, surveillance, and environmental risk management.\n\n### Bug Bounty Program\nSecurity researchers are encouraged to report vulnerabilities through our Bug Bounty Program for responsible disclosure and rewards.\n\n### Compliance & Certifications\nCopernicus maintains compliance with industry standards, including SOC 2 and GDPR. Customers can access security reports and documentation via our Security Portal.\n\n### Conclusion\nCopernicus prioritizes security, privacy, and compliance. For inquiries, contact your account representative or visit our Security Portal.\n\n## Examples\n\n### Example 1: GDPR Compliance\n**Reference Answer**: Copernicus maintains compliance with industry standards, including SOC 2 and GDPR. Customers can access security reports and documentation via our Security Portal.\n\n**Model Answer 1**: Yes, Copernicus is GDPR compliant and provides compliance documentation via the Security Portal. \n**Score: 1.0** (fully correct)\n\n**Model Answer 2**: Yes, Copernicus follows GDPR standards.\n**Score: 0.75** (mostly correct but lacks detail about compliance reports)\n\n**Model Answer 3**: Copernicus may comply with GDPR but does not provide documentation.\n**Score: 0.5** (partially correct, speculative about compliance reports)\n\n**Model Answer 4**: Copernicus does not follow GDPR standards.\n**Score: 0.0** (factually incorrect)\n\n### Example 2: Encryption in Transit\n**Reference Answer**: The Copernicus Product Security Policy states that data is stored with strong encryption (AES-256) and that network security measures include web application firewalls and strict ingress/egress controls. However, the policy does not explicitly mention encryption of data in transit (e.g., TLS encryption). A review is needed to confirm whether data transmission is encrypted.\n\n**Model Answer 1**: Data is encrypted at rest using AES-256, but a review is needed to confirm encryption in transit.\n**Score: 1.0** (fully correct)\n\n**Model Answer 2**: Yes, Copernicus encrypts data in transit and at rest.\n**Score: 0.5** (partially correct, assumes transit encryption without confirmation)\n\n**Model Answer 3**: All data is protected with encryption.\n**Score: 0.25** (vague and lacks clarity on encryption specifics)\n\n**Model Answer 4**: Data is not encrypted in transit.\n**Score: 0.0** (factually incorrect)\n\nReference Answer: {{item.explanation}}\nModel Answer: {{sample.output_json.explanation}}\n"
435 }
436 ],
437 "model": "gpt-4o-2024-08-06"
438 },
439 "compliant": {
440 "name": "compliant",
441 "type": "string_check",
442 "reference": "{{item.compliant}}",
443 "operation": "eq",
444 "input": "{{sample.output_json.compliant}}"
445 }
446 },
447 "calculate_output": "0.5 * compliant + 0.5 * explanation"
448 },
449 "response_format": {
450 "type": "json_schema",
451 "json_schema": {
452 "name": "security_assistant",
453 "strict": true,
454 "schema": {
455 "type": "object",
456 "properties": {
457 "compliant": {
458 "type": "string"
459 },
460 "explanation": {
461 "type": "string"
462 }
463 },
464 "required": [
465 "compliant",
466 "explanation"
467 ],
468 "additionalProperties": false
469 }
470 }
471 },
472 "hyperparameters": {
473 "reasoning_effort": "medium"
474 }
475 }
476 }
477}'
478```
479
480
481This request returns a [fine-tuning job object](https://developers.openai.com/api/reference/resources/fine_tuning), which includes a job `id`. Use this ID to monitor the progress of your job and retrieve the fine-tuned model when the job is complete.
482
483To qualify for [data sharing inference pricing](https://developers.openai.com/api/docs/pricing#fine-tuning), make sure to [share evaluation and fine-tuning data](https://help.openai.com/en/articles/10306912-sharing-feedback-evaluation-and-fine-tuning-data-and-api-inputs-and-outputs-with-openai#h_c93188c569) with OpenAI before creating the job. You can verify the job was marked as shared by confirming `shared_with_openai` is set to `true`.
484
485### Monitoring your fine-tune job
486
487Fine-tuning jobs take some time to complete, and RFT jobs tend to take longer than SFT or DPO jobs. To monitor the progress of your fine-tune job, use the [fine-tuning dashboard](https://platform.openai.com/finetune) or the [API](https://developers.openai.com/api/reference/resources/fine_tuning).
488
489#### Reward metrics
490
491For reinforcement fine-tuning jobs, the primary metrics are the per-step **reward** metrics. These metrics indicate how well your model is performing on the training data. They're calculated by the graders you defined in your job configuration. These are two separate top-level reward metrics:
492
493- `train_reward_mean`: The average reward across the samples taken from all data points in the current step. Because the specific data points in a batch change with each step, `train_reward_mean` values across different steps are not directly comparable and the specific values can fluctuate drastically from step to step.
494- `valid_reward_mean`: The average reward across the samples taken from all data points in the validation set, which is a more stable metric.
495
496
497
498Find a full description of all training metrics in the [training metrics](#training-metrics) section.
499
500#### Pausing and resuming jobs
501
502To evaluate the current state of the model when your job is only partially finished, **pause** the job to stop the training process and produce a checkpoint at the current step. You can use this checkpoint to evaluate the model on a held-out test set. If the results look good, **resume** the job to continue training from that checkpoint. Learn more in [pausing and resuming jobs](#pausing-and-resuming-jobs).
503
504#### Evals integration
505
506Reinforcement fine-tuning jobs are integrated with our [evals product](https://developers.openai.com/api/docs/guides/evals). When you make a reinforcement fine-tuning job, a new eval is automatically created and associated with the job. As validation steps are performed, we combine the input prompts, model samples, and grader outputs to make a new [eval run](https://developers.openai.com/api/docs/guides/evals#creating-an-eval-run) for that step.
507
508Learn more about the evals integration in the [appendix](#evals-integration-details) section below.
509
510## Evaluate the results
511
512By the time your fine-tuning job finishes, you should have a decent idea of how well the model is performing based on the mean reward value on the validation set. However, it's possible that the model has either _overfit_ to the training data or has learned to [reward hack](https://en.wikipedia.org/wiki/Reward_hacking) your grader, which allows it to produce high scores without actually being correct. Before deploying your model, inspect its behavior on a representative set of prompts to ensure it behaves how you expect.
513
514Understanding the model's behavior can be done quickly by inspecting the evals associated with the fine-tuning job. Specifically, pay close attention to the run made for the final training step to see the end model's behavior. You can also use the evals product to compare the final run to earlier runs and see how the model's behavior has changed over the course of training.
515
516### Try using your fine-tuned model
517
518Evaluate your newly optimized model by using it! When the fine-tuned model finishes training, use its ID in either the [Responses](https://developers.openai.com/api/reference/resources/responses) or [Chat Completions](https://developers.openai.com/api/reference/resources/chat) API, just as you would an OpenAI base model.
519
520
521
522Use your model in the Playground
523
524
525
5261. Navigate to your fine-tuning job in [the dashboard](https://platform.openai.com/finetune).
5271. In the right pane, navigate to **Output model** and copy the model ID. It should start with `ft:…`
5281. Open the [Playground](https://platform.openai.com/playground).
5291. In the **Model** dropdown menu, paste the model ID. Here, you should also see other fine-tuned models you've created.
5301. Run some prompts and see how your fine-tuned performs!
531
532
533
534
535
536
537
538Use your model with an API call
539
540
541
542```bash
543curl https://api.openai.com/v1/responses \
544 -H "Content-Type: application/json" \
545 -H "Authorization: Bearer $OPENAI_API_KEY" \
546 -d '{
547 "model": "ft:gpt-4.1-nano-2025-04-14:openai::BTz2REMH",
548 "input": "What is 4+4?"
549 }'
550```
551
552
553
554### Use checkpoints if needed
555
556Checkpoints are models you can use that are created before the final step of the training process. For RFT, OpenAI creates a full model checkpoint at each validation step and keeps the three with the highest `valid_reward_mean` scores. Checkpoints are useful for evaluating the model at different points in the training process and comparing performance at different steps.
557
558
559
560Find checkpoints in the dashboard
561
562
563
5641. Navigate to the [fine-tuning dashboard](https://platform.openai.com/finetune).
5651. In the left panel, select the job you want to investigate. Wait until it succeeds.
5661. In the right panel, scroll to the list of checkpoints.
5671. Hover over any checkpoint to see a link to launch in the Playground.
5681. Test the checkpoint model's behavior by prompting it in the Playground.
569
570
571
572
573
574
575
576Query the API for checkpoints
577
578
579
5801. Wait until a job succeeds, which you can verify by [querying the status of a job](https://developers.openai.com/api/reference/resources/fine_tuning).
5811. [Query the checkpoints endpoint](https://developers.openai.com/api/reference/resources/fine_tuning/subresources/jobs/subresources/checkpoints/methods/list) with your fine-tuning job ID to access a list of model checkpoints for the fine-tuning job.
5821. Find the `fine_tuned_model_checkpoint` field for the name of the model checkpoint.
5831. Use this model just like you would the final fine-tuned model.
584
585The checkpoint object contains `metrics` data to help you determine the usefulness of this model. As an example, the response looks like this:
586
587```json
588{
589 "object": "fine_tuning.job.checkpoint",
590 "id": "ftckpt_zc4Q7MP6XxulcVzj4MZdwsAB",
591 "created_at": 1519129973,
592 "fine_tuned_model_checkpoint": "ft:gpt-3.5-turbo-0125:my-org:custom-suffix:96olL566:ckpt-step-2000",
593 "metrics": {
594 "full_valid_loss": 0.134,
595 "full_valid_mean_token_accuracy": 0.874
596 },
597 "fine_tuning_job_id": "ftjob-abc123",
598 "step_number": 2000
599}
600```
601
602Each checkpoint specifies:
603
604- `step_number`: The step at which the checkpoint was created (where each epoch is number of steps in the training set divided by the batch size)
605- `metrics`: An object containing the metrics for your fine-tuning job at the step when the checkpoint was created
606
607
608
609## Safety checks
610
611Before launching in production, review and follow the following safety information.
612
613
614
615### How we assess for safety
616
617
618
619Once a fine-tuning job is completed, we assess the resulting model’s behavior across 13 distinct safety categories. Each category represents a critical area where AI outputs could potentially cause harm if not properly controlled.
620
621| Name | Description |
622| :--------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
623| advice | Advice or guidance that violates our policies. |
624| harassment/threatening | Harassment content that also includes violence or serious harm towards any target. |
625| hate | Content that expresses, incites, or promotes hate based on race, gender, ethnicity, religion, nationality, sexual orientation, disability status, or caste. Hateful content aimed at non-protected groups (e.g., chess players) is harassment. |
626| hate/threatening | Hateful content that also includes violence or serious harm towards the targeted group based on race, gender, ethnicity, religion, nationality, sexual orientation, disability status, or caste. |
627| highly-sensitive | Highly sensitive data that violates our policies. |
628| illicit | Content that gives advice or instruction on how to commit illicit acts. A phrase like "how to shoplift" would fit this category. |
629| propaganda | Praise or assistance for ideology that violates our policies. |
630| self-harm/instructions | Content that encourages performing acts of self-harm, such as suicide, cutting, and eating disorders, or that gives instructions or advice on how to commit such acts. |
631| self-harm/intent | Content where the speaker expresses that they are engaging or intend to engage in acts of self-harm, such as suicide, cutting, and eating disorders. |
632| sensitive | Sensitive data that violates our policies. |
633| sexual/minors | Sexual content that includes an individual who is under 18 years old. |
634| sexual | Content meant to arouse sexual excitement, such as the description of sexual activity, or that promotes sexual services (excluding sex education and wellness). |
635| violence | Content that depicts death, violence, or physical injury. |
636
637Each category has a predefined pass threshold; if too many evaluated examples in a given category fail, OpenAI blocks the fine-tuned model from deployment. If your fine-tuned model does not pass the safety checks, OpenAI sends a message in the fine-tuning job explaining which categories don't meet the required thresholds. You can view the results in the moderation checks section of the fine-tuning job.
638
639
640
641
642
643
644
645### How to pass safety checks
646
647
648
649In addition to reviewing any failed safety checks in the fine-tuning job object, you can retrieve details about which categories failed by querying the [fine-tuning API events endpoint](https://developers.openai.com/api/reference/resources/fine_tuning/subresources/jobs/methods/list). Look for events of type `moderation_checks` for details about category results and enforcement. This information can help you narrow down which categories to target for retraining and improvement. The [model spec](https://cdn.openai.com/spec/model-spec-2024-05-08.html#overview) has rules and examples that can help identify areas for additional training data.
650
651While these evaluations cover a broad range of safety categories, conduct your own evaluations of the fine-tuned model to ensure it's appropriate for your use case.
652
653
654
655
656
657## Next steps
658
659Now that you know the basics of reinforcement fine-tuning, explore other fine-tuning methods.
660
661[Supervised fine-tuning
662
663
664
665 Fine-tune a model by providing correct outputs for sample inputs.](https://developers.openai.com/api/docs/guides/supervised-fine-tuning)
666
667[Vision fine-tuning
668
669
670
671 Learn to fine-tune for computer vision with image inputs.](https://developers.openai.com/api/docs/guides/vision-fine-tuning)
672
673[Direct preference optimization
674
675
676
677 Fine-tune a model using direct preference optimization (DPO).](https://developers.openai.com/api/docs/guides/direct-preference-optimization)
678
679## Appendix
680
681### Training metrics
682
683Reinforcement fine-tuning jobs publish per-step training metrics as [fine-tuning events](https://developers.openai.com/api/reference/resources/fine_tuning). Pull these metrics through the [API](https://developers.openai.com/api/reference/resources/fine_tuning/subresources/jobs/methods/list) or view them as graphs and charts in the [fine-tuning dashboard](https://platform.openai.com/finetune).
684
685Learn more about training metrics below.
686
687
688
689#### Full example training metrics
690
691
692
693Below is an example metric event from a real reinforcement fine-tuning job. The various fields in this payload will be discussed in the following sections.
694
695```json
696{
697 "object": "fine_tuning.job.event",
698 "id": "ftevent-Iq5LuNLDsac1C3vzshRBuBIy",
699 "created_at": 1746679539,
700 "level": "info",
701 "message": "Step 10/20 , train mean reward=0.42, full validation mean reward=0.68, full validation mean parse error=0.00",
702 "data": {
703 "step": 10,
704 "usage": {
705 "graders": [
706 {
707 "name": "basic_model_grader",
708 "type": "score_model",
709 "model": "gpt-4o-2024-08-06",
710 "train_prompt_tokens_mean": 241.0,
711 "valid_prompt_tokens_mean": 241.0,
712 "train_prompt_tokens_count": 120741.0,
713 "valid_prompt_tokens_count": 4820.0,
714 "train_completion_tokens_mean": 138.52694610778443,
715 "valid_completion_tokens_mean": 140.5,
716 "train_completion_tokens_count": 69402.0,
717 "valid_completion_tokens_count": 2810.0
718 }
719 ],
720 "samples": {
721 "train_reasoning_tokens_mean": 3330.017964071856,
722 "valid_reasoning_tokens_mean": 1948.9,
723 "train_reasoning_tokens_count": 1668339.0,
724 "valid_reasoning_tokens_count": 38978.0
725 }
726 },
727 "errors": {
728 "graders": [
729 {
730 "name": "basic_model_grader",
731 "type": "score_model",
732 "train_other_error_mean": 0.0,
733 "valid_other_error_mean": 0.0,
734 "train_other_error_count": 0.0,
735 "valid_other_error_count": 0.0,
736 "train_sample_parse_error_mean": 0.0,
737 "valid_sample_parse_error_mean": 0.0,
738 "train_sample_parse_error_count": 0.0,
739 "valid_sample_parse_error_count": 0.0,
740 "train_invalid_variable_error_mean": 0.0,
741 "valid_invalid_variable_error_mean": 0.0,
742 "train_invalid_variable_error_count": 0.0,
743 "valid_invalid_variable_error_count": 0.0
744 }
745 ]
746 },
747 "scores": {
748 "graders": [
749 {
750 "name": "basic_model_grader",
751 "type": "score_model",
752 "train_reward_mean": 0.4471057884231537,
753 "valid_reward_mean": 0.675
754 }
755 ],
756 "train_reward_mean": 0.4215686274509804,
757 "valid_reward_mean": 0.675
758 },
759 "timing": {
760 "step": {
761 "eval": 101.69386267662048,
762 "sampling": 226.82190561294556,
763 "training": 402.43121099472046,
764 "full_iteration": 731.5038568973541
765 },
766 "graders": [
767 {
768 "name": "basic_model_grader",
769 "type": "score_model",
770 "train_execution_latency_mean": 2.6894934929297594,
771 "valid_execution_latency_mean": 4.141402995586395
772 }
773 ]
774 },
775 "total_steps": 20,
776 "train_mean_reward": 0.4215686274509804,
777 "reasoning_tokens_mean": 3330.017964071856,
778 "completion_tokens_mean": 3376.0019607843137,
779 "full_valid_mean_reward": 0.675,
780 "mean_unresponsive_rewards": 0.0,
781 "model_graders_token_usage": {
782 "gpt-4o-2024-08-06": {
783 "eval_cached_tokens": 0,
784 "eval_prompt_tokens": 4820,
785 "train_cached_tokens": 0,
786 "train_prompt_tokens": 120741,
787 "eval_completion_tokens": 2810,
788 "train_completion_tokens": 69402
789 }
790 },
791 "full_valid_mean_parse_error": 0.0,
792 "valid_reasoning_tokens_mean": 1948.9
793 },
794 "type": "metrics"
795 },
796```
797
798
799
800
801
802
803
804#### Score metrics
805
806
807
808The top-level metrics to watch are `train_reward_mean` and `valid_reward_mean`, which indicate the average reward assigned by your graders across all samples in the training and validation datasets, respectively.
809
810Additionally, if you use a [multi-grader](https://developers.openai.com/api/reference/resources/graders) configuration, per-grader train and validation reward metrics will be published as well. These metrics are included under the `event.data.scores` object in the fine-tuning events object, with one entry per grader. The per-grader metrics are useful for understanding how the model is performing on each individual grader, and can help you identify if the model is overfitting to one grader or another.
811
812From the fine-tuning dashboard, the individual grader metrics will be displayed in their own graph below the overall `train_reward_mean` and `valid_reward_mean` metrics.
813
814
815
816
817
818
819
820
821
822#### Usage metrics
823
824
825
826An important characteristic of a reasoning model is the number of reasoning tokens it uses before responding to a prompt. Often, during training, the model will drastically change the average number of reasoning tokens it uses to respond to a prompt. This is a sign that the model is changing its behavior in response to the reward signal. The model may learn to use fewer reasoning tokens to achieve the same reward, or it may learn to use more reasoning tokens to achieve a higher reward.
827
828You can monitor the `train_reasoning_tokens_mean` and `valid_reasoning_tokens_mean` metrics to see how the model is changing its behavior over time. These metrics are the average number of reasoning tokens used by the model to respond to a prompt in the training and validation datasets, respectively. You can also view the mean reasoning token count in the fine-tuning dashboard under the "Reasoning Tokens" chart.
829
830
831
832If you are using [model graders](https://developers.openai.com/api/docs/guides/graders#model-graders), you will likely want to monitor the token usage of these graders. Per-grader token usage statistics are available under the `event.data.usage.graders` object, and are broken down into:
833
834- `train_prompt_tokens_mean`
835- `train_prompt_tokens_count`
836- `train_completion_tokens_mean`
837- `train_completion_tokens_count`.
838
839The `_mean` metrics represent the average number of tokens used by the grader to process all prompts in the current step, while the `_count` metrics represent the total number of tokens used by the grader across all samples in the current step. The per-step token usage is also displayed on the fine-tuning dashboard under the "Grading Token Usage" chart.
840
841
842
843
844
845
846
847
848
849#### Timing metrics
850
851
852
853We include various metrics that help you understand how long each step of the training process is taking and how different parts of the training process are contributing to the per-step timing.
854
855These metrics are available under the `event.data.timing` object, and are broken down into `step` and `graders` fields.
856
857The `step` field contains the following metrics:
858
859- `sampling`: The time taken to sample the model outputs (rollouts) for the current step.
860- `training`: The time taken to train the model (backpropagation) for the current step.
861- `eval`: The time taken to evaluate the model on the full validation set.
862- `full_iteration`: The total time taken for the current step, including the above 3 metrics plus any additional overhead.
863
864The step timing metrics are also displayed on the fine-tuning dashboard under the "Per Step Duration" chart.
865
866
867
868The `graders` field contains timing information that details the time taken to execute each grader for the current step. Each grader will have its own timing under the `train_execution_latency_mean` and `valid_execution_latency_mean` metrics, which represent the average time taken to execute the grader on the training and validation datasets, respectively.
869
870Graders are executed in parallel with a concurrency limit, so it is not always clear how individual grader latency adds up to the total time taken for grading. However, it is generally true that graders which take longer to execute individually will cause a job to execute more slowly. This means that slower model graders will cause the job to take longer to complete, and more expensive python code will do the same. The fastest graders generally are `string_check` and `text_similarity` as those are executed local to the training loop.
871
872
873
874
875
876### Evals integration details
877
878Reinforcement fine-tuning jobs are directly integrated with our [evals product](https://developers.openai.com/api/docs/guides/evals). When you make a reinforcement fine-tuning job, a new eval is automatically created and associated with the job.
879
880As validation steps are performed, the input prompts, model samples, grader outputs, and more metadata will be combined to make a new [eval run](https://developers.openai.com/api/docs/guides/evals#creating-an-eval-run) for that step. At the end of the job, you will have one run for each validation step. This allows you to compare the performance of the model at different steps, and to see how the model's behavior has changed over the course of training.
881
882You can find the eval associated with your fine-tuning job by viewing your job on the fine-tuning dashboard, or by finding the `eval_id` field on the [fine-tuning job object](https://developers.openai.com/api/reference/resources/fine_tuning).
883
884The evals product is useful for inspecting the outputs of the model on specific data points, to get an understanding for how the model is behaving in different scenarios. It can help you figure out which slice of your dataset the model is performing poorly on which can help you identify areas for improvement in your training data.
885
886The evals product can also help you find areas of improvement for your graders by finding areas where the grader is either overly lenient or overly harsh on the model outputs.
887
888### Pausing and resuming jobs
889
890You can pause a fine-tuning job at any time by using the [fine-tuning jobs API](https://developers.openai.com/api/reference/resources/fine_tuning). Calling the pause API will tell the training process to create a new model snapshot, stop training, and put the job into a "Paused" state. The model snapshot will go through a normal safety screening process after which it will be available for you to use throughout the OpenAI platform as a normal fine-tuned model.
891
892If you wish to continue the training process for a paused job, you can do so by using the [fine-tuning jobs API](https://developers.openai.com/api/reference/resources/fine_tuning). This will resume the training process from the last checkpoint created when the job was paused and will continue training until the job is either completed or paused again.
893
894### Grading with Tools
895
896If you are training your model to [perform tool calls](https://developers.openai.com/api/docs/guides/function-calling), you will need to:
897
8981. Provide the set of tools available for your model to call on each data point in the RFT training dataset. More info here in the [dataset API reference](https://developers.openai.com/api/reference/resources/fine_tuning).
8992. Configure your grader to assign rewards based on the contents of the tool calls made by the model. Information on grading tools calls can be found [here in the grading docs](https://developers.openai.com/api/docs/guides/graders/#sample-namespace)
900
901### Billing details
902
903Reinforcement fine-tuning jobs are billed based on the amount of time spent training, as well as the number of tokens used by the model during training. We only bill for time spent in the core training loop, not for time spent preparing the training data, validating datasets, waiting in queues, running safety evals, or other overhead.
904
905Details on exactly how we bill for reinforcement fine-tuning jobs can be found in this [help center article](https://help.openai.com/en/articles/11323177-billing-guide-for-the-reinforcement-fine-tuning-api).
906
907### Training errors
908
909Reinforcement fine-tuning is a complex process with many moving parts, and there are many places where things can go wrong. We publish various error metrics to help you understand what is going wrong in your job, and how to fix it. In general, we try to avoid failing a job entirely unless a particularly serious error occurs. When errors do occur, they often happen during the grading step. Errors during grading often happen either to the model outputting a sample that the grader doesn't know how to handle, the grader failing to execute properly due to some sort of system error, or due to a bug in the grading logic itself.
910
911The error metrics are available under the `event.data.errors` object, and are aggregated into counts and rates rolled up per-grader. We also display rates and counts of errors on the fine-tuning dashboard.
912
913
914
915#### Grader errors
916
917
918
919#### Generic grading errors
920
921The grader errors are broken down into the following categories, and they exist in both `train_` (for training data) and `valid_` (for validation data) versions:
922
923- `sample_parse_error_mean`: The average number of samples that failed to parse correctly. This often happens when the model fails to output valid JSON or adhere to a provided response format correctly. A small percentage of these errors, especially early in the training process, is normal. If you see a large number of these errors, it is likely that the response format of the model is not configured correctly or that your graders are misconfigured and looking for incorrect fields.
924- `invalid_variable_error_mean`: These errors occur when you attempt to reference a variable via a template that cannot be found either in the current data point or in the current model sample. This can happen if the model fails to provide output in the correct response format, or if your grader is misconfigured.
925- `other_error_mean`: This is a catch-all for any other errors that occur during grading. These errors are often caused by bugs in the grading logic itself, or by system errors that occur during grading.
926
927#### Python grading errors
928
929- `python_grader_server_error_mean`: These errors occur when our system for executing python graders in a remote sandbox experiences system errors. This normally happens due to reasons outside of your control, like networking failures or system outages. If you see a large number of these errors, it is likely that there is a system issue that is causing the errors. You can check the [OpenAI status page](https://status.openai.com/) for more information on any ongoing issues.
930- `python_grader_runtime_error_mean`: These errors occur when the python grader itself fails to execute properly. This can happen for a variety of reasons, including bugs in the grading logic, or if the grader is trying to access a variable that doesn't exist in the current context. If you see a large number of these errors, it is likely that there is a bug in your grading logic that needs to be fixed. If a large enough number of these errors occur, the job will fail and we will show you a sampling of tracebacks from the failed graders.
931
932#### Model grading errors
933
934- `model_grader_server_error_mean`: These errors occur when we fail to sample from a model grader. This can happen for a variety of reasons, but generally means that either the model grader was misconfigured, that you are attempting to use a model that is not available to your organization, or that there is a system issue that is happening at OpenAI.
