15 15
16## When to use reinforcement fine-tuning16## When to use reinforcement fine-tuning
17 17
18Agentic workflows are designed to make decisions that are both correct and verifiable. RFT can help by providing explicit rubrics and using code‑based or LLM‑based graders to measure functional success, factual accuracy, or policy compliance.18An agentic workflow is designed to make decisions that are both correct and verifiable. RFT can help by providing explicit rubrics and using code‑based or LLM‑based graders to measure functional success, factual accuracy, or policy compliance.
19 19
20Across early users, three clear use cases have emerged:20Across early users, three clear use cases have emerged:
21 21
121 121
122 122
123 123
124#### Production-ready API snippets that compile and pass AST checks
125
126
127
128Use case
129
130
131
132> **Company**: [Runloop](https://www.runloop.ai) is a platform for AI-powered coding agents to be deployed into production and built with public and custom benchmarking capabilities to refine performance.
133>
134> **Problem to solve**: Runloop wanted to improve model performance at using third-party APIs, such as the Stripe API, which can be large and complex without a human in the loop. If they could train a model to use the Stripe API, Runloop could turn economically impactful business cases into working code.
135>
136> **Objective**: Their goal was teaching the model to master usage of the Stripe API, including writing complete code snippets for arbitrary user requests by either adapting information from existing integration guides, merging information from multiple guides, or inferring information not explicitly stated in the guides. They used RFT with two primary rewards:
137>
138> 1. Reward the model for outputting the answer in a Markdown format that aligns with expectation of how a "dynamic" integration guide should look.
139> 1. Reward the model for producing "correct" code snippets by validating the outputted code via AST Grep. This allows them to confirm the model is making the correct Stripe SDK calls with the correct parameters and in some cases even in the correct order.
140
141
142
143
144
145
146
147Grader code
148
149
150
151````python
152# Note this file gets uploaded to the OpenAI API as a grader
153from ast_grep_py import SgRoot
154from pydantic import BaseModel, Field # type: ignore
155from typing import Any
156import re
157
158SUPPORTED_LANGUAGES = ['typescript', 'javascript', 'ts', 'js']
159
160class CodeBlock(BaseModel):
161 language: str = Field(
162 description="Programming language of the code block (e.g., 'python', 'javascript')",
163 examples=["python", "javascript", "typescript"]
164 )
165 path: str = Field(
166 description="Target file path where the code should be written",
167 examples=["main.py", "src/app.js", "index.html"]
168 )
169 code: str = Field(
170 description="Actual code content extracted from the code block"
171 )
172
173class ASTGrepPattern(BaseModel):
174 file_path_mask: str = Field(..., description="The file path pattern to match against")
175 pattern: str = Field(..., description="The main AST grep pattern to search for")
176 additional_greps: list[str] | None = Field(
177 default=None,
178 description="Additional patterns that must also be present in the matched code"
179 )
180
181def extract_code_blocks(llm_output: str) -> list[CodeBlock]:
182 # Regular expression to match code blocks with optional language and path
183 try:
184 pattern = r"```(\w+\s+)?([\w./-]+)?\n([\s\S]*?)\n```"
185 matches = list(re.finditer(pattern, llm_output, re.DOTALL))
186
187 print(f"Found {len(matches)} code blocks in the LLM output")
188
189 # Check if any code blocks were found
190 if not matches:
191 raise Exception("No code blocks found in the LLM response")
192
193 code_blocks: list[CodeBlock] = []
194 for match in matches:
195 language = match.group(1) or ""
196 path = match.group(2) or ""
197 code = match.group(3)
198
199 # Clean the path and language
200 path = path.strip()
201 language = language.strip()
202
203 # If path is relative (doesn't start with /), prefix with /home/user/testbed/
204 if path and not path.startswith("/"):
205 original_path = path
206 path = f"/home/user/testbed/{path}"
207 print(
208 f"Converting relative path '{original_path}' to absolute path '{path}'"
209 )
210
211 code_blocks.append(
212 CodeBlock(language=language, path=path, code=code.strip())
213 )
214
215 # Check for missing language or path in code blocks
216 missing_language = [
217 i for i, block in enumerate(code_blocks) if not block.language
218 ]
219 missing_path = [i for i, block in enumerate(code_blocks) if not block.path]
220
221 if missing_language:
222 print(
223 f"WARNING: Code blocks at positions {missing_language} are missing language identifiers"
224 )
225 raise Exception(
226 f"Code blocks at positions {missing_language} are missing language identifiers"
227 )
228
229 if missing_path:
230 print(
231 f"WARNING: Code blocks at positions {missing_path} are missing file paths"
232 )
233 raise Exception(
234 f"Code blocks at positions {missing_path} are missing file paths"
235 )
236
237 paths = [block.path for block in code_blocks if block.path]
238 print(
239 f"Successfully extracted {len(code_blocks)} code blocks with paths: {', '.join(paths)}"
240 )
241
242 except Exception as e:
243 print(f"Error extracting code blocks: {str(e)}")
244 raise
245
246 return code_blocks
247
248
249def calculate_ast_grep_score(code_blocks: list[CodeBlock], ast_greps: Any) -> float:
250 # Convert ast_greps to list if it's a dict
251 if isinstance(ast_greps, dict):
252 ast_greps = [ast_greps]
253
254 # Parse each grep pattern into the Pydantic model
255 parsed_patterns: list[ASTGrepPattern] = []
256 for grep in ast_greps:
257 try:
258 pattern = ASTGrepPattern(**grep)
259 parsed_patterns.append(pattern)
260 except Exception as e:
261 print(f"Error parsing AST grep pattern: {e}")
262 return 0.0
263
264 if not parsed_patterns:
265 return 0.0
266
267 total_score = 0.0
268 pattern_count = len(parsed_patterns)
269
270 # Filter code blocks to only include TypeScript and JavaScript files
271 supported_blocks = [
272 block for block in code_blocks
273 if block.language.lower() in SUPPORTED_LANGUAGES
274 ]
275
276 if not supported_blocks:
277 print("No TypeScript or JavaScript code blocks found to analyze")
278 return 0.0
279
280 for pattern in parsed_patterns:
281 # Find matching code blocks based on path prefix
282 matching_blocks = [
283 block for block in supported_blocks
284 if block.path.startswith(pattern.file_path_mask)
285 ]
286
287 if not matching_blocks:
288 print(f"No matching code blocks found for path prefix: {pattern.file_path_mask}")
289 continue
290
291 pattern_found = False
292 for block in matching_blocks:
293 try:
294 # Create AST root for the code block
295 root = SgRoot(block.code, block.language)
296 node = root.root()
297
298 # Check main pattern
299 matches = node.find(pattern=pattern.pattern)
300 if not matches:
301 continue
302
303 # If we have additional greps, check them too
304 if pattern.additional_greps:
305 all_additional_found = True
306 for additional_grep in pattern.additional_greps:
307 if additional_grep not in block.code:
308 all_additional_found = False
309 break
310
311 if not all_additional_found:
312 continue
313
314 # If we get here, we found a match with all required patterns
315 pattern_found = True
316 break
317
318 except Exception as e:
319 print(f"Error processing code block {block.path}: {e}")
320 continue
321
322 if pattern_found:
323 total_score += 1.0
324
325 # Return average score across all patterns
326 return total_score / pattern_count if pattern_count > 0 else 0.0
327
328def grade_format(output_text: str) -> float:
329 # Find <plan> and </plan> tags
330 plan_start = output_text.find('<plan>')
331 plan_end = output_text.find('</plan>')
332
333 # Find <code> and </code> tags
334 code_start = output_text.find('<code>')
335 code_end = output_text.find('</code>')
336
337 reward = 0.0
338
339 if plan_start == -1 or plan_end == -1 or code_start == -1 or code_end == -1:
340 print(f'missing plan or code tags. format reward: {reward}')
341 return reward
342 reward += 0.1 # total: 0.1
343
344 if not (plan_start < plan_end < code_start < code_end):
345 print(f'tags present but not in the correct order. format reward: {reward}')
346 return reward
347 reward += 0.1 # total: 0.2
348
349 # Check if there are any stray tags
350 plan_tags = re.findall(r'</?plan>', output_text)
351 code_tags = re.findall(r'</?code>', output_text)
352
353 if len(plan_tags) != 2 or len(code_tags) != 2:
354 print(f'found stray plan or code tags. format reward: {reward}')
355 return reward
356 reward += 0.2 # total: 0.4
357
358 # Extract content after </code> tag
359 after_tags = output_text[code_end + len('</code>'):].strip()
360 if after_tags:
361 print(f'found text after code tags. format reward: {reward}')
362 return reward
363 reward += 0.2 # total: 0.6
364
365 # Extract content inside <plan> tags
366 plan_content = output_text[plan_start + len('<plan>'):plan_end].strip()
367 if not plan_content:
368 print(f'no plan content found. format reward: {reward}')
369 return reward
370 reward += 0.1 # total: 0.7
371
372 # Extract content inside <code> tags
373 code_content = output_text[code_start + len('<code>'):code_end].strip()
374 if not code_content:
375 print(f'no code content found. format reward: {reward}')
376 return reward
377 reward += 0.1 # total: 0.8
378
379 # Extract content between </plan> and <code> tags
380 between_tags = output_text[plan_end + len('</plan>'):code_start].strip()
381 if between_tags:
382 print(f'found text between plan and code tags. format reward: {reward}')
383 return reward
384 reward += 0.2 # total: 1.0
385
386 if reward == 1.0:
387 print(f'global format reward: {reward}')
388
389 return reward
390
391def grade(sample: Any, item: Any) -> float:
392 try:
393 output_text = sample["output_text"]
394
395 format_reward = grade_format(output_text)
396 if format_reward < 1.0:
397 return format_reward
398
399 # Extract code content for grading
400 code_start = output_text.find('<code>')
401 code_end = output_text.find('</code>')
402 code_to_grade: str = output_text[code_start + len('<code>'):code_end].strip()
403 code_blocks: list[CodeBlock] = []
404 try:
405 code_blocks = extract_code_blocks(code_to_grade)
406 except Exception as e:
407 print(f'error extracting code blocks: {e}')
408 return 0.5
409
410 ast_greps = item["reference_answer"]["ast_greps"]
411 ast_grep_score = calculate_ast_grep_score(code_blocks, ast_greps)
412
413 return (format_reward + ast_grep_score) / 2.0
414 except Exception as e:
415 print(f"Error during grading: {str(e)}")
416 return 0.0
417````
418
419
420
421
422
423
424
425Results
426
427
428
429> Looking at the total reward (format and AST Grep) together, Runloop has seen improvements of on average **12%** of the RFT model compared to the base o3-mini model on the benchmark.
430>
431> They implement two types of tests, one providing explicit content from the integration guides (assessing reasoning and instruction following) and one without (assessing knowledge recall). Both variants saw improvement of over **8%**.
432>
433> “OpenAIs RFT platform gives us access to the best generalized reasoning models in the world, with the toolset to supercharge that reasoning on problem domains important to our business.”
434>
435> —[Runloop](https://www.runloop.ai/)
436
437
438
439#### Correct handling of conflicts and dupes in a schedule manager124#### Correct handling of conflicts and dupes in a schedule manager
440 125
441 126
636 321
637### 3. Apply complex rules correctly322### 3. Apply complex rules correctly
638 323
639This use case involves pulling verifiable facts or entities from unstructured inputs into clearly defined schemas (e.g., JSON objects, condition codes, medical codes, legal citations, or financial metrics).324This use case involves pulling verifiable facts or entities from unstructured inputs into defined schemas (for example, JSON objects, condition codes, medical codes, legal citations, or financial metrics).
640 325
641Successful extraction tasks typically benefit from precise, continuous grading methodologies—like span-level F1 scores, fuzzy text-matching metrics, or numeric accuracy checks—to evaluate how accurately the extracted information aligns with ground truth. Define explicit success criteria and detailed rubrics. Then, the model can achieve reliable, repeatable improvements.326Successful extraction tasks typically benefit from precise, continuous grading methodologies—like span-level F1 scores, fuzzy text-matching metrics, or numeric accuracy checks—to evaluate how accurately the extracted information aligns with ground truth. Define explicit success criteria and detailed rubrics. Then, the model can achieve reliable, repeatable improvements.
642 327
773 458
774**Before implementing RFT, we strongly recommended creating and running an eval for the task you intend to fine-tune on**. If the model you intend to fine-tune scores at either the absolute minimum or absolute maximum possible score, then RFT won’t be useful to you.459**Before implementing RFT, we strongly recommended creating and running an eval for the task you intend to fine-tune on**. If the model you intend to fine-tune scores at either the absolute minimum or absolute maximum possible score, then RFT won’t be useful to you.
775 460
776RFT works by reinforcing better answers to provided prompts. If we can’t distinguish the quality of different answers (i.e., if they all receive the minimum or maximum possible score), then there's no training signal to learn from. However, if your eval scores somewhere in the range between the minimum and maximum possible scores, there's enough data to work with.461RFT works by reinforcing better answers to provided prompts. If we can’t distinguish the quality of different answers (that is, if they all receive the minimum or maximum possible score), then there's no training signal to learn from. However, if your eval scores somewhere in the range between the minimum and maximum possible scores, there's enough data to work with.
777 462
778An effective eval reveals opportunities where human experts consistently agree but current frontier models struggle, presenting a valuable gap for RFT to close. [Get started with evals](https://developers.openai.com/api/docs/guides/evals).463An effective eval reveals opportunities where human experts consistently agree but current frontier models struggle, presenting a valuable gap for RFT to close. [Get started with evals](https://developers.openai.com/api/docs/guides/evals).
779 464
781 466
782To see improvements in your fine-tuned model, there are two main places to revisit and refine: making sure your task is well defined, and making your grading scheme more robust.467To see improvements in your fine-tuned model, there are two main places to revisit and refine: making sure your task is well defined, and making your grading scheme more robust.
783 468
784### Reframe or clarify your task469<a id="reframe-or-clarify-your-task"></a>
470
471### Revise or clarify your task
785 472
786Good tasks give the model a fair chance to learn and let you quantify improvements.473Good tasks give the model a fair chance to learn and let you quantify improvements.
787 474
788- **Start with a task the model can already solve occasionally**. RFT works by sampling many answers, keeping what looks best, and nudging the model toward those answers. If the model never gets the answer correct today, it cannot improve.475- **Start with a task the model can already solve occasionally**. RFT works by sampling many answers, keeping what looks best, and nudging the model toward those answers. If the model never gets the answer correct today, it cannot improve.
789- **Make sure each answer can be graded**. A grader must read an answer and produce a score without a person in the loop. We support multiple [grader types](https://developers.openai.com/api/docs/guides/graders), including custom Python graders and LLM judges. If you can't write code to judge the answer with an available grader, RFT is not the right tool.476- **Make sure each answer can be graded**. A grader must read an answer and produce a score without a person in the loop. We support multiple [grader types](https://developers.openai.com/api/docs/guides/graders), including custom Python graders and LLM judges. If you can't write code to judge the answer with an available grader, RFT is not the right tool.
790- **Remove doubt about the “right” answer**. If two careful people often disagree on the solution, the task is too fuzzy. Rewrite the prompt, add context, or split the task into clearer parts until domain experts agree.477- **Remove doubt about the “right” answer**. If two careful people often disagree on the solution, the task is too fuzzy. Rewrite the prompt, add context, or split the task into clearer parts until domain experts agree.
791- **Limit lucky guesses**. If the task is multiple choice with one obvious best pick, the model can win by chance. Add more classes, ask for short open‑ended text, or tweak the format so guessing is costly.478- **Limit lucky guesses**. If the task is multiple choice with one best option, the model can win by chance. Add more classes, ask for short open‑ended text, or tweak the format so guessing is costly.
792 479
793### Strengthen your grader480### Strengthen your grader
794 481
