Evaluate agent workflows

The OpenAI Platform offers a suite of evaluation tools to help you ensure your agents perform consistently and accurately.

Use this page as the decision point for the evaluation surfaces that matter most for agent workflows.

Start with traces when you are still debugging behavior

Trace grading is the fastest way to identify workflow-level issues. A trace captures the end-to-end record of model calls, tool calls, guardrails, and handoffs for one run. Graders let you score those traces with structured criteria so you can find regressions and failure modes at scale.

Use trace grading when you want to answer questions like:

Did the agent pick the right tool?
Did a handoff happen when it should have?
Did the workflow violate an instruction or safety policy?
Did a prompt or routing change improve the end-to-end behavior?

Trace-grading workflow

Open Logs > Traces in the dashboard.
Inspect a representative workflow trace from Agent Builder or an SDK-based app with tracing enabled.
Create a grader and run it against the selected traces.
Use the results to refine prompts, tool surfaces, routing logic, or guardrails.

For code-first SDK workflows, start with Integrations and observability to get high-signal traces before you formalize graders.

Move to datasets and eval runs when you need repeatability

Once you know what “good” looks like, move from individual traces to repeatable datasets and eval runs. This is the right step when you want to benchmark changes, compare prompts, or run larger-scale evaluations over time.

If you need advanced features such as evaluation against external models, evaluation APIs, or larger-scale batch evaluation, use Evals alongside datasets.

<a href="/openai/api/history/docs/api/2026-04-07-0551..2026-04-08-0551/docs/guides/evaluation-getting-started/" target="_blank" rel="noreferrer"

Operate a flywheel of continuous improvement using evaluations. Evaluate against external models, interact with evals via API, and more. Use your dataset to automatically improve your prompts.

<a href="https://cookbook.openai.com/examples/evaluation/building_resilient_prompts_using_an_evaluation_flywheel" target="_blank" rel="noreferrer"

Operate a flywheel of continuous improvement using evaluations.

guides/agent-evals.md +26 −6

~~1# Agent evals~~1# Evaluate agent workflows

2 2

3The OpenAI Platform offers a suite of evaluation tools to help you ensure your agents perform consistently and accurately.3The OpenAI Platform offers a suite of evaluation tools to help you ensure your agents perform consistently and accurately.

4 4

~~5For identifying errors at the workflow-level, we recommend our [trace grading](https://developers.openai.com/api/docs/guides/trace-grading) functionality.~~5Use this page as the decision point for the evaluation surfaces that matter most for agent workflows.

6 6

~~7For an easy way to build and iterate on your evals, we recommend exploring [Datasets](https://developers.openai.com/api/docs/guides/evaluation-getting-started).~~7## Start with traces when you are still debugging behavior

8 8

9If you need advanced features such as evaluation against external models, want to interact with your eval runs via API, or want to run evaluations on a larger scale, consider using [Evals](https://developers.openai.com/api/docs/guides/evals) instead.9Trace grading is the fastest way to identify workflow-level issues. A trace captures the end-to-end record of model calls, tool calls, guardrails, and handoffs for one run. Graders let you score those traces with structured criteria so you can find regressions and failure modes at scale.

10 10

~~11## Next steps~~11Use trace grading when you want to answer questions like:

12 12

~~13For more inspiration, visit the [OpenAI Cookbook](https://developers.openai.com/cookbook), which contains example code and links to third-party resources, or learn more about our tools for evals:~~13- Did the agent pick the right tool?

14- Did a handoff happen when it should have?

15- Did the workflow violate an instruction or safety policy?

16- Did a prompt or routing change improve the end-to-end behavior?

18### Trace-grading workflow

201. Open **Logs** > **Traces** in the dashboard.

212. Inspect a representative workflow trace from Agent Builder or an SDK-based app with tracing enabled.

223. Create a grader and run it against the selected traces.

234. Use the results to refine prompts, tool surfaces, routing logic, or guardrails.

25For code-first SDK workflows, start with [Integrations and observability](https://developers.openai.com/api/docs/guides/agents/integrations-observability#tracing) to get high-signal traces before you formalize graders.

27## Move to datasets and eval runs when you need repeatability

29Once you know what “good” looks like, move from individual traces to repeatable datasets and eval runs. This is the right step when you want to benchmark changes, compare prompts, or run larger-scale evaluations over time.

31If you need advanced features such as evaluation against external models, evaluation APIs, or larger-scale batch evaluation, use [Evals](https://developers.openai.com/api/docs/guides/evals) alongside datasets.

33## Related evaluation surfaces

14 34

15<a35<a

16 href="/api/docs/guides/evaluation-getting-started"36 href="/api/docs/guides/evaluation-getting-started"