Vision fine-tuning

Vision fine-tuning uses image inputs for supervised fine-tuning to improve the model's understanding of image inputs. This guide will take you through this subset of SFT, and outline some of the important considerations for fine-tuning with image inputs.

OpenAI is winding down the fine-tuning platform. The platform is no longer accessible to new users, but existing users of the fine-tuning platform will be able to create training jobs for the coming months.
All fine-tuned models will remain available for inference until their base models are deprecated. The full timeline is here.

Data format

Just as you can send one or many image inputs and create model responses based on them, you can include those same message types within your JSONL training data files. Images can be provided either as HTTP URLs or data URLs containing Base64-encoded images.

Here's an example of an image message on a line of your JSONL file. Below, the JSON object is expanded for readability, but typically this JSON would appear on a single line in your data file:

{
  "messages": [
    {
      "role": "system",
      "content": "You are an assistant that identifies and describes artworks."
    },
    {
      "role": "user",
      "content": "Describe this artwork."
    },
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "https://api.nga.gov/iiif/a2e6da57-3cd1-4235-b20e-95dcaefed6c8/full/!800,800/0/default.jpg"
          }
        }
      ]
    },
    {
      "role": "assistant",
      "content": "This appears to be a traditional painted artwork with a central human subject."
    }
  ]
}

Uploading training data for vision fine-tuning follows the same process described here.

Image data requirements

Size

• Your training file can contain a maximum of 50,000 examples that contain images (not including text examples).
• Each example can have at most 10 images.
• Each image can be at most 10 MB.

Format

• Images must be JPEG, PNG, or WEBP format.
• Your images must be in the RGB or RGBA image mode.
• You cannot include images as output from messages with the assistant role.

Content moderation policy

We scan your images before training to ensure that they comply with our usage policy. This may introduce latency in file validation before fine-tuning begins.

Images containing the following will be excluded from your dataset and not used for training:

• People
• Faces
• Children
• CAPTCHAs

What to do if your images get skipped

Your images can get skipped during training for the following reasons:

• contains CAPTCHAs, contains people, contains faces, contains children
  • Remove the image. For now, we cannot fine-tune models with images containing these entities.
• inaccessible URL
  • Ensure that the image URL is publicly accessible.
• image too large
  • Please ensure that your images fall within our dataset size limits.
• invalid image format
  • Please ensure that your images fall within our dataset format.

Best practices

Reducing training cost

If you set the detail parameter for an image to low, the image is resized to 512 by 512 pixels and is only represented by 85 tokens regardless of its size. This will reduce the cost of training. See here for more information.

{
  "type": "image_url",
  "image_url": {
    "url": "https://api.nga.gov/iiif/a2e6da57-3cd1-4235-b20e-95dcaefed6c8/full/!800,800/0/default.jpg",
    "detail": "low"
  }
}

Control image quality

To control the fidelity of image understanding, set the detail parameter of image_url to low, high, or auto for each image. This will also affect the number of tokens per image that the model sees during training time, and will affect the cost of training. See here for more information.

Safety checks

Before launching in production, review and follow the following safety information.

How we assess for safety

Once a fine-tuning job is completed, we assess the resulting model’s behavior across 13 distinct safety categories. Each category represents a critical area where AI outputs could potentially cause harm if not properly controlled.

Each category has a predefined pass threshold; if too many evaluated examples in a given category fail, OpenAI blocks the fine-tuned model from deployment. If your fine-tuned model does not pass the safety checks, OpenAI sends a message in the fine-tuning job explaining which categories don't meet the required thresholds. You can view the results in the moderation checks section of the fine-tuning job.

How to pass safety checks

In addition to reviewing any failed safety checks in the fine-tuning job object, you can retrieve details about which categories failed by querying the fine-tuning API events endpoint. Look for events of type moderation_checks for details about category results and enforcement. This information can help you narrow down which categories to target for retraining and improvement. The model spec has rules and examples that can help identify areas for additional training data.

While these evaluations cover a broad range of safety categories, conduct your own evaluations of the fine-tuned model to ensure it's appropriate for your use case.

Next steps

Now that you know the basics of vision fine-tuning, explore these other methods as well.

[

](https://developers.openai.com/api/docs/guides/supervised-fine-tuning)

[

](https://developers.openai.com/api/docs/guides/direct-preference-optimization)

[

](https://developers.openai.com/api/docs/guides/reinforcement-fine-tuning)