model-customization.md

title	titleSuffix	description	author	manager	ms.service	ms.topic	ms.date	ms.collection	ms.author	ms.custom
Create a custom Image Analysis model	Azure AI services	Learn how to create and train a custom model to do image classification and object detection that's specific to your use case.	PatrickFarley	nitinme	azure-ai-vision	how-to	09/25/2024	ce-skilling-fresh-tier2, ce-skilling-ai-copilot	pafarley	devx-track-python

Create a custom Image Analysis model

[!INCLUDE model-customization-deprecation]

Image Analysis 4.0 allows you to train a custom model using your own training images. By manually labeling your images, you can train a model to apply custom tags to the images (image classification) or detect custom objects (object detection). Image Analysis 4.0 models are especially effective at few-shot learning, so you can get accurate models with less training data.

This guide shows you how to create and train a custom image classification model. The few differences between training an image classification model and object detection model are noted.

Note

Model customization is available through the REST API and Vision Studio, but not through the client language SDKs.

Prerequisites

• An Azure subscription. You can create one for free.
• Once you have your Azure subscription, create a Vision resource in the Azure portal to get your key and endpoint. If you're following this guide using Vision Studio, you must create your resource in the East US region. After it deploys, select Go to resource. Copy the key and endpoint to a temporary location for later use.
• An Azure Storage resource. Create a Storage resource.
• A set of images with which to train your classification model. You can use the set of sample images on GitHub. Or, you can use your own images. You only need about 3-5 images per class.

Note

We don't recommend you use custom models for business critical environments due to potential high latency. When customers train custom models in Vision Studio, those custom models belong to the Vision resource that they were trained under and the customer is able to make calls to those models using the Analyze Image API. When they make these calls, the custom model is loaded in memory and the prediction infrastructure is initialized. While this happens, customers might experience longer than expected latency to receive prediction results.

Vision Studio

Create a new custom model

Begin by going to Vision Studio and selecting the Image analysis tab. Then select the Customize models tile.

:::image type="content" source="../media/customization/customization-tile.png" alt-text="Screenshot of the Customize models tile.":::

Then, sign in with your Azure account and select your Vision resource. If you don't have one, you can create one from this screen.

:::image type="content" source="../media/customization/select-resource.png" alt-text="Screenshot of the select resource screen.":::

Prepare training images

You need to upload your training images to an Azure Blob Storage container. Go to your storage resource in the Azure portal and navigate to the Storage browser tab. Here you can create a blob container and upload your images. Put them all at the root of the container.

Add a dataset

To train a custom model, you need to associate it with a Dataset where you provide images and their label information as training data. In Vision Studio, select the Datasets tab to view your datasets.

To create a new dataset, select add new dataset. In the popup window, enter a name and select a dataset type for your use case. Image classification models apply content labels to the entire image, while Object detection models apply object labels to specific locations in the image. Product recognition models are a subcategory of object detection models that are optimized for detecting retail products.

:::image type="content" source="../media/customization/create-dataset.png" alt-text="Screenshoot of dialog box to Create new dataset.":::

Then, select the container from the Azure Blob Storage account where you stored the training images. Check the box to allow Vision Studio to read and write to the blob storage container. This is a necessary step to import labeled data. Create the dataset.

Create an Azure Machine Learning labeling project

You need a COCO file to convey the labeling information. An easy way to generate a COCO file is to create an Azure Machine Learning project, which comes with a data-labeling workflow.

In the dataset details page, select Add a new Data Labeling project. Name it and select Create a new workspace. That opens a new Azure portal tab where you can create the Azure Machine Learning project.

:::image type="content" source="../media/customization/dataset-details.png" alt-text="Screenshot of Create Azure Machine Learning Data Labeling project.":::

Once the Azure Machine Learning project is created, return to the Vision Studio tab and select it under Workspace. The Azure Machine Learning portal will then open in a new browser tab.

Create labels

To start labeling, follow the Please add label classes prompt to add label classes.

:::image type="content" source="../media/customization/azure-machine-learning-home-page.png" alt-text="Screenshot showing button to Add label classes.":::

:::image type="content" source="../media/customization/azure-machine-learning-label-categories.png" alt-text="Screenshot showing button to Add label categories.":::

After you add all the class labels, save them, select Start on the project, and then select Label data at the top.

:::image type="content" source="../media/customization/azure-machine-learning-start.png" alt-text="Screenshot showing button to Start labeling data.":::

Manually label training data

Choose Start labeling and follow the prompts to label all of your images. When you're finished, return to the Vision Studio tab in your browser.

Now select Add COCO file, then select Import COCO file from an Azure ML Data Labeling project. This imports the labeled data from Azure Machine Learning.

The COCO file you created is now stored in the Azure Storage container that you linked to this project. You can now import it into the model customization workflow. Select it from the drop-down list. Once the COCO file is imported into the dataset, the dataset can be used for training a model.

Note

If you have a ready-made COCO file you want to import, go to the Datasets tab and select Add COCO files to this dataset. You can choose to add a specific COCO file from a blob storage account or import from the Azure Machine Learning labeling project.

Currently, Microsoft is addressing an issue that causes COCO file import to fail with large datasets when initiated in Vision Studio. To train using a large dataset, it's recommended to use the REST API instead.

[!INCLUDE coco-files]

Train the custom model

To start training a model with your COCO file, go to the Custom models tab and select Add a new model. Enter a name for the model and select Image classification or Object detection as the model type.

:::image type="content" source="../media/customization/start-model-training.png" alt-text="Screenshot of dialog box to Train a new model.":::

Select your dataset, which is now associated with the COCO file containing the labeling information.

Then select a time budget and train the model. For small examples, you can use a 1 hour budget.

:::image type="content" source="../media/customization/train-model.png" alt-text="Screenshot showing training details to review.":::

It might take some time for the training to complete. Image Analysis 4.0 models can be accurate with only a small set of training data, but they take longer to train than previous models.

Evaluate the trained model

After training is complete, you can view the model's performance evaluation. The following metrics are used:

• Image classification: Average Precision, Accuracy Top 1, Accuracy Top 5
• Object detection: Mean Average Precision @ 30, Mean Average Precision @ 50, Mean Average Precision @ 75

If an evaluation set isn't provided when training the model, the reported performance is estimated based on part of the training set. We strongly recommend you use an evaluation dataset (using the same process as above) to have a reliable estimation of your model performance.

:::image type="content" source="../media/customization/training-result.png" alt-text="Screenshot showing evaluation results.":::

Test the custom model in Vision Studio

Once you've built a custom model, you can test by selecting the Try it out button on the model evaluation screen.

:::image type="content" source="../media/customization/custom-try-it-out.png" alt-text="Screenshot of the model evaluation screen with Try it out button outlined.":::

This takes you to the Extract common tags from images page. Choose your custom model from the drop-down menu and upload a test image.

:::image type="content" source="../media/customization/quick-test.png" alt-text="Screenshot of selecting test model in Vision Studio.":::

The prediction results appear in the right column.

REST API

Prepare training data

The first thing you need to do is create a COCO file from your training data. See the following specification.

[!INCLUDE coco-files]

Upload to storage

Upload your COCO file to a blob storage container, ideally the same blob container that holds the training images themselves.

Create your training dataset

The datasets/<dataset-name> API lets you create a new dataset object that references the training data. Make the following changes to the cURL command:

1. Replace <endpoint> with your Azure AI Vision endpoint.
2. Replace <dataset-name> with a name for your dataset.
3. Replace <subscription-key> with your Azure AI Vision key.
4. In the request body, set "annotationKind" to either "imageClassification" or "imageObjectDetection", depending on your project.
5. In the request body, set the "annotationFileUris" array to an array of strings that show the URI locations of your COCO files in blob storage.

curl.exe -v -X PUT "<endpoint>/computervision/datasets/<dataset-name>?api-version=2023-02-01-preview" -H "Content-Type: application/json" -H "Ocp-Apim-Subscription-Key: <subscription-key>" --data-ascii "
{
'annotationKind':'imageClassification',
'annotationFileUris':['<URI>']
}"

Create and train a model

The models/<model-name> API lets you create a new custom model and associate it with an existing dataset. It also starts the training process. Make the following changes to the cURL command:

1. Replace <endpoint> with your Azure AI Vision endpoint.
2. Replace <model-name> with a name for your model.
3. Replace <subscription-key> with your Azure AI Vision key.
4. In the request body, set "trainingDatasetName" to the name of the dataset from the previous step.
5. In the request body, set "modelKind" to either "Generic-Classifier" or "Generic-Detector", depending on your project.

curl.exe -v -X PUT "<endpoint>/computervision/models/<model-name>?api-version=2023-02-01-preview" -H "Content-Type: application/json" -H "Ocp-Apim-Subscription-Key: <subscription-key>" --data-ascii "
{
'trainingParameters': {
    'trainingDatasetName':'<dataset-name>',
    'timeBudgetInHours':1,
    'modelKind':'Generic-Classifier',
    }
}"

Evaluate the model's performance on a dataset

The models/<model-name>/evaluations/<eval-name> API evaluates the performance of an existing model. Make the following changes to the cURL command:

1. Replace <endpoint> with your Azure AI Vision endpoint.
2. Replace <model-name> with the name of your model.
3. Replace <eval-name> with a name that can be used to uniquely identify the evaluation.
4. Replace <subscription-key> with your Azure AI Vision key.
5. In the request body, set "testDatasetName" to the name of the dataset you want to use for evaluation. If you don't have a dedicated dataset, you can use the same dataset you used for training.

curl.exe -v -X PUT "<endpoint>/computervision/models/<model-name>/evaluations/<eval-name>?api-version=2023-02-01-preview" -H "Content-Type: application/json" -H "Ocp-Apim-Subscription-Key: <subscription-key>" --data-ascii "
{
'evaluationParameters':{
    'testDatasetName':'<dataset-name>'
    },
}"

The API call returns a ModelPerformance JSON object, which lists the model's scores in several categories. The following metrics are used:

• Image classification: Average Precision, Accuracy Top 1, Accuracy Top 5
• Object detection: Mean Average Precision @ 30, Mean Average Precision @ 50, Mean Average Precision @ 75

Test the custom model on an image

The imageanalysis:analyze API does ordinary Image Analysis operations. By specifying some parameters, you can use this API to query your own custom model instead of the prebuilt Image Analysis models. Make the following changes to the cURL command:

1. Replace <endpoint> with your Azure AI Vision endpoint.
2. Replace <model-name> with the name of your model.
3. Replace <subscription-key> with your Azure AI Vision key.
4. In the request body, set "url" to the URL of a remote image you want to test your model on.

curl.exe -v -X POST "<endpoint>/computervision/imageanalysis:analyze?model-name=<model-name>&api-version=2023-02-01-preview" -H "Content-Type: application/json" -H "Ocp-Apim-Subscription-Key: <subscription-key>" --data-ascii "
{'url':'https://learn.microsoft.com/azure/ai-services/computer-vision/media/quickstarts/presentation.png'
}"

The API call returns an ImageAnalysisResult JSON object, which contains all the detected tags for an image classifier, or objects for an object detector, with their confidence scores.

{
  "kind": "imageAnalysisResult",
  "metadata": {
    "height": 900,
    "width": 1260
  },
  "customModelResult": {
    "classifications": [
      {
        "confidence": 0.97970027,
        "label": "hemlock"
      },
      {
        "confidence": 0.020299695,
        "label": "japanese-cherry"
      }
    ],
    "objects": [],
    "imageMetadata": {
      "width": 1260,
      "height": 900
    }
  }
}

---

Related content

In this guide, you created and trained a custom image classification model using Image Analysis. Next, learn more about the Analyze Image 4.0 API, so you can call your custom model from an application using REST.

• Model customization concepts
• Call the Analyze Image API