Tutorial - How to convert OpenVINO IR model locally to run on DepthAI

In this tutorial, you’ll learn how to convert OpenVINO IR models into the format required to run on DepthAI, even on a low-powered Raspberry Pi. I’ll introduce you to the OpenVINO toolset, the Open Model Zoo (where we’ll download the face-detection-retail-0004 model), and show you how to generate the files needed to run model inference on your DepthAI board.

model image

Haven’t heard of OpenVINO or the Open Model Zoo? I’ll start with a quick introduction of why we need these tools.

What is OpenVINO?

Under-the-hood, DepthAI uses the Intel tecnnology to perform high-speed model inference. However, you can’t just dump your neural net into the chip and get high-performance for free. That’s where OpenVINO comes in. OpenVINO is a free toolkit that converts a deep learning model into a format that runs on Intel Hardware. Once the model is converted, it’s common to see Frames Per Second (FPS) improve by 25x or more. Are a couple of small steps worth a 25x FPS increase? Often, the answer is yes!

What is the Open Model Zoo?

The Open Model Zoo is a library of freely-available pre-trained models. The Zoo also contains scripts for downloading those models into a compile-ready format to run on DepthAI.

DepthAI is able to run many of the object detection models in the Zoo.

Install OpenVINO

DepthAI requires OpenVINO version 2020.1. Let’s get a package for our OS and meeting this version with the following command:

apt-get update
apt-get install -y software-properties-common
add-apt-repository -y ppa:deadsnakes/ppa
apt-get update
apt-get install -y wget pciutils python3.8 libpng-dev libcairo2-dev libpango1.0-dev libglib2.0-dev libgtk2.0-dev libswscale-dev libavcodec-dev libavformat-dev
cd
mkdir openvino_install && cd openvino_install
wget http://registrationcenter-download.intel.com/akdlm/irc_nas/16345/l_openvino_toolkit_p_2020.1.023.tgz
tar --strip-components=1 -zxvf l_openvino_toolkit_p_2020.1.023.tgz
./install_openvino_dependencies.sh
./install.sh # when finished, you can go ahead and do "rm -r ~/openvino_install"

Now, first screen we’ll wee is EULA, just hit Enter, scroll through and type accept.

Next one is agreement to Intel Software Improvement Program, it’s not relevant so you can choose whether consent (1) or not (2)

Next, you may see the Missing Prerequisites screen showing that Intel® Graphics Compute Runtime for OpenCL™ Driver is missing - you can go ahead and ignore this warning.

Finally, we’ll see the install summary - please verify that it has a correct location pointed out - /opt/intel. If all looks good, go ahead and proceed (1). If the missing prerequisites screen appears again, feel free to skip it.

Let’s verify that a correct version is installed on your host. Check your version by running the following from a terminal session:

cat /opt/intel/openvino/inference_engine/version.txt

You should see output similar to:

Thu Jan 23 19:14:14 MSK 2020
d349c3ba4a2508be72f413fa4dee92cc0e4bc0e1
releases_2020_1_InferenceEngine_37988

Verify that you see releases_2020_1 in your output. If you do, move on. If you are on a different version, goto the OpenVINO site and download the 2020.1 version for your OS:

openvino_version

Check if the Model Downloader is installed

When installing OpenVINO, you can choose to perform a smaller install to save disk space. This custom install may not include the model downloader script. Lets check if the downloader was installed. In a terminal session, type the following:

find /opt/intel/ -iname downloader.py

Move on if you see the output below:

/opt/intel/openvino_2020.1.023/deployment_tools/open_model_zoo/tools/downloader/downloader.py

Didn’t see any output? Don’t fret if downloader.py isn’t found. We’ll install this below.

Install Open Model Zoo Downloader

If the downloader tools weren’t found, we’ll install the tools by cloning the Open Model Zoo Repo and installing the tool dependencies.

Start a terminal session and run the following commands in your terminal:

apt-get install -y git curl
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py
rm get-pip.py
cd ~
git clone https://github.com/opencv/open_model_zoo.git
cd open_model_zoo
git checkout tags/2020.1
cd tools/downloader
python3 -m pip install --user -r ./requirements.in

This clones the repo into a ~/open_model_zoo directory, checks out the required 2020.1 version, and installs the downloader dependencies.

Create an OPEN_MODEL_DOWNLOADER environment variable

Typing the full path to downloader.py can use a lot of keystrokes. In an effort to extend your keyboard life, let’s store the path to this script in an environment variable.

Run the following in your terminal:

export OPEN_MODEL_DOWNLOADER='INSERT PATH TO YOUR downloader.py SCRIPT'

Where INSERT PATH TO YOUR downloader.py SCRIPT can be found via:

find /opt/intel/ -iname downloader.py
find ~ -iname downloader.py

For example, if you installed open_model_zoo yourself:

export OPEN_MODEL_DOWNLOADER="$HOME/open_model_zoo/tools/downloader/downloader.py"

Download the face-detection-retail-0004 model

We’ve installed everything we need to download models from the Open Model Zoo! We’ll now use the Model Downloader to download the face-detection-retail-0004 model files. Run the following in your terminal:

$OPEN_MODEL_DOWNLOADER --name face-detection-retail-0004 --output_dir ~/open_model_zoo_downloads/

This will download the model files to ~/open_model_zoo_downloads/. Specifically, the model files we need are located at:

~/open_model_zoo_downloads/intel/face-detection-retail-0004/FP16

You’ll see two files within the directory:

ls -lh 
total 1.3M
-rw-r--r-- 1 root root 1.2M Jul 28 12:40 face-detection-retail-0004.bin
-rw-r--r-- 1 root root 100K Jul 28 12:40 face-detection-retail-0004.xml

The model is in the OpenVINO Intermediate Representation (IR) format:

  • face-detection-retail-0004.xml - Describes the network topology
  • face-detection-retail-0004.bin - Contains the weights and biases binary data.

This means we are ready to compile the model for the MyriadX!

Compile the model

The MyriadX chip used on our DepthAI board does not use the IR format files directly. Instead, we need to generate two files:

  • face-detection-retail-0004.blob - We’ll create this file with the myriad_compile command.
  • face-detection-retail-0004.json - A blob_file_config file in JSON format. This describes the format of the output tensors. You can read more about this file structure and examples here

We’ll start by creating the blob file.

Locate myriad_compile

Let’s find where myriad_compile is located. In your terminal, run:

find /opt/intel/ -iname myriad_compile

You should see the output similar to this

find /opt/intel/ -iname myriad_compile
/opt/intel/openvino_2020.1.023/deployment_tools/inference_engine/lib/intel64/myriad_compile

Since it’s such a long path, let’s store the myriad_compile executable in an environment variable (just like OPEN_MODEL_DOWNLOADER):

export MYRIAD_COMPILE=$(find /opt/intel/ -iname myriad_compile)

Activate OpenVINO environment

In order to use myriad_compile tool, we need to activate our OpenVINO environment.

First, let’s find setupvars.sh file

find /opt/intel/ -name "setupvars.sh"
/opt/intel/openvino_2020.1.023/opencv/setupvars.sh
/opt/intel/openvino_2020.1.023/bin/setupvars.sh

We’re interested in bin/setupvars.sh file, so let’s go ahead and source it to activate the environment:

source /opt/intel/openvino_2020.1.023/bin/setupvars.sh
[setupvars.sh] OpenVINO environment initialized

If you see [setupvars.sh] OpenVINO environment initialized then your environment should be initialized correctly

Run myriad_compile

$MYRIAD_COMPILE -m ~/open_model_zoo_downloads/intel/face-detection-retail-0004/FP16/face-detection-retail-0004.xml -ip U8 -VPU_MYRIAD_PLATFORM VPU_MYRIAD_2480 -VPU_NUMBER_OF_SHAVES 4 -VPU_NUMBER_OF_CMX_SLICES 4

You should see:

Inference Engine: 
	API version ............ 2.1
	Build .................. 37988
	Description ....... API
Done

Where’s the blob file? It’s located in the same folder as face-detection-retail-0004.xml:

ls -lh ~/open_model_zoo_downloads/intel/face-detection-retail-0004/FP16/
total 2.6M
-rw-r--r-- 1 root root 1.2M Jul 28 12:40 face-detection-retail-0004.bin
-rw-r--r-- 1 root root 1.3M Jul 28 12:50 face-detection-retail-0004.blob
-rw-r--r-- 1 root root 100K Jul 28 12:40 face-detection-retail-0004.xml

Create the blob config file

The MyriadX needs both a blob file (which we just created) and a blob_file_config in JSON format. We’ll create this config file manually. In your terminal:

cd ~/open_model_zoo_downloads/intel/face-detection-retail-0004/FP16/
touch face-detection-retail-0004.json

Copy and paste the following into face-detection-retail-0004.json:

{
    "tensors":
    [
        {
            "output_tensor_name": "out",
            "output_dimensions": [1, 1, 100, 7],
            "output_entry_iteration_index": 2,
            "output_properties_dimensions": [3],
            "property_key_mapping":
            [
                [],
                [],
                [],
                ["id", "label", "confidence", "left", "top", "right", "bottom"]
            ],
            "output_properties_type": "f16"
        }
    ]
}

What do these values mean?

  • output_dimensions - The neural net has an output format of [1, 1, N, 7] where N is the number of detected bounding boxes. Each detection has the format: ["id", "label", "confidence", "left", "top", "right", "bottom"]. We specify 100 instead of N to indicate we’ll handle up to 100 bounding boxes, which should be more than enough possible faces to detect.
  • output_entry_iteration_index - Indicates the array position we’ll use to iterate over detections. N is in the 2nd position (index start = 0).
  • property_key_mapping - Maps the properties to friendly names we can access when running model inference in our too-be-created Python script.
  • output_properties_type - We’re using a model with f16 precision.

Run and display the model output

With both blob and a json blob config file, we’re ready to roll! To verify that the model is running correctly, let’s modify a bit the program we’ve created in Hello World! tutorial

In particular, let’s change the create_pipeline invocation to load our model. Remember to replace the paths to correct ones that you have!

pipeline = depthai.create_pipeline(config={
    'streams': ['previewout', 'metaout'],
    'ai': {
-        'blob_file': consts.resource_paths.blob_fpath,
+        'blob_file': "/root/open_model_zoo_downloads/intel/face-detection-retail-0004/FP16/face-detection-retail-0004.bin",
-        'blob_file_config': consts.resource_paths.blob_config_fpath
+        'blob_file_config': "/root/open_model_zoo_downloads/intel/face-detection-retail-0004/FP16/face-detection-retail-0004.json"
    }
})

And that’s all!

You should see output annotated output similar to:

model image

Substitute your face for mine, of course.

Reviewing the flow

The flow we walked through works for other pre-trained object detection models in the Open Model Zoo:

  1. Download the model:
     $OPEN_MODEL_DOWNLOADER --name [INSERT MODEL NAME] --output_dir ~/open_model_zoo_downloads/
    
  2. Create the MyriadX blob file:
     $MYRIAD_COMPILE -m [INSERT PATH TO MODEL XML FILE] -ip U8 -VPU_MYRIAD_PLATFORM VPU_MYRIAD_2480 -VPU_NUMBER_OF_SHAVES 4 -VPU_NUMBER_OF_CMX_SLICES 4
    
  3. Create a JSON config file based on the model output.
  4. Use this model in your script

You’re on your way! You can find the complete code for this tutorial on GitHub.