Our platform supports computer vision (CV) functions to be performed on the device itself. While you can't run OpenCV, you can use many of its supported functions. With DepthAI, you can:
Crop, rotate, warp/dewarp, mirror, flip, transform perspective, etc. with ImageManip
Track objects (Kalman filter, Hungarian algorithm) with ObjectTracker Out-of-the-box support for Yolo and MobileNet object detectors.
Perceive stereo depth (Census Tranform, Cost Matching and Aggregation) with StereoDepth
If you would like to use any other CV functions, see below guide on how to implement and run CV functions efficiently on the device's hardware-accelerated blocks.
For the sake of this guide, we will create a simple model that concatenates three frames into one. This is a simple example, but you can use the same procedure to create more complex models.
We first need to create a Python class that extends PyTorch's nn.Module. We can then put our NN logic into the forward function of the created class. In the example of frame concatenation, we can use torch.cat function to concatenate multiple frames:
For a more complex module, please refer to Harris corner detection in PyTorch demo by Kunal Tyagi.Keep in mind that VPU supports only FP16, which means that max value is 65504. When multiplying a few values you can quickly overflow if you don't properly normalize/divide values.
Since PyTorch isn't directly supported by OpenVINO, we first need to export the model to onnx format and then to OpenVINO. PyTorch has integrated support for onnx, so exporting to onnx is as simple as:
Python
1# For 300x300 frames2X = torch.ones((1,3,300,300), dtype=torch.float32)3torch.onnx.export(4 CatImgs(),5(X, X, X),# Dummy input for shape6"path/to/model.onnx",7 opset_version=12,8 do_constant_folding=True,9)
This will export the concatenate model into onnx format. We can visualize the created model using Netron app:
When exporting the model to onnx, PyTorch isn't very efficient. It creates tons of unnecessary operations/layers which increase the size of your network (which can lead to lower FPS). That's why we recommend using onnx-simplifier, a simple python package that removes unnecessary operations/layers.
Here is an example of how significant the simplification was using the onnx-simplifier. On the left, there's a blur model (from Kornia) exported directly from PyTorch, and on the right, there's a simplified network of the same functionality:
Now that we have a (simplified) onnx model, we can convert it to OpenVINO and then to the .blob format. For additional information about converting models, see conversion guide.This would usually be done first by using OpenVINO's model optimizer to convert from onnx to IR format (.bin/.xml) and then using Compile tool to compile to .blob. But we could also use blobconverter to convert from onnx directly to .blob.Blobconverter just does both of these steps at once - without the need of installing OpenVINO. You can compile your onnx model like this:
Kornia, "State-of-the-art and curated Computer Vision algorithms for AI.", has a set of common computer vision algorithms implemented in PyTorch. This allows users to do something similar to:
During our testing, we have found that several algorithms aren't supported by either the OpenVINO framework or by the VPU. We have submitted an Issue for Sobel filter already.
