# Computer Vision

Our platform supports computer vision (CV) functions to be performed on the device itself. While you can't run
[OpenCV](https://github.com/opencv/opencv), you can use many of its supported functions. With DepthAI, you can:

 * Crop, rotate, warp/dewarp, mirror, flip, transform perspective, etc. with
   [ImageManip](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/image_manip.md)
 * Detect edges (Sobel filter) with
   [EdgeDetector](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/edge_detector.md)
 * Detect and track features with
   [FeatureTracker](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/feature_tracker.md)
 * Track objects (Kalman filter, Hungarian algorithm) with
   [ObjectTracker](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/object_tracker.md) Out-of-the-box support
   for Yolo and MobileNet object detectors.
 * Perceive stereo depth (Census Tranform, Cost Matching and Aggregation) with
   [StereoDepth](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/stereo_depth.md)

If you would like to use any other CV functions, see below guide on how to implement and run CV functions efficiently on the
device's hardware-accelerated blocks.

## Run your own CV functions on-device

Demos:

 * [Frame concatenation](https://github.com/luxonis/oak-examples/tree/master/gen2-custom-models/generate_model#concatenate-frames)
   - using [PyTorch](https://pytorch.org/)
 * [Laplacian edge detection](https://github.com/luxonis/oak-examples/tree/master/gen2-custom-models/generate_model#blur-frames) -
   using Kornia
 * [Frame blurring](https://github.com/luxonis/oak-examples/tree/master/gen2-custom-models/generate_model#corner-detection) -
   using Kornia
 * [Tutorial on running custom models on OAK](https://rahulrav.com/blog/depthai_camera.html) by Rahul Ravikumar
 * [Harris corner detection in PyTorch](https://github.com/kunaltyagi/pytorch_harris/) by Kunal Tyagi

## Create a custom model with PyTorch

For the sake of this guide, we will create a simple model that concatenates three frames into one. This is a simple example, but
you can use the same procedure to create more complex models.

### TL;DR

If you are only interested in the implementation

[TL;DR](https://github.com/luxonis/oak-examples/blob/master/gen2-custom-models/generate_model/pytorch_concat.py)

### Create PyTorch NN module

We first need to create a Python class that extends PyTorch's
[nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html). We can then put our NN logic into the forward
function of the created class. In the example of frame concatenation, we can use
[torch.cat](https://pytorch.org/docs/master/generated/torch.cat.html#torch-cat) function to concatenate multiple frames:

```python
class CatImgs(nn.Module):
    def forward(self, img1, img2, img3):
        return torch.cat((img1, img2, img3), 3)
```

For a more complex module, please refer to [Harris corner detection in PyTorch](https://github.com/kunaltyagi/pytorch_harris/)
demo by Kunal Tyagi.

Keep in mind that VPU supports only
[FP16](https://en.wikipedia.org/wiki/Half-precision_floating-point_format#Half_precision_examples), which means that max value is
65504. When multiplying a few values you can quickly overflow if you don't properly normalize/divide values.

### Export the NN module to onnx

Since PyTorch isn't directly supported by OpenVINO, we first need to export the model to [onnx format](https://onnx.ai/) and then
to OpenVINO. PyTorch has [integrated support for onnx](https://pytorch.org/docs/stable/onnx.html), so exporting to onnx is as
simple as:

```python
# For 300x300 frames
X = torch.ones((1, 3, 300, 300), dtype=torch.float32)
torch.onnx.export(
    CatImgs(),
    (X, X, X), # Dummy input for shape
    "path/to/model.onnx",
    opset_version=12,
    do_constant_folding=True,
)
```

This will export the concatenate model into onnx format. We can visualize the created model using [Netron
app](https://netron.app/):

### Simplify onnx model

When exporting the model to onnx, PyTorch isn't very efficient. It creates tons of unnecessary operations/layers which increase
the size of your network (which can lead to lower FPS). That's why we recommend using
[onnx-simplifier](https://github.com/daquexian/onnx-simplifier), a simple python package that removes unnecessary
operations/layers.

```python
import onnx
from onnxsim import simplify

onnx_model = onnx.load("path/to/model.onnx")
model_simplified, check = simplify(onnx_model)
onnx.save(model_simplified, "path/to/simplified/model.onnx")
```

Here is an example of how significant the simplification was using the onnx-simplifier. On the left, there's a blur model (from
Kornia) exported directly from PyTorch, and on the right, there's a simplified network of the same functionality:

### Convert to OpenVINO/blob

Now that we have a (simplified) onnx model, we can convert it to OpenVINO and then to the .blob format. For additional information
about converting models, see [conversion guide](https://docs.luxonis.com/software-v3/ai-inference/conversion.md).

This would usually be done first by using [OpenVINO's model
optimizer](https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html) to convert
from onnx to IR format (.bin/.xml) and then using [Compile
tool](https://docs.openvinotoolkit.org/latest/openvino_inference_engine_tools_compile_tool_README.html) to compile to .blob. But
we could also use blobconverter to convert from onnx directly to .blob.

Blobconverter just does both of these steps at once - without the need of installing OpenVINO. You can compile your onnx model
like this:

```python
import blobconverter

blobconverter.from_onnx(
    model="/path/to/model.onnx",
    output_dir="/path/to/output/model.blob",
    data_type="FP16",
    shaves=6,
    use_cache=False,
    optimizer_params=[]
)
```

> **Online blobconverter**
> You can also use the
> [online blobconverter](https://blobconverter.luxonis.com/)
> to convert your model.

### Use the .blob in your pipeline

You can now use your .blob model with the
[NeuralNetwork](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/neural_network.md) node. Check
[oak-examples/custom-models](https://github.com/luxonis/oak-examples/tree/master/gen2-custom-models) to run the demo applications
that use these custom models.

## Kornia

[Kornia](https://kornia.readthedocs.io/en/latest/), "State-of-the-art and curated Computer Vision algorithms for AI.", has a set
of common computer vision algorithms implemented in PyTorch. This allows users to do something similar to:

```python
import kornia

class Model(nn.Module):
    def forward(self, image):
        return kornia.filters.gaussian_blur2d(image, (9, 9), (2.5, 2.5))
```

and use the exact same procedure as described in [Create a custom model with
PyTorch](#Computer%2520Vision-Create%2520a%2520custom%2520model%2520with%2520PyTorch) to achieve [frame
blurring](https://github.com/luxonis/oak-examples/blob/master/gen2-custom-models/generate_model/kornia_blur.py), as shown below:

> **Algorithm support**
> During our testing, we have found that
> **several algorithms aren't supported**
> by either the OpenVINO framework or by the VPU. We have submitted an
> [Issue](https://github.com/openvinotoolkit/openvino/issues/7557)
> for
> [Sobel filter](https://kornia.readthedocs.io/en/latest/filters.html?highlight=sobel#kornia.filters.Sobel)
> already.