On-device programming

While regular (firmware) on-device development is not possible due to closed nature of native tooling, we still expose a couple of alternative ways of running custom code:

  1. Scripting - Using Python3.9 with Script node

  2. Creating your own NN model to run more computationally heavy features

  3. Creating custom OpenCL kernels

Using Script node

Using Script node allows you to run custom python scripts on the device itself, which allows users greater flexibility when constructing pipelines.

Script node is also very useful when using multiple neural networks in series and you need to process the output of the 1st one before feeding an image to the second one. Example here would be face age/gender recognition demo - first NN would detect faces, pass detections to the Script node which would create ImageManip configurations to crop the original frame and feed the face age/gender recognition NN only the cropped face frame.

For running computationally heavy features (eg. image filters), due to performance reasons you might want to avoud using Script node and rather go with one of the 2 options described below.

Creating custom NN models

You can create custom models with your favourite NN library, convert the model into OpenVINO and then compile it into the .blob. More information on this topic can be found on Converting model to MyriadX blob documentation.

Here are two implementations of this workflow:

Creating custom OpenCL kernel

Creating custom NN models has some limitations, for example unsupported layers by OpenVINO/VPU. To avoid these limitations, you could consider creating custom OpenCL kernel and compile it for the VPU. This kernel will run on SHAVE core(s) on the VPU. One should also take into the account that this option is not super user friendly. We plan on creating a tutorial on how to develop these and run them on OAK cameras.