Getting started with OAK IoT devices¶

Note

This tutorial is intended for OAK-D-IoT-75 and OAK-D-IoT-40, which both have an ESP32 on the board.

Code examples¶

Code examples can be found in our esp32-spi-message-demo github repository.

Overview¶

Overview explained:

Robotics Vision Core 2 (RVC2) is the SoC+circuitry on the OAK camera, where you can run your pipeline
Robotics Vision Core 2 (RVC2) is connected to the host (eg. PC)
Robotics Vision Core 2 (RVC2) can communicate Messages with the ESP32 via SPI (using SPIOut / SPIOut node)
ESP32 can receive these messages using the depthai-spi-api library (which is an ESP-IDF component).
On ESP32 you can run post-processing of the messages and optionally send the results to a server (if connected to a WiFi network) or to a Bluetooth device (eg. a smartphone)

Usually, developers would develop their solution with the device connected via USB-C, and use potentially use OTA updates via ESP32 in production.

Limitations¶

Currently, the bottleneck for some use cases is the SPI throughput, which is about 1MB/s (8Mbps). This means that you can stream frames over the SPI, just not high-resolution / unencoded streams.

As of now we only support ESP-IDF as the development framework. In the future, we might support Arduino.

Common use cases for the ESP32¶

TL;DR:

Sending metadata results (spatial coordinates, NN results, tracklets etc.)
OTA updates
Capturing and streaming video/images
System information logging
Sending data to the cloud (eg. AWS/Azure/GCP or any other IOT platform)
Sending data to a Bluetooth device (eg. smartphone)

Explained in more detail:

You could run spatial object detections and/or object tracking pipeline on the VPU and send tracklets with spatial data over the SPI to the ESP32. On ESP32 you could run some simple filtering and/or NN result decoding and then send final results to the cloud
ESP32 can also flash DepthAI bootloader and/or pipeline, which means OTA (over-the-air) updates are supported.
We have an example for both sending a whole image and part of an image through the SPI from the VPU to the ESP32
Pipeline on the VPU could be configured to send system information (using SystemLogger node) to the ESP32 which would forward them to a logging platform
ESP32 also has Bluetooth capabilities, so you could forward the data from the ESP32 to a smartphone

How to get started with the development¶

Install the ESP-IDF, instructions here.
After setting up the environmental variables, you can build any demo from the esp32-spi-message-demo repository with idf.py build.
After building, you can flash your ESP32 using idf.py -p PORT flash monitor (replace PORT with the ESP32 port, eg. /dev/ttyUSB0). You might need to change the permission of the port with sudo chmod 777 PORT so idf.py can access it.
After flashing the ESP32, you can start the pipeline. If you have used a demo ESP32 code, you should run the corresponding python script that’s inside the demo folder.

Got questions?

Head over to Discussion Forum for technical support or any other questions you might have.