IMU and Video Sync

This example demonstrates the use of the DepthAI Sync node to synchronize IMU (Inertial Measurement Unit) data with video frames from a color camera. It highlights the capability to process and display the latest rotation vector from the IMU alongside the video stream in real-time.

Similar samples:

Demo

Command Line

1~/depthai-python/examples/Sync $ python3 imu_video_synced.py
2IMU type: BNO086, firmware version: 3.9.7
3
4Device timestamp imu: 0:00:05.379914
5Device timestamp video:0:00:05.385096
6Quaternion: i: -0.0549 j: -0.0335 k: 0.0018 real: 0.9979
7
8
9Device timestamp imu: 0:00:05.410274
10Device timestamp video:0:00:05.418425
11Quaternion: i: -0.0549 j: -0.0334 k: 0.0018 real: 0.9979
12
13
14Device timestamp imu: 0:00:05.445439
15Device timestamp video:0:00:05.451753
16Quaternion: i: -0.0548 j: -0.0334 k: 0.0018 real: 0.9979
17
18
19Device timestamp imu: 0:00:05.475084
20Device timestamp video:0:00:05.485082
21Quaternion: i: -0.0547 j: -0.0334 k: 0.0018 real: 0.9979
22
23
24Device timestamp imu: 0:00:05.510046
25Device timestamp video:0:00:05.518411
26Quaternion: i: -0.0546 j: -0.0334 k: 0.0018 real: 0.9979

Setup

Please run the install script to download all required dependencies. Please note that this script must be ran from git context, so you have to download the depthai-python repository first and then run the script

Command Line

1git clone https://github.com/luxonis/depthai-python.git
2cd depthai-python/examples
3python3 install_requirements.py

For additional information, please follow the installation guide.

Source code

Python

C++

Python

GitHub

1import depthai as dai
2import numpy as np
3import cv2
4from datetime import timedelta
5
6device = dai.Device()
7
8imuType = device.getConnectedIMU()
9imuFirmwareVersion = device.getIMUFirmwareVersion()
10print(f"IMU type: {imuType}, firmware version: {imuFirmwareVersion}")
11
12if imuType != "BNO086":
13    print("Rotation vector output is supported only by BNO086!")
14    exit(0)
15
16pipeline = dai.Pipeline()
17
18color = pipeline.create(dai.node.ColorCamera)
19imu = pipeline.create(dai.node.IMU)
20sync = pipeline.create(dai.node.Sync)
21xoutImu = pipeline.create(dai.node.XLinkOut)
22xoutImu.setStreamName("imu")
23
24xoutGrp = pipeline.create(dai.node.XLinkOut)
25xoutGrp.setStreamName("xout")
26
27color.setCamera("color")
28
29imu.enableIMUSensor(dai.IMUSensor.ROTATION_VECTOR, 120)
30imu.setBatchReportThreshold(1)
31imu.setMaxBatchReports(10)
32
33sync.setSyncThreshold(timedelta(milliseconds=10))
34sync.setSyncAttempts(-1)
35
36color.video.link(sync.inputs["video"])
37imu.out.link(sync.inputs["imu"])
38
39sync.out.link(xoutGrp.input)
40
41
42with device:
43    device.startPipeline(pipeline)
44    groupQueue = device.getOutputQueue("xout", 3, True)
45    while True:
46        groupMessage = groupQueue.get()
47        imuMessage = groupMessage["imu"]
48        colorMessage = groupMessage["video"]
49        print()
50        print("Device timestamp imu: " + str(imuMessage.getTimestampDevice()))
51        print("Device timestamp video:" + str(colorMessage.getTimestampDevice()))
52        latestRotationVector = imuMessage.packets[-1].rotationVector
53        imuF = "{:.4f}"
54        print(f"Quaternion: i: {imuF.format(latestRotationVector.i)} j: {imuF.format(latestRotationVector.j)} "
55        f"k: {imuF.format(latestRotationVector.k)} real: {imuF.format(latestRotationVector.real)}")
56        print()
57        cv2.imshow("video", colorMessage.getCvFrame())
58        if cv2.waitKey(1) == ord("q"):
59            break

How it Works

Initialize the DepthAI device.
Check the connected IMU type and firmware version.
Create a pipeline and add a ColorCamera and IMU node.
Set up the Sync node to synchronize the IMU data with the video frames.
Link the output of the ColorCamera and IMU nodes to the Sync node.
Start the pipeline and continuously receive synchronized data.
Display the video frames and print the IMU rotation vector data, including quaternion values.

Need assistance?

Head over to Discussion Forum for technical support or any other questions you might have.

ON THIS PAGE