CloudSoftwareHardware
Platform

ON THIS PAGE

  • Frame Synchronization

Frame Synchronization

There are 2 way to synchronize messages from different sensors (frames, IMU packet, ToF, etc.);
Hardware syncing
Software syncing

Hardware syncing

Allows precise synchronization (< 10µs) across multiple camera sensors and potentially with other hardware, e.g. flash LED, external IMU, or other cameras.

FSYNC signal

FSYNC/FSIN (frame sync) signal is a pulse that is driven high at the start of each frame capture. Its length is not proportional to the exposure time. It can be either input or output. It operates in 1.8V logic.On stereo cameras (OAK-D*), we want stereo camera pair (monochrome cameras) to be perfectly in sync, so one camera sensor (eg. left) has FSYNC set to INPUT, while the other camera sensor (eg. right) has FSYNC set to OUTPUT. In such configuration the right camera drives left camera.

Synchronizing frames externally

If we would like to drive cameras with an outside signal, we would need to set FSIN as INPUT for camera sensors.All Series 2 OAK PoE models have an M8 I/O connector which exposes FSIN signal (and also STROBE). We have developed FSYNC Y-Adapter that allows you to sync (daisy-chain) multiple OAK cameras together.
Py
1# Example: we have 3 cameras on ports A,B, and C
2cam_A.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
3cam_B.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
4cam_C.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
You can also control FSIN line via GPIO from within Script, see example here.

Sensor FSYNC support

As noted above the paragraph, only some sensors support FSYNC syncing. There are 2 types of FSYNC syncing:
  • Continuous streaming with external syncing, configured with CameraControl.setFrameSyncMode(). In this mode, the FSIN signal is expected to arrive at a continuous rate matching the configured sensor FPS, and trigger can't arrive at arbitrary times as that would disrupt internal sensor operations (leading to bad frames, etc). It can only correct for very small amounts of drift over time.
  • Snapshot mode with external syncing, configured with CameraControl.setExternalTrigger(). In this mode, trigger can arrive to the sensor at any time, and the sensor will take the photo/snapshot.
SensorShutterSupport
OV9282, OV9782GlobalFSYNC input/output, both continuous mode and snapshot (arbitrary external trigger) supported
OV7251GlobalShould have the same hardware support as OV9*82, but not implemented in FW as of now
AR0234GlobalFSYNC input, both continuous and snapshot mode supported
IMX378, IMX477, IMX577, IMX380RollingFSYNC input, only continuous mode supported for rolling shutter sensors. Hardware also supports FSYNC output, but not implemented in FW yet
IMX582RollingSimilar to IMX378, but not yet tested
IMX296 (RPi GS Camera)GlobalArbitrary external trigger supported on XTR/XTRIG pin. Pulse length determines exposure time (sensor feature). Global Arbitrary external trigger supported on XTR/XTRIG pin. Pulse length determines exposure time (sensor feature).

External FSYNC Example

Older devices
Newer devices

Older devices

Here's an example of how to use external FSYNC signal to trigger camera sensors. You can use any Series 2 OAK-D PoE model to trigger the FSYNC. We used M8 breakout board to expose the GND/FSYNC lines.
In this example (script here), sensors were set to Snapshot mode, as we were triggering the signal with a switch button. Only stereo cameras (2x OV9282) were triggered by the button, as IMX378 color camera does not support snapshot mode. If we were to use OV9782 color camera, it could be triggered by the button as well.

Strobe signal

STROBE signal is an output from the image sensor, and is active (high) during the exposure of the image sensor. It would be used to eg. drive an external LED lighting for illumination - so lighting would only be active during exposure times, instead constantly on, which would decrease power consumption and heating of the lighting.We have used STROBE signal on Pro version of OAK cameras (which have on-board illumination IR LED and IR laser dot projector) to drive the laser/LED.

Strobe demo

Cameras with M8 connector allow you to connect external lighting to the STROBE signal, as demonstrated in the video below (blog post here):

Frame capture graphs

Frame timestamp is assigned to the frame at the MIPI SoF (start of frame) event, when the sensor starts streaming the frame (MIPI readout).For global shutter sensors, this follows immediately after the exposure for the whole frame was finished, so we can say the timestamp assigned is aligned with end of exposure-window (within a margin of few microseconds). Here's an example graph of the global shutter sensor timings, which demonstrates when timestamp is assigned to the frame:

Global shutter sensor timings

For rolling shutter, the example graph looks a bit different. MIPI SoF follows after the first row of the image was fully exposed and it's being streamed, but the following rows are still exposing or may have not started exposing yet (depending on exposure time).

Rolling shutter sensor timings

Below there's an example graph of rolling shutter sensor (IMX378) at 1080p and 30fps (33.3ms frame time). MIPI readout time varies between sensors/resolutions, but for IMX378 it's 16.54ms at 1080P, 23.58ms at 4K, and 33.04ms at 12MP.

OAK-FFC hardware syncing

On OAK-FFC-4P, we have 4 camera ports; A (rgb), B (left), C (right), and D (cam_d). A & D are 4-lane MIPI, and B & C are 2-lane MIPI. Each pair (A&D and B&C) share an I2C bus, and the B&C bus is configured for HW syncing left+right cameras by default.For A&D ports, you need to explicitly enable hardware syncing:
Py
1cam_A.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT)
2cam_D.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

Arducam FFC camera syncing

Arducam FFC cameras have a 22-pin connector, which don't have lines for FSIN/STROBE. As seen below, to connect Arducam FFC camera to our OAK-FFC baseboard you need to use 26-to-22 pin converter connector which only exposes STROBE/FSIN lines via test pads. To sync these cameras, you could either solder a wire from test pad to the camera module's FSIN header pin, or connect all FSIN header pins together, as done here.

Connecting FSIN/STROBE

As mentioned, all Series 2 OAK PoE models have an M8 I/O connector with FSYNC/STROBE signal. But if you won't be using these, you will likely need to solder a wire to the PCB on your device. Most PCB designs are open-source (on depthai-hardware repository), so you can easily check where FSIN/STROBE signals are on the PCB.

OAK-FFC-4P FSIN

As shown on image above, on OAK-FFC-4P you can enable connection of FSIN_4LANE and FSIN_2LANE with the MXIO6. The script below will sync together all 4 cameras that are connected to the OAK-FFC-4P.
Python
1# CAM_A will drive FSIN signal for all other cameras:
2cam_A.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) # 4LANE
3cam_B.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT) # 2LANE
4cam_C.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) # 2LANE
5cam_D.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) # 4LANE
6
7# AND importantly to tie the FSIN signals of A+D and B+C pairs, by setting a GPIO:
8# OAK-FFC-4P requires driving MXIO6 high (FSIN_MODE_SELECT) to connect together
9# the A+D FSIN group (4-lane pair) with the B+C group (2-lane pair)
10config = dai.Device.Config()
11config.board.gpio[6] = dai.BoardConfig.GPIO(dai.BoardConfig.GPIO.OUTPUT,
12                                            dai.BoardConfig.GPIO.Level.HIGH)
13
14with dai.Device(config) as device:
15    device.startPipeline(pipeline)
Additional info can be found in this forum discussion.

Series 2 USB OAKs

FSIN lines on DM9098 board (OAK-D S2, OAK-D W, OAK-D Pro, OAK-D Pro W):

USB OAK-1* FSIN

FSIN test pad on NG9093 board (OAK-1, OAK-1 W, OAK-1 Lite, OAK-1 Lite W, OAK-1 Max):

OAK-D-Lite FSIN

Note that stereo camera pair and color cameras aren't connected together.