# Frame Synchronization There are 2 way to synchronize messages from different sensors (frames, IMU packet, ToF, etc.); * [Hardware syncing](#Hardware%20syncing): multi-sensor sub-ms accuracy, hardware trigger * [Software syncing](#Software%20syncing): based on timestamp/sequence numbers #### Software syncing There are two approaches for software syncing: * [Sequence number syncing](#Frame%2520Synchronization-Sequence%2520number%2520syncing) - for streams set to the same FPS, sub-ms accuracy can be achieved * [Timestamp syncing](#Frame%2520Synchronization-Timestamp%2520syncing) - for streams with different FPS, syncing with other sensors either onboard (eg. IMU) or also connected to the host computer (eg. USB ToF sensor) ### Sequence number syncing If we want to synchronize multiple messages from the same OAK, such as: * Camera frames from [ColorCamera](https://docs.luxonis.com/software/depthai-components/nodes/color_camera.md#colorcamera) or [MonoCamera](https://docs.luxonis.com/software/depthai-components/nodes/mono_camera.md#monocamera) (color, left and right frames) * Messages generated from camera frames (NN results, disparity/depth, edge detections, tracklets, encoded frames, tracked features, etc.) We can use sequence number syncing, [demos here](https://github.com/luxonis/oak-examples/tree/master/gen2-syncing#message-syncing). Each frame from ColorCamera/MonoCamera will get assigned a sequence number, which then also gets copied to message generated from that frame. For sequence number syncing FPS of all cameras need to be the same. On host or inside script node you can get message's sequence number like this: ```py # Get the message from the queue message = queue.get() # message can be ImgFrame, NNData, Tracklets, ImgDetections, TrackedFeatures... seqNum = message.getSequenceNum() ``` Through firmware sync, we're monitoring for drift and aligning the capture timestamps of all cameras (left, right, color), which are taken at the MIPI Start-of-Frame (SoF) event. The Left/Right global shutter cameras are driven by the same clock, started by broadcast write on I2C, so no drift will happen over time, even when running freely without a hardware sync. The RGB rolling shutter has a slight difference in clocking/frame-time, so when we detect a small drift, we're modifying the frame-time (number of lines) for the next frame by a small amount to compensate. If sensors are set to the same FPS (default is 30), the above two approaches are already integrated into depthai and enabled by default, which allows us to achieve sub-ms delay between all frames + messages generated by these frames! ```bash [Seq 325] RGB timestamp: 0:02:33.549449 [Seq 325] Disparity timestamp: 0:02:33.549402 ----------- [Seq 326] RGB timestamp: 0:02:33.582756 [Seq 326] Disparity timestamp: 0:02:33.582715 ----------- [Seq 327] RGB timestamp: 0:02:33.616075 [Seq 327] Disparity timestamp: 0:02:33.616031 ``` Disparity and color frame timestamps indicate that we achieve well below sub-ms accuracy. ### Timestamp syncing As opposed to sequence number syncing, timestamp syncing can sync: * streams with different FPS * IMU results with other messages * messages with other devices connected to the computer, as timestamps are [synced to the host computer clock](https://docs.luxonis.com/software-v3/depthai/depthai-components/device.md) DepthAI 2.24 introduces Sync node which can be used to sync messages from different streams, or messages from different sensors (eg. IMU and color frames). See [Sync node](https://docs.luxonis.com/software/depthai-components/nodes/sync.md) for more details. The sync node does not currently support multiple device syncing, so if you want to sync messages from multiple devices, you should use the manual approach. Feel free to check the [demo here](https://github.com/luxonis/oak-examples/tree/master/gen2-syncing#imu--rgb--depth-timestamp-syncing) which uses timestamps to sync IMU, color and disparity frames together, with all of these streams producing messages at different FPS. In case of multiple streams having different FPS, there are 2 options on how to sync them: 1. Removing some messages from faster streams to get the synced FPS of the slower stream 2. Duplicating some messages from slower streams to get the synced FPS of the fastest stream Timestamps are assigned to the frame at the MIPI Start-of-Frame (SoF) events, [more details here](https://docs.luxonis.com/hardware/platform/deploy/frame-sync.md). ```py # Get the message from the queue message = queue.get() timestamp = message.getTimestamp() # Timestamp synced with the host computer clock # If message is ImgFrame, you can select start/mid/end of frame exposure # Can also use .START or .END imgFrame.getTimestamp(dai.CameraExposureOffset.MIDDLE) ``` #### Hardware syncing Allows precise synchronization (< 10µs) across multiple camera sensors and potentially with other hardware, e.g. flash LED, external IMU, or other cameras. ## FSYNC signal FSYNC/FSIN (frame sync) signal is a pulse that is driven high at the start of each frame capture. Its length is not proportional to the exposure time. It can be either input or output. It operates in 1.8V logic. On stereo cameras (OAK-D*), we want stereo camera pair (monochrome cameras) to be perfectly in sync, so one camera sensor (eg. left) has FSYNC set to INPUT, while the other camera sensor (eg. right) has FSYNC set to OUTPUT. In such configuration the right camera drives left camera. > **FSYNC signal output** > At the moment, only OV9282/OV9782 can output FSYNC signal, while IMX378/477/577/etc should also have the capability, but isn't yet supported (so these can not drive FSYNC signal, only be driven by it). AR0234 has input-only FSYNC trigger. ### Synchronizing frames externally If we would like to drive cameras with an outside signal, we would need to set FSIN as INPUT for camera sensors. All [Series 2 OAK PoE models](https://docs.luxonis.com/hardware.md) have an M8 I/O connector which exposes FSIN signal (and also STROBE). We have developed [FSYNC Y-Adapter](https://docs.luxonis.com/hardware/products/FSYNC%2520Y-Adapter.md) that allows you to sync (daisy-chain) multiple OAK cameras together. ```py # Example: we have 3 cameras on ports A,B, and C cam_A.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) cam_B.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) cam_C.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) ``` You can also control FSIN line via GPIO from within [Script](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/script.md), see example [here](https://gist.github.com/Erol444/a9189a8215371ff9f4cf4472960e1d66). ### Sensor FSYNC support As noted above the paragraph, only some sensors support FSYNC syncing. There are 2 types of FSYNC syncing: * Continuous streaming with external syncing, configured with CameraControl.setFrameSyncMode(). In this mode, the FSIN signal is expected to arrive at a continuous rate matching the configured sensor FPS, and trigger can't arrive at arbitrary times as that would disrupt internal sensor operations (leading to bad frames, etc). It can only correct for very small amounts of drift over time. * Snapshot mode with external syncing, configured with CameraControl.setExternalTrigger(). In this mode, trigger can arrive to the sensor at any time, and the sensor will take the photo/snapshot. | Sensor | Shutter | Support | | --- | --- | --- | | [OV9282](https://docs.luxonis.com/hardware/sensors/OV9282.md), [OV9782](https://docs.luxonis.com/hardware/sensors/OV9782.md) | Global | FSYNC input/output, both continuous mode and snapshot (arbitrary external trigger) supported | | [OV7251](https://docs.luxonis.com/hardware/sensors/OV7251.md) | Global | Should have the same hardware support as OV9*82, but not implemented in FW as of now | | [AR0234](https://docs.luxonis.com/hardware/sensors/AR0234.md) | Global | FSYNC input, both continuous and snapshot mode supported | | [IMX378](https://docs.luxonis.com/hardware/sensors/IMX378.md), [IMX477](https://docs.luxonis.com/hardware/sensors/IMX477.md), [IMX577](https://docs.luxonis.com/hardware/sensors/IMX577.md), IMX380 | Rolling | FSYNC input, only continuous mode supported for rolling shutter sensors. Hardware also supports FSYNC output, but not implemented in FW yet | | [IMX582](https://docs.luxonis.com/hardware/sensors/IMX582.md) | Rolling | Similar to IMX378, but not yet tested | | IMX296 (RPi GS Camera) | Global | Arbitrary external trigger supported on XTR/XTRIG pin. Pulse length determines exposure time (sensor feature). Global Arbitrary external trigger supported on XTR/XTRIG pin. Pulse length determines exposure time (sensor feature). | ### External FSYNC Example #### Older devices Here's an example of how to use external FSYNC signal to trigger camera sensors. You can use any Series 2 OAK-D PoE model to trigger the FSYNC. We used M8 breakout board to expose the GND/FSYNC lines. In this example ([script here](https://gist.github.com/Erol444/0138af63378dc8de5b3f7d80db1ea1a5)), sensors were set to Snapshot mode, as we were triggering the signal with a switch button. Only stereo cameras (2x OV9282) were triggered by the button, as IMX378 color camera does not support snapshot mode. If we were to use OV9782 color camera, it could be triggered by the button as well. #### Newer devices Devices that use the new M8 connector (like the [OAK-D ToF](https://docs.luxonis.com/hardware/products/OAK-D%2520ToF.md)) will expect a 5V trigger signal. By default, pin 2 on the M8 connector (FSYNC) is pulled high internally. When the pin is pulled high, it will stream frames continuously (when camera is either in INPUT or OUTPUT mode). To trigger frames externally, you will need to pull the pin low (eg. to pin 8 - GND) to stop the frame streaming, then pull it high again to start streaming. This way you can send 5V pulses to the FSYNC pin to trigger the frame capture. > **FSYNC Triggering** > Only global shutter (OV9282, OV9782, AR0234...) cameras support FSYNC triggering in photo/snapshot mode. Rolling shutter cameras (IMX378, IMX477, IMX577, etc) don't support it. ### Strobe signal STROBE signal is an output from the image sensor, and is active (high) during the exposure of the image sensor. It would be used to eg. drive an external LED lighting for illumination - so lighting would only be active during exposure times, instead constantly on, which would decrease power consumption and heating of the lighting. We have used STROBE signal on Pro version of OAK cameras (which have on-board illumination IR LED and IR laser dot projector) to drive the laser/LED. ### Strobe demo Cameras with M8 connector allow you to connect external lighting to the STROBE signal, as demonstrated in the video below ([blog post here](https://discuss.luxonis.com/blog/5491-external-strobe-illumination-with-oak-camera)): ### Frame capture graphs Frame timestamp is assigned to the frame at the MIPI SoF (start of frame) event, when the sensor starts streaming the frame (MIPI readout). For global shutter sensors, this follows immediately after the exposure for the whole frame was finished, so we can say the timestamp assigned is aligned with end of exposure-window (within a margin of few microseconds). Here's an example graph of the global shutter sensor timings, which demonstrates when timestamp is assigned to the frame: #### Global shutter sensor timings For rolling shutter, the example graph looks a bit different. MIPI SoF follows after the first row of the image was fully exposed and it's being streamed, but the following rows are still exposing or may have not started exposing yet (depending on exposure time). #### Rolling shutter sensor timings Below there's an example graph of rolling shutter sensor (IMX378) at 1080p and 30fps (33.3ms frame time). MIPI readout time varies between sensors/resolutions, but for IMX378 it's 16.54ms at 1080P, 23.58ms at 4K, and 33.04ms at 12MP. ### OAK-FFC hardware syncing On [OAK-FFC-4P](https://docs.luxonis.com/hardware/products/OAK-FFC%25204P.md), we have 4 camera ports; A (rgb), B (left), C (right), and D (cam_d). A & D are 4-lane MIPI, and B & C are 2-lane MIPI. Each pair (A&D and B&C) share an I2C bus, and the B&C bus is configured for HW syncing left+right cameras by default. For A&D ports, you need to explicitly enable hardware syncing: ```py cam_A.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT) cam_D.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) ``` ### Arducam FFC camera syncing Arducam FFC cameras have a 22-pin connector, which don't have lines for FSIN/STROBE. As seen below, to connect Arducam FFC camera to our OAK-FFC baseboard you need to use 26-to-22 pin converter connector which only exposes STROBE/FSIN lines via test pads. To sync these cameras, you could either solder a wire from test pad to the camera module's FSIN header pin, or connect all FSIN header pins together, as done [here](https://discuss.luxonis.com/d/934-ffc-4p-hardware-synchronization/3). ### Connecting FSIN/STROBE As mentioned, all [Series 2 OAK PoE models](https://docs.luxonis.com/hardware.md) have an M8 I/O connector with FSYNC/STROBE signal. But if you won't be using these, you will likely need to solder a wire to the PCB on your device. Most PCB designs are open-source (on [oak-hardware](https://github.com/luxonis/oak-hardware) repository), so you can easily check where FSIN/STROBE signals are on the PCB. ## OAK-FFC-4P FSIN As shown on image above, on [OAK-FFC-4P](https://docs.luxonis.com/hardware/products/OAK-FFC%25204P.md) you can enable connection of FSIN_4LANE and FSIN_2LANE with the MXIO6. The script below will sync together all 4 cameras that are connected to the OAK-FFC-4P. ```python # CAM_A will drive FSIN signal for all other cameras: cam_A.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) # 4LANE cam_B.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT) # 2LANE cam_C.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) # 2LANE cam_D.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) # 4LANE # AND importantly to tie the FSIN signals of A+D and B+C pairs, by setting a GPIO: # OAK-FFC-4P requires driving MXIO6 high (FSIN_MODE_SELECT) to connect together # the A+D FSIN group (4-lane pair) with the B+C group (2-lane pair) config = dai.Device.Config() config.board.gpio[6] = dai.BoardConfig.GPIO(dai.BoardConfig.GPIO.OUTPUT, dai.BoardConfig.GPIO.Level.HIGH) with dai.Device(config) as device: device.startPipeline(pipeline) ``` Additional info can be found in [this forum discussion](https://discuss.luxonis.com/d/934-ffc-4p-hardware-synchronization/3). > **CAM_B and CAM_C** > **CAM_B and CAM_C** > (2-lane MIPI ports) share the same I2C bus, which means if both of them have the same sensor (FFC module) connected to them, they will be > **in sync regardless** > of any setting (as I2C command "start exporute" will arrive at the same time). This mechanism is used accross OAK-D cameras to sync stereo camera pair. ## Series 2 USB OAKs FSIN lines on DM9098 board ([OAK-D S2](https://docs.luxonis.com/hardware/products/OAK-D%2520S2.md), [OAK-D W](https://docs.luxonis.com/hardware/products/OAK-D%2520W.md), [OAK-D Pro](https://docs.luxonis.com/hardware/products/OAK-D%2520Pro.md), [OAK-D Pro W](https://docs.luxonis.com/hardware/products/OAK-D%2520Pro%2520W.md)): ## USB OAK-1* FSIN FSIN test pad on NG9093 board ([OAK-1](https://docs.luxonis.com/hardware/products/OAK-1.md), [OAK-1 W](https://docs.luxonis.com/hardware/products/OAK-1%2520W.md), [OAK-1 Lite](https://docs.luxonis.com/hardware/products/OAK-1%2520Lite.md), [OAK-1 Lite W](https://docs.luxonis.com/hardware/products/OAK-1%2520Lite%2520W.md), [OAK-1 Max](https://docs.luxonis.com/hardware/products/OAK-1%2520Max.md)): ## OAK-D-Lite FSIN Note that stereo camera pair and color cameras aren't connected together.