StereoDepth

How to place it

1pipeline = dai.Pipeline()
2stereo = pipeline.create(dai.node.StereoDepth)

Inputs and Outputs

• Debug outputs:
  • debugDispLrCheckIt1: ImgFrame
  • debugDispLrCheckIt2: ImgFrame
  • debugExtDispLrCheckIt1: ImgFrame
  • debugExtDispLrCheckIt2: ImgFrame
  • debugDispCostDump: ImgFrame
  • confidenceMap: ImgFrame

Internal block diagram of StereoDepth node

Currently configurable blocks

• Stereo Mode:
  • Left-Right Check:
    • Left-Right Check or LR-Check is used to remove incorrectly calculated disparity pixels due to occlusions at object borders (Left and Right camera views are slightly different).
      1. Computes disparity by matching in R->L direction.
      2. Computes disparity by matching in L->R direction.
      3. Combines results from steps 1 and 2, running on Shave: each pixel d = disparity_LR(x,y) is compared with disparity_RL(x-d,y). If the difference is above a threshold, the pixel at (x,y) in the final disparity map is invalidated.
    • You can use debugDispLrCheckIt1 and debugDispLrCheckIt2 debug outputs for debugging/fine-tuning purposes.
  • Extended Disparity:
    • Extended disparity mode allows detecting closer distance objects for the given baseline. This increases the maximum disparity search from 96 to 191, meaning the range is now: [0..190].
      1. Computes disparity on the original size images (e.g. 1280x720).
      2. Computes disparity on 2x downscaled images (e.g. 640x360).
      3. Combines the two level disparities on Shave, effectively covering a total disparity range of 191 pixels (in relation to the original resolution).
    • You can use debugExtDispLrCheckIt1 and debugExtDispLrCheckIt2 debug outputs for debugging/fine-tuning purposes.
  • Subpixel Disparity:
    • Subpixel mode improves the precision and is especially useful for long-range measurements. It also helps for better estimating surface normals.
    • In this mode, stereo cameras perform: 94 depth steps * 8 subpixel depth steps + 2 (min/max values) = 754 depth steps.
    • For comparison of normal disparity vs. subpixel disparity images, click here.
• Depth Filters:
  • Depth Filtering / Depth Post-Processing is performed at the end of the depth pipeline. It helps with noise reduction and overall depth quality.
• Mesh files:
  • Mesh files (homography matrix) are generated using the camera intrinsics, distortion coeffs, and rectification rotations. These files help in overcoming the distortions in the camera increasing the accuracy and also help in when wide FOV lens are used.
  • Currently, mesh files are generated only for stereo cameras on the host during calibration. The generated mesh files are stored in depthai/resources which users can load to the device. This process will be moved to on device in the upcoming releases.
• Confidence Threshold:
  • Confidence threshold: Stereo depth algorithm searches for the matching feature from the right camera point to the left image (along the 96 disparity levels). During this process, it computes the cost for each disparity level and chooses the minimal cost between two disparities and uses it to compute the confidence at each pixel. Stereo node will output disparity/depth pixels only where depth confidence is below the confidence threshold (lower the confidence value means better depth accuracy).
  • LR check threshold: Disparity is considered for the output when the difference between LR and RL disparities is smaller than the LR check threshold.

Limitations

• Median filtering is disabled when subpixel mode is set to 4 or 5 bits.
• For RGB-depth alignment the RGB camera has to be placed on the same horizontal line as the stereo camera pair.
• RGB-depth alignment doesn't work when using disparity shift.

Changing camera calibration during runtime

Depth Presets

Depth Presets

• Left-right consistency check
• 2x decimation with pixel skipping
• Speckle filtering (range: 200, threshold: 2)
• 3 hardware resources for post-processing

Stereo depth FPS

Usage

1pipeline = dai.Pipeline()
2stereo = pipeline.create(dai.node.StereoDepth)
3
4# Better handling for occlusions:
5stereo.setLeftRightCheck(False)
6# Closer-in minimum depth, disparity range is doubled:
7stereo.setExtendedDisparity(False)
8# Better accuracy for longer distance, fractional disparity 32-levels:
9stereo.setSubpixel(False)
10
11# Define and configure MonoCamera nodes beforehand
12left.out.link(stereo.left)
13right.out.link(stereo.right)

Examples of functionality

• Stereo Depth
• RGB Depth alignment - align depth to color camera
• RGB & Yolo with spatial data

Reference

Disparity

• 0 - maximum confidence that it holds a valid value
• 255 - minimum confidence, so there is more chance that the value is incorrect

Calculate depth using disparity map

1depth = focal_length_in_pixels * baseline / disparity_in_pixels

1import depthai as dai
2
3with dai.Device() as device:
4  calibData = device.readCalibration()
5  intrinsics = calibData.getCameraIntrinsics(dai.CameraBoardSocket.CAM_C)
6  print('Right mono camera focal length in pixels:', intrinsics[0][0])

1focal_length_in_pixels = image_width_in_pixels * 0.5 / tan(HFOV * 0.5 * PI/180)
2
3# With 400P mono camera resolution where HFOV=71.9 degrees
4focal_length_in_pixels = 640 * 0.5 / tan(71.9 * 0.5 * PI / 180) = 441.25
5
6# With 800P mono camera resolution where HFOV=71.9 degrees
7focal_length_in_pixels = 1280 * 0.5 / tan(71.9 * 0.5 * PI / 180) = 882.5

1# For OAK-D @ 400P mono cameras and disparity of eg. 50 pixels
2depth = 441.25 * 7.5 / 50 = 66.19 # cm
3
4# For OAK-D @ 800P mono cameras and disparity of eg. 10 pixels
5depth = 882.5 * 7.5 / 10 = 661.88 # cm

Min stereo depth distance

1min_distance = focal_length_in_pixels * baseline / disparity_in_pixels = 882.5 * 7.5cm / 95 = 69.67cm

1. Changing the resolution to 640x400, instead of the standard 1280x800.
2. Enabling Extended Disparity.

1min_distance = focal_length_in_pixels * baseline / disparity_in_pixels = 882.5 * 7.5cm / 190 = 34.84cm

Disparity shift to lower min depth perception

• Because of the inverse relationship between disparity and depth, MaxZ will decrease much faster than MinZ as the disparity shift is increased. Therefore, it is advised not to use a larger than necessary disparity shift.
• Tradeoff in reducing the MinZ this way is that objects at distances farther away than MaxZ will not be seen.
• Because of the point above, we only recommend using disparity shift when MaxZ is known, such as having a depth camera mounted above a table pointing down at the table surface.
• Output disparity map is not expanded, only the depth map. So if disparity shift is set to 50, and disparity value obtained is 90, the real disparity is 140.

• (+) Is faster, as it doesn't require an extra computation, which means there's also no extra latency
• (-) Reduces the MaxZ (significantly), while extended disparity only reduces MinZ.

Max stereo depth distance

1Dm = (baseline/2) * tan((90 - HFOV / HPixels)*pi/180)

1# For OAK-D (7.5cm baseline)
2Dm = (7.5/2) * tan((90 - 71.9/1280)*pi/180) = 3825.03cm = 38.25 meters
3
4# For OAK-D-CM4 (9cm baseline)
5Dm = (9/2) * tan((90 - 71.9/1280)*pi/180) = 4590.04cm = 45.9 meters

Depth perception accuracy

Limitation

• BL [cm] - Baseline of stereo cameras.
• Dv [cm] - Minimum distace where both cameras see an object (thus where depth can be calculated).
• B [pixels] - Width of the band where depth cannot be calculated.
• W [pixels] - Width of mono in pixels camera or amount of horizontal pixels, also noted as HPixels in other formulas.
• D [cm] - Distance from the camera plane to an object (see image Measuring real-world object dimensions).
• F [cm] - Width of image at the distance D.

• F = 2 * D * tan(HFOV/2)
• Dv = (BL/2) * tan(90 - HFOV/2)

1B = 2 * Dv * tan(HFOV/2) * W / F
2B = 2 * Dv * tan(HFOV/2) * W / (2 * D * tan(HFOV/2))
3B = W * Dv / D  # pixels

1Dv = 7.5 / 2 * tan(90 - 72/2) = 3.75 * tan(54°) = 5.16 cm
2B = 640 * 5.16 / 100 = 33 # pixels

Measuring real-world object dimensions

1distance = math.sqrt(detection.spatialCoordinates.x ** 2 + detection.spatialCoordinates.y ** 2 + detection.spatialCoordinates.z ** 2) # mm