# RGB-D RGBD refers to the type of image data that includes both color (RGB) and depth (D) information. ### Applications of RGB-D ### Colorized Pointcloud Create on-device pointclouds [Colorized Pointcloud](https://docs.luxonis.com/software-v3/depthai/examples/pointcloud_visualization.md) ### Spatial AI Combine depth with neural networks [Spatial AI](https://docs.luxonis.com/software-v3/depthai/perception/spatial-ai.md) ## Aligning RGB and Depth Images Core element of RGB-D pipeline is alignment. The idea is to align each pixel in the depth image with the corresponding pixel in the color image. This is necessary because the depth image perspective is different from the color image perspective due to the different camera positions. The alignment can be done using reprojection. By default, the depth is aligned to the `rectified_left` image during the stereo-matching process. We can project this image to 3D space (using acquired depth information) and then reproject it back to the color image plane using color camera intrinsics. ## Alignment using DepthAI DepthAI provides a few ways to achieve alignment of RGB and depth images which all offer the same functionality, but have their own benefits and drawbacks. ### StereoDepth and GPUStereo - for stereo cameras The [StereoDepth](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/stereo_depth.md) node offers alignment capabilities out of the box. The node takes in a pair of stereo images (`left` and `right`) and outputs a depth map. By default, the depth image is aligned with the `rectified_left` image, but can be changed using the `setDepthAlign()` method. ```python stereo.setDepthAlign(dai.CameraBoardSocket.RGB) # Provided that the RGB camera is connected to RGB/CAM_A port ``` On RVC4, [GPUStereo](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/gpu_stereo.md) is another stereo depth option. It generates disparity and depth from a synchronized stereo pair; when you need that depth reprojected into the RGB camera plane, pair it with [ImageAlign](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/image_align.md). ### Examples * [RGB Depth Alignment](https://docs.luxonis.com/software-v3/depthai/examples/rgb_depth_aligned.md) ### ImageAlign - general alignment The ImageAlign node is a more general-purpose node that can be used to align any two images. The node takes in two images (`input` and `inputAlignTo`) and outputs the aligned image. For RGB-D, this node allows us to align the [Time-of-Flight (ToF) depth](https://docs.luxonis.com/hardware/platform/features/depth/#Depth%20Perception-Time-of-Flight%20Depth) to RGB. ### Examples * [ToF example (RVC2)](https://docs.luxonis.com/software-v3/depthai/examples/rvc2/tof/tof.md) * [RGB-ToF alignment (source code)](https://github.com/luxonis/depthai-core/blob/main/examples/python/RVC2/ToF/tof_align.py) * [RGB-Stereo alignment](https://github.com/luxonis/depthai-python/blob/main/examples/ImageAlign/depth_align.py) > **Efficiency** > Use [StereoDepth](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/stereo_depth.md) when you want built-in stereo depth alignment, as it avoids a separate [ImageAlign](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/image_align.md) reprojection step. On RVC4, [GPUStereo](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/gpu_stereo.md) remains a valid stereo source for RGB-D pipelines when paired with ImageAlign. ### RGB-D on Wide FOV cameras When using WFOV cameras, the alignment can be tricky due to the large distortion in the images caused by the lens. By default, the depth image is undistorted during the rectification-undistortion process, but the color image is not. This can lead to misalignment between the two images. To solve this, the color image needs to be undistorted as well. Using [Camera](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/camera.md) node. The node can be used to undistort the color image using the intrinsic parameters of the camera. #### Python ```python cam = pipeline.create(dai.node.Camera) cam.setBoardSocket(dai.CameraBoardSocket.RGB) cam.setMeshSource(dai.CameraProperties.WarpMeshSource.CALIBRATION) ``` #### C++ ```cpp auto cam = pipeline.create(); cam->setBoardSocket(dai::CameraBoardSocket::RGB); cam->setMeshSource(dai::CameraProperties::WarpMeshSource::CALIBRATION); ``` The undistortion can be done manually using the intrinsic parameters of the camera and OpenCV. ```python alpha = 0 stereo.setAlphaScaling(alpha) rgb_w = camRgb.getResolutionWidth() rgb_h = camRgb.getResolutionHeight() rgbIntrinsics = np.array(calibData.getCameraIntrinsics(rgbCamSocket, rgb_w, rgb_h)) rgb_d = np.array(calibData.getDistortionCoefficients(rgbCamSocket)) rgb_new_cam_matrix, _ = cv2.getOptimalNewCameraMatrix(rgbIntrinsics, rgb_d, (rgb_w, rgb_h), alpha) map_x, map_y = cv2.initUndistortRectifyMap(rgbIntrinsics, rgb_d, None, rgb_new_cam_matrix, (rgb_w, rgb_h), cv2.CV_32FC1) frameRgb = cv2.remap(frameRgb, map_x, map_y, cv2.INTER_LINEAR) ``` ### Full code Example using OpenCV undistortion [Full code](https://gist.githubusercontent.com/Erol444/bf0fa3debde7e4479477b03e1a98867e/raw/7541837302af9c7b304a50102eb64ce8c938c8fa/rgb-depth-align-alpha1.py) When undistorting an image captured by a wide FOV camera, the warping will cause some parts of the image to be cut off. This results in the loss of FOV which is the primary selling point of WFOV cameras. To maximize the FOV, the `alpha` parameter [0-1] can be used to scale the undistorted image [[guide](https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html)]. ### Setting alpha parameter * `alpha = 0` - No scaling, the undistorted image will be the same size as the original image. * `alpha = 1` - Maximum scaling, the undistorted image will be the largest possible image that fits in the original image. #### StereoDepth undistortion ```python stereo.setAlphaScaling(alpha) ``` #### Camera undistortion ```python camRgb.setCalibrationAlpha(alpha) ``` #### OpenCV undistortion ```python rgb_new_cam_matrix, _ = cv2.getOptimalNewCameraMatrix(rgbIntrinsics, rgb_d, (rgb_w, rgb_h), alpha) ``` ### Need assistance? Head over to [Discussion Forum](https://discuss.luxonis.com/) for technical support or any other questions you might have.