# RGB-D

RGBD refers to the type of image data that includes both color (RGB) and depth (D) information.

### Applications of RGB-D

### Colorized Pointcloud

Create on-device pointclouds

[Colorized Pointcloud](https://docs.luxonis.com/software/depthai/examples/pointcloud_visualization.md)

### Spatial AI

Combine depth with neural networks

[Spatial AI](https://docs.luxonis.com/software/perception/spatial-ai.md)

## Aligning RGB and Depth Images

Core element of RGB-D pipeline is alignment. The idea is to align each pixel in the depth image with the corresponding pixel in
the color image. This is necessary because the depth image perspective is different from the color image perspective due to the
different camera positions.

The alignment can be done using reprojection. By default, the depth is aligned to the rectified_left image during the
stereo-matching process. We can project this image to 3D space (using acquired depth information) and then reproject it back to
the color image plane using color camera intrinsics.

## Alignment using DepthAI

DepthAI provides a few ways to achieve alignment of RGB and depth images which all offer the same functionality, but have their
own benefits and drawbacks.

### StereoDepth - for stereo cameras

The [StereoDepth](https://docs.luxonis.com/software/depthai-components/nodes/stereo_depth.md) node offers alignment capabilities
out of the box. The node takes in a pair of stereo images (left and right) and outputs a depth map. By default, the depth image is
aligned with the rectified_left image, but can be changed using the setDepthAlign() method.

```python
stereo.setDepthAlign(dai.CameraBoardSocket.RGB) # Provided that the RGB camera is connected to RGB/CAM_A port
```

### Examples

 * [RGB Depth Alignment](https://docs.luxonis.com/software/depthai/examples/rgb_depth_aligned.md)

### ImageAlign - general alignment

The [ImageAlign](https://docs.luxonis.com/software/depthai-components/nodes/image_align.md) node is a more general-purpose node
that can be used to align any two images. The node takes in two images (input and inputAlignTo) and outputs the aligned image. For
RGB-D, this node allows us to align the [Time-of-Flight (ToF)
depth](https://docs.luxonis.com/hardware/platform/features/depth.md#Depth%20Perception-Time-of-Flight%20Depth) to RGB.

### Examples

 * [RGB-ToF alignment](https://docs.luxonis.com/software/depthai/examples/tof_align.md)
 * [RGB-Stereo alignment](https://github.com/luxonis/depthai-python/blob/main/examples/ImageAlign/depth_align.py)

> **Efficiency**
> Use
> [StereoDepth](https://docs.luxonis.com/software/depthai-components/nodes/stereo_depth.md)
> node for stereo cameras, as it is more efficient than using
> [ImageAlign](https://docs.luxonis.com/software/depthai-components/nodes/image_align.md)
> due to optimizations in the reprojection step.

### RGB-D on Wide FOV cameras

When using WFOV cameras, the alignment can be tricky due to the large distortion in the images caused by the lens. By default, the
depth image is undistorted during the rectification-undistortion process, but the color image is not. This can lead to
misalignment between the two images. To solve this, the color image needs to be undistorted as well.

### Using Camera node

Using [Camera](https://docs.luxonis.com/software/depthai-components/nodes/camera.md) node. The node can be used to undistort the
color image using the intrinsic parameters of the camera.

#### Python

```python
cam = pipeline.create(dai.node.Camera)
cam.setBoardSocket(dai.CameraBoardSocket.RGB)
cam.setMeshSource(dai.CameraProperties.WarpMeshSource.CALIBRATION)
```

#### C++

```cpp
auto cam = pipeline.create<dai::node::Camera>();
cam->setBoardSocket(dai::CameraBoardSocket::RGB);
cam->setMeshSource(dai::CameraProperties::WarpMeshSource::CALIBRATION);
```

### Manual undistortion

The undistortion can be done manually using the intrinsic parameters of the camera and OpenCV.

```python
alpha = 0
stereo.setAlphaScaling(alpha)

rgb_w = camRgb.getResolutionWidth()
rgb_h = camRgb.getResolutionHeight()
rgbIntrinsics = np.array(calibData.getCameraIntrinsics(rgbCamSocket, rgb_w, rgb_h))
rgb_d = np.array(calibData.getDistortionCoefficients(rgbCamSocket))
rgb_new_cam_matrix, _ = cv2.getOptimalNewCameraMatrix(rgbIntrinsics, rgb_d, (rgb_w, rgb_h), alpha)
map_x, map_y = cv2.initUndistortRectifyMap(rgbIntrinsics, rgb_d, None, rgb_new_cam_matrix, (rgb_w, rgb_h), cv2.CV_32FC1)

frameRgb = cv2.remap(frameRgb, map_x, map_y, cv2.INTER_LINEAR)
```

### Full code

Example using OpenCV undistortion

[Full
code](https://gist.githubusercontent.com/Erol444/bf0fa3debde7e4479477b03e1a98867e/raw/7541837302af9c7b304a50102eb64ce8c938c8fa/rgb-depth-align-alpha1.py)

### Alpha and maximizing FOV

When undistorting an image captured by a wide FOV camera, the warping will cause some parts of the image to be cut off. This
results in the loss of FOV which is the primary selling point of WFOV cameras. To maximize the FOV, the alpha parameter [0-1] can
be used to scale the undistorted image [[guide](https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html)].

### Setting alpha parameter

 * alpha = 0 - No scaling, the undistorted image will be the same size as the original image.

 * alpha = 1 - Maximum scaling, the undistorted image will be the largest possible image that fits in the original image.

#### StereoDepth undistortion

```python
stereo.setAlphaScaling(alpha)
```

#### Camera undistortion

```python
camRgb.setCalibrationAlpha(alpha)
```

#### OpenCV undistortion

```python
rgb_new_cam_matrix, _ = cv2.getOptimalNewCameraMatrix(rgbIntrinsics, rgb_d, (rgb_w, rgb_h), alpha)
```

### Need assistance?

Head over to [Discussion Forum](https://discuss.luxonis.com/) for technical support or any other questions you might have.