# Manual Calibration ### Stereo camera recalibration Estimate intrinsics and extrinsics of all camera sensors [Stereo camera recalibration](#Manual%2520Calibration-Stereo%2520camera%2520recalibration) ### ToF recalibration Calibrate for ToF extrinsics [ToF recalibration](#Manual%2520Calibration-ToF%2520recalibration) > **Auto-calibrate your camera** > If you only need to calibrate > **extrinsics** > , you can do so dynamically by using our > [Dynamic Calibration Node](https://docs.luxonis.com/software-v3/depthai/depthai-components/host_nodes/dynamic_calibration.md) > . ## About Camera Calibration Camera calibration is the process of determining the intrinsic, extrinsic, and distortion parameters of a camera. These parameters enable accurate mapping of 3D points to the 2D sensor and help correct lens distortions. ### Intrinsic Parameters Intrinsic parameters define the internal characteristics of a camera: * Focal Length (fx, fy): Sets the field of view. A higher focal length narrows the view, while a lower one widens it. * Optical Center (cx, cy): The principal point where the optical axis intersects the sensor. A slight offset (around 20 pixels) is acceptable. This occurs due to minor lens-sensor alignment variations, like small shifts or tilts. Proper calibration compensates for these offsets, so they don't affect the camera's accuracy or performance. * Distortion Coefficients: Adjust for lens imperfections: * Radial Distortion (k1, k2, k3, k4, k5, k6): Corrects barrel or pincushion effects. * Tangential Distortion (p1, p2): Compensates for lens-sensor misalignment. * Thin Prism/Skew Distortion (s1, s2, s3, s4, τx, τy): Addresses tilt issues, particularly in wide-angle lenses. ### Extrinsic Parameters Extrinsic parameters describe the camera's position and orientation relative to the scene or other cameras: * Rotation: The orientation of the camera. * Translation: The camera's position relative to a reference point. These parameters are essential for applications involving multiple cameras, such as stereo vision, as they allow each camera's image to align with the others. ### Calibration Process Calibration is the process of accurately determining the camera's intrinsic and extrinsic parameters. A robust calibration procedure typically includes: 1. Marker-Based Calibration: Using Charuco markers (a combination of chessboard and Aruco markers) positioned at various distances and angles for high accuracy. 2. Controlled Environment: Performing calibration in a stable environment to cover all relevant poses and angles. This approach helps achieve consistently reproducible calibration results. 3. Model-Specific Customization: Adapting the calibration process to the specific camera model for enhanced reliability. ### Distortion Models Different distortion models are used to correct the unique distortions for each lens type. These models are based on [OpenCV's distortion model framework](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html) but may be adapted for specific camera configurations. * Normal Field of View (NFOV): Uses a standard perspective model to correct radial (k1, k2, k3, …) and tangential (p1, p2) distortions. * Wide Field of View (WFOV): Utilizes an extended perspective model, which can have up to 14 total distortion parameters: 1. Radial Distortion: k1, k2, k3, k4, k5, k6 2. Tangential Distortion: p1, p2 3. Thin Prism: s1, s2, s3, s4 4. Tilt Parameters: τx, τy > **Note:** > In practice, we do not activate all 14 parameters. For wide-angle lenses, we enable two additional parameters (often τx and τy) to address the tilt we observe. This > *extended perspective model* > handles our wide-field-of-view (WFOV) lenses very well. * Fisheye Lenses: Employ a specialized model for extreme, circular distortions. We generally avoid fisheye calibration because OpenCV's fisheye support doesn't meet all our needs, and the extended perspective model already covers our WFOV applications effectively. For additional details, please refer to the [OpenCV Documentation on Camera Calibration and Distortion Models](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html). ## Stereo camera recalibration Using the intrinsic, extrinsic, and distortion parameters described in the [Camera Calibration](#Manual%2520Calibration-About%2520Camera%2520Calibration) section, stereo calibration aligns both cameras relative to each other. From these parameters, the system computes rectification matrices, enabling the StereoDepth node to produce accurate stereo disparity and depth estimates. > All OAK cameras, except the > [Modular line](https://docs.luxonis.com/hardware/platform/deploy/ffc.md) > , are calibrated before shipment, so recalibration is not required. For OAK FFC camera modules, camera calibration is required after mounting the cameras in the desired configuration. The video below walks through the calibration steps. For more details on calibration options, see the sections below and run ./calibrate.py --help to print all available calibration options. ### Prerequisites If you don't yet have the depthai repository on your computer, you need to clone it and install the requirements: ```bash git clone https://github.com/luxonis/depthai.git --branch main cd depthai git submodule update --init --recursive python3 install_requirements.py ``` #### Prepare Charuco Board We recommend displaying the Charuco board on a TV or a large flat monitor. Larger screens are better because they allow more Charuco markers to be visible in the image, which typically improves calibration accuracy. Depending on the screen size, we suggest displaying the following Charuco board in full-screen: * [24" screen Charuco board](https://drive.google.com/file/d/1phwh_6jrwDddYP2Efb-Mw93e1H_d3j4F/view?usp=drive_link) * [28" screen Charuco board](https://drive.google.com/file/d/1FV2jNvzMkOBqdP_a8ULEGKOZxR5c-rWU/view?usp=drive_link) * [32" screen Charuco board](https://drive.google.com/file/d/1oR_WLaL2iXKXkWci68s55C38Bb74PJer/view?usp=drive_link) * [36" screen Charuco board](https://drive.google.com/file/d/1483YOjJro9FAKP0X0gweeoV1x4B5Np4b/view?usp=drive_link) * [42" screen Charuco board](https://drive.google.com/file/d/1W05dPQvk3F8YM8jpMG_GhubG46AbXtWX/view?usp=drive_link) * [50" screen Charuco board](https://drive.google.com/file/d/1qb4z0omvHa0z1G3AtqMa9gCdykPQiuKX/view?usp=drive_link) * [55" screen Charuco board](https://drive.google.com/file/d/13hSTA2NUSoKyZFqzE4NRw6csfO7fMHX7/view?usp=drive_link) * [65" screen Charuco board](https://drive.google.com/file/d/1Hc3L3w9g4otGoJvRtgJmA3-Sg1RLjVGF/view?usp=drive_link) * [75" screen Charuco board](https://drive.google.com/file/d/1pyGY5CPhn09Vdxb9wH90SoBFKzu4OYww/view?usp=drive_link) If you have a different screen size, round down to the nearest size. #### Display the Charuco Board When displaying the Charuco board, the markers and squares should be sharp and clearly visible. Keep in mind: * Avoid too bright or dim screens. * Avoid direct bright lights or sun on the screen. * Display the Charuco board in full-screen. Note the size of the squares on the displayed Charuco board. This is the SQUARE_SIZE_IN_CM argument for the calibration script. ### Calibration Please select the appropriate calibration procedure for your device: #### Compact devices #### Run the Calibration Script Replace the placeholder argument values with valid entries: ```bash python3 calibrate.py -s [SQUARE_SIZE_IN_CM] --board [BOARD] -nx [squaresX] -ny [squaresY] ``` For example, for calibrating the OAK-D S2 on a 32" screen: ```bash python3 calibrate.py -s 3.76 --board OAK-D-S2 -nx 17 -ny 9 ``` #### Modular devices #### Prepare the Board Config Modular devices (OAK-FFC-4P, 3P, 6P) will need a custom board config file, which describes the camera configuration for your device. Use one of the board .json files from [here](https://github.com/luxonis/depthai-boards/tree/main/boards) as a template for your custom board config. > It is best to use a template which has a similar camera configuration to your custom board. In the board config, we define the cameras, their sockets, and their positions relative to the other cameras. Each camera should have the following information: | Feature | Details | | --- | --- | | Camera board socket | Defined on the board PCB (e.g., `CAM_A`, `CAM_B`, `CAM_C`, `CAM_D`, ...) | | HFOV of the camera module | [Hardware catalog](https://docs.luxonis.com/hardware.md) | | Type of camera | Color/Mono | | Camera extrinsics | Translation [x, y, z] and Rotation [r, p, y] | Example for OAK-FFC-4P with two OV9282 (PY003) cameras in stereo setup with 14.8cm baseline, with IMX378 (PY052) between them, 5cm from the right mono camera: ```json { "board_config": { "name": "Custom FFC", "revision": "R1M0E1", "cameras": { "CAM_C": { "name": "right", "hfov": 71.86, "type": "mono", "extrinsics": { "to_cam": "CAM_B", "specTranslation": { "x": 14.8, "y": 0, "z": 0 }, "rotation": { "r": 0, "p": 0, "y": 0 } } }, "CAM_B": { "name": "left", "hfov": 71.86, "type": "mono", "extrinsics": { "to_cam": "CAM_A", "specTranslation": { "x": -9.8, "y": 0, "z": 0 }, "rotation": { "r": 0, "p": 0, "y": 0 } } }, "CAM_A": { "name": "middle", "hfov": 69, "type": "color" } }, "stereo_config": { "left_cam": "CAM_B", "right_cam": "CAM_C" } } } ``` Another example using OAK-FFC-4P, but this time without the color camera: ```json { "board_config": { "name": "Custom FFC", "revision": "R1M0E1", "cameras": { "CAM_C": { "name": "right", "hfov": 71.86, "type": "mono", "extrinsics": { "to_cam": "CAM_B", "specTranslation": { "x": 14.8, "y": 0, "z": 0 }, "rotation": { "r": 0, "p": 0, "y": 0 } } }, "CAM_B": { "name": "left", "hfov": 71.86, "type": "mono" } }, "stereo_config": { "left_cam": "CAM_B", "right_cam": "CAM_C" } } } ``` > Make sure to place your custom board config in the > `resources/depthai_boards/boards/` > folder of the > `depthai` > repository. #### Run the Calibration Script After setting up the board config, run the calibration with the name of the JSON config as the board name. ```bash python3 calibrate.py -s [SQUARE_SIZE_IN_CM] -brd OAK-FFC-4P.json -nx [squaresX] -ny [squaresY] ``` Example command for the 17x9 Charuco board measured above: ```bash python3 calibrate.py -s 3.76 -nx 17 -ny 9 -brd OAK-FFC-4P.json ``` | Argument | Argument alias | Argument Description | | --- | --- | --- | | `-s` | `--squareSizeCm` | Measured square size of the printed charuco board in centimeters | | `-brd` | `--board` | Name of the camera (from [depthai-boards](https://github.com/luxonis/depthai-boards/tree/main/boards), not case-sensitive), or path to a custom .json board config | | `-nx` | `--squaresX` | Number of squares in X direction. SquaresX is specified in [Prepare Charuco Board](#Manual%2520Calibration-Stereo%2520camera%2520recalibration-Prerequisites-Prepare%2520Charuco%2520Board), depending on your screen size. | | `-ny` | `--squaresY` | Number of squares in Y direction. SquaresY is specified in [Prepare Charuco Board](#Manual%2520Calibration-Stereo%2520camera%2520recalibration-Prerequisites-Prepare%2520Charuco%2520Board), depending on your screen size. | | `-cm` | `--cameraMode` | Camera mode, either `perspective` (default) or `fisheye`. | | `-mdmp` | `--minDetectedMarkersPercent` | Minimum percentage of detected markers in a frame, to consider the frame valid. Default is 0.5 (50%). If you want to be more strict, you can increase this value, but it can cause longer time to get enough valid frames. | | `-ep` | `--maxEpipolarError` | Maximum epipolar error in pixels, to consider the frame valid. Default is 0.7. If you want to be more strict, you can decrease this value. | For a full list of arguments and usage examples, run: ```bash python3 calibrate.py --help ``` #### Camera Positioning During Calibration We suggest capturing the calibration from different angles and distances, as it will assist the calibration algorithm in finding the best possible calibration. 1. Close to the screen: Ensure the calibration board covers almost the entire field of view (FOV). Take 5 images to cover the entire FOV of the camera: * Front view, with the calibration board in the middle of the FOV. * Without moving the camera, rotate to align the camera FOV with the calibration board edges. Take 4 images, one for each edge of the screen. 2. Close to the screen, from the side: Capture 4 or more images of the calibration board tilted, covering the majority of the FOV. Move the camera to the top, bottom, left, and right sides of the screen. Varying distances can be used. 3. Middle distance: The calibration board should cover about 40% of the FOV. Take 5 images to cover the entire FOV of the camera: * Front view, with the calibration board in the middle of the FOV. * Similar to the close to the screen position, rotate the camera to align the FOV with the calibration board edges. 4. Far from the screen: The calibration board covers only a small part of the FOV. In total, take 9 images to cover the entire FOV of the camera: * Frontal view, with the calibration board in the middle of the FOV. * Take 4 images aligning the camera FOV with all 4 edges, similar to the close and middle distance positions. * Also, take 4 images aligning with the corners of the screen. #### Running the Processing Stage After capturing images, you can run the calibration processing stage by pressing the s key. The script shows epipolar lines for each image, and you should verify that they are aligned correctly. Once all images are checked, the calibration result, if successful, is flashed to the device EEPROM. Each captured image is saved in the dataset folder, so you can rerun the calibration process without capturing the images again. If you want to re-run the calibration process on the captured images, use the -m process argument: ```bash python3 calibrate.py -s [SQUARE_SIZE_IN_CM] --board [BOARD] -nx [squaresX] -ny [squaresY] -m process ``` Calibration results are stored in the resources/ folder and can be used later for testing or debugging. You can also load or flash this local calibration file to the device. See the [calibration load example](https://docs.luxonis.com/software/depthai/examples/calibration_load.md#calibration-load) for more details. ### Test Depth For testing depth quality, use OAK Viewer by following the [OAK Viewer instructions](https://docs.luxonis.com/software-v3/depthai/tools/oak-viewer.md). ### Troubleshooting * If calibration fails with error: High reprojection error!, the usual cause is a misconfigured board config, often due to incorrect specified HFOV of the used camera module. * If despite successful calibration, the depth is still incorrect, perhaps your left and right cameras are swapped. Retry the calibration with changed board config, or swap the board sockets the cameras are plugged into. ## ToF recalibration The Time-of-Flight (ToF) calibration is essential for aligning the ToF sensor with other cameras in the system. This procedure focuses on obtaining extrinsic parameters, crucial for ensuring coordinated operation among different cameras. > Note that this calibration does not enhance the depth accuracy, as that aspect is managed by the device's firmware. ### Calibration Procedure If you have already installed the DepthAI repository, update it for TOF calibration with the following commands: ```bash git checkout new_tof_calib git submodule update --init --recursive ``` After updating the boards, install the updated DepthAI Python libraries: ```bash python3 ./install_requirements.py ``` To start the calibration process, run calibrate.py with the appropriate parameters for the Charuco board you are using. For example: ```bash python3 calibrate.py -db -nx 12 -ny 9 -c 1 -cd 0 -s 6 -ms 4.7 -brd OAK-D-SR-POE ``` Parameters explained: * -db: Indicates the default board, meaning you are using Charuco markers. * -nx: Number of Charuco markers in the x direction. * -ny: Number of Charuco markers in the y direction. * -c: Number of pictures taken each time the polygon is displayed (optional). * -cd: Countdown time before a picture is taken in seconds (optional). * -s: Size of the square around the Charuco marker in centimeters. * -ms: Size of the markers in centimeters. * -brd: Board of the device (in this case, OAK-D-SR-POE). If you encounter errors such as division by zero or Failed to detect markers in the image dataset/rgb/rgb_p3_10.png, delete the image with poor Charuco board detection from all camera folders, then run the same command again with the added -m process parameter. This starts only the processing stage, so you do not have to retake the board images. ```bash python3 calibrate.py -db -nx 12 -ny 9 -c 1 -cd 0 -s 6 -ms 4.7 -brd OAK-D-SR-POE -m process ```