# MultiDeviceCalibration

`MultiDeviceCalibration` estimates the relative positions and orientations of OAK devices. Save its result and apply it to a
pipeline to combine camera data in one coordinate system.

For camera placement and a visual overview, see [Multi-Device
Calibration](https://docs.luxonis.com/hardware/platform/depth/multi-device-calibration.md).

> **Beta API**
> `dai.beta.node.MultiDeviceCalibration`, its messages and `dai.beta.MultiDeviceCalibrationHandler` are a [Beta feature](https://docs.luxonis.com/software-v3/depthai/examples/beta-features.md) added in DepthAI v3.11.0. The API may still change between releases.

## Calibrate two devices

Use two devices with factory-calibrated stereo cameras on `CAM_B` and `CAM_C`. Keep both devices still and point their cameras at
a shared, static, textured scene. The stereo calibration supplies the known distance needed to estimate real-world positions.

Replace the IP addresses below. The example collects 10 image groups and saves the result to `multi_device_calibration.json`. Its
five-second synchronization tolerance is only suitable for a static scene; it does not synchronize exposures.

#### Python

```python
import depthai as dai
from datetime import timedelta

with dai.Pipeline(createImplicitDevice=False) as pipeline:
    calibration = pipeline.create(dai.beta.node.MultiDeviceCalibration)
    calibration.setSampleCount(10)
    calibration.sync.setSyncThreshold(timedelta(seconds=5))

    for ip in ["192.168.1.11", "192.168.1.12"]:
        device = pipeline.addDevice(ip)
        for socket in (dai.CameraBoardSocket.CAM_B, dai.CameraBoardSocket.CAM_C):
            camera = pipeline.create(dai.node.Camera, device).build(socket, sensorFps=5)
            calibration.addCamera(device.getDeviceId(), socket, camera.requestFullResolutionOutput(fps=5))

    controlQueue = calibration.inputControl.createInputQueue()
    resultQueue = calibration.calibrationOutput.createOutputQueue()

    pipeline.start()
    controlQueue.send(dai.beta.MultiDeviceCalibrationControl.start())
    result = resultQueue.get(timedelta(minutes=3))
    if result is None or not result.passed:
        raise RuntimeError(result.info if result is not None else "Calibration timed out")
    if not result.getHandler().toJsonFile("multi_device_calibration.json"):
        raise RuntimeError("Could not save calibration")
    print(f"Confidence: {result.dataConfidence:.3f}, Sampson error: {result.sampsonError:.4f}")
```

#### C++

The C++ snippet configures the node and queues. See the complete C++ examples below for running calibration and saving the result.

```cpp
dai::Pipeline pipeline(false);
auto calibration = pipeline.create<dai::beta::node::MultiDeviceCalibration>();
calibration->setSampleCount(10);
calibration->sync->setSyncThreshold(std::chrono::seconds(5));

for(const auto& ip : {"192.168.1.11", "192.168.1.12"}) {
    auto device = pipeline.addDevice(ip);
    for(auto socket : {dai::CameraBoardSocket::CAM_B, dai::CameraBoardSocket::CAM_C}) {
        auto camera = pipeline.create<dai::node::Camera>(device)->build(socket, std::nullopt, 5.0f);
        calibration->addCamera(device->getDeviceId(), socket, *camera->requestFullResolutionOutput(std::nullopt, 5.0f));
    }
}

auto controlQueue = calibration->inputControl.createInputQueue();
auto resultQueue = calibration->calibrationOutput.createOutputQueue();
```

## Inputs and Outputs

 * `inputControl` — [Start, stop or
   reset](https://docs.luxonis.com/software-v3/depthai/depthai-components/messages/multi_device_calibration_control.md) a
   calibration run.
 * `inputs` — camera streams registered with `addCamera()`, grouped by `calibration.sync`. Each group must contain a frame from
   every camera.
 * `calibrationOutput` — one
   [result](https://docs.luxonis.com/software-v3/depthai/depthai-components/messages/multi_device_calibration_result.md) per run.
   Check `passed`; on failure, read `info`.

Configure the node before starting. After a result, send `reset()` before changing settings or running again. `stop()` aborts
collection and returns the node to idle.

## Using the result

Calibration does not apply itself or flash the devices. Load the saved graph into your application pipeline:

```python
handler = dai.beta.MultiDeviceCalibrationHandler("multi_device_calibration.json")
pipeline = dai.Pipeline(createImplicitDevice=False)
pipeline.setMultiDeviceCalibration(handler.getGraph())
```

Add your devices and processing nodes to this pipeline. Their camera frames now carry a shared coordinate system for
[PointCloud](https://docs.luxonis.com/software-v3/depthai/depthai-components/host_nodes/pointcloud.md) or
[Stitching](https://docs.luxonis.com/software-v3/depthai/depthai-components/host_nodes/stitching.md). See [Multi-Device
Pipelines](https://docs.luxonis.com/software-v3/depthai/depthai-components/multi-device.md) for applying or clearing calibration
at runtime.

## Configuration

Set these options while the node is idle, before sending `start()`.

| Method | Purpose |
| --- | --- |
| `addCamera(deviceId, socket, output)` | Register a camera stream. Each device/socket pair can be registered once. |
| `setSampleCount(count)` / `getSampleCount()` | Set or read the number of complete image groups collected per run. Default: `10`;
minimum: `1`. |
| `setKnownDistance(...)` | Supply a measured distance between two registered camera centers on different devices. Needed when no
calibrated stereo pair provides scale. |
| `setStereoPair(deviceId, leftSocket, rightSocket)` | Select the registered stereo pair used to provide a known baseline. |
| `setInitialGuess(...)` | Supply an approximate relative device pose to help initialize calibration. |
| `setDeviceCalibration(deviceId, handler)` | Supply a device's calibration instead of reading it from hardware. Required for
recorded streams without a connected device. |
| `sync.setSyncThreshold(duration)` / `sync.getSyncThreshold()` | Set or read the allowed timestamp difference between frames in a
group. |

```python
calibration.setSampleCount(20)
print(calibration.getSampleCount())  # 20
calibration.sync.setSyncThreshold(timedelta(seconds=5))  # static scene only
```

For one camera per device, supply the actual device IDs, registered sockets and measured distance:

```python
calibration.setKnownDistance(
    idA, dai.CameraBoardSocket.CAM_A,
    idB, dai.CameraBoardSocket.CAM_A,
    80.0, dai.LengthUnit.CENTIMETER,
)
```

## Calibration handler

`result.getHandler()` returns a `MultiDeviceCalibrationHandler` for a successful result. Use it to save, load or inspect
calibration:

| Method | Purpose |
| --- | --- |
| `MultiDeviceCalibrationHandler(path)` | Load a saved calibration file. |
| `getGraph()` | Return the calibration graph to pass to `pipeline.setMultiDeviceCalibration()`. |
| `toJsonFile(path)` | Save calibration to a JSON file. Returns whether saving succeeded. |
| `toJson()` / `fromJson(data)` | Export or load calibration as a JSON object. |
| `getDeviceSocket(deviceId)` | Return the device's origin socket, or `None` if it is absent from the graph. |
| `getExtrinsicsToOrigin(deviceId, socket)` | Return the transform from a device's local origin to the shared origin, or `None` if
the device is absent. |

The handler handles unit conversion when saving and loading. Graph translations are in meters; JSON translations are in
centimeters.

## Troubleshooting

| Symptom | Cause and fix |
| --- | --- |
| No result within the timeout | Check that every registered camera streams. For a static scene, increase the sync threshold so
complete groups can form. |
| `passed` is `False` with `DynamicCalibration failed: ... optimization failed` | Check overlap, texture and lighting. Keep the
scene still; try more samples or an initial pose estimate. |
| `no explicit or live calibration is available for device ...` | The stream comes from a replay or from a device without
calibration. Provide one with `setDeviceCalibration()`. |
| `known-distance endpoints must both be registered cameras` | `setKnownDistance()` names a (device, socket) that was not passed
to `addCamera()`. |
| A configuration call raises `cannot be changed unless the node is idle` | The node has already run. Send `reset()` before
changing the configuration. |
| `MultiDeviceCalibration needs at least two cameras on two devices` | Only one device, or only one camera, was registered. |

## Examples

 * [Multi-device
   calibration](https://github.com/luxonis/depthai-core/blob/main/examples/python/MultiDeviceCalibration/multi_device_calibration.py)
   - The minimal flow: `CAM_B` and `CAM_C` of every device, scale from the factory stereo calibration, result saved to JSON.
 * [Multi-device calibration with
   constraints](https://github.com/luxonis/depthai-core/blob/main/examples/python/MultiDeviceCalibration/multi_device_calibration_constraints.py)
   - A single camera per device with a measured distance, initial guesses, a previous result as seed and calibration loaded from
   files.

C++ versions of both live in
[examples/cpp/MultiDeviceCalibration](https://github.com/luxonis/depthai-core/tree/main/examples/cpp/MultiDeviceCalibration).

## Reference

### dai::beta::node::MultiDeviceCalibration

Kind: class

Estimates metric cross-device calibration from synchronized camera images.

The node is host-only and one-shot. Configuration is supplied before the start control command; the node emits a pure-data result
and enters its completed state. A reset command clears the cycle and permits another run.

#### Subnode < dai::node::Sync > sync

Kind: variable

Synchronized camera streams, keyed internally by device ID and socket.

#### InputMap & inputs

Kind: variable

#### Input inputControl

Kind: variable

Explicit lifecycle control.

#### Input syncInput

Kind: variable

Synchronized image groups.

#### Output calibrationOutput

Kind: variable

Pure-data calibration result.

#### MultiDeviceCalibration()

Kind: function

#### ~MultiDeviceCalibration()

Kind: function

#### void addCamera(Node::Output & cameraOutput)

Kind: function

Register a live camera stream. The device ID and socket are taken from the Camera node that owns the output, so the output must
come directly from a built Camera created on a device. For outputs of other nodes (ImageManip, host nodes, replayed streams) use
the explicit overload.

#### void addCamera(std::string deviceId, CameraBoardSocket socket, Node::Output & cameraOutput)

Kind: function

Register an image stream with an explicit identity. Use this when the output does not come directly from a Camera node, for
example for replayed or recorded streams. Image metadata is not a reliable source of cross-device identity, so the identity is
passed in.

#### void setSampleCount(std::size_t sampleCount)

Kind: function

Set the number of complete synchronized groups required before solving.

#### std::size_t getSampleCount()

Kind: function

#### void setKnownDistance(std::string fromDeviceId, CameraBoardSocket fromSocket, std::string toDeviceId, CameraBoardSocket toSocket, float distance, LengthUnit unit)

Kind: function

Supply a metric camera-center distance for an endpoint pair.

#### void setInitialGuess(const MultiDeviceExtrinsics & guess)

Kind: function

Supply an optional initial pose estimate between two devices. The edge may connect any two sockets known to the devices'
calibrations, registered or not; the node converts it to the devices' calibration origins when the run starts. One guess per
device pair, the last one set wins. Sockets unknown to a device's calibration fail the run with a descriptive result.

#### void setInitialGuesses(const std::vector< MultiDeviceExtrinsics > & guesses)

Kind: function

Supply several initial guesses at once, for example a previous result's graph.

#### void setInitialGuesses(const MultiDeviceCalibrationHandler & previous)

Kind: function

Seed the run with every edge of a previous calibration.

#### void setStereoPair(std::string deviceId, CameraBoardSocket leftSocket, CameraBoardSocket rightSocket)

Kind: function

Optional. Restrict metric scale recovery to an explicit factory-calibrated stereo pair. By default every pair of registered
cameras on the same device may anchor the metric scale.

#### void setDeviceCalibration(std::string deviceId, const CalibrationHandler & calibration)

Kind: function

Override live calibration for recorded/offline streams and tests.

#### void buildInternal()

Kind: function

Function called from within the create function to build the node. This function is useful for initialization, setting up inputs
and outputs = stuff that cannot be perform in the constuctor.

### Need assistance?

Head over to [Discussion Forum](https://discuss.luxonis.com/) for technical support or any other questions you might have.