# annotation

Python API: `luxonis_ml.data.datasets.annotation`

Annotation schemas used by Luxonis Data Format datasets.

This module owns the record and annotation payload contracts accepted by
[LuxonisDataset.add](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/luxonis_dataset.md)
and produced by format-specific parsers. The schemas are implemented as [pydantic
models](https://pydantic.dev/docs/validation/latest/concepts/models/), so input dictionaries are validated and normalized before
they are written to LDF parquet shards.

Table of Contents

 * [Record
   Model](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
 * [Coordinates and
   Instances](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
 * [Classification](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
 * [Bounding
   Boxes](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
 * [Keypoints](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
 * [Segmentation](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
 * [Instance
   Segmentation](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
 * [Arrays](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
 * [Metadata and
   Categories](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)

## Record Model

Dataset ingestion starts with
[DatasetRecord](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md).
A record points to media, optionally assigns a task name, and optionally carries an annotation payload validated by
[Detection](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md).

Single-source records use `"file"`:

```python
{
    "file": "path/to/image.jpg",
    "task_name": "detection",
    "annotation": {
        "class": "car",
        "boundingbox": {"x": 0.1, "y": 0.2, "w": 0.3, "h": 0.4},
    },
}
```

Multi-source records use `"files"`:

```python
{
    "files": {
        "rgb": "path/to/rgb.png",
        "depth": "path/to/depth.png",
    },
    "task_name": "detection",
    "annotation": {
        "class": "person",
        "boundingbox": {"x": 0.1, "y": 0.1, "w": 0.3, "h": 0.4},
    },
}
```

Task names group annotations that should be consumed together. If no `task_name` is provided, the empty string `""` is used.
Loader label keys therefore follow `"task_name/task_type"` and default-task keys start with `"/"`.

## Coordinates and Instances

Spatial annotations use image-normalized coordinates. For an image with width W and height H, an absolute point (x, y) is stored
as (x ⁄ W, y ⁄ H).

When multiple annotation types describe the same physical object, use the same `instance_id` so bounding boxes, keypoints, and
instance masks can be associated even when yielded as separate records.

> **Warning**
> If `instance_id` is omitted and related annotation types are yielded in separate records, association falls back to insertion order. That is only reliable when every record is emitted consistently.

[Detection.scale_to_boxes](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
supports box-relative annotations. When enabled, keypoints and segmentation values are interpreted relative to the bounding box
and rescaled to image-normalized coordinates before storage.

## Classification

Classification assigns a class to the whole sample or instance:

```python
{"class": "vehicle"}
```

Classification is represented internally as the `classification` task type. Any detection that provides a class name contributes a
classification target, even when the same detection also contains boxes, masks, keypoints, arrays, or metadata. When loaded,
classes are usually returned as one-hot vectors with shape (C).

## Bounding Boxes

[BBoxAnnotation](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
stores normalized `xywh` boxes, where `x` and `y` are the top-left corner and `w` and `h` are width and height:

```python
{
    "class": "car",
    "instance_id": 17,
    "boundingbox": {
        "x": 0.20,
        "y": 0.10,
        "w": 0.35,
        "h": 0.25,
    },
}
```

Loader output combines boxes into (N, 5) arrays with rows [c, x, y, w, h], where c is the class index.

## Keypoints

[KeypointAnnotation](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
stores keypoints as `(x, y, visibility)` triplets. Coordinates are normalized and visibility follows the COCO convention:

 * 0: not visible or not labeled.
 * 1: occluded.
 * 2: visible.

```python
{
    "class": "car",
    "instance_id": 17,
    "keypoints": {
        "keypoints": [
            (0.10, 0.20, 2),
            (0.30, 0.40, 1),
        ],
    },
}
```

For K keypoints and N instances, loader output uses shape (N, 3⋅K).

## Segmentation

[SegmentationAnnotation](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
supports polygon, binary-mask, and run-length encoded inputs.

Polyline segmentation stores normalized polygon points. The final point is implicitly connected to the first one:

```python
{
    "class": "road",
    "segmentation": {
        "height": 720,
        "width": 1280,
        "points": [
            (0.10, 0.10),
            (0.90, 0.10),
            (0.80, 0.80),
            (0.20, 0.80),
        ],
    },
}
```

Binary masks are two-dimensional arrays where foreground pixels are 1 and background pixels are 0:

```python
{
    "class": "road",
    "segmentation": {
        "mask": binary_mask,
    },
}
```

Run-length encoded masks use COCO RLE. The `counts` value may be an uncompressed list of integers or a compressed byte string:

```python
{
    "class": "road",
    "segmentation": {
        "height": 720,
        "width": 1280,
        "counts": [120, 8, 200, 12],
    },
}
```

> **Note**
> Numpy masks are converted to RLE internally. RLE input is primarily for compatibility with datasets that already store masks in that format.

Semantic segmentation loader output uses channel-first masks with shape (C, H, W).

## Instance Segmentation

[InstanceSegmentationAnnotation](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
uses the same mask encodings as semantic segmentation, but stores one mask per instance. A detection may include both a bounding
box and an instance mask:

```python
{
    "class": "car",
    "instance_id": 17,
    "boundingbox": {"x": 0.20, "y": 0.10, "w": 0.35, "h": 0.25},
    "instance_segmentation": {
        "height": 720,
        "width": 1280,
        "points": [
            (0.20, 0.10),
            (0.55, 0.10),
            (0.55, 0.35),
            (0.20, 0.35),
        ],
    },
}
```

Instance-mask loader output uses shape (N, H, W).

## Arrays

[ArrayAnnotation](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
references arbitrary `.npy` data synchronized with a sample:

```python
{
    "class": "embedding",
    "array": {
        "path": "path/to/embedding.npy",
    },
}
```

Arrays are useful for modality-specific targets or auxiliary data that should be stored with the dataset but does not fit standard
spatial schemas.

## Metadata and Categories

Metadata stores flexible key-value values. Use
[Category](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
to mark a string as categorical metadata rather than free-form text:

```python
from luxonis_ml.data import Category

{
    "metadata": {
        "text": "ABC-123",
        "text_color": Category("white"),
        "track_id": 42,
    },
}
```

Categorical metadata can be encoded as integers by
[LuxonisLoader](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/loaders/luxonis_loader.md),
or kept as strings when loader configuration requests it.

OCR annotations commonly store recognized text and categorical visual properties:

```python
{
    "metadata": {
        "text": "ABC-123",
        "color": Category("red"),
    },
}
```

Embedding and re-identification datasets commonly store identifiers or other lookup keys:

```python
{
    "metadata": {
        "id": 42,
        "color": Category("red"),
    },
}
```

> **Important**
> Metadata and arrays have no universal geometric semantics. Built-in augmentations can discard values associated with boxes that leave the image, but arbitrary values are otherwise preserved unless a custom augmentation explicitly handles them.

## Classes

### Annotation

Base class for an annotation.

#### Methods

##### combine_to_numpy

```python
def combine_to_numpy(annotations: list[Annotation], classes: list[int], n_classes: int) -> np.ndarray:
```

Combine multiple annotations into a single numpy array.

Parameters

 * `annotations` (`list[Annotation]`): Annotations to combine.
 * `classes` (`list[int]`): Class IDs corresponding to each annotation.
 * `n_classes` (`int`): Total number of classes.

Returns

 * `np.ndarray`: Combined annotation representation.

### ArrayAnnotation

Custom annotation backed by an array file.

All instances of this annotation must have the same shape.

#### Methods

##### combine_to_numpy

```python
def combine_to_numpy(annotations: list[ArrayAnnotation], classes: list[int], n_classes: int) -> np.ndarray:
```

Combine array annotations into instance-class-indexed arrays.

Parameters

 * `annotations` (`list[ArrayAnnotation]`): Array annotations to combine.
 * `classes` (`list[int]`): Class IDs associated with the annotations.
 * `n_classes` (`int`): Total number of known classes.

Returns

 * `np.ndarray`: Combined arrays of shape (N, C, …) where C is the number of classes and N is the number of instances.

##### to_numpy

```python
def to_numpy(self) -> np.ndarray:
```

Load the array from the file path.

#### Attributes

##### path

Path to the array saved as a `.npy` file.

### BBoxAnnotation

Bounding box annotation.

Values are normalized based on the image size.

#### Methods

##### combine_to_numpy

```python
def combine_to_numpy(annotations: list[BBoxAnnotation], classes: list[int], n_classes: int | None = None) -> np.ndarray:
```

Combine bounding box annotations into rows with class IDs.

Parameters

 * `annotations` (`list[BBoxAnnotation]`): Bounding box annotations to combine.
 * `classes` (`list[int]`): Class IDs associated with the annotations.
 * `n_classes` (`int | None`): Unused class count kept for API compatibility.

Returns

 * `np.ndarray`: An array of shape (N, 5) where N is the number of bounding box annotations and each row is in the format
   `[class_id, x, y, w, h]`.

##### to_numpy

```python
def to_numpy(class_id: int) -> np.ndarray:
```

Convert the bounding box annotation to row format.

Parameters

 * `class_id` (`int`): The numeric class ID of the annotation.

Returns

 * `np.ndarray`: An array of shape (5,) in the format `[class_id, x, y, w, h]`.

#### Attributes

##### h

Normalized bounding box height.

##### w

Normalized bounding box width.

##### x

Normalized top-left x coordinate.

##### y

Normalized top-left y coordinate.

### Category

Category label for metadata values.

This class is used to distinguish categorical metadata values from free-form string values.

### ClassificationAnnotation

Dummy wrapper annotation for classification tasks.

There is no explicit annotation field for classification tasks, instead the class name of a detection is interpreted as the class
label and interpreted as belonging to the entire image.

Multiple classification annotations are multi-hot encoded into a single vector with length equal to the total number of classes.

#### Methods

##### combine_to_numpy

```python
def combine_to_numpy(annotations: list[ClassificationAnnotation], classes: list[int], n_classes: int) -> np.ndarray:
```

Combine classification annotations into a multi-hot label vector.

Parameters

 * `annotations` (`list[ClassificationAnnotation]`): Classification annotations to combine.
 * `classes` (`list[int]`): Class IDs associated with the annotations.
 * `n_classes` (`int`): Total number of known classes.

Returns

 * `np.ndarray`: Multi-hot class label vector of shape (N,) where N is the total number of classes.

### DatasetRecord

Dataset record containing file paths and an optional annotation.

#### Methods

##### decode_metadata

```python
def decode_metadata(value: Any) -> Params:
```

##### to_parquet_rows

```python
def to_parquet_rows(self) -> Iterable[ParquetRecord]:
```

Recursively convert the dataset record and all its annotations and sub-annotations to parquet rows.

Yields

 * Annotation data rows.

##### validate_files

```python
def validate_files(values: dict[str, Any]) -> dict[str, Any]:
```

##### validate_task_name

```python
def validate_task_name(values: dict[str, Any]) -> dict[str, Any]:
```

##### validate_task_name_valid_identifier

```python
def validate_task_name_valid_identifier(self) -> Self:
```

#### Attributes

##### all_file_paths

All file paths associated with the dataset record.

> **Deprecated**
> Deprecated since version 0.9.0:Use `list(record.files.values())` instead.

##### annotation

Optional detection associated with the dataset record.

##### file

The file path of the dataset record.

This property is provided for convenience when the dataset record has exactly one file.

Raises

 * `ValueError`: If the dataset record has zero or multiple files.

##### files

File paths keyed by source name.

##### sample_metadata

##### task_name

The name of the task to which the record belongs.

### Detection

Detection record containing annotations and metadata for one object.

It describes a single detected object in an image and can contain various types of annotations and metadata as well as nested
sub-detections for hierarchical annotations.

When `scale_to_boxes` is enabled, keypoints and segmentation data are interpreted relative to the bounding box and rescaled to
image-normalized coordinates.

> **Example**
> ```pycon
>>> detection = Detection(
...     class_name="person",
...     instance_id=1,
...     metadata={
...         "category": Category("adult"),
...     },
...     boundingbox={
...         "x": 0.1,
...         "y": 0.2,
...         "w": 0.3,
...         "h": 0.4,
...     },
...     instance_segmentation={
...         "mask": np.array([[0, 1], [1, 0]]),
...     },
...     sub_detections={
...         "face": {
...             "class_name": "face",
...             "boundingbox": {
...                 "x": 0.2,
...                 "y": 0.3,
...                 "w": 0.1,
...                 "h": 0.1,
...             },
...             "keypoints": {
...                 "keypoints": [
...                     (0.25, 0.35, 2),  # left eye
...                     (0.3, 0.35, 2),  # right eye
...                 ],
...             },
...             "metadata": {
...                 "expression": Category("happy"),
...                 "eye_color": Category("blue"),
...             },
...         },
...     },
... )
```

#### Methods

##### get_task_types

```python
def get_task_types(self) -> set[str]:
```

Get all the task type associated with this detection.

> **Example**
> ```pycon
>>> detection = Detection(
...     class_name="cat",
...     boundingbox=BBoxAnnotation(x=0.1, y=0.2, w=0.3, h=0.4),
...     metadata={"color": "black"},
... )
>>> sorted(detection.get_task_types())
['boundingbox', 'classification', 'metadata/color']
```

Returns

 * `set[str]`: Annotation task types and metadata keys.

#### Attributes

##### array

Optional array annotation.

##### boundingbox

Optional bounding box annotation.

##### class_name

optional class name for the detection. Input data may use the `"class"` alias.

##### instance_id

Instance identifier. If not provided, the instance IDs will correspond to the order in which the detections were added to the
dataset. Note that this might lead to incorrect pairing of instance annotations if individual detection types are added separately
and in an inconsistent order across records:

```pycon
# Without specifying `instance_id`, the
# bounding box and keypoint annotation will
# not be correctly paired as they are added in separate
# detections and in a different order.
def generator():
    yield {
        "file": ...,
        "annotation": {"boundingbox": bbox1},
    }
    yield {
        "file": ...,
        "annotation": {"keypoints": kpts2},
    }
    yield {
        "file": ...,
        "annotation": {"boundingbox": bbox2},
    }
    yield {
        "file": ...,
        "annotation": {"keypoints": kpts1},
    }
```

It is recommended to provide instance IDs if possible and to avoid generating annotations individually in separate detections:

```pycon
# This is the correct way
def generator():
    yield {
        "file": ...,
        "annotation": {
            "instance_id": 1,
            "boundingbox": bbox1
            "keypoints": kpts1,
        },
    }
    yield {
        "file": ...,
        "annotation": {
            "instance_id": 2,
            "boundingbox": bbox2
            "keypoints": kpts2,
        },
    }
```

##### instance_segmentation

Optional instance segmentation annotation.

##### keypoints

Optional keypoint annotation.

##### metadata

Metadata values keyed by metadata name.

##### scale_to_boxes

Whether annotation coordinates should be rescaled from bounding-box-relative coordinates.

##### segmentation

Optional semantic segmentation annotation.

##### sub_detections

Nested detections keyed by sub-detection name.

### InstanceSegmentationAnnotation

Instance segmentation annotation.

Subclass of
[SegmentationAnnotation](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
used to distinguish instance segmentation annotations from semantic segmentation annotations.

The array representation of a single instance segmentation annotation is the same as that of a semantic segmentation annotation,
but multiple instance segmentation annotations are combined into an array of shape (N, H, W) where the leading dimension N
corresponds to the number of instance annotations instead of the number of classes as in semantic segmentation.

#### Methods

##### combine_to_numpy

```python
def combine_to_numpy(annotations: list[InstanceSegmentationAnnotation], classes: list[int] | None = None, n_classes: int | None = None) -> np.ndarray:
```

Combine instance segmentation annotations into instance masks.

> **Note**
> As opposed to semantic segmentation, overlapping annotations are allowed and are not resolved in any way. One pixel can belong to multiple instances and will be marked as 1 in each instance mask it belongs to.

Parameters

 * `annotations` (`list[InstanceSegmentationAnnotation]`): Instance segmentation annotations to combine.
 * `classes` (`list[int] | None`): Unused class IDs kept for API compatibility.
 * `n_classes` (`int | None`): Unused class count kept for API compatibility.

Returns

 * `np.ndarray`: Combined instance segmentation masks of shape (N, H, W) where N is the number of instances.

### KeypointAnnotation

Keypoint annotation.

The coordinates are normalized to [0, 1] based on the image size.

#### Methods

##### combine_to_numpy

```python
def combine_to_numpy(annotations: list[KeypointAnnotation], classes: list[int] | None = None, n_classes: int | None = None) -> np.ndarray:
```

Combine keypoint annotations into flattened keypoint rows.

Parameters

 * `annotations` (`list[KeypointAnnotation]`): Keypoint annotations to combine.
 * `classes` (`list[int] | None`): Unused class IDs kept for API compatibility.
 * `n_classes` (`int | None`): Unused class count kept for API compatibility.

Returns

 * `np.ndarray`: An array of shape (N, 3K) where N is the number of keypoint annotations and K is the number of keypoints per
   annotation. Flattened keypoint rows. Each row contains keypoint coordinates and visibility in the format [x1, y1, v1, x2, y2,
   v2, …] where (xi, yi, vi) are the coordinates and visibility of the i-th keypoint.

##### to_numpy

```python
def to_numpy(self) -> np.ndarray:
```

Convert the keypoint annotation to flattened row format.

Returns

 * `np.ndarray`: An array of shape (3K,) where K is the number of keypoints. The format of the array is [x1, y1, v1, x2, y2, v2,
   …] where (xi, yi, vi) are the coordinates and visibility of the i-th keypoint.

#### Attributes

##### keypoints

Keypoints in `(x, y, visibility)` format. Visibility follows the COCO convention:

 * 0: Not visible or not labeled.
 * 1: Occluded.
 * 2: Visible.

### SegmentationAnnotation

Run-length encoded segmentation mask.

The encoded mask uses COCO-style [run-length encoding](https://en.wikipedia.org/wiki/Run-length_encoding).

This class support parsing segmentation masks from multiple input formats:

 * Run-length encoding (RLE) directly as a list of counts or as a byte string.
 * Binary mask arrays as numpy arrays or saved as `.npy` or `.png` files.
 * Polygons as lists of normalized points together with the image width and height.

> **Example**
> ```pycon
>>> rle = SegmentationAnnotation(
...     height=4,
...     width=4,
...     counts=b'11213ON0'
... )
>>> mask = SegmentationAnnotation(
...     mask=np.array(
...         [
...            [0, 1, 0, 0],
...            [1, 1, 0, 0],
...            [0, 0, 0, 0],
...            [0, 0, 1, 1],
...         ]
...     )
... )
>>> np.array_equal(rle.to_numpy(), mask.to_numpy())
True
```

> **Note**
> When providing the RLE as a list of counts instead of encoded bytes make sure the counts follow FORTRAN (column-major) order as expected by the COCO RLE format.

#### Methods

##### combine_to_numpy

```python
def combine_to_numpy(annotations: list[SegmentationAnnotation], classes: list[int], n_classes: int) -> np.ndarray:
```

Combine segmentation annotations into class masks.

> **Note**
> In case of overlapping annotations, the first mask in the list takes precedence.

Parameters

 * `annotations` (`list[SegmentationAnnotation]`): Segmentation annotations to combine.
 * `classes` (`list[int]`): Class IDs associated with the annotations.
 * `n_classes` (`int`): Total number of known classes.

Returns

 * `np.ndarray`: Combined semantic segmentation masks of shape (C, H, W).

##### to_numpy

```python
def to_numpy(self) -> np.ndarray:
```

Convert the segmentation annotation to a binary mask.

Returns

 * `np.ndarray`: Binary mask of shape (H, W).

#### Attributes

##### counts

Run-length encoded mask data.

##### height

The height of the segmentation mask.

##### width

The width of the segmentation mask.

## Functions

### load_annotation

```python
def load_annotation(task_type: Literal['classification', 'boundingbox', 'keypoints', 'segmentation', 'instance_segmentation',
'array'], data: dict[str, Any]) -> Annotation:
```

Load an annotation from serialized data.

Parameters

 * `task_type` (`Literal['classification', 'boundingbox', 'keypoints', 'segmentation', 'instance_segmentation', 'array']`): The type of the annotation task.
 * `data` (`dict[str, Any]`): Serialized annotation data.

Returns

 * `Annotation`: An instance of the appropriate [Annotation](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md) subclass based on the task type.

Raises

 * `ValueError`: If the task type is unknown.

## Attributes

### KeypointVisibility

Keypoint visibility following the COCO convention.

The values indicate the visibility of a keypoint in an image:

 * 0: Not visible or not labeled.
 * 1: Occluded.
 * 2: Visible.

### NormalizedFloat

A float value normalized to the range [0, 1].