# data_utils

Python API: `luxonis_ml.data.utils.data_utils`

## Functions

| Name | Description |
| --- | --- |
|
[`find_duplicates`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/data_utils.md)
| Collect duplicate UUID and annotation information. |
|
[`get_class_distributions`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/data_utils.md)
| Return class distribution information for non-classification tasks. |
|
[`get_duplicates_info`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/data_utils.md)
| Return duplicate UUID and annotation information. |
|
[`get_heatmaps`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/data_utils.md)
| Generate heatmaps for boxes, keypoints, and segmentation masks. |
|
[`get_missing_annotations`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/data_utils.md)
| Return file paths that have no annotations. |
|
[`infer_task`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/data_utils.md)
| |
|
[`merge_uuids`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/data_utils.md)
| Merge multiple UUIDs into a single deterministic UUID, independent of order. |
|
[`rgb_to_bool_masks`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/data_utils.md)
| Convert an RGB segmentation mask to boolean class masks. |
|
[`warn_on_duplicates`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/data_utils.md)
| Log warnings for duplicate UUIDs and annotations. |

### function `find_duplicates`

```python
def find_duplicates(df: pl.LazyFrame) -> dict[str, list[dict[str, Any]]]
```

Collect duplicate UUID and annotation information.

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `df` | `pl.LazyFrame` | Dataset information. | required |

Returns

 * `Dictionary with`: * `"duplicate_uuids"`: List of dictionaries containing a duplicated `"uuid"` and all `"files"` associated
   with it.
    * `"duplicate_annotations"`: List of dictionaries containing `"file_name"`, `"task_type"`, `"task_name"`, serialized
      `"annotation"` (or `"<binary mask>"` for segmentation), and duplicate `"count"`.

### function `get_class_distributions`

```python
def get_class_distributions(
    df: pl.LazyFrame,
) -> dict[str, dict[str, list[dict[str, Any]]]]
```

Return class distribution information for non-classification tasks.

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `df` | `pl.LazyFrame` | Dataset information. | required |

Returns

| Type | Description |
| --- | --- |
| `dict[str, dict[str, list[dict[str, Any]]]]` | Class counts grouped by task name and task type. |

### function `get_duplicates_info`

```python
def get_duplicates_info(df: pl.LazyFrame) -> dict[str, Any]
```

Return duplicate UUID and annotation information.

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `df` | `pl.LazyFrame` | Dataset information. | required |

Returns

 * `Dictionary with duplicate UUID and annotation details`: * `"duplicate_uuids"`: List of `{"uuid": str, "files": list[str]}`
   entries.
    * `"duplicate_annotations"`: List of entries with `"file_name"`, `"task_name"`, `"task_type"`, `"annotation"`, and `"count"`.

### function `get_heatmaps`

```python
def get_heatmaps(
    df: pl.LazyFrame,
    sample_size: int | None = None,
    downsample_factor: int = 5,
) -> dict[str, dict[str, list[list[int]]]]
```

Generate heatmaps for boxes, keypoints, and segmentation masks.

Heatmaps are accumulated on a fixed 15×15 grid over normalized image coordinates. Bounding boxes contribute their center points,
keypoints contribute visible points, and segmentation masks contribute foreground pixels after optional downsampling.

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `df` | `pl.LazyFrame` | Dataset information. | required |
| `sample_size` | `int | None` | Optional number of samples used to generate heatmaps. | `None` |
| `downsample_factor` | `int` | Factor used to downsample segmentation masks. A value of 5 keeps every fifth row and column. | `5`
|

Returns

| Type | Description |
| --- | --- |
| `dict[str, dict[str, list[list[int]]]]` | Heatmaps grouped by task name and task type. Each heatmap is a 15×15 nested list of
counts. |

### function `get_missing_annotations`

```python
def get_missing_annotations(df: pl.LazyFrame) -> list[str]
```

Return file paths that have no annotations.

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `df` | `pl.LazyFrame` | Dataset information. | required |

Returns

| Type | Description |
| --- | --- |
| `list[str]` | File paths with no annotations. |

### function `infer_task`

```python
def infer_task(
    old_task: str,
    class_name: str | None,
    current_classes: dict[str, dict[str, int]],
) -> str
```

### function `merge_uuids`

```python
def merge_uuids(uuids: Iterable[str]) -> uuid.UUID
```

Merge multiple UUIDs into a single deterministic UUID, independent of order.

> **Examples**
> ```pycon
>>> first = "00000000-0000-0000-0000-000000000001"
>>> second = "00000000-0000-0000-0000-000000000002"
>>> merge_uuids([first, second]) == merge_uuids([second, first])
True
>>> isinstance(merge_uuids([first]), uuid.UUID)
True
```

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `uuids` | `Iterable[str]` | UUID strings to merge. | required |

Returns

| Type | Description |
| --- | --- |
| `uuid.UUID` | Deterministic merged UUID. |

### function `rgb_to_bool_masks`

```python
def rgb_to_bool_masks(
    segmentation_mask: np.ndarray,
    class_colors: dict[str, RGB],
    add_background_class: bool = False,
) -> Iterator[tuple[str, np.ndarray]]
```

Convert an RGB segmentation mask to boolean class masks.

> **Example**
> ```pycon
>>> segmentation_mask = np.array(
...     [
...         [[0, 0, 0], [255, 0, 0], [0, 255, 0]],
...         [[0, 0, 0], [0, 255, 0], [0, 0, 255]],
...     ],
...     dtype=np.uint8,
... )
>>> class_colors = {
...     "red": (255, 0, 0),
...     "green": (0, 255, 0),
...     "blue": (0, 0, 255),
... }
>>> for class_name, mask in rgb_to_bool_masks(
...     segmentation_mask,
...     class_colors,
...     add_background_class=True,
... ):
...     print(class_name, np.array2string(mask, separator=", "))
background [[ True, False, False],
            [ True, False, False]]
red        [[False,  True, False],
            [False, False, False]]
green      [[False, False,  True],
            [False, True, False]]
blue       [[False, False, False],
            [False, False,  True]]
```

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `segmentation_mask` | `np.ndarray` | RGB segmentation mask where each pixel color identifies its class. | required |
| `class_colors` | `dict[str, RGB]` | Mapping from class names to RGB colors. | required |
| `add_background_class` | `bool` | Whether to add a `"background"` mask for pixels that do not belong to any class. The
background mask is yielded first. | `False` |

Returns

| Type | Description |
| --- | --- |
| `Iterator[tuple[str, np.ndarray]]` | Iterator of class names and their boolean masks. |

### function `warn_on_duplicates`

```python
def warn_on_duplicates(df: pl.LazyFrame)
```

Log warnings for duplicate UUIDs and annotations.

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `df` | `pl.LazyFrame` | Dataset information. | required |
