# data_utils

Python API: `luxonis_ml.data.utils.data_utils`

## Functions

### find_duplicates

```python
def find_duplicates(df: pl.LazyFrame) -> dict[str, list[dict[str, Any]]]:
```

Collect duplicate UUID and annotation information.

Parameters

 * `df` (`pl.LazyFrame`): Dataset information.

Returns

 * `Dictionary with`: * `"duplicate_uuids"`: List of dictionaries containing a duplicated `"uuid"` and all `"files"` associated
   with it.
    * `"duplicate_annotations"`: List of dictionaries containing `"file_name"`, `"task_type"`, `"task_name"`, serialized
      `"annotation"` (or `"<binary mask>"` for segmentation), and duplicate `"count"`.

### get_class_distributions

```python
def get_class_distributions(df: pl.LazyFrame) -> dict[str, dict[str, list[dict[str, Any]]]]:
```

Return class distribution information for non-classification tasks.

Parameters

 * `df` (`pl.LazyFrame`): Dataset information.

Returns

 * `dict[str, dict[str, list[dict[str, Any]]]]`: Class counts grouped by task name and task type.

### get_duplicates_info

```python
def get_duplicates_info(df: pl.LazyFrame) -> dict[str, Any]:
```

Return duplicate UUID and annotation information.

Parameters

 * `df` (`pl.LazyFrame`): Dataset information.

Returns

 * `Dictionary with duplicate UUID and annotation details`: * `"duplicate_uuids"`: List of `{"uuid": str, "files": list[str]}`
   entries.
    * `"duplicate_annotations"`: List of entries with `"file_name"`, `"task_name"`, `"task_type"`, `"annotation"`, and `"count"`.

### get_heatmaps

```python
def get_heatmaps(df: pl.LazyFrame, sample_size: int | None = None, downsample_factor: int = 5) -> dict[str, dict[str, list[list[int]]]]:
```

Generate heatmaps for boxes, keypoints, and segmentation masks.

Heatmaps are accumulated on a fixed 15×15 grid over normalized image coordinates. Bounding boxes contribute their center points,
keypoints contribute visible points, and segmentation masks contribute foreground pixels after optional downsampling.

Parameters

 * `df` (`pl.LazyFrame`): Dataset information.
 * `sample_size` (`int | None`): Optional number of samples used to generate heatmaps.
 * `downsample_factor` (`int`): Factor used to downsample segmentation masks. A value of 5 keeps every fifth row and column.

Returns

 * `dict[str, dict[str, list[list[int]]]]`: Heatmaps grouped by task name and task type. Each heatmap is a 15×15 nested list of
   counts.

### get_missing_annotations

```python
def get_missing_annotations(df: pl.LazyFrame) -> list[str]:
```

Return file paths that have no annotations.

Parameters

 * `df` (`pl.LazyFrame`): Dataset information.

Returns

 * `list[str]`: File paths with no annotations.

### infer_task

```python
def infer_task(old_task: str, class_name: str | None, current_classes: dict[str, dict[str, int]]) -> str:
```

### merge_uuids

```python
def merge_uuids(uuids: Iterable[str]) -> uuid.UUID:
```

Merge multiple UUIDs into a single deterministic UUID, independent of order.

> **Examples**
> ```pycon
>>> first = "00000000-0000-0000-0000-000000000001"
>>> second = "00000000-0000-0000-0000-000000000002"
>>> merge_uuids([first, second]) == merge_uuids([second, first])
True
>>> isinstance(merge_uuids([first]), uuid.UUID)
True
```

Parameters

 * `uuids` (`Iterable[str]`): UUID strings to merge.

Returns

 * `uuid.UUID`: Deterministic merged UUID.

### rgb_to_bool_masks

```python
def rgb_to_bool_masks(segmentation_mask: np.ndarray, class_colors: dict[str, RGB], add_background_class: bool = False) ->
Iterator[tuple[str, np.ndarray]]:
```

Convert an RGB segmentation mask to boolean class masks.

> **Example**
> ```pycon
>>> segmentation_mask = np.array([
...     [[0, 0, 0], [255, 0, 0], [0, 255, 0]],
...     [[0, 0, 0], [0, 255, 0], [0, 0, 255]],
...     ], dtype=np.uint8)
>>> class_colors = {
...     "red": (255, 0, 0),
...     "green": (0, 255, 0),
...     "blue": (0, 0, 255),
... }
>>> for class_name, mask in rgb_to_bool_masks(
...     segmentation_mask,
...     class_colors,
...     add_background_class=True,
... ):
...     print(class_name, np.array2string(mask, separator=", "))
background [[ True, False, False],
            [ True, False, False]]
red        [[False,  True, False],
            [False, False, False]]
green      [[False, False,  True],
            [False, True, False]]
blue       [[False, False, False],
            [False, False,  True]]
```

Parameters

 * `segmentation_mask` (`np.ndarray`): RGB segmentation mask where each pixel color identifies its class.
 * `class_colors` (`dict[str, RGB]`): Mapping from class names to RGB colors.
 * `add_background_class` (`bool`): Whether to add a `"background"` mask for pixels that do not belong to any class. The
   background mask is yielded first.

Returns

 * `Iterator[tuple[str, np.ndarray]]`: Iterator of class names and their boolean masks.

### warn_on_duplicates

```python
def warn_on_duplicates(df: pl.LazyFrame):
```

Log warnings for duplicate UUIDs and annotations.

Parameters

 * `df` (`pl.LazyFrame`): Dataset information.