# base_dataset

Python API: `luxonis_ml.data.datasets.base_dataset`

## Classes

| Name | Description |
| --- | --- |
|
[`BaseDataset`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Base class for datasets in the Luxonis MLOps ecosystem. |

### class `BaseDataset`

```python
class BaseDataset(ABC)
```

Base class for datasets in the Luxonis MLOps ecosystem.

Properties

| Name | Type | Description |
| --- | --- | --- |
|
[`identifier`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| `str` | The unique identifier for the dataset. |
|
[`version`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| `Version` | The version of the underlying LDF. |

Methods

| Name | Description |
| --- | --- |
|
[`add`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Write annotations to parquet files. |
|
[`delete_dataset`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Delete local files belonging to the dataset. |
|
[`exists`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Check whether a dataset exists. |
|
[`get_class_names`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Return class names per task. |
|
[`get_classes`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Get class names and IDs per task. |
|
[`get_n_classes`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Return number of classes per task. |
|
[`get_n_keypoints`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Return the number of keypoints for each task. |
|
[`get_skeletons`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Return keypoint skeletons for each task. |
|
[`get_source_names`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Return input source names for the dataset. |
|
[`get_task_names`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Return task names for the dataset. |
|
[`get_tasks`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Return task names and task types. |
|
[`make_splits`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Create dataset splits for training, validation, and testing. |
|
[`set_classes`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Set classes for one or more tasks. |
|
[`set_skeletons`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Set keypoint skeleton semantics for tasks that use keypoints. |
|
[`set_tasks`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Set dataset tasks. |
|
[`update_source`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| Update the dataset source definition. |

#### property `identifier`

```python
identifier: str
```

The unique identifier for the dataset.

#### property `version`

```python
version: Version
```

The version of the underlying LDF.

#### method `add`

```python
def add(generator: DatasetIterator, batch_size: int = 1000000)
```

Write annotations to parquet files.

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `generator` | `DatasetIterator` | Iterator yielding `DatasetRecord` instances or dictionaries that can be converted to
`DatasetRecord`. | required |
| `batch_size` | `int` | Number of records to buffer before processing. Lower values reduce peak memory usage. | `1000000` |

#### method `delete_dataset`

```python
def delete_dataset()
```

Delete local files belonging to the dataset.

#### static method `exists`

```python
def exists(dataset_name: str) -> bool
```

Check whether a dataset exists.

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `dataset_name` | `str` | Dataset name to check. | required |

Returns

| Type | Description |
| --- | --- |
| `bool` | `True` if the dataset exists, `False` otherwise. |

#### method `get_class_names`

```python
def get_class_names() -> dict[str, list[str]]
```

Return class names per task.

Returns

 * `Class names keyed by task name`:```python
   {
       "vehicles": ["red", "green", "blue"],
       "brands": ["audi", "bmw", "mercedes"],
   }
   ```

#### method `get_classes`

```python
def get_classes() -> dict[str, dict[str, int]]
```

Get class names and IDs per task.

Returns

 * `Mapping from class names to class IDs grouped by task name`:```python
   {
       "color": {"red": 0, "green": 1, "blue": 2},
       "brand": {"audi": 0, "bmw": 1, "mercedes": 2},
   }
   ```

#### method `get_n_classes`

```python
def get_n_classes() -> dict[str, int]
```

Return number of classes per task.

Returns

| Type | Description |
| --- | --- |
| `dict[str, int]` | Mapping from task names to class counts. |

#### method `get_n_keypoints`

```python
def get_n_keypoints() -> dict[str, int]
```

Return the number of keypoints for each task.

Returns

| Type | Description |
| --- | --- |
| `dict[str, int]` | Number of keypoints keyed by task name. |

#### method `get_skeletons`

```python
def get_skeletons() -> dict[str, tuple[list[str], list[tuple[int, int]]]]
```

Return keypoint skeletons for each task.

Returns

| Type | Description |
| --- | --- |
| `dict[str, tuple[list[str], list[tuple[int, int]]]]` | Keypoint labels and edges keyed by task name. |

#### method `get_source_names`

```python
def get_source_names() -> list[str]
```

Return input source names for the dataset.

Returns

| Type | Description |
| --- | --- |
| `list[str]` | Source names used to identify input data. |

#### method `get_task_names`

```python
def get_task_names() -> list[str]
```

Return task names for the dataset.

This is equivalent to
[get_tasks](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
but returns only the task names.

Returns

| Type | Description |
| --- | --- |
| `list[str]` | Task names. |

#### method `get_tasks`

```python
def get_tasks() -> dict[str, list[str]]
```

Return task names and task types.

Returns

| Type | Description |
| --- | --- |
| `dict[str, list[str]]` | Task types keyed by task name. |

#### method `make_splits`

```python
def make_splits(
    splits: Mapping[str, Sequence[PathType]] | Mapping[str, float] | tuple[float, float, float] | None = None,
    *,
    replace_old_splits: bool = False,
)
```

Create dataset splits for training, validation, and testing.

> **Note**
> Although `"train"`, `"val"`, and `"test"` are the conventional split names, you can use any split names you want by providing a mapping to the `splits` argument. This can be useful for combining records from multiple sources (`"train_real"`, `"train_synth"`) or for creating fully custom splits.

Args

 * `splits` (`Mapping[str, Sequence[PathType]] | Mapping[str, float] | tuple[float, float, float] | None`): A mapping defining the
   splits. Can be one of the following: * A mapping of split names to lists of file paths.
    * A mapping of split names to ratios.
    * A tuple of three ratios for train, val, and test splits. A ratio is a number from 0 to 1, and the ratios sum to 1. `1` and
      `1.0` both work.
 * `replace_old_splits` (`bool`): Whether to replace old splits with new ones. If `False` (default), new splits are added to the
   existing splits. If `True`, the existing splits are discarded first.

Raises

| Type | Description |
| --- | --- |
| `TypeError` | If the mapping values are neither ratios nor filepath lists. The method warns and skips each element of a filepath
list that is not a filepath. |
| `ValueError` | If `splits` is provided but is empty. |
| `ValueError` | If the ratios are outside the range from 0 to 1 or do not sum to 1. |
| `ValueError` | If split ratios are used but all the data already belongs to a split while `replace_old_splits` is `False`. |
| `FileNotFoundError` | If the dataset is empty. |

#### method `set_classes`

```python
def set_classes(classes: list[str] | dict[str, int], task: str | None = None)
```

Set classes for one or more tasks.

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `classes` | `list[str] | dict[str, int]` | Class names, or class IDs keyed by class name. If class names are provided, IDs are
assigned alphabetically starting from 0. A class named `"background"` is always assigned ID 0. | required |
| `task` | `str | None` | Optional task to update. If omitted, all tasks are updated. | `None` |

#### method `set_skeletons`

```python
def set_skeletons(
    labels: list[str] | None = None,
    edges: list[tuple[int, int]] | None = None,
    task: str | None = None,
)
```

Set keypoint skeleton semantics for tasks that use keypoints.

For example:

```python
dataset.set_skeletons(
    labels=["right hand", "right shoulder", ...],
    edges=[[0, 1], [4, 5], ...]
)
```

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `labels` | `list[str] | None` | Optional keypoint names. | `None` |
| `edges` | `list[tuple[int, int]] | None` | Optional edges between keypoints. | `None` |
| `task` | `str | None` | Optional task to update. If omitted, all keypoint tasks are updated. | `None` |

Raises

| Type | Description |
| --- | --- |
| `ValueError` | If neither `labels` nor `edges` are provided. |

#### method `set_tasks`

```python
def set_tasks(tasks: dict[str, list[str]])
```

Set dataset tasks.

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `tasks` | `dict[str, list[str]]` | Mapping from task names to task types. | required |

#### method `update_source`

```python
def update_source(source: LuxonisSource)
```

Update the dataset source definition.

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `source` | `LuxonisSource` | Source definition to store. | required |

## Module Attributes

| Name | Type | Description |
| --- | --- | --- |
|
[`DatasetIterator`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| `TypeAlias` | |
|
[`DATASETS_REGISTRY`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/base_dataset.md)
| `Registry[type[BaseDataset]]` | |

### type alias `DatasetIterator`

```python
DatasetIterator: TypeAlias = Iterator[dict | DatasetRecord]
```

### constant `DATASETS_REGISTRY`

```python
DATASETS_REGISTRY: Registry[type[BaseDataset]] = Registry(name='datasets')
```
