# parquet

Python API: `luxonis_ml.data.utils.parquet`

Parquet row helpers for Luxonis Data Format annotations.

Most users should interact with
[DatasetRecord.sample_metadata](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/ldf/annotation.md)
and
[LoaderOutput.metadata](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/typing.md).
This module documents the lower-level storage shape: `sample_metadata` is stored in parquet as serialized JSON text and decoded
back to a dictionary when records are loaded or exported.

## Classes

| Name | Description |
| --- | --- |
|
[`ParquetFileManager`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/parquet.md)
| Manage append-style writes across partitioned parquet files. |

### class `ParquetFileManager`

```python
class ParquetFileManager
```

Manage append-style writes across partitioned parquet files.

Rows are buffered in memory and flushed to the current parquet file. A new file is selected every `num_rows` writes. Filenames are
zero-padded numeric counters such as `0000000000.parquet`.

Existing parquet shards that do not have `sample_metadata` are read as if every row contained `"{}"`. This keeps older LDF
datasets compatible with record-level metadata.

Attributes

| Name | Description |
| --- | --- |
| `buffer` | In-memory column buffer for the current file. |
| `current_file` | Path to the current parquet file. |
| `dir` | Directory containing parquet files. |
| `num` | Current parquet file index. |
| `num_rows` | Maximum rows written to one parquet file. |
| `parquet_files` | Existing parquet files discovered in `dir`. |
| `row_count` | Number of rows currently buffered or loaded from the current file. |

Methods

| Name | Description |
| --- | --- |
|
[`__init__`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/parquet.md)
| Manage writing rows into partitioned parquet files. |
|
[`remove_duplicate_uuids`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/parquet.md)
| |
|
[`write`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/parquet.md)
| Write a row to the current parquet file. |

#### method `__init__`

```python
def __init__(directory: PathType, num_rows: int = 100000)
```

Manage writing rows into partitioned parquet files.

> **Example**
> ```pycon
>>> record = {
...     "file": "image.jpg",
...     "source_name": "image",
...     "task_name": "detection",
...     "class_name": "car",
...     "instance_id": 0,
...     "task_type": "boundingbox",
...     "annotation": "{}",
...     "sample_metadata": "{}",
... }
>>> manager = ParquetFileManager(
...     "/tmp/ldf-parquet-example", num_rows=2
... )
>>> manager.num_rows
2
```

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `directory` | `PathType` | Local directory where parquet files are stored. | required |
| `num_rows` | `int` | Maximum rows per parquet file before a new file is created. | `100000` |

#### method `remove_duplicate_uuids`

```python
def remove_duplicate_uuids(overwrite_uuids: set[str])
```

#### method `write`

```python
def write(uuid: str, data: ParquetRecord, group_id: str)
```

Write a row to the current parquet file.

Args

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `uuid` | `str` | Unique row identifier. | required |
| `data` | `ParquetRecord` | Annotation row data. | required |
| `group_id` | `str` | Unique identifier for the sample group. | required |

## Module Attributes

| Name | Type | Description |
| --- | --- | --- |
| [`DEFAULT_METADATA`](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/utils/parquet.md) | `str` | Serialized empty metadata object used for old or missing metadata rows. |

### constant `DEFAULT_METADATA`

```python
DEFAULT_METADATA: str = '{}'
```

Serialized empty metadata object used for old or missing metadata rows.
