# parquet

Python API: `luxonis_ml.data.utils.parquet`

Parquet row helpers for Luxonis Data Format annotations.

Most users should interact with
[DatasetRecord.sample_metadata](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/datasets/annotation.md)
and
[LoaderOutput.metadata](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/typing.md).
This module documents the lower-level storage shape: `sample_metadata` is stored in parquet as serialized JSON text and decoded
back to a dictionary when records are loaded or exported.

## Classes

### ParquetFileManager

Manage append-style writes across partitioned parquet files.

Rows are buffered in memory and flushed to the current parquet file. A new file is selected every `num_rows` writes. Filenames are
zero-padded numeric counters such as `0000000000.parquet`.

Existing parquet shards that do not have `sample_metadata` are read as if every row contained `"{}"`. This keeps older LDF
datasets compatible with record-level metadata.

#### Methods

##### init

```python
def __init__(directory: PathType, num_rows: int = 100000):
```

Manage writing rows into partitioned parquet files.

> **Example**
> ```pycon
>>> record = {
...     "file": "image.jpg",
...     "source_name": "image",
...     "task_name": "detection",
...     "class_name": "car",
...     "instance_id": 0,
...     "task_type": "boundingbox",
...     "annotation": "{}",
...     "sample_metadata": "{}",
... }
>>> manager = ParquetFileManager(
...     "/tmp/ldf-parquet-example", num_rows=2
... )
>>> manager.num_rows
2
```

Parameters

 * `directory` (`PathType`): Local directory where parquet files are stored.
 * `num_rows` (`int`): Maximum rows per parquet file before a new file is created.

##### remove_duplicate_uuids

```python
def remove_duplicate_uuids(overwrite_uuids: set[str]):
```

##### write

```python
def write(uuid: str, data: ParquetRecord, group_id: str):
```

Write a row to the current parquet file.

Parameters

 * `uuid` (`str`): Unique row identifier.
 * `data` (`ParquetRecord`): Annotation row data.
 * `group_id` (`str`): Unique identifier for the sample group.

#### Attributes

##### buffer

In-memory column buffer for the current file.

##### current_file

Path to the current parquet file.

##### dir

Directory containing parquet files.

##### num

Current parquet file index.

##### num_rows

Maximum rows written to one parquet file.

##### parquet_files

Existing parquet files discovered in `dir`.

##### row_count

Number of rows currently buffered or loaded from the current file.

### ParquetRecord

Single annotation row written to parquet.

> **Example**
> ```python
{
    "file": "/data/images/frame_001.jpg",
    "source_name": "image",
    "task_name": "detection",
    "class_name": "person",
    "instance_id": 0,
    "task_type": "boundingbox",
    "annotation": '{"x":0.1,"y":0.2,"w":0.3,"h":0.4}',
    "sample_metadata": '{"record_id":123,"camera":"left"}',
}
```

#### Attributes

##### annotation

Optional serialized annotation JSON.

##### class_name

Optional class name.

##### file

Image or source file path.

##### instance_id

Optional instance identifier.

##### sample_metadata

Serialized JSON object for record-level metadata. Empty metadata is stored as `"{}"`.

##### source_name

Source component name.

##### task_name

Task name.

##### task_type

Optional task type.

## Attributes

### DEFAULT_METADATA

Serialized empty metadata object used for old or missing metadata rows.