# LuxonisParser

## Overview

`LuxonisParser` converts supported third-party dataset layouts into the Luxonis Data Format (LDF). It validates the input
structure, detects the format when possible, converts its annotations, and creates a `LuxonisDataset` with split definitions.

Import it from the public data package:

```python
from luxonis_ml.data import LuxonisParser

dataset = LuxonisParser(
    "path/to/dataset",
    dataset_name="my_dataset",
).parse()
```

Pass a `DatasetType` when detection is ambiguous or when the same directory layout can represent different tasks:

```python
from luxonis_ml.data import LuxonisParser
from luxonis_ml.enums import DatasetType

dataset = LuxonisParser(
    "path/to/yolo_dataset",
    dataset_name="people_pose",
    dataset_type=DatasetType.YOLOV8KEYPOINTS,
    task_name="pose",
).parse()
```

`task_name` can also map classes to task groups:

```python
parser = LuxonisParser(
    "path/to/dataset",
    task_name={
        "head": "head_pose",
        "neck": "head_pose",
        "torso": "body_pose",
        "leg": "body_pose",
    },
)
```

## Input Locations

The parser accepts:

 * a local directory
 * a local ZIP archive
 * `gs://...` or `gcs://...` paths in Google Cloud Storage
 * `s3://...` paths in S3-compatible storage
 * `roboflow://workspace/project/version/format` dataset references
 * `ultralytics://username/datasets/slug` Ultralytics Platform references

Select a specific Ultralytics dataset version with `?v=<version>`:

```python
dataset = LuxonisParser(
    "ultralytics://username/datasets/people?v=3",
    dataset_name="people",
).parse()
```

Roboflow sources require `ROBOFLOW_API_KEY`; Ultralytics Platform sources require `ULTRALYTICS_API_KEY`. Google Cloud Storage and
S3 use the credentials supported by `LuxonisFileSystem`. Use `save_dir` to choose where a remote source is downloaded before
parsing.

> **Note**
> A ZIP archive can contain the dataset layout at its root. When an archive has one top-level directory and that directory contains a recognizable dataset layout, the parser automatically uses it as the dataset root.

## Supported Formats

| Format | `DatasetType` value | Imported annotations |
| --- | --- | --- |
| COCO JSON, FiftyOne or Roboflow layout | `coco` | Bounding boxes, semantic and instance segmentation, keypoints |
| Pascal VOC XML | `voc` | Bounding boxes |
| YOLO Darknet TXT | `darknet` | Bounding boxes |
| YOLOv4 PyTorch TXT | `yolov4` | Bounding boxes |
| MT YOLOv6 | `yolov6` | Bounding boxes |
| YOLOv8-v12 bounding boxes | `yolov8` | Bounding boxes |
| YOLOv8-v12 instance segmentation | `yolov8instancesegmentation` | Instance segmentation |
| YOLOv8-v12 keypoints | `yolov8keypoints` | Keypoints |
| Ultralytics NDJSON detection | `ultralytics-ndjson` | Bounding boxes |
| Ultralytics NDJSON instance segmentation | `ultralytics-ndjson-instancesegmentation` | Instance segmentation |
| Ultralytics NDJSON keypoints | `ultralytics-ndjson-keypoints` | Keypoints |
| CreateML JSON | `createml` | Bounding boxes |
| TensorFlow Object Detection CSV | `tfcsv` | Bounding boxes |
| Unity SOLO | `solo` | Bounding boxes, semantic and instance segmentation, keypoints |
| Classification directory | `clsdir` | Classification |
| FiftyOne classification | `fiftyone-classification` | Classification |
| Segmentation mask directory | `segmask` | Semantic segmentation |
| Native LDF export | `native` | LDF records supported by the native parser |

Class-bearing annotations also populate the corresponding LDF classification target.

For YOLOv8-v12 data, explicitly select the bounding-box, instance-segmentation, or keypoint dataset type because those variants
can share the same directory structure.

## Expected Directory Layouts

The parser recognizes complete dataset directories and, for supported formats, individual split directories. Validation split
names such as `valid` and `validation` are normalized to `val` in the resulting LDF dataset.

### COCO JSON

FiftyOne COCO exports keep media in `data/` and annotations in `labels.json`:

```plaintext
dataset_dir/
├── train/
│   ├── data/
│   │   ├── img1.jpg
│   │   └── img2.jpg
│   └── labels.json
├── validation/
│   ├── data/
│   └── labels.json
└── test/
    ├── data/
    └── labels.json
```

Roboflow COCO exports keep each split's images beside `_annotations.coco.json`:

```plaintext
dataset_dir/
├── train/
│   ├── img1.jpg
│   ├── img2.jpg
│   └── _annotations.coco.json
├── valid/
└── test/
```

### YOLOv8-v12

Roboflow-style exports use `images/` and `labels/` inside each split:

```plaintext
dataset_dir/
├── train/
│   ├── images/
│   │   ├── img1.jpg
│   │   └── img2.jpg
│   └── labels/
│       ├── img1.txt
│       └── img2.txt
├── valid/
├── test/
└── data.yaml
```

The Ultralytics layout groups all split directories under top-level `images/` and `labels/` directories:

```plaintext
dataset_dir/
├── images/
│   ├── train/
│   ├── val/
│   └── test/
├── labels/
│   ├── train/
│   ├── val/
│   └── test/
└── data.yaml
```

### Ultralytics NDJSON

An NDJSON dataset contains one `.ndjson` manifest. Its first line is the dataset header with `class_names`; each following line
describes one image and its split and annotations.

```plaintext
dataset_dir/
├── dataset.ndjson
├── train/
├── val/
└── test/
```

Image records can point to local files or remote URLs. A manifest containing only remote image references does not need image
directories, and you can pass the manifest file itself to `LuxonisParser`.

### Classification Directory

A split-based classification dataset uses one directory per class:

```plaintext
dataset_dir/
├── train/
│   ├── car/
│   │   ├── img1.jpg
│   │   └── img2.jpg
│   └── motorcycle/
├── valid/
└── test/
```

A flat classification directory places the class directories directly under the dataset root. The parser creates random splits for
this form. An optional `info.json` file may also be present.

### Segmentation Mask Directory

Each `.jpg` image has a matching `_mask.png` file. `_classes.csv` maps grayscale pixel values to class names:

```plaintext
dataset_dir/
├── train/
│   ├── img1.jpg
│   ├── img1_mask.png
│   └── _classes.csv
├── valid/
└── test/
```

```csv
Pixel Value,Class
0,background
1,car
2,motorcycle
```

### Native LDF Export

The native parser reads the layout produced by `LuxonisDataset.export(..., dataset_type=DatasetType.NATIVE)`:

```plaintext
dataset_dir/
├── metadata.json
├── train/
│   ├── annotations.json
│   └── images/
├── val/
│   ├── annotations.json
│   └── images/
└── test/
    ├── annotations.json
    └── images/
```

See the [parser API
reference](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/parsers.md)
for the exact layouts and parser-specific options for Pascal VOC, Darknet, YOLOv4, YOLOv6, CreateML, TensorFlow CSV, SOLO,
FiftyOne classification, and native records.

## Split Behavior

When the source contains split directories and no override is provided, their assignments are preserved. A source represented by
one split or a flat layout is divided into `train`, `val`, and `test` using the default 80:10:10 ratio.

Override the assignments with floating-point ratios that sum to `1.0`:

```python
dataset = parser.parse(
    split_ratios={"train": 0.7, "val": 0.2, "test": 0.1}
)
```

Ratio-based splitting shuffles and redistributes all parsed samples. Integer values request absolute counts. For a source with
existing split directories, records are sampled from the corresponding original split so the split boundaries remain intact:

```python
dataset = parser.parse(
    split_ratios={"train": 1000, "val": 100, "test": 50}
)
```

When a requested count exceeds the available records in that split, all available records are used.

## Command-Line Interface

Use `luxonis_ml data parse` for the same workflow:

```bash
luxonis_ml data parse path/to/dataset --name my_dataset
luxonis_ml data parse path/to/yolo_dataset \
    --name people_pose \
    --type yolov8keypoints \
    --task-name pose
luxonis_ml data parse path/to/dataset \
    --train 0.8 \
    --val 0.1 \
    --test 0.1
luxonis_ml data parse path/to/dataset \
    --train 1000 \
    --val 100 \
    --test 50
```

Run `luxonis_ml data parse --help` for storage selection, download location, format selection, warning reporting, and deletion
options.