# augmentations

Python API: `luxonis_ml.data.augmentations`

Augmentation engines and custom transforms for LDF samples.

This package provides the augmentation interface used by
[LuxonisLoader](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/loaders/luxonis_loader.md).
The default implementation is
[AlbumentationsEngine](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/albumentations_engine.md),
which adapts LDF labels to Albumentations targets before transformation and converts them back after transformation.

Table of Contents

 * [Configuration](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations.md)
 * [Custom
   Transforms](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations.md)
 * [Engine
   Interface](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations.md)
 * [Tips and
   Tricks](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations.md)
   * [Rotated Bounding Boxes Are Too
   Large](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations.md)

## Configuration

Augmentation configuration is a list of records. Each record contains a `name` identifying an Albumentations transform or a
transform registered in
[TRANSFORMATIONS](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom.md),
optional `params`, optional `use_for_resizing`, and optional stage filtering through `apply_on_stages`. When `apply_on_stages` is
omitted, the transform applies to `"train"`.

```python
[
    {"name": "HorizontalFlip", "params": {"p": 0.5}},
    {
        "name": "Mosaic4",
        "params": {"height": 640, "width": 640, "p": 1.0},
    },
]
```

The engine groups transforms by behavior rather than preserving the exact input order:

 1. Batch transforms, such as
    [MixUp](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/mixup.md)
    or
    [Mosaic4](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/mosaic.md).
 2. Spatial transforms, such as Albumentations dual transforms.
 3. Custom basic transforms.
 4. Pixel-only transforms.

Resize handling is part of the engine. A transform marked with `use_for_resizing` is used as the resize stage; otherwise the
engine falls back to a regular resize or
[LetterboxResize](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/letterbox_resize.md),
depending on the loader's aspect-ratio setting. If the selected resize transform has probability `p < 1`, it stays in the resize
stage and the remaining probability mass is filled by the default resize in an always-on `OneOf`. The resize stage is applied
before pixel-only transforms when downscaling saves work, and after pixel-only transforms when upscaling or preserving size.

Standard Albumentations flip transforms such as `HorizontalFlip`, `VerticalFlip`, and `Transpose` flip keypoint coordinates but do
not swap semantic left/right keypoint labels. For symmetric keypoint structures, use the Luxonis custom transforms
[HorizontalSymmetricKeypointsFlip](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/symmetric_keypoints_flip.md),
[VerticalSymmetricKeypointsFlip](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/symmetric_keypoints_flip.md),
and
[TransposeSymmetricKeypoints](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/symmetric_keypoints_flip.md).

Batch transforms multiply the number of source samples required by the loader. For example, a pipeline that contains
[MixUp](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/mixup.md)
and
[Mosaic4](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/mosaic.md)
requires 8 = 2⋅4 samples for each augmented output.

Custom augmentation engines can be added by subclassing
[AugmentationEngine](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/base_engine.md).
Subclasses are automatically registered in
[AUGMENTATION_ENGINES](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/base_engine.md).

[LuxonisLoader](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/loaders/luxonis_loader.md)
builds the engine for you. Pass the configuration as a list of records, or as the path of a YAML or JSON file that holds the same
list:

```python
from luxonis_ml.data import LuxonisDataset, LuxonisLoader

loader = LuxonisLoader(
    LuxonisDataset("parking_lot"),
    view="train",
    augmentation_engine="albumentations",
    augmentation_config="augmentations.yaml",
    height=256,
    width=320,
    keep_aspect_ratio=True,
    color_space="RGB",
)

for sample in loader:
    images = sample.images
    labels = sample.labels
```

## Custom Transforms

Custom transforms follow Albumentations conventions. Subclass an appropriate base class such as `DualTransform` or
`ImageOnlyTransform`, implement the target methods needed by your labels, register the class in
[luxonis_ml.data.augmentations.custom.TRANSFORMATIONS](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom.md),
and reference the class name in loader configuration. The [Albumentations
guide](https://albumentations.ai/docs/4-advanced-guides/creating-custom-transforms/) explains the base classes and the target
methods.

To combine several samples into one, subclass
[BatchTransform](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/batch_transform.md)
instead of an Albumentations base class.
[CutMix](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/cutmix.md),
[MixUp](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/mixup.md),
and
[Mosaic4](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/mosaic.md)
are the built-in examples. Registration and configuration work the same way.

```python
from albumentations import DualTransform
from luxonis_ml.data.augmentations.custom import TRANSFORMATIONS

class CustomTransform(DualTransform):
    def apply(self, image, **kwargs):
        return image

    def apply_to_mask(self, mask, **kwargs):
        return mask

    def apply_to_bboxes(self, bboxes, **kwargs):
        return bboxes

    def apply_to_keypoints(self, keypoints, **kwargs):
        return keypoints

TRANSFORMATIONS.register(module=CustomTransform)

augmentation_config = [
    {"name": "CustomTransform", "params": {"p": 1.0}},
]
```

## Engine Interface

A custom engine should subclass
[AugmentationEngine](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/base_engine.md)
and implement:

 * `__init__` to consume output size, class count, configuration, aspect-ratio behavior, pipeline stage, and target metadata;
 * `apply` to transform a batch of images and labels and return the transformed values;
 * `batch_size` to tell
   [LuxonisLoader](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/loaders/luxonis_loader.md)
   how many source samples are needed per augmented output.

Engines may also override
[AugmentationEngine.applied_augmentations](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/base_engine.md)
to report the configured paths and runtime parameters of their latest call.

## Tips and Tricks

### Rotated Bounding Boxes Are Too Large

`Affine`, `Rotate`, `SafeRotate`, and `ShiftScaleRotate` can leave bounding boxes much larger than the objects in them once they
rotate or shear the image. Tall or wide objects, such as standing people, show it the most.

The sample before augmentation.

Rotated with the default `rotate_method="largest_box"`.

The same rotation with `rotate_method="ellipse"`.

A box says nothing about the shape of the object inside it, so after a rotation or a shear Albumentations has to guess the new
box. The `rotate_method` parameter of these transforms selects the guess:

 * `"largest_box"` (default)`: Takes the box around the four transformed corners, as if the object filled its whole box. The box
   is never too small, but often too large. A w×h box rotated by θ becomes w|cosθ| + h|sinθ| wide.
 * `"ellipse"`: Takes the box around the transformed ellipse inscribed in the box. The box is tighter, but cuts off the corners of
   objects that fill their box. The same box becomes √(w2cos2θ + h2sin2θ) wide.

A 120×315 person box rotated by 20° becomes 221 pixels wide with `"largest_box"` and 156 with `"ellipse"`. A square object rotated
by 45° spans 1.41w, but `"ellipse"` gives it w. Without rotation or shear, both methods give the same box.

Use `"ellipse"` when objects do not reach the corners of their boxes, as with people or animals. Keep `"largest_box"` for
rectangular objects such as cars or screens. [Towards Rotation Invariance in Object Detection](https://arxiv.org/abs/2109.13488)
shows that oversized boxes can make a detector worse than training without rotation at all.

```python
[
    {
        "name": "Affine",
        "params": {
            "rotate": [-30, 30],
            "shear": [-15, 15],
            "rotate_method": "ellipse",
        },
    },
]
```

Check the augmented boxes before you train:

```bash
luxonis_ml data inspect <dataset> --aug-config augmentations.yaml
```

Add `--list-augmentations` to print the sampled rotation and scale of each `Affine`. Albumentations does not report the sampled
shear.

## Child Pages

 * [custom](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom.md)
 * [albumentations_engine](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/albumentations_engine.md)
 * [base_engine](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/base_engine.md)
 * [batch_compose](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/batch_compose.md)
 * [batch_transform](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/batch_transform.md)
 * [utils](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/utils.md)