# augmentations

Python API: `luxonis_ml.data.augmentations`

Augmentation engines and custom transforms for LDF samples.

This package provides the augmentation interface used by
[LuxonisLoader](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/loaders/luxonis_loader.md).
The default implementation is
[AlbumentationsEngine](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/albumentations_engine.md),
which adapts LDF labels to Albumentations targets before transformation and converts them back after transformation.

Table of Contents

 * [Configuration](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations.md)
 * [Custom
   Transforms](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations.md)
 * [Engine
   Interface](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations.md)
 * [Tips and
   Tricks](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations.md)
   * [Rotated Bounding Boxes Are Too
   Large](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations.md)

## Configuration

Augmentation configuration is a list of records. Each record contains a `name` identifying an Albumentations transform or a
transform registered in
[TRANSFORMATIONS](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom.md),
optional `params`, optional `use_for_resizing`, and optional stage filtering through `apply_on_stages`. When `apply_on_stages` is
omitted, the transform applies to `"train"`.

```python
[
    {"name": "HorizontalFlip", "params": {"p": 0.5}},
    {
        "name": "Mosaic4",
        "params": {"height": 640, "width": 640, "p": 1.0},
    },
]
```

The engine groups transforms by behavior rather than preserving the exact input order:

 1. Batch transforms, such as
    [MixUp](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/mixup.md)
    or
    [Mosaic4](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/mosaic.md).
 2. Spatial transforms, such as Albumentations dual transforms.
 3. Custom basic transforms.
 4. Pixel-only transforms.

Resize handling is part of the engine. A transform marked with `use_for_resizing` is used as the resize stage; otherwise the
engine falls back to a regular resize or
[LetterboxResize](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/letterbox_resize.md),
depending on the loader's aspect-ratio setting. If the selected resize transform has probability `p < 1`, it stays in the resize
stage and the remaining probability mass is filled by the default resize in an always-on `OneOf`. The resize stage is applied
before pixel-only transforms when downscaling saves work, and after pixel-only transforms when upscaling or preserving size.

Standard Albumentations flip transforms such as `HorizontalFlip`, `VerticalFlip`, and `Transpose` flip keypoint coordinates but do
not swap semantic left/right keypoint labels. For symmetric keypoint structures, use the Luxonis custom transforms
[HorizontalSymmetricKeypointsFlip](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/symmetric_keypoints_flip.md),
[VerticalSymmetricKeypointsFlip](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/symmetric_keypoints_flip.md),
and
[TransposeSymmetricKeypoints](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/symmetric_keypoints_flip.md).

Batch transforms multiply the number of source samples required by the loader. For example, a pipeline that contains
[MixUp](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/mixup.md)
and
[Mosaic4](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/mosaic.md)
requires 8 = 2⋅4 samples for each augmented output.

Custom augmentation engines can be added by subclassing
[AugmentationEngine](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/base_engine.md).
Subclasses are automatically registered in
[AUGMENTATION_ENGINES](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/base_engine.md).

[LuxonisLoader](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/loaders/luxonis_loader.md)
builds the engine for you. Pass the configuration as a list of records, or as the path of a YAML or JSON file that holds the same
list:

```python
from luxonis_ml.data import LuxonisDataset, LuxonisLoader

loader = LuxonisLoader(
    LuxonisDataset("parking_lot"),
    view="train",
    augmentation_engine="albumentations",
    augmentation_config="augmentations.yaml",
    height=256,
    width=320,
    keep_aspect_ratio=True,
    color_space="RGB",
)

for sample in loader:
    images = sample.images
    labels = sample.labels
```

## Custom Transforms

Custom transforms follow Albumentations conventions. Subclass an appropriate base class such as `DualTransform` or
`ImageOnlyTransform`, implement the target methods needed by your labels, register the class in
[luxonis_ml.data.augmentations.custom.TRANSFORMATIONS](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom.md),
and reference the class name in loader configuration. The [Albumentations
guide](https://albumentations.ai/docs/4-advanced-guides/creating-custom-transforms/) explains the base classes and the target
methods.

To combine several samples into one, subclass
[BatchTransform](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/batch_transform.md)
instead of an Albumentations base class.
[CutMix](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/cutmix.md),
[MixUp](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/mixup.md),
and
[Mosaic4](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom/mosaic.md)
are the built-in examples. Registration and configuration work the same way.

```python
from albumentations import DualTransform
from luxonis_ml.data.augmentations.custom import TRANSFORMATIONS

class CustomTransform(DualTransform):
    def apply(self, image, **kwargs):
        return image

    def apply_to_mask(self, mask, **kwargs):
        return mask

    def apply_to_bboxes(self, bboxes, **kwargs):
        return bboxes

    def apply_to_keypoints(self, keypoints, **kwargs):
        return keypoints

TRANSFORMATIONS.register(module=CustomTransform)

augmentation_config = [
    {"name": "CustomTransform", "params": {"p": 1.0}},
]
```

## Engine Interface

A custom engine should subclass
[AugmentationEngine](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/base_engine.md)
and implement:

 * `__init__` to consume output size, class count, configuration, aspect-ratio behavior, pipeline stage, and target metadata;
 * `apply` to transform a batch of images and labels and return the transformed values;
 * `batch_size` to tell
   [LuxonisLoader](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/loaders/luxonis_loader.md)
   how many source samples are needed per augmented output.

Engines may also override
[AugmentationEngine.applied_augmentations](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/base_engine.md)
to report the configured paths and runtime parameters of their latest call.

## Tips and Tricks

### Rotated Bounding Boxes Are Too Large

`Affine`, `Rotate`, `SafeRotate`, and `ShiftScaleRotate` can leave bounding boxes much larger than the objects in them once they
rotate or shear the image. Tall or wide objects, such as standing people, show it the most.

![https://raw.githubusercontent.com/luxonis/luxonis-ml/30a7530c1725538e1e9816b3d188e9df400db636/luxonis_ml/data/augmentations/media/bbox_rotation_original.png](https://raw.githubusercontent.com/luxonis/luxonis-ml/30a7530c1725538e1e9816b3d188e9df400db636/luxonis_ml/data/augmentations/media/bbox_rotation_original.png)

The sample before augmentation.

![https://raw.githubusercontent.com/luxonis/luxonis-ml/30a7530c1725538e1e9816b3d188e9df400db636/luxonis_ml/data/augmentations/media/bbox_rotation_largest_box.png](https://raw.githubusercontent.com/luxonis/luxonis-ml/30a7530c1725538e1e9816b3d188e9df400db636/luxonis_ml/data/augmentations/media/bbox_rotation_largest_box.png)

Rotated with the default `rotate_method="largest_box"`.

![https://raw.githubusercontent.com/luxonis/luxonis-ml/30a7530c1725538e1e9816b3d188e9df400db636/luxonis_ml/data/augmentations/media/bbox_rotation_ellipse.png](https://raw.githubusercontent.com/luxonis/luxonis-ml/30a7530c1725538e1e9816b3d188e9df400db636/luxonis_ml/data/augmentations/media/bbox_rotation_ellipse.png)

The same rotation with `rotate_method="ellipse"`.

A box says nothing about the shape of the object inside it, so after a rotation or a shear Albumentations has to guess the new
box. The `rotate_method` parameter of these transforms selects the guess:

 * `"largest_box"` (default)`: Takes the box around the four transformed corners, as if the object filled its whole box. The box
   is never too small, but often too large. A w×h box rotated by θ becomes w|cosθ| + h|sinθ| wide.
 * `"ellipse"`: Takes the box around the transformed ellipse inscribed in the box. The box is tighter, but cuts off the corners of
   objects that fill their box. The same box becomes √(w2cos2θ + h2sin2θ) wide.

A 120×315 person box rotated by 20° becomes 221 pixels wide with `"largest_box"` and 156 with `"ellipse"`. A square object rotated
by 45° spans 1.41w, but `"ellipse"` gives it w. Without rotation or shear, both methods give the same box.

Use `"ellipse"` when objects do not reach the corners of their boxes, as with people or animals. Keep `"largest_box"` for
rectangular objects such as cars or screens. [Towards Rotation Invariance in Object Detection](https://arxiv.org/abs/2109.13488)
shows that oversized boxes can make a detector worse than training without rotation at all.

```python
[
    {
        "name": "Affine",
        "params": {
            "rotate": [-30, 30],
            "shear": [-15, 15],
            "rotate_method": "ellipse",
        },
    },
]
```

Check the augmented boxes before you train:

```bash
luxonis_ml data inspect <dataset> --aug-config augmentations.yaml
```

Add `--list-augmentations` to print the sampled rotation and scale of each `Affine`. Albumentations does not report the sampled
shear.

## Child Pages

 * [custom](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/custom.md)
 * [albumentations_engine](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/albumentations_engine.md)
 * [base_engine](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/base_engine.md)
 * [batch_compose](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/batch_compose.md)
 * [batch_transform](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/batch_transform.md)
 * [utils](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/augmentations/utils.md)
