# precision_dfl_segmentation_loss

Python API: `luxonis_train.attached_modules.losses.precision_dfl_segmentation_loss`

The YOLOv8 detection loss with a mask term for instance segmentation.

## Classes

### PrecisionDFLSegmentationLoss

Instance segmentation loss for
[PrecisionSegmentBBoxHead](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/precision_seg_bbox_head.md).

The loss adds a mask term to
[PrecisionDFLDetectionLoss](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/losses/precision_dfl_detection_loss.md).
For each positive anchor, the loss builds mask logits from the mask coefficients of the anchor and the prototypes of the head. A
binary cross-entropy compares these logits with the target mask of the assigned box, inside that box only.

 * `Inputs:`: * `features` (`list[Tensor]`): [B, 4**reg*max + nclasses, Hi, Wi] per scale, the distance bin logits followed by the
   class logits
    * `prototypes` (`Tensor`): [B, nmasks, 2H*0, 2W*0]
    * `mask_coefficients` (`Tensor`): [B, nmasks, N], for the N anchors of all scales
    * `target_boundingbox` (`Tensor`): [Ngt, 6], `[batch_index, class, x, y, w, h]`, `xywh` normalized, with `x` and `y` at the
      top-left corner
    * `target_instance_segmentation` (`Tensor`): [Ngt, H, W], one mask for each target box, in the same order
 * `Outputs:`: * `Tensor`: scalar total loss
    * `dict[str, Tensor]`: scalar sub-losses `class`, `iou`, `dfl`, `seg`, detached and without the weights
 * `Formula:`: The terms Lcls, Liou, and Ldfl are those of
   [PrecisionDFLDetectionLoss](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/losses/precision_dfl_detection_loss.md).
   P is the set of positive anchors in all images of the batch. For anchor a, the mask logit map ma = ∑kca, k pk is the sum of the
   prototypes pk, weighted by the mask coefficients ca, k of the anchor. The prototypes have the size h×w. ga is the target mask
   of the assigned box, at the size h×w. Aa is the area of that box as a fraction of the image area. cropa sets the pixels outside
   the box to `0`: Lseg = (1)/(|P|)∑a ∈ P(1)/(Aa h**w)∑x, ycropa(BCE(ma, ga))x, y L = λclsLcls + λboxLiou + λdflLdfl + λboxLseg
   The mask term uses the box weight `bbox_loss_weight`.

> **References**
> * Source: Reimplemented from [Real-Time Flying Object Detection with YOLOv8](https://arxiv.org/abs/2305.09972) and [YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications](https://arxiv.org/abs/2209.02976) and [PP-YOLOE: An evolved version of YOLO](https://arxiv.org/abs/2203.16250).
 * License: Apache-2.0 (this project)

> **Notes**
> The `seg` term and the total loss are NaN when the batch has no positive anchor. The mask term divides by the number of positive anchors, also when that number is `0`. The notes of [PrecisionDFLDetectionLoss](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/losses/precision_dfl_detection_loss.md) on the anchor cache and on a `reg_max` of `1` apply here too.

> **Example**
> Attached to a `PrecisionSegmentBBoxHead` in `model.nodes`:

```yaml
- name: PrecisionSegmentBBoxHead
  inputs: [RepPANNeck]
  losses:
    - name: PrecisionDFLSegmentationLoss
```

 * `Compatible with:`: * Used by:
   [InstanceSegmentationModel](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/config/predefined_models/instance_segmentation/v1/model.md)
    * Nodes:
      [PrecisionSegmentBBoxHead](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/precision_seg_bbox_head.md)

#### Methods

##### init

```python
def __init__(tal_topk: int = 10, class_loss_weight: float = 0.5, bbox_loss_weight: float = 7.5, dfl_loss_weight: float = 1.5, skip_stal: bool = False, **kwargs):
```

Initialize the loss with the settings of the detection terms.

The mask term has no settings of its own. The loss needs a `node` of type
[PrecisionSegmentBBoxHead](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/precision_seg_bbox_head.md).
When a config uses the predefined
[InstanceSegmentationModel](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/config/predefined_models/instance_segmentation/v1/model.md)
and `trainer.smart_cfg_auto_populate` is `True`, the config changes the three weights. It sets each weight to its default times
`trainer.accumulate_grad_batches`. These values replace the weights in the `loss_params` of the model.

Parameters

 * `tal_topk` (`int`): The `topk` of
   [TaskAlignedAssigner](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/assigners/tal_assigner.md),
   the largest number of positive anchors for each target box.
 * `class_loss_weight` (`float`): Weight of the classification term.
 * `bbox_loss_weight` (`float`): Weight of the CIoU box term and of the mask term.
 * `dfl_loss_weight` (`float`): Weight of the DFL term.
 * `skip_stal` (`bool`): Whether to turn off Small-Target-Aware Label Assignment (STAL) in the assigner.
 * `**kwargs`: Keyword arguments forwarded to
   [PrecisionDFLDetectionLoss](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/losses/precision_dfl_detection_loss.md),
   such as `node` and `final_loss_weight`.

##### compute_segmentation_loss

```python
def compute_segmentation_loss(fg_mask: Tensor, gt_masks: Tensor, gt_idx: Tensor, bboxes: Tensor, batch_ids: Tensor, proto: Tensor, pred_masks: Tensor) -> Tensor:
```

Compute the mask term of the whole batch.

For each positive anchor of an image, the method multiplies the mask coefficients with the prototypes into mask logits. It
computes the binary cross-entropy between these logits and the target mask of the assigned box. It sets the loss outside the box
to `0`. The loss of the anchor is the mean over all mask pixels. The method divides this mean by the area of the box as a fraction
of the image area. An image with no positive anchor adds a zero that depends on `proto` and `pred_masks`, so both stay in the
autograd graph. The method reads the image scale that
[forward](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/losses/precision_dfl_segmentation_loss.md)
caches, so
[forward](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/losses/precision_dfl_segmentation_loss.md)
must run first.

Parameters

 * `fg_mask` (`Tensor`): Boolean mask of the positive anchors, of shape `[B, N]`.
 * `gt_masks` (`Tensor`): Target masks of all images, of shape `[N_gt, h, w]`, at the prototype size. The masks of one image are
   in the order of its target boxes.
 * `gt_idx` (`Tensor`): For each anchor, the index of the assigned box among the target boxes of its image, of shape `[B, N]`.
 * `bboxes` (`Tensor`): Assigned boxes in `xyxy` pixels of the input image, of shape `[B, N, 4]`.
 * `batch_ids` (`Tensor`): Image index of each target mask, of shape `[N_gt, 1]`.
 * `proto` (`Tensor`): Mask prototypes of shape `[B, n_masks, h, w]`.
 * `pred_masks` (`Tensor`): Mask coefficients of shape `[B, N, n_masks]`.

Returns

 * `Tensor`: The scalar sum of the anchor losses, divided by the number of positive anchors. `NaN` when `fg_mask` has no positive
   anchor.

##### forward

```python
def forward(features: list[Tensor], prototypes: Tensor, mask_coefficients: Tensor, target_boundingbox: Tensor, target_instance_segmentation: Tensor) -> tuple[Tensor, dict[str, Tensor]]:
```

Compute the detection loss and the mask loss of one batch.

First, the method prepares `target_instance_segmentation`:

 * When it is empty, the method replaces it with an empty tensor of shape `[0, h, w]`, the prototype size.
 * When its size is not the prototype size, the method resizes it to `[h, w]` with nearest interpolation.

The class, box, and DFL terms are the same as in
[PrecisionDFLDetectionLoss.forward](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/losses/precision_dfl_detection_loss.md).
The first call caches the anchor points and the image scale in the same way. The method also keeps the index of the assigned
target box of each anchor. It passes this index and the assigned boxes in pixels to
[compute_segmentation_loss](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/losses/precision_dfl_segmentation_loss.md).

Parameters

 * `features` (`list[Tensor]`): One tensor per scale, of shape `[B, 4 * reg_max + n_classes, H_i, W_i]`. The `features` output of
   the node.
 * `prototypes` (`Tensor`): Mask prototypes of shape `[B, n_masks, h, w]`. The `prototypes` output of the node.
 * `mask_coefficients` (`Tensor`): Mask coefficients of shape `[B, n_masks, N]`, for the `N` anchors of all scales. The
   `mask_coefficients` output of the node.
 * `target_boundingbox` (`Tensor`): Target boxes of shape `[N_gt, 6]`, with rows `[batch_index, class, x, y, w, h]`. The
   coordinates are `xywh` normalized to `[0, 1]`, with `x` and `y` at the top-left corner. The `boundingbox` label of the task.
 * `target_instance_segmentation` (`Tensor`): Target masks of shape `[N_gt, H, W]`, one for each row of `target_boundingbox`, in
   the same order. The `instance_segmentation` label of the task.

Returns

 * `tuple[Tensor, dict[str, Tensor]]`: The scalar weighted total loss, and a dictionary that maps `"class"`, `"iou"`, `"dfl"`, and
   `"seg"` to the detached terms before the weights. The total loss and `"seg"` are `NaN` when no anchor is positive. The total
   loss and `"dfl"` have the shape `[1]` when `reg_max` of the node is `1` and an anchor is positive.

#### Attributes

##### node

##### supported_tasks