# precision_recall_curve

Python API: `luxonis_train.attached_modules.metrics.precision_recall_curve`

Precision, recall, and F1 over a grid of confidence thresholds.

The metric runs its own NMS on the pre-NMS candidates of the head. Thus the confidence threshold of the head does not limit the
curves.

## Classes

### PrecisionRecallCurve

Detection precision, recall, and F1 over a confidence grid.

 * `Inputs:`: * `detections_pre_nms` (`Tensor`): [B, N, 5 + nclasses + E], the NMS input of the head, boxes in `xyxy` pixels. E is
   the number of extra values of a keypoint or segmentation head, else `0`.
    * `target_boundingbox` (`Tensor`): [Nt, 6], `[batch, class, x, y, w, h]`, normalized
 * `Outputs:`: * `confidence`, `precision`, `recall`, `f1` (`Tensor`): [T] over the threshold grid
    * `max_f1`, `confidence_at_max_f1` (`Tensor`): scalar
 * `Formula:`: For each batch, the metric runs NMS with `nms_conf_threshold`, `nms_iou_threshold`, and `max_detections`. It visits
   the kept detections of an image from the highest score to the lowest. A detection is a true positive when a free target of its
   class has an IoU of at least `matching_iou_threshold` with it. The detection then takes the free target with the highest IoU.
   For the threshold c, the detections with a score of at least c count: P(c) = (TP(c))/(TP(c) + FP(c)), R(c) = (TP(c))/(Nt),
   F1(c) = (2 P(c) R(c))/(P(c) + R(c)) Nt is the number of targets of all batches. P is `1` when no detection reaches c, so the
   curve does not drop to `0` at the high thresholds. R is `0` when there are no targets, and F1 is `0` when P + R = 0.

> **References**
> * Source: This project.
 * License: Apache-2.0 (this project)

> **Notes**
> * The node must be a [BaseDetectionHead](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/base_detection_head.md). The constructor calls [BaseDetectionHead.request_detections_pre_nms](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/base_detection_head.md), so the node adds the candidates to its packet in evaluation mode.
 * [get_loggable_values](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/precision_recall_curve.md)
   selects `max_f1` as the logged value of the metric.
   [get_artifacts](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/precision_recall_curve.md)
   draws the curves.

> **Example**
> Attached to a `EfficientBBoxHead` in `model.nodes`:

```yaml
- name: EfficientBBoxHead
  inputs: [RepPANNeck]
  metrics:
    - name: PrecisionRecallCurve
```

 * `Compatible with:`: * Nodes: *
   [EfficientBBoxHead](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/efficient_bbox_head.md)
       * [EfficientKeypointBBoxHead](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/efficient_keypoint_bbox_head.md)
       * [PrecisionBBoxHead](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/precision_bbox_head.md)
       * [PrecisionSegmentBBoxHead](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/precision_seg_bbox_head.md)

#### Methods

##### init

```python
def __init__(*, confidence_thresholds: list[float] | None = None, num_thresholds: int | None = None, min_confidence: float | None = None, max_confidence: float | None = None, matching_iou_threshold: float = 0.5, nms_conf_threshold: float = 0.001, nms_iou_threshold: float | None = None, max_detections: int | None = None, **kwargs):
```

Build the threshold grid and resolve the NMS settings.

Parameters

 * `confidence_thresholds` (`list[float] | None`): An explicit grid of at least two strictly increasing values in `[0, 1]`. `None`
   builds a grid of `num_thresholds` evenly spaced values from `min_confidence` to `max_confidence`.
 * `num_thresholds` (`int | None`): The number of values of the built grid, at least `2`. `None` selects `101`.
 * `min_confidence` (`float | None`): The first value of the built grid, in `[0, 1]`. `None` selects `0.0`.
 * `max_confidence` (`float | None`): The last value of the built grid, in `[0, 1]` and above `min_confidence`. `None` selects
   `1.0`.
 * `matching_iou_threshold` (`float`): The smallest IoU, in `[0, 1]`, at which a detection matches a target of its class.
 * `nms_conf_threshold` (`float`): The score floor of NMS, in `[0, 1]`. NMS drops the candidates below the floor, so they never
   count, not even at the first threshold. The metric uses the larger of this value and the first value of the grid. A lower floor
   keeps more candidates, and NMS then takes more time. A floor of `0` keeps every candidate with a score above `0`.
 * `nms_iou_threshold` (`float | None`): The IoU threshold of NMS, in `[0, 1]`. `None` selects `iou_thres` of the node.
 * `max_detections` (`int | None`): The largest number of detections that NMS keeps for each image, above `0`. `None` selects
   `max_det` of the node.
 * `**kwargs`: Keyword arguments forwarded to
   [BaseMetric](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md).
   They must hold `node`.

Raises

 * `ValueError`: When `confidence_thresholds` comes with one of `num_thresholds`, `min_confidence`, or `max_confidence`. Also when
   a value is outside its range, and when the explicit grid is not strictly increasing.

##### build_curve_figure

```python
def build_curve_figure(values: dict[str, Tensor]) -> Figure:
```

Build a figure with three curve plots side by side.

 * Left: the precision over the recall.
 * Middle: the precision over the confidence.
 * Right: the recall over the confidence.

The confidence axes span the threshold grid. All the other axes span `[0, 1]`.

Parameters

 * `values` (`dict[str, Tensor]`): The result of
   [compute](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/precision_recall_curve.md).
   The method reads `"confidence"`, `"precision"`, and `"recall"`.

Returns

 * `Figure`: A `matplotlib` figure of `12x4` inches with a constrained layout.

##### compute

```python
def compute(self) -> dict[str, Tensor]:
```

Compute precision, recall, and F1 at each confidence threshold.

The class docstring gives the formulas.

> **Example**
> One detection with the score `0.8` covers the only target. At the threshold `0.9`, no detection counts. The precision there is `1` and the recall is `0`:

```pycon
>>> import torch
>>> from torch import Size
>>> from luxonis_train.nodes import EfficientBBoxHead
>>> head = EfficientBBoxHead(
...     n_heads=1,
...     n_classes=1,
...     input_shapes=[{"features": [Size([1, 8, 8, 8])]}],
...     original_in_shape=Size([3, 64, 64]),
... )
>>> metric = PrecisionRecallCurve(
...     node=head, confidence_thresholds=[0.0, 0.5, 0.9]
... )
>>> candidates = torch.tensor([[[0.0, 0.0, 32.0, 32.0, 1.0, 0.8]]])
>>> target = torch.tensor([[0.0, 0.0, 0.0, 0.0, 0.5, 0.5]])
>>> metric.update(candidates, target)
>>> values = metric.compute()
>>> values["precision"].tolist(), values["recall"].tolist()
([1.0, 1.0, 1.0], [1.0, 1.0, 0.0])
```

Returns

 * `dict[str, Tensor]`: The curves and their best point. * `"confidence"` holds the threshold grid, of shape `[T]`.
    * `"precision"`, `"recall"`, and `"f1"` hold the values at the thresholds, each of shape `[T]`.
    * `"max_f1"` holds the largest F1, a scalar.
    * `"confidence_at_max_f1"` holds the first threshold that reaches `max_f1`, a scalar.

##### get_artifact_names

```python
def get_artifact_names(self) -> tuple[str, ...]:
```

Return the name of the curve image.

Returns

 * `tuple[str, ...]`: `("curves",)`, the key that
   [get_artifacts](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/precision_recall_curve.md)
   uses.

##### get_artifacts

```python
def get_artifacts(values: dict[str, Tensor]) -> dict[str, Tensor]:
```

Draw the curves of
[build_curve_figure](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/precision_recall_curve.md)
as one image.

The method closes the figure after it draws it.

Parameters

 * `values` (`dict[str, Tensor]`): The result of
   [compute](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/precision_recall_curve.md).

Returns

 * `dict[str, Tensor]`: The key `"curves"`, with a `uint8` RGB image of shape `[3, 400, 1200]`.

##### get_loggable_values

```python
def get_loggable_values(values: dict[str, Tensor]) -> tuple[Tensor, dict[str, Tensor]]:
```

Select `max_f1` as the logged value of the metric.

The trainer logs only the returned values, so the curves stay out of the scalar logs.

Parameters

 * `values` (`dict[str, Tensor]`): The result of
   [compute](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/precision_recall_curve.md).

Returns

 * `tuple[Tensor, dict[str, Tensor]]`: `values["max_f1"]`, and a dictionary with the key `"confidence_at_max_f1"`.

##### update

```python
def update(detections_pre_nms: Tensor, target_boundingbox: Tensor):
```

Run NMS on one batch and add its counts to the states.

For each threshold, the method adds the true positives and the false positives of the batch to `true_positives` and
`false_positives`. It adds the number of targets to `target_count`. First, it replaces each state that `reset` made in inference
mode with a normal tensor.

Parameters

 * `detections_pre_nms` (`Tensor`): The candidates of the head, of shape `[B, N, 5 + n_classes + E]`. Each row holds the `xyxy`
   box in pixels, a constant `1`, the class scores, and `E` extra values that the metric ignores.
 * `target_boundingbox` (`Tensor`): The target boxes of shape `[N_t, 6]`, as `[batch, class, x, y, w, h]` with normalized values.
   The method moves them to the device of the candidates.

#### Attributes

##### false_positives

##### node

##### supported_tasks

##### target_count

##### thresholds

##### true_positives

## Attributes

### ARTIFACT_HEIGHT

### ARTIFACT_WIDTH

### DEFAULT_MAX_CONFIDENCE

### DEFAULT_MIN_CONFIDENCE

### DEFAULT_NUM_THRESHOLDS