# base_metric

Python API: `luxonis_train.attached_modules.metrics.base_metric`

The base class of all metrics and the marker for metric states.

[BaseMetric](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
combines
[BaseAttachedModule](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/base_attached_module.md)
with the `torchmetrics` `Metric` class.
[MetricState](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
marks the class attributes that
[BaseMetric](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
registers as metric states. `DistReduceFx` is the type of the reduction that merges one state across processes, and `MetricResult`
is the type of the result of
[BaseMetric.compute](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md).

## Classes

### BaseMetric

Base class for all metrics.

A metric is a
[BaseAttachedModule](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/base_attached_module.md)
and a `torchmetrics` `Metric`. Every subclass registers itself in the
[METRICS](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/registry.md)
registry under its class name, unless its class statement passes `register=False`. A config names a registered metric by that
string. A subclass implements
[update](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
and
[compute](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md).
It declares its states with
[MetricState](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md),
or with `add_state` in its `__init__`. The example of
[MetricState](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
shows a complete subclass.

[BaseAttachedModule.get_parameters](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/base_attached_module.md)
describes how
[run_update](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
fills the parameters of
[update](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
from predictions and labels.

The trainer calls
[run_update](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
on each validation and test batch. At the end of the epoch, it calls
[compute](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
and logs the images of
[get_artifacts](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md).
It then calls `reset` and logs the values that
[get_loggable_values](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
selects.

Two metrics are equal only when they are the same object, and the hash of a metric is its `id`. In `torchmetrics`, `==` builds a
new composed metric, and the hash reads the states.

#### Methods

##### init

```python
def __init__(**kwargs):
```

Initialize the metric and register its metric states.

The method reads the type hints of the class and of its base classes. It registers each attribute whose `Annotated` hint holds a
[MetricState](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
with the `add_state` method of `torchmetrics`.
[MetricState](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
describes how the default value and the reduction of a state follow from the marker.

Parameters

 * `**kwargs`: Keyword arguments forwarded to
   [BaseAttachedModule](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/base_attached_module.md),
   such as `node`. The remaining arguments reach the `torchmetrics` `Metric`, which accepts only its own options, such as
   `sync_on_compute`, and raises `ValueError` for any other name.

Raises

 * `ValueError`: When a metric state has no `default` and its type is neither `Tensor` nor `list[...]`. Also when the `default` of
   a metric state is a list that is not empty.

##### compute

```python
def compute(self) -> Tensor | tuple[Tensor, dict[str, Tensor]] | dict[str, Tensor]:
```

Compute the value of the metric from its states.

Returns

 * `Tensor | tuple[Tensor, dict[str, Tensor]] | dict[str, Tensor]`: The result in one of three forms. * The main value as a
   `Tensor`.
    * A tuple of the main value and a dictionary of sub-metrics.
    * A dictionary of sub-metrics only. The trainer then logs no value under the name of the metric, so the metric cannot be the
      main metric.

##### get_artifact_names

```python
def get_artifact_names(self) -> tuple[str, ...]:
```

Return the stable names emitted by
[get_artifacts](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md).

Returns

 * `tuple[str, ...]`: The keys of the dictionary that
   [get_artifacts](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
   returns.

##### get_artifacts

```python
def get_artifacts(values: MetricResult) -> dict[str, Tensor]:
```

Render images from a computed result.

Parameters

 * `values` (`MetricResult`): The result of
   [compute](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md).

Returns

 * `dict[str, Tensor]`: The images of shape `[C, H, W]`, keyed by the names that
   [get_artifact_names](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
   returns.

##### get_loggable_values

```python
def get_loggable_values(values: MetricResult) -> MetricResult:
```

Select the part of a computed result to log.

Override this when
[compute](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md)
returns data that belongs in artifacts rather than scalar logs.

Parameters

 * `values` (`MetricResult`): The result of
   [compute](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md).

Returns

 * `MetricResult`: The values to log, in one of the forms of
   [compute](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md).

##### get_predefined_model_params_aliases

```python
def get_predefined_model_params_aliases(task: Task | None = None) -> dict[str, str]:
```

Return the constructor names of predefined model parameters.

A predefined model can add `per_class_metrics` to the `params` of each of its metrics. When
[Nodes](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/lightning/utils.md)
builds the metric, it looks the key up in the returned dictionary and passes the value under the parameter name that it finds.
When the dictionary has no such key,
[Nodes](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/lightning/utils.md)
drops the value and logs a warning. It drops a `None` value without a lookup.

This implementation ignores `task` and returns the class attribute `predefined_model_params_aliases`. The attribute is empty on
[BaseMetric](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md),
and
[MIoU](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/mean_iou.md)
maps `per_class_metrics` to `per_class`.
[MeanAveragePrecision](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/mean_average_precision/mean_average_precision.md)
is not a subclass. It defines its own method, because its mapping depends on the task.

> **Example**
> ```pycon
>>> from luxonis_train.attached_modules.metrics import MIoU
>>> BaseMetric.get_predefined_model_params_aliases()
{}
>>> MIoU.get_predefined_model_params_aliases()
{'per_class_metrics': 'per_class'}
```

Parameters

 * `task` (`Task | None`): The task of the node that the metric attaches to, or `None` when the node has no task.

Returns

 * `dict[str, str]`: The predefined model parameter names, mapped to the parameter names of the constructor.

##### run_update

```python
def run_update(inputs: Packet[Tensor], labels: Labels):
```

Resolve the inputs of [update](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md) and add one batch.

[BaseAttachedModule.get_parameters](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/base_attached_module.md) documents how parameter names select predictions and labels.

Parameters

 * `inputs` (`Packet[Tensor]`): The output packet of the node.
 * `labels` (`Labels`): The labels of the batch, keyed `<task_name>/<label>`.

##### update

```python
def update(*args: Tensor | list[Tensor]):
```

Add one batch to the metric state.

Parameters

 * `*args` (`Tensor | list[Tensor]`): The inputs of the batch. An implementation replaces them with named parameters.

#### Attributes

##### predefined_model_params_aliases

### MetricState

Marker for a class attribute that becomes a metric state.

Put an instance into the `Annotated` type hint of a class attribute of a [BaseMetric](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md) subclass. [BaseMetric.init](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md) finds each such attribute in the class and in its base classes. It registers the attribute with the `add_state` method of `torchmetrics`, so the state is an attribute of the metric instance. The first argument of `Annotated` is the type of the state.

`torchmetrics` treats a state like a buffer: `.to()` moves it. `reset` gives a tensor state a copy of its default and empties a list state.

> **Example**
> A subclass with two tensor states. `register=False` keeps the class out of the [METRICS](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/registry.md) registry.

```pycon
>>> from typing import Annotated
>>> import torch
>>> from torch import Tensor
>>> class PositiveRate(BaseMetric, register=False):
...     positives: Annotated[Tensor, MetricState()]
...     total: Annotated[Tensor, MetricState(default=0)]
...
...     def update(self, predictions: Tensor) -> None:
...         self.positives += (predictions > 0).sum()
...         self.total += predictions.numel()
...
...     def compute(self) -> Tensor:
...         return self.positives / self.total
>>> metric = PositiveRate()
>>> metric.update(torch.tensor([1.0, -1.0, 2.0, 3.0]))
>>> metric.compute().item()
0.75
>>> metric.reset()
>>> metric.positives.item(), metric.total.item()
(0.0, 0)
```

#### Attributes

##### default

The value of the state after initialization and after `reset`. A number becomes a zero-dimensional tensor, so `0` gives an `int64` state and `0.0` a `float32` state. `None` selects `0.0` for a `Tensor` attribute and `[]` for a `list[...]` attribute. For an attribute of any other type, `None` makes [BaseMetric.init](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/metrics/base_metric.md) raise `ValueError`. A list default must be empty, else `add_state` raises `ValueError`. Each instance receives a tensor or list default as it is, not a copy. All instances of the class then share one state object. An in-place change such as `+=` or `append` in one metric also changes the others. `reset` gives one metric a new tensor, so that metric no longer shares a tensor state. `reset` empties a list state in place, so the list stays shared and empties for all metrics. A number or `None` gives each instance its own state.

##### dist_reduce_fx

The reduction that merges the state of all processes. A string selects `torch.sum`, `torch.mean`, `torch.cat`, `torch.min`, or `torch.max` over dimension `0`. A callable receives the gathered state: the tensor states stacked along a new first dimension, or one list with the items of all processes. `None` keeps the gathered state as it is. The default `...` selects `"cat"` for a list state and `"sum"` for a tensor state.

##### persistent

Whether the `state_dict` of the metric holds the state.

## Attributes

### DistReduceFx

### MetricResult