# keypoints

Python API: `luxonis_train.utils.keypoints`

Math helpers for keypoints.

The module holds the keypoint sigmas, the object keypoint similarity, and the conversions between keypoints and bounding boxes.

## Functions

### compute_pose_oks

```python
def compute_pose_oks(predictions: Tensor, targets: Tensor, sigmas: Tensor, gt_bboxes: Tensor | None = None, pose_area: Tensor | None = None, eps: float = 1e-09, area_factor: float = 0.53, use_cocoeval_oks: bool = True) -> Tensor:
```

Compute the object keypoint similarity of each target-prediction pair.

For one image, the similarity of target t and prediction p is the mean of this term over the visible keypoints of t:

exp⎛⎝ − (d2i)/(2 (2σi)2 A)⎞⎠

In the term, di is the distance between the two keypoints i, and σi is the sigma of keypoint i. A is the pose area of t. A target
keypoint is visible when its third value is greater than `0`. A target without visible keypoints gets a similarity of `0`. With
`use_cocoeval_oks` set to `False`, the exponent is − d2i ⁄ (2(A**σi)2) instead.

> **References**
> * COCO keypoint evaluation: [https://cocodataset.org/#keypoints-eval](https://cocodataset.org/#keypoints-eval)
 * `computeOks` in `pycocotools/cocoeval.py`:
   [https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L229](https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L229)

> **Examples**
> ```pycon
>>> import torch
>>> targets = torch.tensor([[[[0.5, 0.5, 2.0]]]])
>>> sigmas = torch.tensor([0.05])
>>> gt_bboxes = torch.tensor([[[0.0, 0.0, 1.0, 1.0]]])
>>> exact = torch.tensor([[[[0.5, 0.5, 1.0]]]])
>>> oks = compute_pose_oks(exact, targets, sigmas, gt_bboxes=gt_bboxes)
>>> oks.round(decimals=3).tolist()
[[[1.0]]]
```

```pycon
>>> far = torch.tensor([[[[5.0, 5.0, 1.0]]]])
>>> oks = compute_pose_oks(far, targets, sigmas, gt_bboxes=gt_bboxes)
>>> oks.round(decimals=3).tolist()
[[[0.0]]]
```

Parameters

 * `predictions` (`Tensor`): The predicted keypoints of shape `[N, M2, n_keypoints, 3]`. The function reads only `x` and `y`, the first two values of each keypoint.
 * `targets` (`Tensor`): The target keypoints of shape `[N, M1, n_keypoints, 3]`, as `(x, y, visibility)`.
 * `sigmas` (`Tensor`): One sigma per keypoint, of shape `[n_keypoints]`.
 * `gt_bboxes` (`Tensor | None`): The target boxes of shape `[N, M1, 4]` in `xyxy` format. Their area times `area_factor` is the pose area. The function reads them only when `pose_area` is `None`.
 * `pose_area` (`Tensor | None`): The pose area of each target, of shape `[N, M1, 1, 1]`. `None` computes it from `gt_bboxes`.
 * `eps` (`float`): A small constant that the function adds to the area and to the visible count. It prevents a division by zero.
 * `area_factor` (`float`): The factor that scales the box area to the pose area.
 * `use_cocoeval_oks` (`bool`): When `True`, use the formula of the COCO evaluation code. When `False`, use the other formula above.

Returns

 * `Tensor`: The similarities of shape `[N, M1, M2]`, in `[0, 1]`.

Raises

 * `ValueError`: When both `pose_area` and `gt_bboxes` are `None`.

### get_center_keypoints

```python
def get_center_keypoints(bboxes: Tensor, *, height: int = 1, width: int = 1) -> Tensor:
```

Make one center keypoint per bounding box.

The FOMO loss and the object keypoint similarity metric use the box centers as the keypoint targets of the FOMO task. That task has no annotated keypoints.

> **Example**
> ```pycon
>>> import torch
>>> bboxes = torch.tensor([[0.0, 1.0, 0.25, 0.5, 0.5, 0.25]])
>>> get_center_keypoints(bboxes).tolist()
[[0.0, 0.5, 0.625, 2.0]]
>>> get_center_keypoints(bboxes, height=200, width=100).tolist()
[[0.0, 50.0, 125.0, 2.0]]
```

Parameters

 * `bboxes` (`Tensor`): The bounding boxes of shape `[N, 6]`, with the columns `(batch_index, class, x, y, w, h)`. `x` and `y` are
   the normalized top-left corner, and `w` and `h` are the normalized size.
 * `height` (`int`): The height that scales `y`, for example the height of an image or of a heatmap. `1` keeps the coordinates
   normalized.
 * `width` (`int`): The width that scales `x`, for example the width of an image or of a heatmap. `1` keeps the coordinates
   normalized.

Returns

 * `Tensor`: The keypoints of shape `[N, 4]`, with the columns `(batch_index, x, y, visibility)`, on the device and of the dtype
   of `bboxes`. `x` and `y` are the box center scaled by `width` and `height`. The visibility is always `2`.

### get_sigmas

```python
def get_sigmas(sigmas: list[float] | None, n_keypoints: int, caller_name: str | None = None) -> Tensor:
```

Validate the given keypoint sigmas or create the default ones.

The sigmas are the per-keypoint scales of the object keypoint similarity in
[compute_pose_oks](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/utils/keypoints.md).
When `sigmas` is `None`, the function selects the defaults:

 * For `17` keypoints, it returns the COCO person sigmas and logs a warning.
 * For any other count, it returns `0.04` for each keypoint and logs an info message.

> **Examples**
> ```pycon
>>> get_sigmas([0.1, 0.2], 2).shape
torch.Size([2])
```

```pycon
>>> get_sigmas([0.1], 2)
Traceback (most recent call last):
ValueError: The length of the sigmas list must be the same ...
```

Parameters

 * `sigmas` (`list[float] | None`): One sigma per keypoint. `None` selects the defaults.
 * `n_keypoints` (`int`): The number of keypoints.
 * `caller_name` (`str | None`): The name of the caller, used as a prefix of the log and error messages. `None` adds no prefix.

Returns

 * `Tensor`: The sigmas as a `float32` tensor of shape `[n_keypoints]`.

Raises

 * `ValueError`: When `sigmas` is given and its length differs from `n_keypoints`.

### insert_class

```python
def insert_class(keypoints: Tensor, bboxes: Tensor) -> Tensor:
```

Insert the class index of each bounding box into its keypoints.

> **Example**
> ```pycon
>>> import torch
>>> keypoints = torch.tensor([[0.0, 0.5, 0.5, 2.0]])
>>> bboxes = torch.tensor([[0.0, 3.0, 0.1, 0.1, 0.2, 0.2]])
>>> insert_class(keypoints, bboxes).tolist()
[[0.0, 3.0, 0.5, 0.5, 2.0]]
```

Parameters

 * `keypoints` (`Tensor`): The keypoints of shape `[N, 1 + 3K]`, where `K` is the number of keypoints. The batch index is in the
   first column, followed by `(x, y, visibility)` triples.
 * `bboxes` (`Tensor`): The bounding boxes of shape `[N, 6]`, in the same instance order, with the class index in the second
   column.

Returns

 * `Tensor`: The keypoints of shape `[N, 2 + 3K]`, with the class index inserted as the second column.