# general

Python API: `luxonis_train.utils.general`

Small helpers used across the package.

The helpers round channel counts, infer upscale factors, inspect signatures, split batched instances, download files to a cache,
and decode text metadata labels.

## Classes

### Counter

A counter that returns the next integer on each call.

The prediction writer that saves the inference renders keeps one counter for all batches. It uses the counter to number the saved
render files, or to select the image path of each image.

> **Example**
> ```pycon
>>> counter = Counter(start=5)
>>> counter(), counter(), counter()
(5, 6, 7)
```

#### Methods

##### init

```python
def __init__(start: int = 0):
```

Initialize the counter.

Parameters

 * `start` (`int`): The first value that the counter returns.

## Functions

### clean_url

```python
def clean_url(url: str) -> str:
```

Strip the query string from a URL and decode percent-escapes.

The function first normalizes the URL with `pathlib.PurePosixPath`. This step collapses repeated slashes to one, removes the `.` path components, and removes a trailing slash. The function decodes the escapes before it cuts the URL at the first `?`, so an escaped `%3F` also cuts the URL.

> **Example**
> ```pycon
>>> clean_url("https://url.com/dir/file%20name.txt?token=abc")
'https://url.com/dir/file name.txt'
>>> clean_url("https://url.com//dir/")
'https://url.com/dir'
```

Parameters

 * `url` (`str`): The URL, for example `"https://url.com/file%20a.txt?auth"`.

Returns

 * `str`: The URL without the first `?` and the text after it, with the `%XX` escapes decoded, for example `"https://url.com/file
   a.txt"`.

### decode_text_metadata_labels

```python
def decode_text_metadata_labels(labels: dict[str, np.ndarray], metadata_types: dict[str, type]) -> dict[str, np.ndarray]:
```

Decode the `str` metadata labels from character codes.

[BaseLoaderTorch](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/loaders/base_loader.md)
converts a string label to a `float32` tensor of its character codes. When it collates a batch of `metadata/text` labels, it pads
the shorter rows with `0`. This function reverses that for every label whose type in `metadata_types` is `str`:

 * Each row becomes the string of its codes up to the first `0`.
 * A one-dimensional array counts as one row.
 * The codes can be integers or integer-valued floats.

The function converts all other labels to arrays with `np.asarray`.

A `str` label stays unchanged in these cases:

 * Its array is empty or has a string or object dtype.
 * A row has a negative value before its first `0`.
 * A row has a fractional value before its first `0`.
 * A row has a value that is not an integer or a float before its first `0`.

> **Example**
> ```pycon
>>> import numpy as np
>>> labels = {
...     "text": np.array([[72.0, 105.0, 0.0], [79.0, 75.0, 33.0]]),
...     "count": np.array([1, 2]),
... }
>>> decoded = decode_text_metadata_labels(
...     labels, {"text": str, "count": int}
... )
>>> decoded["text"].tolist()
['Hi', 'OK!']
>>> decoded["count"].tolist()
[1, 2]
```

Parameters

 * `labels` (`dict[str, np.ndarray]`): Label names mapped to their label arrays.
 * `metadata_types` (`dict[str, type]`): Label names mapped to the type of their metadata values. The function does not decode a label that is missing here.

Returns

 * `dict[str, np.ndarray]`: The same label names. A decoded `str` label is an array of strings, one for each row.

### get_attribute_check_none

```python
def get_attribute_check_none(obj: object, attribute: str) -> Any:
```

Get the private attribute `_<attribute>` and reject `None`.

A property uses it to expose a value that the constructor can leave unset.

> **Examples**
> ```pycon
>>> class Person:
...     def __init__(self, age: int | None = None):
...         self._age = age
...
...     @property
...     def age(self):
...         return get_attribute_check_none(self, "age")
```

```pycon
>>> mike = Person(20)
>>> print(mike.age)
20
```

```pycon
>>> amanda = Person()
>>> print(amanda.age)
Traceback (most recent call last):
ValueError: attribute 'age' was not set
```

Parameters

 * `obj` (`object`): The object that holds the attribute.
 * `attribute` (`str`): The attribute name without the leading underscore.

Returns

 * `Any`: The value of `obj._<attribute>`.

Raises

 * `AttributeError`: When `obj` has no attribute `_<attribute>`.
 * `ValueError`: When the value is `None`.

### get_batch_instances

```python
def get_batch_instances(batch_index: int, bboxes: Tensor, payload: Tensor | None = None) -> Tensor:
```

Select the rows of one image from batched instance data.

> **Example**
> ```pycon
>>> import torch
>>> bboxes = torch.tensor([[0, 1], [0, 2], [1, 3]])
>>> get_batch_instances(1, bboxes).tolist()
[[3]]
>>> payload = torch.tensor([10, 20, 30])
>>> get_batch_instances(0, bboxes, payload).tolist()
[10, 20]
```

Parameters

 * `batch_index` (`int`): The index of the image in the batch.
 * `bboxes` (`Tensor`): The bounding boxes of the whole batch, of shape `[N, C]`, with the batch index in the first column.
 * `payload` (`Tensor | None`): A tensor of shape `[N, ...]` with one row per row of `bboxes`, in the same order. `None` selects from `bboxes` itself.

Returns

 * `Tensor`: The rows whose batch index equals `batch_index`. From `bboxes` they come without the first column, of shape `[n, C - 1]`. From `payload` they come with all columns, so a batch index column in `payload` stays.

### get_signature

```python
def get_signature(func: Callable, exclude: Collection[str] | None = None) -> dict[str, Parameter]:
```

Get the parameters of a function without the excluded ones.

The function always leaves out the parameters `self` and `kwargs`. It also leaves out the names in `exclude`.

> **Example**
> ```pycon
>>> def forward(self, x, y=1, **kwargs): ...
>>> list(get_signature(forward))
['x', 'y']
>>> list(get_signature(forward, exclude=["y"]))
['x']
```

Parameters

 * `func` (`Callable`): The function or method to inspect.
 * `exclude` (`Collection[str] | None`): More parameter names to leave out. `None` excludes only `"self"` and `"kwargs"`.

Returns

 * `dict[str, Parameter]`: The remaining parameter names, in signature order, mapped to their `inspect.Parameter` objects.

### get_with_default

```python
def get_with_default(value: T | None, action_name: str, caller_name: str | None = None, *, default: T) -> T:
```

Return `value`, or `default` when `value` is `None`.

When the function returns `default`, it logs an info message that names `action_name`.

> **Example**
> ```pycon
>>> get_with_default(0.4, "area factor", default=0.53)
0.4
```

Parameters

 * `value` (`T | None`): The value to return when it is not `None`.
 * `action_name` (`str`): What the value is for, named in the log message.
 * `caller_name` (`str | None`): The name of the caller, used as a prefix of the log message. `None` adds no prefix.
 * `default` (`T`): The value to return when `value` is `None`.

Returns

 * `T`: `value` when it is not `None`, else `default`.

### infer_upscale_factor

```python
def infer_upscale_factor(in_size: tuple[int, int] | int, orig_size: tuple[int, int] | int) -> int:
```

Infer the number of doublings from one size to another.

The function returns the exponent n with o = i⋅2n, where i is the input size and o is the original size. The result is not the factor 2n itself. n is negative when the original size is smaller. The segmentation heads use n as the number of upsampling steps, or compute the scale factor 2n from it.

> **Examples**
> ```pycon
>>> infer_upscale_factor((32, 32), (128, 128))
2
>>> infer_upscale_factor(16, 64)
2
>>> infer_upscale_factor(64, 16)
-2
```

```pycon
>>> infer_upscale_factor((32, 64), (128, 128))
Traceback (most recent call last):
ValueError: Width and height upscale factors are different. ...
```

Parameters

 * `in_size` (`tuple[int, int] | int`): The input size as `(height, width)`, or one integer for both.
 * `orig_size` (`tuple[int, int] | int`): The original size as `(height, width)`, or one integer for both.

Returns

 * `int`: The exponent n.

Raises

 * `ValueError`: When the width ratio or the height ratio is not a power of two, when the two exponents differ, or when a size is
   not positive.

### instances_from_batch

```python
def instances_from_batch(bboxes: Tensor, *args: Tensor, batch_size: int | None = None) -> Iterator[Tensor | tuple[Tensor, ...]]:
```

Yield the instances of each image from batched instance data.

The batch index is in the first column of `bboxes`. The extra tensors in `args` have one row per row of `bboxes`, in the same
order. The function selects their rows with the batch index of `bboxes` and keeps all their columns. The object keypoint
similarity metric passes the target keypoints as an extra tensor.

The function yields one item for each image index from `0` to `batch_size - 1`. When `batch_size` is `None` or `0`, the number of
items is the largest batch index plus one. In that case, the function yields no items for the images without instances at the end
of the batch.

When `bboxes` is empty, the function yields new empty tensors from `torch.empty_like`, with the shapes of the inputs. The bounding
boxes keep the batch index column. Empty input yields no items when `batch_size` is `None` or `0`.

> **Examples**
> ```pycon
>>> import torch
>>> bboxes = torch.tensor([[0, 1], [0, 2], [1, 3]])
>>> keypoints = torch.tensor([[10], [20], [30]])
>>> for bbox, kpt in instances_from_batch(bboxes, keypoints):
...     print(bbox.tolist(), kpt.tolist())
[[1], [2]] [[10], [20]]
[[3]] [[30]]
```

```pycon
>>> [b.tolist() for b in instances_from_batch(bboxes, batch_size=3)]
[[[1], [2]], [[3]], []]
```

Parameters

 * `bboxes` (`Tensor`): The bounding boxes of the whole batch, of shape `[N, C]`, with the batch index in the first column.
 * `*args` (`Tensor`): Extra tensors of shape `[N, ...]`, in the same order as `bboxes`.
 * `batch_size` (`int | None`): The number of images to yield. `None` or `0` infers it from the largest batch index.

Returns

 * `Iterator[Tensor | tuple[Tensor, ...]]`

Yields

 * Without extra tensors, the bounding boxes of one image with the batch index column removed, of shape `[n, C - 1]`. With extra tensors, a tuple of those bounding boxes followed by the matching rows of each tensor in `args`.

Raises

 * `ValueError`: When a tensor in `args` has a different length than `bboxes`. The error occurs when the iteration starts, not when the code calls the function.

### make_divisible

```python
def make_divisible(x: float, divisor: int) -> int:
```

Round `x` up to the nearest multiple of `divisor`.

The function computes ⌈x ⁄ d⌉⋅d, where d is `divisor`. The backbones and necks use it to round channel counts.

> **Example**
> ```pycon
>>> make_divisible(37, 8)
40
>>> make_divisible(40, 8)
40
```

Parameters

 * `x` (`float`): The value to round.
 * `divisor` (`int`): The number the result is a multiple of.

Returns

 * `int`: The smallest multiple of `divisor` that is not less than `x`.

### safe_download

```python
def safe_download(url: PathType | None, file: str | None = None, cache_dir: PathType = '.cache/luxonis_train', retry: int = 3, force: bool = False) -> Path | None:
```

Download a remote file into the cache and return its local path.

The function returns a `pathlib.Path` unchanged. It converts a `str` without a remote protocol to a `pathlib.Path`. It does not
check that a local file exists.

The function saves a remote file as `cache_dir/<version>/<file>`, where `<version>` is the `luxonis_train` version. It creates
that directory when it is missing. It reuses a file that is already there and logs a warning, unless `force` is `True`. Before the
download, it logs an info message with the local path and the URL. The logged URL comes from
[clean_url](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/utils/general.md)
and has no query string.

The function downloads `s3`, `gcs`, and `gs` URLs with `LuxonisFileSystem.download`, and all other URLs with
`torch.hub.download_url_to_file`. After a failed attempt, it logs the traceback and tries again, at most `retry` more times. When
all attempts fail, it logs a warning and returns `None`.

> **Example**
> ```pycon
>>> from pathlib import Path
>>> safe_download("weights/model.ckpt") == Path("weights/model.ckpt")
True
>>> safe_download(None) is None
True
```

Parameters

 * `url` (`PathType | None`): The URL or path of the file. `None` returns `None`.
 * `file` (`str | None`): The name of the saved file. `None` takes the file name from `url`.
 * `cache_dir` (`PathType`): The root of the cache directory.
 * `retry` (`int`): The number of repeated attempts after a failed download.
 * `force` (`bool`): When `True`, download again even when the file is in the cache.

Returns

 * `Path | None`: The local path of the file, or `None` when every attempt failed. For `s3`, `gcs`, and `gs` URLs, the path that `LuxonisFileSystem.download` returns.

### to_shape_packet

```python
def to_shape_packet(packet: Packet[Tensor]) -> Packet[Size]:
```

Convert a packet of tensors to a packet of their shapes.

[LuxonisOutput](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/lightning/luxonis_output.md) uses it in its string form to show the output shapes of the nodes. The model uses it when it builds the nodes, to give each node the shapes of its inputs.

> **Example**
> ```pycon
>>> import torch
>>> features = [torch.zeros(1, 3), torch.zeros(2, 4)]
>>> to_shape_packet({"features": features, "boxes": torch.zeros(5, 4)})
{'features': [torch.Size([1, 3]), torch.Size([2, 4])],
 'boxes': torch.Size([5, 4])}
```

Parameters

 * `packet` (`Packet[Tensor]`): A packet whose values are tensors or lists of tensors.

Returns

 * `Packet[Size]`: A packet with the same keys. A tensor becomes its `torch.Size`, and a list of tensors becomes a list of
   `torch.Size` objects.

### url2file

```python
def url2file(url: str) -> str:
```

Get the file name from a URL.

> **Example**
> ```pycon
>>> url2file("https://url.com/dir/file.txt?token=abc")
'file.txt'
```

Parameters

 * `url` (`str`): The URL, for example `"https://url.com/file.txt?auth"`.

Returns

 * `str`: The last path component of the URL after [clean_url](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/utils/general.md), for example `"file.txt"`.

## Attributes

### T