# base_benchmark

Python API: `modelconverter.platforms.base_benchmark`

Platform-agnostic scaffolding for on-device benchmarking.

Every platform measures throughput and latency with its own runtime, but the surrounding work is the same: resolve the model to
either a path or URL of a model file or a HubAI slug, run it once per configuration, and report the collected numbers. That shared
part lives in
[Benchmark](https://docs.luxonis.com/software-v3/ai-inference/conversion/rvc-conversion/offline/modelconverter/modelconverter-api-reference/platforms/base_benchmark.md),
which the per-platform benchmarks subclass.

## Classes

### Benchmark

Base class for benchmarking a converted model on a device.

Subclasses implement the platform-specific measurement in
[benchmark](https://docs.luxonis.com/software-v3/ai-inference/conversion/rvc-conversion/offline/modelconverter/modelconverter-api-reference/platforms/base_benchmark.md)
and describe what to measure it with in
[default_configuration](https://docs.luxonis.com/software-v3/ai-inference/conversion/rvc-conversion/offline/modelconverter/modelconverter-api-reference/platforms/base_benchmark.md)
and
[all_configurations](https://docs.luxonis.com/software-v3/ai-inference/conversion/rvc-conversion/offline/modelconverter/modelconverter-api-reference/platforms/base_benchmark.md).

#### Methods

##### init

```python
def __init__(model_path: str):
```

Resolve the model to benchmark.

Parameters

 * `model_path` (`str`): Either a local path or a URL of a model file ending in one of the supported extensions, or a HubAI model
   slug of the form `[team_name/]model_name:variant[:model_instance]`. A URL is downloaded first.

Raises

 * `ValueError`: If `model_path` is neither a supported model file nor a slug of a model available on HubAI.

##### benchmark

```python
def benchmark(configuration: Configuration) -> Result:
```

Run a single benchmark with the given configuration.

Parameters

 * `configuration` (`Configuration`): The options to benchmark with.

Returns

 * `Result`: The measured metrics. Must contain a `"fps"` entry and may contain a `"latency"` entry.

##### print_results

```python
def print_results(results: list[tuple[Configuration, Result]]):
```

Print the benchmark results as a table.

The table is transposed relative to the results: one column per run, one row per configuration option and measured metric.

Parameters

 * `results` (`list[tuple[Configuration, Result]]`): The configuration/result pairs to print. Must not be empty.

##### run

```python
def run(full: bool = True, save: bool = False, **kwargs: ConfigValue):
```

Benchmark the model and report the results.

Parameters

 * `full` (`bool`): If `True`, benchmark every configuration in
   [all_configurations](https://docs.luxonis.com/software-v3/ai-inference/conversion/rvc-conversion/offline/modelconverter/modelconverter-api-reference/platforms/base_benchmark.md).
   If `False`, benchmark only
   [default_configuration](https://docs.luxonis.com/software-v3/ai-inference/conversion/rvc-conversion/offline/modelconverter/modelconverter-api-reference/platforms/base_benchmark.md).
 * `save` (`bool`): If `True`, the results are written to a CSV file in addition to being printed.
 * `**kwargs` (`ConfigValue`): Overrides for individual configuration options. An override of an option with a non-`None` default
   is cast to the type of that default, unless the default is a boolean. In a full run, an override is only applied to
   configurations that do not set the option themselves.

##### save_results

```python
def save_results(results: list[tuple[Configuration, Result]]):
```

Save the benchmark results to a CSV file.

The file is named `<model_name>_benchmark_results.csv` and is written to the current working directory. A nested `power` column is
first split into `power_system` and `power_processor`.

Parameters

 * `results` (`list[tuple[Configuration, Result]]`): The configuration/result pairs to save. Must not be empty.

#### Attributes

##### all_configurations

Return the configurations a full benchmark run sweeps.

##### default_configuration

Return the configuration a plain benchmark run uses.

##### model_name

##### model_path

## Functions

### get_option

```python
def get_option(configuration: Configuration, key: str, option_type: type[OptionT]) -> OptionT:
```

Read one benchmark option and check its type.

Parameters

 * `configuration` (`Configuration`): The options of a single benchmark run.
 * `key` (`str`): The name of the option.
 * `option_type` (`type[OptionT]`): The necessary type of the option.

Returns

 * `OptionT`: The value of the option.

Raises

 * `TypeError`: If the option is missing or has a different type.

### get_optional_option

```python
def get_optional_option(configuration: Configuration, key: str, option_type: type[OptionT]) -> OptionT | None:
```

Read one benchmark option that may also be unset.

Parameters

 * `configuration` (`Configuration`): The options of a single benchmark run.
 * `key` (`str`): The name of the option.
 * `option_type` (`type[OptionT]`): The necessary type of the option.

Returns

 * `OptionT | None`: The value of the option, or `None` if it is not set.

Raises

 * `TypeError`: If the option has a different type.

## Attributes

### Configuration

### ConfigValue

### OptionT

### Result