# Benchmarking

## Overview

The ModelConverter benchmark command measures converted models on physical RVC2, RVC3, and RVC4 devices. It reports throughput
and, when the selected backend exposes it, inference latency. RVC4 can also report device power, DSP, memory, and CPU utilization.

Benchmark inputs are generated randomly from the model's tensor metadata. The results measure runtime performance, not model
accuracy.

> **Note**
> ModelConverter does not currently provide a Hailo benchmark implementation.

## Installation

Install ModelConverter with its benchmark dependencies:

```bash
pip install "modelconv[bench]"
```

See the [ModelConverter
guide](https://docs.luxonis.com/software-v3/ai-inference/conversion/rvc-conversion/offline/modelconverter.md) for the base
installation and Docker requirements.

## Run a Benchmark

The `--model-path` option is required:

```bash
modelconverter benchmark <platform> --model-path <path_url_or_slug>
```

The command prints a table containing the configuration and measured results. Add `--save` to also write
`<model_name>_benchmark_results.csv` in the current working directory.

For example, benchmark a public HubAI model on RVC4:

```bash
modelconverter benchmark rvc4 \
  --model-path yolov6-nano:r2-coco-512x288 \
  --save
```

## Supported Platforms and Model Sources

Accepted inputs depend on the platform and, for RVC4, the benchmark backend:

| Platform and backend | Supported `--model-path` values |
| --- | --- |
| RVC2, DepthAI | A `.blob`, an NN Archive (`.tar.xz`), or a HubAI model slug. |
| RVC3, OpenVINO VPUX | A compiled `.blob`. |
| RVC4, DepthAI (default) | An NN Archive (`.tar.xz`) or a HubAI model slug. A standalone `.dlc` is not supported by this backend.
|
| RVC4, SNPE | A `.dlc`, an NN Archive (`.tar.xz`), or a HubAI model slug. |

File inputs can be local paths or remote URLs. NN Archives use the [Luxonis NN Archive
format](https://docs.luxonis.com/software-v3/ai-inference/nn-archive.md).

HubAI slugs use this form:

```text
[team_name/]model_name:variant[:instance]
```

For example, `luxonis/yolov6-nano:r2-coco-512x384` identifies a model and variant, while the optional final segment selects a
specific instance. Set `HUBAI_API_KEY` when accessing a private model or team.

## CLI Options

| Option | Description | Default | Applies to |
| --- | --- | --- | --- |
| `--model-path` | Model path, remote URL, or HubAI slug. | Required | All platforms |
| `--full` | Sweep the platform's predefined configurations instead of running one configuration. | `false` | All platforms |
| `--save` | Save the results to CSV in addition to printing them. | `false` | All platforms |
| `--repetitions` | Number of benchmark reports when time-based execution is disabled. | `10` | RVC2; RVC4 DepthAI |
| `--benchmark-time` | Duration in seconds. A positive value takes precedence over `--repetitions`. | `20` | RVC2; RVC4 DepthAI |
| `--num-threads` | Number of neural-network inference threads. | `2` | RVC2; RVC4 DepthAI |
| `--num-messages` | Messages included in each benchmark report. | `50` | RVC2; RVC4 DepthAI |
| `--requests` | Concurrent OpenVINO inference requests. | `1` | RVC3 |
| `--profile` | SNPE/DepthAI performance profile. | `balanced` | RVC4 |
| `--runtime` | Inference runtime: `dsp` or `cpu`. | `dsp` | RVC4 |
| `--num-images` | Number of generated inputs processed by the SNPE backend. | `500` | RVC4 SNPE |
| `--device-ip` | Address of the RVC4 device. | Auto-detect | RVC4 |
| `--device-id` | DepthAI device ID or ADB serial. Takes precedence over `--device-ip`. | Auto-detect | RVC4 |
| `--dai-benchmark` / `--no-dai-benchmark` | Select the DepthAI or SNPE RVC4 backend. | DepthAI | RVC4 |
| `--device-monitor` / `--no-device-monitor` | Collect available device utilization measurements. | Enabled | RVC4 |

Run `modelconverter benchmark --help` to see the option values supported by the installed release.

### Single and Full Runs

Without `--full`, ModelConverter runs one benchmark with the defaults and any overrides you provide.

With `--full`, it performs a platform-specific sweep:

 * RVC2 tests one, two, and three inference threads.
 * RVC3 tests one through five concurrent inference requests.
 * RVC4 tests each supported performance profile with one and two inference threads. This sweep is intended for the DepthAI
   backend because SNPE benchmarking does not use `--num-threads`.

Because full RVC4 sweeps run many configurations and device monitoring is enabled by default, they can take considerably longer
than a single benchmark.

### Duration Control

RVC2 and the RVC4 DepthAI backend use time-based execution by default. A positive `--benchmark-time` takes precedence over
`--repetitions`.

To run a fixed number of reports instead, disable time-based execution with a non-positive duration:

```bash
modelconverter benchmark rvc4 \
  --model-path <archive_or_slug> \
  --benchmark-time -1 \
  --repetitions 50
```

RVC3 runs a fixed internal workload of 1,000 asynchronous inferences. Use `--requests` to control its concurrency.

## RVC4 Backends

### DepthAI

DepthAI is the default RVC4 backend. It accepts an NN Archive or HubAI model slug, runs a DepthAI benchmark pipeline, and reports
mean throughput and mean per-inference latency.

The `--profile`, `--runtime`, `--num-threads`, `--num-messages`, `--benchmark-time`, and `--repetitions` options affect this
backend.

Use it for normal RVC4 measurements and for results comparable to other DepthAI workloads:

```bash
modelconverter benchmark rvc4 \
  --model-path ./model.tar.xz \
  --profile high_performance \
  --benchmark-time 30
```

### SNPE

Pass `--no-dai-benchmark` to execute `snpe-parallel-run` directly on the RVC4 device. This backend also accepts a standalone
`.dlc` and reports throughput; per-inference latency is reported as unavailable.

```bash
modelconverter benchmark rvc4 \
  --model-path ./model.dlc \
  --no-dai-benchmark \
  --runtime dsp \
  --profile balanced \
  --num-images 500
```

The SNPE backend needs the DLC input tensor metadata. ModelConverter first tries `snpe-dlc-info` on the host, then on the
connected device. For an NN Archive, it can fall back to the archive's input metadata.

ModelConverter creates at most 100 distinct random input samples on the device and hard-links any additional entries requested by
`--num-images`, limiting storage use. If `snpe-parallel-run` is killed because the device runs out of memory, reduce
`--num-images`.

## RVC4 Device Selection and Monitoring

For DepthAI-only benchmarking with monitoring disabled, normal DepthAI connectivity is sufficient:

```bash
modelconverter benchmark rvc4 \
  --model-path ./model.tar.xz \
  --no-device-monitor
```

The SNPE backend always requires shell and file-transfer access to the device. The default DepthAI run also requires it because
`--device-monitor` is enabled by default. ModelConverter prefers ADB and falls back to SSH when `--device-ip` is supplied and ADB
is unavailable.

Select a device with either:

 * `--device-ip <address>`
 * `--device-id <depthai_id_or_adb_serial>`

If both are supplied and identify different devices, `--device-id` takes precedence and ModelConverter warns about the mismatch.
If neither is supplied, the first suitable device is selected.

The RVC4 device monitor samples the counters available on the device, including:

 * system and processor power,
 * DSP utilization,
 * used RAM,
 * CPU utilization and frequency, and
 * thermal-zone temperatures.

The results include an idle baseline where available. Missing counters are reported as unavailable rather than preventing the
benchmark.

## Results

Every platform reports frames per second (`fps`). RVC2, RVC3, and RVC4 DepthAI runs also report inference latency in milliseconds.
RVC4 SNPE runs report latency as unavailable.

When RVC4 device monitoring is enabled, the console table adds the available system power, processor power, DSP utilization, used
memory, and CPU utilization. Saved CSV files also contain the collected result fields and the configuration used for each run.

Use identical hardware, device software, runtime, performance profile, and benchmark settings when comparing results. Throughput
can vary with device state and background workloads.

## Examples

### RVC2 NN Archive

Run an RVC2 archive for 60 seconds and save the result:

```bash
modelconverter benchmark rvc2 \
  --model-path ./model.tar.xz \
  --benchmark-time 60 \
  --save
```

### RVC3 Concurrency

Benchmark an RVC3 blob with four concurrent inference requests:

```bash
modelconverter benchmark rvc3 \
  --model-path ./model.blob \
  --requests 4
```

Use `--full` to compare one through five requests automatically.

### RVC4 HubAI Model with Monitoring

Benchmark a HubAI model on a selected device and retain the default device metrics:

```bash
modelconverter benchmark rvc4 \
  --model-path yolov6-nano:r2-coco-512x288 \
  --device-ip 192.168.1.50 \
  --device-monitor \
  --save
```

## Manual SNPE Benchmarking

For custom automation or lower-level control, invoke Qualcomm's SNPE tools directly instead of `modelconverter benchmark`:

 1. Inspect the DLC with `snpe-dlc-info` to determine its input tensor names, shapes, and data types.
 2. Create matching `.raw` inputs and an `input_list.txt` file.
 3. Run
    [`snpe-parallel-run`](https://docs.qualcomm.com/nav/home/SNPE_general_tools.html?product=1601111740010412#snpe-parallel-run)
    or
    [`snpe-throughput-net-run`](https://docs.qualcomm.com/nav/home/SNPE_general_tools.html?product=1601111740010412#snpe-throughput-net-run)
    on the device.

Refer to Qualcomm's [SNPE general tools
documentation](https://docs.qualcomm.com/nav/home/SNPE_general_tools.html?product=1601111740010412) for the options and input
formats supported by the installed SNPE release.