# Inference

## Overview

Models converted for RVC Platforms can be deployed on OAK devices to perform inference. The following section guides you through
setting up a simple inference pipeline for a desired AI model. We utilize DepthAI to build the inference pipeline as a sequence
of:

 * [Built-in nodes](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes.md) (execution location depends on the
   node and platform), and
 * [Host nodes](https://docs.luxonis.com/software-v3/depthai/depthai-components/host_nodes.md) (run on the host).

The nodes of both kinds can be connected when their input and output message types are compatible. The Built-in nodes are stable,
optimized, and ensure efficient performance on Luxonis devices, while the Host nodes offer a greater flexibility and can be
customized to meet specific use case. Please check out the [DepthAI
Nodes](https://docs.luxonis.com/software-v3/ai-inference/inference/depthai-nodes.md) library for our in-house collection of Python
host nodes.

The inference pipeline can be defined manually, node-by-node. However, we also offer a certain degree of automation of pipeline
creation based on the relevant NN Archive (for example, automatically connecting the neural network with a parser responsible for
decoding its outputs). Please see below for more information.

> **Note**
> If the model of choice is not converted for a desired RVC platform, please refer to the [Conversion](https://docs.luxonis.com/software-v3/ai-inference/conversion.md) section.

## Alternative standalone RVC4 inference options

These runtimes load models directly in the app, outside the DepthAI `NeuralNetwork` node, and support hardware acceleration on the
device's Hexagon Tensor Processor (HTP).

### ONNX Runtime with QNN

Load ONNX models through the standard ONNX Runtime API and accelerate supported operations with the QNN Execution Provider.

[Run ONNX models](https://docs.luxonis.com/software-v3/ai-inference/inference/onnx-runtime-qnn.md)

### llama.cpp

Run GGUF language and vision-language models locally with CPU or Hexagon acceleration, including questions about camera images.

[Run language and vision-language models](https://docs.luxonis.com/software-v3/ai-inference/inference/llamacpp.md)

## Installation

Creation of an inference pipeline requires DepthAI (v3) library. Usage of our custom host nodes (e.g. for model output decoding)
requires DepthAI Nodes library. The current DepthAI Nodes package requires Python 3.10 or newer and DepthAI 3.9 or newer.
Installing it also installs its DepthAI dependency:

```bash
pip install depthai-nodes
```

## Inference Pipeline

We present here a simple inference pipeline template. It consists of four main sections that we describe in more detail below:

 * Camera,
 * Model and Parser(s);
 * Queue(s);
 * Results.

```python
import depthai as dai
from depthai_nodes.node import ParsingNeuralNetwork

model = "..." # HubAI model identifier, or use dai.NNArchive("model.tar.xz")

# Create pipeline
with dai.Pipeline() as pipeline:

    # Camera
    camera = pipeline.create(dai.node.Camera).build()

    # Model and Parser(s)
    nn_with_parser = pipeline.create(ParsingNeuralNetwork).build(
        camera, model
    )

    # Queue(s)
    parser_output_queue = nn_with_parser.out.createOutputQueue()

    # Start pipeline
    pipeline.start()

    while pipeline.isRunning():

        # Results
        parser_output = parser_output_queue.get()
        ...
```

> **Note**
> The template assumes a single-head model with parser metadata and a model variant compatible with the connected device. You can limit inference FPS with `pipeline.create(ParsingNeuralNetwork).build(camera, model, fps=15)`.

### Camera

The inference pipeline starts with the [Camera](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/camera.md)
node. It is the source of image frames that get inferenced on. The node can be added to a pipeline as follows:

```python
camera_node = pipeline.create(dai.node.Camera).build()
```

### Model and Parser(s)

Inference consists of two steps. First, the model makes predictions on the input data. Second, a postprocessing node, also known
as a parser, is utilized to process the model output(s). This step is optional and the raw model output is returned if skipped.
You can find more information about the available parsers in the [DepthAI Nodes API
reference](https://docs.luxonis.com/software-v3/ai-inference/inference/depthai-nodes/depthai-nodes-api-reference/node.md) library.

A model is set up using the
[NeuralNetwork](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/neural_network.md) node. A model-parser pair
can be established:

 * Automatically, using the `ParsingNeuralNetwork` node, or
 * Manually, initializing them as independent nodes and linking them together.

The former automatically links the model outputs with the appropriate parsers as defined in the relevant [NN
Archive](https://docs.luxonis.com/software-v3/ai-inference/nn-archive.md). This abstracts away all the configuration details and
is thus the preferred way of interaction with parsers. The created nodes (independent or not) can be used the same way as standard
DepthAI nodes by linking them to other nodes or define them as pipeline queues.

#### Automatic Setup

The `ParsingNeuralNetwork` node wraps a standard `NeuralNetwork` node and selects native DepthAI parsers from the archive
metadata. Use `HostParsingNeuralNetwork` with the same build arguments to select the Python host parsers instead. This changes the
parser implementation, not the underlying inference node; unavailable native parsers do not automatically fall back to Python. It
can be imported from the `depthai_nodes` package as:

```python
from depthai_nodes.node import ParsingNeuralNetwork
```

and instantiated directly from either:

(1) NN Archive object, or

```python
# Set up NN Archive
nn_archive = dai.NNArchive("path/to/NNArchiveName.tar.xz")

# Set up model (with parser(s)) and link it to camera output
nn_with_parser = pipeline.create(ParsingNeuralNetwork).build(
    camera_node, nn_archive
)
```

(2) HubAI, by specifying the model identifier (a unique identifier of a model on the HubAI platform. Find more information at the
[Model Upload/Download](https://docs.luxonis.com/cloud/hubai/model-registry/upload-download.md) section).

```python
# Set up the HubAI model identifier
model = "..."

# Set up model with parser(s)
nn_with_parser = pipeline.create(ParsingNeuralNetwork).build(
    camera_node, model
)
```

You can also pass a `dai.NNModelDescription`. A plain string is treated as a model identifier, so wrap local paths in
`dai.NNArchive(...)`. For downloaded models, the wrapper fills in the connected device's platform when none is specified. Local
archives must already target the correct platform. It configures an output from the supplied camera and links it to the model
input.

> **Note**
> If you plan to use a private HubAI model, make sure to configure your Luxonis Hub API Key. Instructions are available on the [API Key Good Practices](https://docs.luxonis.com/software-v3/oak-apps/apikey-good-practices.md) page. Once set up correctly, the API key will be applied automatically to authenticate your requests with the HubAI platform.

#### Manual Setup

The model and the parser can also be instantiated as independent nodes.

First, import a `DepthAI Nodes` parser of interest or implement a parser of your own.

```python
# For example: from depthai_nodes.node import ClassificationParser
# Or implement your own parser (skeleton):
class ParserNode(dai.node.ThreadedHostNode):
    def __init__(self) -> None:
        super().__init__()
        self.input = self.createInput()
        self.out = self.createOutput()
    def build(self) -> "ParserNode":
        return self
    def run(self) -> None:
        while self.isRunning():
            nn_out_raw = self.input.get()
            nn_out_processed = ... # replace with your postprocessing and message creation
            self.out.send(nn_out_processed)
```

Second, initialize the model and the parser as individual nodes by calling the `create()` method on the pipeline:

```python
model = pipeline.create(dai.node.NeuralNetwork)
parser = pipeline.create(ParserNode)
```

For built-in Python parsers, the supported initialization arguments and configuration dictionaries vary by parser. Configure them
before starting the pipeline, either:

 * at initialization, configuration can be set by passing the parameter values as arguments to the `create()` method: `parser =
   pipeline.create(<ParserNode>, <ParameterName>=<ParameterValue>, ...)` If configuring multiple parameters, you can arrange them
   into a `dict` and pass it as an argument to the parser `build()` method: `parser =
   pipeline.create(ParserNode).build(config_dict)`;
 * after initialization, one can change configuration by using the setter methods: `parser.<SetterMethodName>(<ParameterValue>)`.
   You can find all the setter methods available for a specific parser on the [DepthAI Nodes API
   Reference](https://docs.luxonis.com/software-v3/ai-inference/inference/depthai-nodes/depthai-nodes-api-reference.md) page.

Third, set the model executable (i.e. the `.blob` for RVC2, or the `.dlc` file for RVC4):

```python
model.setModelPath("path/to/model_executable")
```

Last, prepare the camera stream and link the independent nodes to constitute a pipeline. Match its size, image type, and layout to
the model:

```python
width, height = 640, 640 # replace with your model input size
camera_stream = camera_node.requestOutput(size=(width, height))
camera_stream.link(model.input)
model.out.link(parser.input)
```

> **Note**
> For custom parsers, see [`BaseParser`](https://docs.luxonis.com/software-v3/ai-inference/inference/depthai-nodes/depthai-nodes-api-reference/node/parsers/base_parser.md). To create and configure parsers from NN Archive metadata in a manual pipeline, use [`ParserGenerator`](https://docs.luxonis.com/software-v3/ai-inference/inference/depthai-nodes/depthai-nodes-api-reference/node/parser_generator.md).

### Queue(s)

Queues are used to obtain data from specific nodes of the pipeline. To obtain the image frame that gets input to the model, you
can use the passthrough queue:

```python
frame_queue = nn_with_parser.passthrough.createOutputQueue()
```

To obtain the (parsed) model output, you can use the output queue(s). The definition depends on the number of model heads:

#### Single-Headed

```python
parser_output_queue = nn_with_parser.out.createOutputQueue()
```

#### Multi-Headed

```python
head0_parser_output_queue = nn_with_parser.getOutput(0).createOutputQueue()
head1_parser_output_queue = nn_with_parser.getOutput(1).createOutputQueue()
...
```

For synchronized multi-head results, use `nn_with_parser.outputs.createOutputQueue()`. It emits `dai.MessageGroup` messages keyed
by string head indexes, such as `"0"` and `"1"`. The `out` property requires one head; `outputs` requires multiple heads.

### Results

After the pipeline is started with `pipeline.start()`, outputs can be obtained from the defined queue(s). You can obtain the input
frame and parsed model outputs as:

```python
while pipeline.isRunning():

    # Get Camera Output
    frame_queue_output = frame_queue.get()
    frame = frame_queue_output.getCvFrame()
    ...

    # Get Parsed Output(s)
    parser_output = parser_output_queue.get()
    ...
```

Reading the frame and result queues sequentially does not itself synchronize them. Match timestamps or sequence numbers when
pairing results with frames.

The supplied parsers return native DepthAI messages, including `dai.ImgDetections`, `dai.SegmentationMask`, and `dai.beta` types
such as `Classifications` and `Keypoints`. See the [message
reference](https://docs.luxonis.com/software-v3/ai-inference/inference/depthai-nodes/depthai-nodes-api-reference/message.md) for
formats and the separate custom containers used for grouping and uploads.

## Examples

Please consult the [OAK Examples](https://docs.luxonis.com/software-v3/ai-inference/inference/oak-examples.md) page.

## Troubleshooting

Below are some common issues and their solutions.

### Setting Model SHAVEs

SHAVEs are the compute cores in the RVC2 VPU that run neural networks. Sometimes, a model is built to use a different number of
SHAVEs than what the target device supports.

If the model is compiled to use more SHAVEs than the device actually has, the pipeline will fail with a RuntimeError similar to:

```bash
NeuralNetwork: Blob compiled for ... shaves, but only ... are available in current configuration
```

The available allocation depends on the pipeline configuration, including inference-thread counts.

Conversely, if the model is compiled to use fewer SHAVEs than are available on the device, it will still run, but you may see a
warning like:

```bash
[14442C103180EECF00] [2.1] [4.736] [NeuralNetwork(2)] [warning] Network compiled for 8 shaves, maximum available 13, compiling for 6 shaves likely will yield in better performance
```

A larger SHAVE count does not necessarily improve throughput. Benchmark the allocation and inference-thread configuration
together.

To fix the number of utilized SHAVES, you can either:

 * if model was exported with legacy
   [Blobconverter](https://docs.luxonis.com/software-v3/ai-inference/conversion/rvc-conversion/online/blobconverter.md), you can
   recompile the model with a matching number of SHAVEs, or
 * if the NN Archive contains a superblob with a suitable compiled variant, recompilation is not needed. Select the SHAVE count on
   the native network before configuring parsers and queues. This requires a superblob, not just any archive exported from Hub AI:

```python
nn_archive = dai.NNArchive("model.tar.xz")
model = pipeline.create(dai.node.NeuralNetwork).build(camera_node, nn_archive)
# Requires a superblob with a compatible six-SHAVE variant.
model.setNNArchive(nn_archive, numShaves=6)
```

> **Note**
> The old `SHAVE` configuration method is no longer supported in DepthAI v3.

Avoid using:

```python
nn = pipeline.create(dai.node.NeuralNetwork)
nn.setNumShaves(6)
```

### Changing Parser Parameters

To modify parser parameters, you first need to access the parser object.

 * Pipelines with a separate parser node: Simply access the parser node directly.
 * Pipelines with a `ParsingNeuralNetwork` node: In this case, the parser is integrated with the AI model. Retrieve it by calling
   the `.getParser()` method on the `ParsingNeuralNetwork` node.

Native and Python parsers can expose different setters; check the selected parser's API. For example, a native
`dai.node.DetectionParser` uses:

```python
parser.setConfidenceThreshold(0.5)
```

### Different Visualization and Model Input Sizes

You can use separate image sizes for model input and for visualization. Use the `ImageManip` node to resize the image before
sending it to the model, while keeping the original resolution for display.

Example:

```python
cam = pipeline.create(dai.node.Camera).build()

# Request specific image size for capture
cam_out = cam.requestOutput(size=(1280, 720))

# Create and configure resize node for model input
resize_node = pipeline.create(dai.node.ImageManip)
resize_node.initialConfig.setOutputSize(640, 640) # use your model input size
cam_out.link(resize_node.inputImage)

# Define model with resized input
nn_with_parser: ParsingNeuralNetwork = pipeline.create(ParsingNeuralNetwork).build(
    resize_node.out, ...
)

# Visualize Using Original Resolution
video_queue = cam_out.createOutputQueue() # high-res stream
detection_queue = nn_with_parser.out.createOutputQueue() # detections on the low-res stream
...
```

Account for cropping or letterboxing when mapping detections back to the display frame. See
[`CoordinatesMapper`](https://docs.luxonis.com/software-v3/ai-inference/inference/depthai-nodes/depthai-nodes-api-reference/node/coordinates_mapper.md)
for supported message transformations.

### Model Location When Downloaded from HubAI

Downloaded models are cached. You can choose the location with `cacheDirectory` in `dai.getModelFromZoo()` or the
`DEPTHAI_ZOO_CACHE_PATH` environment variable; it is not necessarily the project root. If a model is already cached, it’s loaded
locally instead of re-downloading. To force a fresh download, can use the `useCached=False` parameter when downloading the model.
Example:

```python
nn_archive = dai.NNArchive(dai.getModelFromZoo(model_description, useCached=False))
nn_with_parser = pipeline.create(ParsingNeuralNetwork).build(
    ..., nn_archive
)
```

See the [DepthAI Python API](https://docs.luxonis.com/software-v3/depthai/api/python.md) for model download and cache options.

## Further Reading

The following section provide additional information and insights into the inference process on RVC4 platform and how NN models
are executed on Qualcomm's Hexagon Tensor Processor (HTP).

### Concurrent Model Execution on RVC4

When running multiple models concurrently on the Hexagon Tensor Processor (HTP) of a Qualcomm SoC, there are some important
considerations to keep in mind regarding resource allocation and scheduling. The HTP shares compute and on-chip memory resources
dynamically. DepthAI's RVC4 SNPE backend accepts `priority_hint` through the NeuralNetwork node's `setBackendProperties()` method.
This requests scheduling priority; it does not guarantee latency or a fixed share of compute. Measure the threading and priority
settings with your workload.

#### Resource allocation on HTP

 * HTP compute cores and V-TCM memory are shared elastically across all concurrent SNPE sessions.
 * Resource allocation is not fixed between multiple models and can vary frame-to-frame.

#### Control knobs

 * Use `priority_hint` in SNPE backend properties to request model scheduling priority. See
   [NeuralNetwork](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/neural_network.md).
 * Priority hints do not provide core affinity or compute quotas. Performance profiles control power/performance trade-offs.

#### Practical guidance

 1. Assume latency and throughput will fluctuate when new/concurrent sessions start or stop.
 2. Use SNPE timing logs such as layer-wise profiling, to measure end-to-end latency instead of guessing resource shares.