# ONNX Runtime with QNN

## Overview

ONNX Runtime with the Qualcomm QNN Execution Provider is an alternative inference path for
[RVC4](https://docs.luxonis.com/hardware/platform/rvc/rvc4.md) devices. It lets a standalone OAK App load an `.onnx` model with
the standard ONNX Runtime API while QNN executes supported operations on the RVC4 Hexagon Tensor Processor (HTP).

This path is available only inside an RVC4 standalone app. It is not available for RVC2 devices or for an application running on a
separate host computer. The app ships the ONNX file, and QNN compiles it for the HTP when the ONNX Runtime session is first
created.

> **When to use this path**
> Use ONNX Runtime with QNN when your application already consumes ONNX models or an integration expects the ONNX Runtime API, and you want hardware acceleration without first converting every model to a native RVC4 artifact. For the most direct integration with a DepthAI pipeline, a model converted for the built-in `NeuralNetwork` node remains the primary deployment path.

Typical use cases include:

 * Running an ONNX-only model from Python code in an OAK App.
 * Accelerating a third-party library that creates ONNX Runtime sessions.
 * Keeping a model in ONNX format while evaluating whether a native conversion is worthwhile.
 * Running auxiliary models, such as text encoders, alongside a native DepthAI neural network.

## Deployment Paths

All four paths below run locally on the RVC4 device, but they have different integration and model-preparation requirements.

| Path | Model artifact | Execution target | Best suited for |
| --- | --- | --- | --- |
| ONNX Runtime CPU | `.onnx` | RVC4 ARM CPU | Compatibility testing, debugging, and fallback |
| ONNX Runtime QNN | `.onnx` with a compiled QNN context created at session startup | HTP | ONNX Runtime integrations and models
that are not yet converted for native DepthAI deployment |
| Native SNPE FP16 | Converted RVC4 model, usually supplied through an NN Archive | HTP | First-class `NeuralNetwork` node
integration with floating-point accuracy |
| Native SNPE INT8 | Quantized, converted RVC4 model with representative calibration | HTP | Highest throughput when the accuracy
and conversion requirements are satisfied |

The QNN route runs in the OAK App process and exchanges NumPy arrays through ONNX Runtime. The native SNPE route runs through
`dai.node.NeuralNetwork`, so frames and inference messages remain part of the DepthAI graph.

## Requirements

You need:

 * An RVC4-based OAK device running Luxonis OS 1.40.0 or newer.
 * A [standalone OAK App](https://docs.luxonis.com/software-v3/oak-apps.md) based on the
   `luxonis/oakapp-base:<version>-onnxruntime` image.
 * The QNN/FastRPC devices and the OS-provided NPU runtime mounted into the app container.
 * `depthai-nodes`, which provides the
   [`onnx_qnn_session`](https://github.com/luxonis/depthai-nodes/blob/main/depthai_nodes/runtime/onnx_qnn.py) helper.
 * An ONNX graph with static shapes and operations supported by the QNN HTP backend.

The ONNX Runtime variant of [`oakapp-base`](https://github.com/luxonis/oakapp-base) includes `onnxruntime`, `onnxruntime-qnn`, the
QNN libraries, and the environment needed to locate the device NPU runtime. Use `/entrypoint.sh` so that this environment is
configured before the Python application starts.

### OAK App configuration

The following `oakapp.toml` contains the QNN-specific settings for a minimal Python app. Replace the base-image version with the
version selected for your application.

```toml
identifier = "com.example.onnx-qnn"
app_version = "1.0.0"
base_image = "luxonis/oakapp-base:1.4.0-onnxruntime"

entrypoint = ["/entrypoint.sh", "python3.12", "-u", "/app/main.py"]

prepare_container = [
    { type = "COPY", source = "requirements.txt", target = "requirements.txt" },
    { type = "RUN", command = "python3.12 -m pip install --no-cache-dir -r /app/requirements.txt" },
]

optional_devices = [
    "/dev/fastrpc-cdsp",
    "/dev/adsprpc-smd",
    "/dev/dma_heap/qcom,system",
    "/dev/dma_heap/system",
]
optional_mounts = [
    "/opt/luxonis/npu-runtime:/opt/luxonis/npu-runtime:ro,rbind",
]
allowed_devices = [{ allow = true, access = "rw" }]
```

The base image already supplies its compatible ONNX Runtime packages. Avoid replacing `onnxruntime` or `onnxruntime-qnn`
indirectly when installing other dependencies; if a package changes them, explicitly restore the versions supplied by the selected
base image. See [`oakapp.toml` configuration](https://docs.luxonis.com/software-v3/oak-apps/configuration.md) for the complete app
manifest reference.

## Prepare the ONNX Model

QNN has stricter graph requirements than the ONNX Runtime CPU Execution Provider. A model that works on the CPU is not
automatically eligible for complete HTP execution.

### Use static shapes

All input dimensions must have fixed positive values; dynamic batch, image, sequence, and other dimensions are not supported by
this path. After fixing the inputs, run ONNX shape inference and verify that the outputs are also static. Data-dependent output
shapes or operations can still prevent complete HTP placement.

For example, fix a model input to NCHW batch size 1 with the ONNX Runtime helper:

```bash
python -m onnxruntime.tools.make_dynamic_shape_fixed \
    --input_name images \
    --input_shape 1,3,288,512 \
    model.onnx model.fixed.onnx
```

If an application needs several shapes, create one fixed model for each shape. For variable-length inputs, a practical approach is
to define a small set of static buckets, pad each input to the nearest bucket, and cache a compiled QNN context for every bucket.

### Check operators and data types

QNN supports a subset of ONNX operators, and the supported data types and attributes vary by operator and backend. Control-flow
operators such as `Loop` and `If` are not supported. Check the ONNX Runtime [QNN supported operator
list](https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider.html#supported-onnx-operators) for the runtime version
used by your base image.

When a graph is not supported completely:

 * Re-export the model with a compatible ONNX opset and static axes.
 * Fold constants and simplify redundant graph operations.
 * Replace or decompose unsupported operators with supported equivalents.
 * Move incompatible preprocessing or postprocessing, such as data-dependent selection, out of the ONNX graph and into application
   code.
 * Preserve the model's input names, order, layout, data types, normalization, and output contract while rewriting it.

Always compare the prepared model against the original with representative inputs. Enabling FP16 execution or changing the graph
can introduce numerical differences.

### Validate without CPU fallback

The QNN helper defaults to CPU fallback so an application can continue when the HTP is unavailable or part of a graph is
unsupported. This can hide a deployment problem and make a session appear accelerated when some or all operations actually run on
the CPU.

During model qualification, set both strict options:

```python
session = onnx_qnn_session(
    "model.fixed.onnx",
    fallback_to_cpu=False,
    runtime_fallback="raise",
)
```

Restore CPU fallback only when degraded operation is acceptable for the application.

## Generic Example

The following example captures a camera frame with DepthAI, prepares a `1x3x288x512` float32 tensor, and invokes the ONNX model
through QNN. Its resize, RGB conversion, normalization, and output processing are placeholders; match them to the contract of your
model.

```python
import cv2
import depthai as dai
import numpy as np

from depthai_nodes.runtime import onnx_qnn_session

MODEL_PATH = "/app/model.fixed.onnx"
MODEL_SIZE = (512, 288)  # width, height

# The helper registers the QNN Execution Provider, selects the HTP backend,
# enables FP16 execution for FP32 graphs, and caches the compiled context.
session = onnx_qnn_session(
    MODEL_PATH,
    fp16=True,
    cache_context=True,
    performance_mode="burst",
    fallback_to_cpu=False,
    runtime_fallback="raise",
)
model_input = session.get_inputs()[0]

with dai.Pipeline() as pipeline:
    camera = pipeline.create(dai.node.Camera).build()
    frame_queue = camera.requestOutput(size=MODEL_SIZE).createOutputQueue()

    pipeline.start()

    while pipeline.isRunning():
        frame_bgr = frame_queue.get().getCvFrame()

        # Example preprocessing only. Follow your model's input contract.
        frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
        input_tensor = frame_rgb.astype(np.float32) / 255.0
        input_tensor = np.transpose(input_tensor, (2, 0, 1))[None, ...]

        outputs = session.run(None, {model_input.name: input_tensor})

        # Decode or consume outputs according to the model contract.
        print([output.shape for output in outputs])
```

The complete helper implementation and its configuration arguments are available in
[`onnx_qnn.py`](https://github.com/luxonis/depthai-nodes/blob/main/depthai_nodes/runtime/onnx_qnn.py).

### Compiled context cache

QNN graph compilation happens during the first session creation and is not part of inference latency. With `cache_context=True`,
`onnx_qnn_session` stores an EPContext model in a `.oak4ort_cache` directory next to the ONNX model when that location is
writable. Set `OAK4ORT_CACHE_DIR` to choose a different writable cache directory.

The cache is specific to the model file, ONNX Runtime and QNN package versions, and provider options. Changing any of them can
trigger a new compilation. Keep the cache on persistent storage if warm startup time matters across app replacement or container
recreation.

## Performance Comparison

> **First startup can be slow**
> Session creation was excluded from the reported inference latency. This can take up to few minutes in some cases. The compiled context cache lets subsequent session creations skip HTP graph compilation.

The table compares mean FPS for single-threaded inference on an OAK4-D with models from the [Model
Zoo](https://docs.luxonis.com/software-v3/ai-inference/model-source/zoo.md).

| Model | ONNX CPU | ONNX QNN FP16 | SNPE FP16 | SNPE INT8 |
| --- | --- | --- | --- | --- |
| `luxonis/deeplab-v3-plus:512x288` | 2.23 | 67.66 | 77.15 | 217.17 |
| `luxonis/depth-anything-v2:vit-s-336x252` | 2.36 | 72.68 | 72.61 | 143.55 |
| `luxonis/paddle-text-recognition:320x48` | 36.81 | 137.69 | 88.88 | 239.53 |
| `luxonis/yoloe-v8-l:640x640` | 0.37 | 25.61 | 27.67 | 78.86 |
| `luxonis/yolov6-nano:r2-coco-512x288` | 17.58 | 442.16 | 436.26 | 557.96 |
| `luxonis/yolov8-large-pose-estimation:coco-640x352` | 0.84 | 53.07 | 55.71 | 149.10 |

Across these models, QNN delivered approximately 3.7x to 69.4x the throughput of single-threaded ONNX Runtime CPU execution. QNN
FP16 is comparable to native SNPE FP16 while SNPE INT8 path is always the fastest one with potential drop in accuracy due to
quantization.

## Limitations

 * RVC4 standalone only: the required QNN libraries, HTP devices, and FastRPC transport are provided by the RVC4 device and app
   container environment.
 * Static graph shapes: every deployed shape needs a separately prepared model and may require a separate compiled context.
 * Partial ONNX support: unsupported operators, attributes, or data types require graph changes or CPU fallback.
 * Startup cost: the first session can spend substantial time compiling the graph; cache the context and design startup health
   checks accordingly.
 * Separate from the DepthAI graph: preprocessing, postprocessing, scheduling, and NumPy data movement are managed by the app
   instead of the built-in `NeuralNetwork` node.
 * Fallback can mask performance: CPU fallback improves compatibility but can make provider selection and latency less obvious.
   Validate in strict mode first.
 * Precision must be validated: `fp16=True` allows QNN to run an FP32 graph with FP16 precision on the HTP. Quantized QDQ models
   keep their quantized behavior. Test application-level accuracy for either option.

## Examples

The following OAK Examples use this deployment path:

 * [YOLO World](https://github.com/luxonis/oak-examples/tree/main/neural-networks/object-detection/yolo-world) fixes the text
   encoder's dynamic dimensions before creating a QNN session.
 * [Open-Vocabulary Object
   Detection](https://github.com/luxonis/oak-examples/tree/main/custom-frontend/open-vocabulary-object-detection) uses static
   input buckets for prompt encoders.
 * [Data Collection](https://github.com/luxonis/oak-examples/tree/main/apps/data-collection) accelerates the ONNX prompt-encoding
   workflow in standalone mode.
 * [Roboflow Workflow](https://github.com/luxonis/oak-examples/tree/main/integrations/roboflow-workflow) routes compatible ONNX
   Runtime sessions through QNN and exposes strict and CPU-only debugging modes.