OverviewHardwareSoftwareCloud
OAK Apps
DepthAI
AI
Software Stack
Troubleshooting

ON THIS PAGE

  • Overview
  • Usage
  • Main steps
  • Export compatibility changes
  • Export to ONNX
  • Create the NN Archive
  • Convert the NN Archive to RVC4
  • Small inference example

Roboflow RF-DETR

Supported on:RVC4

Overview

RF-DETR is Roboflow's transformer-based object detection and instance segmentation model family. On Luxonis devices, the current deployment workflow targets RVC4.This integration shows the main path from an official RF-DETR checkpoint to on-device inference:The linked tutorial is validated with RF-DETR 1.7.1 and RF-DETR Nano, but the same general approach can be used for other RF-DETR variants as long as the relevant modules are present.If you only need a ready-to-run model, RF-DETR Nano is already available in the Model Zoo: RF-DETR Nano model card and RF-DETR Nano Instance Segmentation model card. The steps below focus primarily on the detection model, but the same process can also be applied to the instance segmentation variant, as demonstrated in the linked notebook.

Usage

You can run the export, archive creation, and HubAI conversion steps in Colab. The final inference step must run on a machine that can access an RVC4 device.

Main steps

  1. Install the required packages and clone the official rf-detr repository at a known-good version.
  2. Apply a small runtime class injection before export so the generated ONNX graph is compatible with the RVC4 conversion flow, without modifying the upstream RF-DETR source tree.
  3. Export the model to ONNX with a fixed 384x384 input and opset 17.
  4. Create an NN Archive that describes the model inputs, outputs, preprocessing, and the RFDETRParser head used at runtime.
  5. Convert the archive to RVC4 with the HubAI SDK.
  6. Run the resulting .rvc4.tar.xz archive with a DepthAI inference pipeline.

Export compatibility changes

Before export, RF-DETR needs a small compatibility patch so the generated ONNX graph can be converted cleanly for RVC4.The tutorial applies these changes at runtime through class injection rather than by editing the upstream RF-DETR source:
  • the windowed DINOv2 backbone is adjusted where image windows are merged back into a full spatial feature map,
  • the MSDeformAttn module is adjusted where multi-scale deformable attention computes sampling locations and attention weights.
These changes do not retrain the model and do not modify the weights. They only express the same computation in a form that is more export- and conversion-friendly.

Export to ONNX

The export step starts from the official RF-DETR model, applies the runtime compatibility patch, and writes the ONNX model:
Python
1from rfdetr import RFDETRNano
2from rvc4_class_injection import apply_rvc4_class_injection
3
4model = RFDETRNano()
5apply_rvc4_class_injection(model, require=True, verbose=True)
6
7model.export(
8    output_dir="onnx_model",
9    shape=(384, 384),
10    batch_size=1,
11    dynamic_batch=False,
12    opset_version=17,
13    format="onnx",
14)

Create the NN Archive

The generated archive should describe the input tensor, the dets and labels outputs, and an RFDETRParser head so runtime parsing can be configured automatically:
Python
1from luxonis_ml.nn_archive.archive_generator import ArchiveGenerator
2
3generator = ArchiveGenerator(
4    archive_name="rfdetr-nano-onnx",
5    save_path=".",
6    cfg_dict=cfg_dict,
7    executables_paths=["onnx_model/rfdetr-nano.onnx"],
8)
9
10archive_path = generator.make_archive()
For a full example of the archive configuration, including preprocessing values and COCO class metadata, refer to the tutorial notebook.

Convert the NN Archive to RVC4

Once the ONNX NNArchive is ready, convert it with HubAI:
Python
1import os
2from hubai_sdk import HubAIClient
3
4client = HubAIClient(api_key=os.environ["HUBAI_API_KEY"])
5
6response = client.convert.RVC4(
7    path="rfdetr-nano-onnx.tar.xz",
8    name="rfdetr-nano-rvc4",
9    quantization_mode="FP16_STANDARD",
10    tool_version="2.41.0",
11    output_dir=".",
12)
13
14print(response.downloaded_path)
This produces the deployable RVC4 archive, typically named rfdetr-nano-onnx.rvc4.tar.xz.

Small inference example

Because the NN Archive already declares RFDETRParser, ParsingNeuralNetwork can wire the parser automatically:
Python
1from pathlib import Path
2
3import depthai as dai
4from depthai_nodes.node.parsing_neural_network import ParsingNeuralNetwork
5
6MODEL_PATH = Path("rfdetr-nano-onnx.rvc4.tar.xz")
7nn_archive = dai.NNArchive(str(MODEL_PATH))
8visualizer = dai.RemoteConnection(httpPort=8082)
9
10with dai.Pipeline() as pipeline:
11    cam = pipeline.create(dai.node.Camera).build(sensorFps=10)
12    cam_out = cam.requestOutput(
13        size=(384, 384),
14        fps=10,
15        type=dai.ImgFrame.Type.BGR888i,
16    )
17
18    nn_with_parser = pipeline.create(ParsingNeuralNetwork).build(
19        input=cam_out,
20        nn_source=nn_archive,
21    )
22
23    visualizer.addTopic(topicName="rgb", output=nn_with_parser.passthrough)
24    visualizer.addTopic(topicName="detections", output=nn_with_parser.out)
25
26    pipeline.start()
27    visualizer.registerPipeline(pipeline)
28
29    while pipeline.isRunning():
30        pipeline.processTasks()
For a complete script with device checks, model validation, and a ready-to-use local visualizer flow, use the tutorial linked above.