ON THIS PAGE

  • Robotics Vision Core 4 (RVC4)
  • Timeline
  • Computer vision engine
  • Image Signal Processor
  • AI
  • AI Power Consumption
  • Jetson comparison
  • Power efficiency
  • Custom applications
  • RVC4-based devices
  • OAK4-S
  • OAK4-D
  • OAK4-D SR
  • OAK4-D LR

Robotics Vision Core 4 (RVC4)

Robotics Vision Core 4 (RVC4 in short) is the fourth generation of our RVC. Main specs:
  • Octa-core ARM CPU running Linux (Kernel 5.15)
  • AI: 48 INT8, 12 FP16 TOPS
  • Computer vision: Stereo depth, frame warp engine, optical flow, feature detection, description matching, template matching
  • ISP: 5 camera streams, HDR, EIS, 3A, up to 3x 8K @ 30FPS
  • Encoding: 4K @ 240FPS decoding, 4K @ 120FPS encoding for H264 and H265. Decoding also supported for VP9, AV1
To stay up to date, consider subscribing to the RVC4 newsletter.

Timeline

RVC4 is currently in the development phase. Our current plan is to release RVC4-based devices in October 2024.

Computer vision engine

  • Stereo depth: Max 720P @ 60FPS, 4bit subpixel by default, 64 disparity search. 8-bit confidence map, spatial consistency, occlusion and texture masking, ±3 pixel lines rectification error tolerance
  • Image warp engine: Throughput of 1080P @ 240FPS
  • Optical flow: Semi-dense: 1080P @ 60FPS, Full dense: VGA @ 60 FPS
  • Feature detection: Harris corner detection, 1080P @ 60FPS
  • Description matching: ORB calculation and inline matching. Descriptor: 256 bits. Max 1080P, 1ms per 500 descriptors (calculation + matching)
  • Template matching: Max 1080P, 1.2ms for 500 templates, max 1024 patches per frame

Image Signal Processor

RVC4's Image Signal Processor (ISP) has the following features:
  • 5 concurrent camera streams
  • High throughput: Up to 3x 8K @ 30FPS, or 1x 108MP @ 30FPS
  • 3A (Auto Exposure, Auto White Balance, Auto Focus)
  • Supports 18 bpp (bits-per-pixel)
  • Low-power camera mode (dedicated low-power island), up to 10FPS at VGA resolution + dedicated low-power NPU (Neural Processing Unit) for image processing
  • Hardware HDR: Staggered HDR, digital overlap, non-overlap
  • Image stabilization (EIS), good low-light performance

AI

RVC4 NN model benchmarks:
Model nameSizeFPSTask
YoloV5m640x640280Object detection
YoloV6n512x2882340Object detection
YoloV7-W6640x640162Object detection
ResNet-50224x224934Classification
ViT-Tiny224x224650Classification
BiSeNetv1-MBNV2512x228647Semantic segmentation
eWaSR512x384309Semantic segmentation

AI Power Consumption

RVC4's AI system is designed to be power-efficient and configurable to the user's needs. The AI system can be configured to run at different power levels (FPS speeds), which will affect the performance of the AI system. The following table shows model FPS and [power consumption] at different FPS speeds:
Model nameLow FPSMedium FPSHigh FPSMax FPS
BiSeNet-MBNV2 (512x288)80 [0.67 W]133 [1.04 W]391 [3.02 W]647 [5 W]
eWaSR ResNet18 (512x384)59 [0.79 W]106 [1.51 W]229 [3.55 W]309 [5.25 W]
MobileVit-xxs (224x224)104 [0.63 W]199 [1 W]277 [1.82 W]488 [3.27 W]
Repvgg_a2 (224x224)181 [1.07 W]327 [2.22 W]466 [3.75 W]1250 [10.4 W]
ResNet101 (224x224)115 [1.05 W]243 [2.4 W]339 [3.57 W]718 [8.62 W]
ResNet50-v2-7 (224x224)145 [0.96 W]260 [1.9 W]380 [2.83 W]934 [7.65 W]
ViT-Tiny patch16 (224x224)124 [0.7 W]228 [1.25 W]300 [1.88 W]615 [4.27 W]
Yolo6N (512x288)190 [0.7 W]307 [1.1 W]702 [3.8 W]2340 [7.5 W]
YoloV5M (640x640)48 [0.93 W]72 [1.75 W]212 [6.05 W]280 [8.4 W]
YoloV7-W6 (640x640)34 [1.05 W]60 [2.48 W]139 [7.45 W]162 [7.85 W]
Power measurements were taken of the whole RVC4 board during 10 second inference runs. So the AI power consumption is a bit less, as the rest of the chip (mainly CPU) is also consuming power.

Jetson comparison

Nvidia's Jetson series is currently the de-facto edge AI platform. We tested the Jetson Orin Nano 8GB (MSRP: $499), which has 40 TOPS (GPU) and 6-core ARM CPU. Below is a 1:1 comparison of RVC4 to Jetson Orin Nano 8GB, both using INT8 precision and the same image shape:
Model nameRVC4 [FPS]Jetson Orin Nano 8GB [FPS]
InceptionV4, BS1691170
InceptionV4, BS32608358
ResNet50, BS11369502
ResNet50, BS3216441191
VGG19, BS1269183
VGG19, BS32560362
Super Resolution, BS136*202
SSD MobileNet V1, BS11910920
SSD MobileNet V1, BS3226882260
UNet Segmentation, BS1323142
YoloV3 Tiny, BS11342563
Conclusion: From results above, the RVC4 provides 1.9x better performance compared to the Jetson Orin Nano 8GB.Looking at Jetson Family Benchmarks (at the bottom), Nvidia reports FPS for BS32 models (Batch Size 32, so 32 images getting inferenced all at once). From our own tests, these numbers are realistic, however, their BS1 models (Batch Size 1, so single image) perform ~2x worse than BS32 ones. If you want real-time performance (not +1 sec latency), you will need to use BS1 models.Looking only at BS1 model comparison, on average, RVC4 provides 2.17x better performance.* As the SoC is brand new, the model optimizer is still being updated, and additional layers will be added to get inferenced on accelerated blocks in the future. For Super resolution model, a few layers got inferenced on the CPU, that's why RVC4 performance was low.

Power efficiency

We also measured the power usage of both RVC4 and the Jetson Orin Nano 8GB. We ran both devices at max performance (so MAX FPS for RVC4), and measured the power usage of the whole board. Power usage of Orin Nano fluctuates a lot, so we took the average of the power usage over 10 seconds. We always took a model with Batch Size=1 (so a single image).
Model nameRVC4 [W]Jetson Orin Nano 8GB [W]
InceptionV49.112
ResNet5010.213
VGG199.511
YoloV3 Tiny9.910
YoloV5 M (416x416)9.311
Conclusion: While being 90% faster, RVC4 is also 15% more power efficient compared to the Jetson Orin Nano 8GB

Custom applications

Users will have full access to the power of the RVC4:
  • Easy development & deployment of custom containerized apps will be possible out-of-the-box via Luxonis Hub
  • Develop and run fast CV pipelines on top of accelerated hardware blocks using Halide
  • Interface with GPIOs and communication interfaces
  • As the RVC4 can also optionally act as a host computer, it will be able to connect other RVC2-based OAK (PoE) cameras to it.

RVC4-based devices

Also called OAK4, are planned to be released in June 2024. From the hardware perspective, OAK4 cameras will have:
  • Both PoE (M12 connector) and USB3 (with screw holes) connectivity, so user can choose which one to use
  • M8 auxiliary connector, just like the OAK PoE cameras
  • Microphones
  • Status indication LED
  • IP67-rated enclosure
We plan to release the following OAK4 devices:
  • OAK4-S (S as in Single/Small), so similar to OAK-1 (PoE), but with RVC4 inside.
  • OAK4-D and all its variants (FoV, sensor types, active/passive stereo)
  • OAK4-D SR Short Range, with a ToF sensor
  • OAK4-D LR Long Range
All devices will come with 128GB storage and 8GB RAM by default. We can also offer 256GB storage and 12GB RAM as an option (in the future perhaps up to 1TB storage / 16GB RAM).

OAK4-S

We got initial prototypes of OAK4-S in January '24, and are working on OS and FW support for it.It is much smaller than the OAK-1 PoE, and will have a wide variety of sensor options. It will also have an IMU, and (optionally) an IR illumination LED for night vision capabilities.
Since the design of the OAK4-S is quite modular (2 PCBAs + SOM, and 3-part enclosure), we will be able to reuse most of the design to also quickly develop RVC4-based OAK Thermal. More details about those two models will be available in the future.

OAK4-D

This device will be quite similar to the OAK-D S2 PoE (and all its variations), but with RVC4 inside, and additional hardware features mentioned above. We plan to release similar variations as for the Series 2 OAK PoE cameras, so normal/wide FOV, different sensor types and passive/active stereo (Pro version).

OAK4-D SR

This is the RVC4-based version of the OAK-D SR PoE. It will have a ToF sensor, and will be IP67-rated.

OAK4-D LR

Will be very similar to the OAK-D LR, but with RVC4 inside, and an M12 connector (+ M8 aux connector) instead of the RJ45 for the ethernet (PoE) connection.