Robotics Vision Core 4 (RVC4)
Robotics Vision Core 4 (RVC4 in short) is the fourth generation of our RVC. Main specs:- Octa-core ARM CPU running Linux (Kernel 5.15)
- AI: 48 INT8, 12 FP16 TOPS
- Computer vision: Stereo depth, frame warp engine, optical flow, feature detection, description matching, template matching
- ISP: 5 camera streams, HDR, EIS, 3A, up to 3x 8K @ 30FPS
- Encoding: 4K @ 240FPS decoding, 4K @ 120FPS encoding for H264 and H265. Decoding also supported for VP9, AV1
Timeline
RVC4 is currently in the development phase. Our current plan is to release RVC4-based devices in October 2024.Computer vision engine
- Stereo depth: Max 720P @ 60FPS, 4bit subpixel by default, 64 disparity search. 8-bit confidence map, spatial consistency, occlusion and texture masking, ±3 pixel lines rectification error tolerance
- Image warp engine: Throughput of 1080P @ 240FPS
- Optical flow: Semi-dense: 1080P @ 60FPS, Full dense: VGA @ 60 FPS
- Feature detection: Harris corner detection, 1080P @ 60FPS
- Description matching: ORB calculation and inline matching. Descriptor: 256 bits. Max 1080P, 1ms per 500 descriptors (calculation + matching)
- Template matching: Max 1080P, 1.2ms for 500 templates, max 1024 patches per frame
Image Signal Processor
RVC4's Image Signal Processor (ISP) has the following features:- 5 concurrent camera streams
- High throughput: Up to 3x 8K @ 30FPS, or 1x 108MP @ 30FPS
- 3A (Auto Exposure, Auto White Balance, Auto Focus)
- Supports 18 bpp (bits-per-pixel)
- Low-power camera mode (dedicated low-power island), up to 10FPS at VGA resolution + dedicated low-power NPU (Neural Processing Unit) for image processing
- Hardware HDR: Staggered HDR, digital overlap, non-overlap
- Image stabilization (EIS), good low-light performance
AI
RVC4 NN model benchmarks (peak performance):Model name | Size | FPS | Task |
---|---|---|---|
YoloV5m | 640x640 | 280 | Object detection |
YoloV6n | 512x288 | 2340 | Object detection |
YoloV7-W6 | 640x640 | 162 | Object detection |
ResNet-50 | 224x224 | 934 | Classification |
ViT-Tiny | 224x224 | 650 | Classification |
BiSeNetv1-MBNV2 | 512x228 | 647 | Semantic segmentation |
eWaSR | 512x384 | 309 | Semantic segmentation |
AI Power Consumption
RVC4's AI system is designed to be power-efficient and configurable to the user's needs. The AI system can be configured to run at different power levels (FPS speeds), which will affect the performance of the AI system. The following table shows model FPS and [power consumption] at different FPS speeds:Model name | Low FPS | Medium FPS | High FPS | Max FPS |
---|---|---|---|---|
BiSeNet-MBNV2 (512x288) | 80 [0.67 W] | 133 [1.04 W] | 391 [3.02 W] | 647 [5 W] |
eWaSR ResNet18 (512x384) | 59 [0.79 W] | 106 [1.51 W] | 229 [3.55 W] | 309 [5.25 W] |
MobileVit-xxs (224x224) | 104 [0.63 W] | 199 [1 W] | 277 [1.82 W] | 488 [3.27 W] |
Repvgg_a2 (224x224) | 181 [1.07 W] | 327 [2.22 W] | 466 [3.75 W] | 1250 [10.4 W] |
ResNet101 (224x224) | 115 [1.05 W] | 243 [2.4 W] | 339 [3.57 W] | 718 [8.62 W] |
ResNet50-v2-7 (224x224) | 145 [0.96 W] | 260 [1.9 W] | 380 [2.83 W] | 934 [7.65 W] |
ViT-Tiny patch16 (224x224) | 124 [0.7 W] | 228 [1.25 W] | 300 [1.88 W] | 615 [4.27 W] |
Yolo6N (512x288) | 190 [0.7 W] | 307 [1.1 W] | 702 [3.8 W] | 2340 [7.5 W] |
YoloV5M (640x640) | 48 [0.93 W] | 72 [1.75 W] | 212 [6.05 W] | 280 [8.4 W] |
YoloV7-W6 (640x640) | 34 [1.05 W] | 60 [2.48 W] | 139 [7.45 W] | 162 [7.85 W] |
Please note that the max FPS values are recorded at peak performance of the RVC4. In most use-cases, the RVC4 will be throttled down to avoid overheating.
Jetson comparison
Nvidia's Jetson series is currently the de-facto edge AI platform. We tested the Jetson Orin Nano 8GB (MSRP: $499), which has 40 TOPS (GPU) and 6-core ARM CPU. Below is a 1:1 comparison of RVC4 to Jetson Orin Nano 8GB, both using INT8 precision and the same image shape:Model name | RVC4 [FPS] | Jetson Orin Nano 8GB [FPS] |
---|---|---|
InceptionV4, BS1 | 691 | 170 |
InceptionV4, BS32 | 608 | 358 |
ResNet50, BS1 | 1369 | 502 |
ResNet50, BS32 | 1644 | 1191 |
VGG19, BS1 | 269 | 183 |
VGG19, BS32 | 560 | 362 |
Super Resolution, BS1 | 36* | 202 |
SSD MobileNet V1, BS1 | 1910 | 920 |
SSD MobileNet V1, BS32 | 2688 | 2260 |
UNet Segmentation, BS1 | 323 | 142 |
YoloV3 Tiny, BS1 | 1342 | 563 |
Power efficiency
We also measured the power usage of both RVC4 and the Jetson Orin Nano 8GB. We ran both devices at max performance (so MAX FPS for RVC4), and measured the power usage of the whole board. Power usage of Orin Nano fluctuates a lot, so we took the average of the power usage over 10 seconds. We always took a model with Batch Size=1 (so a single image).Model name | RVC4 [W] | Jetson Orin Nano 8GB [W] |
---|---|---|
InceptionV4 | 9.1 | 12 |
ResNet50 | 10.2 | 13 |
VGG19 | 9.5 | 11 |
YoloV3 Tiny | 9.9 | 10 |
YoloV5 M (416x416) | 9.3 | 11 |
Custom applications
Users will have full access to the power of the RVC4:- Easy development & deployment of custom containerized apps will be possible out-of-the-box via Luxonis Hub
- Develop and run fast CV pipelines on top of accelerated hardware blocks using Halide
- Interface with GPIOs and communication interfaces
- As the RVC4 can also optionally act as a host computer, it will be able to connect other RVC2-based OAK (PoE) cameras to it.
OAK4 cameras
From the hardware perspective, OAK4 cameras will have:- Both PoE (M12 connector) and USB3 (with screw holes) connectivity, so user can choose which one to use
- M8 auxiliary connector, just like the OAK PoE cameras
- Microphones
- Status indication LED
- IP67-rated enclosure
- 48MP color sensor by default (IMX586)
- OAK4 S (S as in Single/Small), so similar to OAK-1 (PoE), but with RVC4 inside.
- OAK4 D and all its variants (FoV, sensor types, active/passive stereo)
- OAK4 SR Short Range, with a ToF sensor
- OAK4 LR Long Range
OAK4 S
![](/static/images/hardware/oak4/oak4-s.webp)
![](/static/images/hardware/oak4/oak1poe-vs-oak4s.jpeg)
Since the design of the OAK4 S is quite modular (2 PCBAs + SOM, and 3-part enclosure), we will be able to reuse most of the design to also quickly develop RVC4-based OAK Thermal. More details about those two models will be available in the future.
![](/static/images/hardware/oak4/oak4-s-schematics.webp)
OAK4 D
![](/static/images/hardware/oak4/oak4-d.webp)
This device will be quite similar to the OAK-D S2 PoE (and all its variations), but with RVC4 inside, and additional hardware features mentioned above. We plan to release similar variations as for the Series 2 OAK PoE cameras, so normal/wide FOV, different sensor types and passive/active stereo (Pro version).
![](/static/images/hardware/oak4/oak4-d-schematics.webp)
OAK4 SR
![](/static/images/hardware/oak4/oak4-d-sr.webp)
This is the RVC4-based version of the OAK-D SR PoE. It will have a ToF sensor, and will be IP67-rated.
![](/static/images/hardware/oak4/oak4-d-sr-schematics.webp)
OAK4 LR
![](/static/images/hardware/oak4/oak4-d-lr.webp)
Will be very similar to the OAK-D LR, but with RVC4 inside, and an M12 connector (+ M8 aux connector) instead of the RJ45 for the ethernet (PoE) connection.
![](/static/images/hardware/oak4/oak4-d-lr-schematics.webp)