In this tutorial we will look at how you can use the full FOV of the image sensor.
When you are using
preview output from ColorCamera, DepthAI will crop the
frames by default to get the desired aspect ratio.
preview stream derives from
video stream, which is cropped (16:9 aspect ratio,
max 4k resolution) from the
isp stream, which has the full FOV.
The image above is the
isp output from the ColorCamera (12MP from IMX378). The blue rectangle represents the cropped 4K
video output, and the yellow rectangle represents a cropped
preview output when the preview size is set to 1:1 aspect ratio
(eg. when using 300x300 MobileNet-SSD NN model).
In other words, you need to use ISP output from the ColorCamera to maximize the image FOV. A challenge occures when your NN model expects different aspect ratio (eg. 1:1) compared to isp output (eg. 4:3). Let’s say we have a MobileNet-SSD which requires 300x300 frames (1:1 aspect ratio) - we have a few options:
Stretch the ISP frame to to the 1:1 aspect ratio of the NN
Apply letterboxing to the ISP frame to get 1:1 aspect ratio frame
Crop the ISP frame to 1:1 aspect ratio and lose some FOV
Change aspect ratio¶
Pros: Preserves full FOV. Cons: Due to stretched frames, NNs accuracy might decrease.
Changing aspect ratio (stretching) can be used Use
camRgb.setPreviewKeepAspectRatio(False). This means the aspect
ratio will not be preserved and the image will be “stretched”. This might be problematic for some off-the-shelf NN models, so some fine-tuning might be required.
Usage example here.
Pros: Preserves full FOV. Cons: Smaller “frame” means less features might decrease NN accuracy.
Letterboxing approach will apply “black bars” above and below
the image to the full FOV (isp) frames, so the aspect ratio will be preserved. You can
achieve this by using ImageManip with
manip.setResizeThumbnail(x,y) (for Mobilenet
The downside of using this method is that your actual image will be smaller, so some features might not be preserved,
which can mean the NN accuracy could decrease.
Usage example here.
Pros: No NN accuracy decrease. Cons: Frame is cropped, so it’s not full FOV.
Cropping the full FOV (isp) frames to match the NN aspect ratio can be used to get the best NN accuracy, but this decreases FOV. Usage example here.
We’re always happy to help with code or other questions you might have.