SpatialDetectionNetwork

Supported on:RVC2 RVC4

Spatial 检测节点类似于 DetectionNetwork 和 SpatialLocationCalculator 的组合。

如何放置它

Python

1modelDescription = dai.NNModelDescription("yolov6-nano")
2pipeline = dai.Pipeline()
3spatialDetectionNetwork = pipeline.create(dai.node.SpatialDetectionNetwork).build(camRgb, stereo, modelDescription)

C++

1dai::NNModelDescription modelDescription("yolov6-nano");
2dai::Pipeline pipeline;
3auto spatialDetectionNetwork = pipeline.create<dai::node::SpatialDetectionNetwork>()->build(camRgb, stereo, modelDescription);

输入和输出

配置空间检测

空间检测节点（SpatialDetectionNetwork）的管道在下面的模式中进行了描述：

空间检测节点本质上只是检测网络（DetectionNetwork）和空间位置计算器（SpatialLocationCalculator）的抽象。它的工作原理是将每个检测到的对象的边界框链接到空间位置计算器。该过程如下：

检测网络负责检测输入帧中的对象。它输出一个检测到的对象列表，每个对象都由一个边界框、标签和置信度分数表示。

深度图与输入帧对齐。这是必需的，因为检测网络（DetectionNetwork）在输入帧上运行，而空间位置计算器（SpatialLocationCalculator）在深度图上运行。

网络输出的边界框被发送到空间位置计算器，并根据 BoundingBoxScaleFactor 进行缩放。这样做是为了确保它包含整个对象。然后，边界框与深度一起用于计算对象的空间坐标。

X 和 Y 坐标取自边界框中心。它们是根据与帧中心偏移量以及该点的深度计算得出的。
对于深度（Z），将考虑缩放后的边界框（ROI）内的每个像素。这将得到一组深度值，然后对这些值进行平均以获得最终的深度值。

平均方法

平均值/均值：ROI 的平均值用于计算。
最小值：ROI 内的最小值用于计算。
最大值：ROI 内的最大值用于计算。
众数：ROI 内最常见的值用于计算。
中位数：ROI 内的中位数用于计算。

默认方法是中位数。

常见错误

大多数错误源于不正确的边界框重叠。缩放后的边界框可能包含背景部分，这会影响深度计算的准确性。

细长对象（如电线杆）的空间坐标可能不准确，因为边界框只有一小部分实际位于检测到的对象上。在这种情况下，最好尽可能使用较小的 BoundingBoxScaleFactor。
带孔的对象 - 呼啦圈、戒指等。要获得正确的深度，边界框应包含整个对象。与其使用中位数深度，不如使用最小值深度方法来排除计算中的背景。或者，可以设置深度阈值以忽略静态环境中的背景。

用法

Python

1with dai.Pipeline() as p:
2    camRgb = p.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_A)
3    monoLeft = p.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_B)
4    monoRight = p.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_C)
5    stereo = p.create(dai.node.StereoDepth)
6    spatialDetectionNetwork = p.create(dai.node.SpatialDetectionNetwork).build(camRgb, stereo, modelDescription, fps=FPS)
7
8    spatialDetectionNetwork.input.setBlocking(False)
9    spatialDetectionNetwork.setBoundingBoxScaleFactor(0.5)
10    spatialDetectionNetwork.setDepthLowerThreshold(100)
11    spatialDetectionNetwork.setDepthUpperThreshold(5000)
12
13    labelMap = spatialDetectionNetwork.getClasses()

C++

1dai::Pipeline pipeline;
2auto camRgb = pipeline.create<dai::node::Camera>()->build(dai::CameraBoardSocket::CAM_A);
3auto monoLeft = pipeline.create<dai::node::Camera>()->build(dai::CameraBoardSocket::CAM_B);
4auto monoRight = pipeline.create<dai::node::Camera>()->build(dai::CameraBoardSocket::CAM_C);
5auto stereo = pipeline.create<dai::node::StereoDepth>();
6auto spatialDetectionNetwork = pipeline.create<dai::node::SpatialDetectionNetwork>()->build(camRgb, stereo, modelDescription, FPS);
7spatialDetectionNetwork->input.setBlocking(false);
8spatialDetectionNetwork->setBoundingBoxScaleFactor(0.5);
9spatialDetectionNetwork->setDepthLowerThreshold(100);
10spatialDetectionNetwork->setDepthUpperThreshold(5000);
11auto labelMap = spatialDetectionNetwork->getClasses();

功能示例

Spatial Detection Network

空间坐标系

OAK 相机使用 RDF（右-下-前）坐标系来表示所有空间坐标。

在 X、Y 坐标系中，帧的中心为 0,0。向下移动时 Y 坐标会增加，向右移动时 X 坐标会增加。

参考

class

dai::node::SpatialDetectionNetwork

#include SpatialDetectionNetwork.hpp

variable

Properties & properties

variable

Subnode< NeuralNetwork > neuralNetwork

variable

Subnode< DetectionParser > detectionParser

variable

Subnode< SpatialLocationCalculator > spatialLocationCalculator

variable

std::unique_ptr< Subnode< ImageAlign > > depthAlign

variable

Input & input

Input message with data to be inferred upon Default queue is blocking with size 5

variable

Output & outNetwork

Outputs unparsed inference results.

variable

Output & passthrough

Passthrough message on which the inference was performed.Suitable for when input queue is set to non-blocking behavior.

variable

Input & inputDepth

Input message with depth data used to retrieve spatial information about detected object Default queue is non-blocking with size 4

variable

Output & out

Outputs ImgDetections message that carries parsed detection results.

variable

Output & passthroughDepth

Passthrough message for depth frame on which the spatial location calculation was performed. Suitable for when input queue is set to non-blocking behavior.

explicit function

SpatialDetectionNetwork(const std::shared_ptr< Device > & device)

explicit function

SpatialDetectionNetwork(std::unique_ptr< Properties > props)

function

SpatialDetectionNetwork(std::unique_ptr< Properties > props, bool confMode)

function

SpatialDetectionNetwork(const std::shared_ptr< Device > & device, std::unique_ptr< Properties > props, bool confMode)

function

std::shared_ptr< SpatialDetectionNetwork > build(const std::shared_ptr< Camera > & inputRgb, const DepthSource & depthSource, const Model & model, std::optional< float > fps, std::optional< dai::ImgResizeMode > resizeMode)

function

std::shared_ptr< SpatialDetectionNetwork > build(const std::shared_ptr< Camera > & inputRgb, const DepthSource & depthSource, const Model & model, const ImgFrameCapability & capability)

function

void setNNArchive(const NNArchive & nnArchive)

function

void setFromModelZoo(NNModelDescription description, bool useCached)

function

void setNNArchive(const NNArchive & nnArchive, int numShaves)

function

void setBlobPath(const std::filesystem::path & path)

Backwards compatibility interface Load network blob into assets and use once pipeline is started.

Parameters

Error: if file doesn't exist or isn't a valid network blob.

Parameters

path: Path to network blob

function

void setBlob(OpenVINO::Blob blob)

Load network blob into assets and use once pipeline is started.

Parameters

blob: Network blob

function

void setBlob(const std::filesystem::path & path)

Same functionality as the setBlobPath(). Load network blob into assets and use once pipeline is started.

Parameters

Error: if file doesn't exist or isn't a valid network blob.

Parameters

path: Path to network blob

function

void setModelPath(const std::filesystem::path & modelPath)

Load network file into assets.

Parameters

modelPath: Path to the model file.

function

void setNumPoolFrames(int numFrames)

Specifies how many frames will be available in the pool

Parameters

numFrames: How many frames will pool have

function

void setNumInferenceThreads(int numThreads)

How many threads should the node use to run the network.

Parameters

numThreads: Number of threads to dedicate to this node

function

void setNumNCEPerInferenceThread(int numNCEPerThread)

How many Neural Compute Engines should a single thread use for inference

Parameters

numNCEPerThread: Number of NCE per thread

function

void setNumShavesPerInferenceThread(int numShavesPerThread)

How many Shaves should a single thread use for inference

Parameters

numShavesPerThread: Number of shaves per thread

function

void setBackend(std::string backend)

Specifies backend to use

Parameters

backend: String specifying backend to use

function

void setBackendProperties(std::map< std::string, std::string > properties)

Set backend properties

Parameters

backendProperties: backend properties map

function

int getNumInferenceThreads()

How many inference threads will be used to run the network

Returns

Number of threads, 0, 1 or 2. Zero means AUTO

function

void setConfidenceThreshold(float thresh)

Specifies confidence threshold at which to filter the rest of the detections.

Parameters

thresh: Detection confidence must be greater than specified threshold to be added to the list

function

float getConfidenceThreshold()

Retrieves threshold at which to filter the rest of the detections.

Returns

Detection confidence

function

void setBoundingBoxScaleFactor(float scaleFactor)

Custom interface Specifies scale factor for detected bounding boxes.

Parameters

scaleFactor: Scale factor must be in the interval (0,1].

function

void setDepthLowerThreshold(uint32_t lowerThreshold)

Specifies lower threshold in depth units (millimeter by default) for depth values which will used to calculate spatial data

Parameters

lowerThreshold: LowerThreshold must be in the interval [0,upperThreshold] and less than upperThreshold.

function

void setDepthUpperThreshold(uint32_t upperThreshold)

Specifies upper threshold in depth units (millimeter by default) for depth values which will used to calculate spatial data

Parameters

upperThreshold: UpperThreshold must be in the interval (lowerThreshold,65535].

function

void setSpatialCalculationAlgorithm(dai::SpatialLocationCalculatorAlgorithm calculationAlgorithm)

Specifies spatial location calculator algorithm: Average/Min/Max

Parameters

calculationAlgorithm: Calculation algorithm.

function

void setSpatialCalculationStepSize(int stepSize)

Specifies spatial location calculator step size for depth calculation. Step size 1 means that every pixel is taken into calculation, size 2 means every second etc.

Parameters

stepSize: Step size.

function

std::optional< std::vector< std::string > > getClasses()

function

void buildInternal()

enum