HomeCloudSoftwareHardware
DepthAI Components
DepthAI Tutorials
DepthAI Perception
Tools
ROS
Neural Networks
DepthAI API References

ON THIS PAGE

  • Pipeline Debugging
  • Debugging with API
  • Debugging with environmental variable
  • CPU usage
  • RAM usage
  • Decreasing RAM consumption

Pipeline Debugging

You can enable logging by changing the debugging level. It's set to warning by default, but more verbose levels can be set to help debug issues. The following levels are available:
Debug LevelInformation
criticalOnly a critical error that stops/crashes the program.
errorErrors will not stop the program, but won't complete the action. Examples: When ImageManip cropping ROI was out of bounds, error will get printed and the cropping won't take place. When NeuralNetwork gets a frame whose shape (width/heigth/channel) isn't that of the .blob.
warnWarnings are printed in cases where user action could improve certain behavior/fix it. Example: When API changes, the old API style will be deprecated and warning will be shown to the user.
infoWill print information about CPU/RAM consumption, temperature, CMX slices and SHAVE core allocation.
debugUseful especially on starting and stopping the pipeline. Debug will print: Information about device initialization eg. Pipeline JSON, firmware/bootloader/OpenVINO version. How device/XLink is being closed/disposed.
traceTrace will print out a Message whenever one is received from the device.
Debugging can be enabled either inside the code (via API), or via environmental variables.

Debugging with API

Python
1with dai.Device() as device: # Initialize device
2    # Set debugging level
3    device.setLogLevel(dai.LogLevel.DEBUG)
4    device.setLogOutputLevel(dai.LogLevel.DEBUG)
Where setLogLevel sets verbosity which filters messages that get sent from the device to the host and setLogOutputLevel sets verbosity which filters messages that get printed on the host (stdout). This difference allows us to capture the log messages internally and not print them to stdout, and use those to eg. display them somewhere else or analyze them.

Debugging with environmental variable DEPTHAI_LEVEL

Using an environment variable to set the debugging level, rather than configuring it directly in code, provides additional detailed information. This includes metrics such as CMX and SHAVE usage, and the time taken by each node in the pipeline to process a single frame.Example of a log message for RGB Preview in INFO mode:
Command Line
1[184430102189660F00] [2.1] [0.675] [system] [info] DepthCamera allocated resources: shaves: [0-12] no cmx slices.
2[184430102189660F00] [2.1] [0.675] [system] [info] SIPP (Signal Image Processing Pipeline) internal buffer size '18432'B, DMA buffer size: '16384'B
3[184430102189660F00] [2.1] [0.711] [system] [info] ImageManip internal buffer size '285440'B, shave buffer size '34816'B
4[184430102189660F00] [2.1] [0.711] [system] [info] ColorCamera allocated resources: no shaves; cmx slices: [13-15]
5ImageManip allocated resources: shaves: [15-15] no cmx slices.
Example of a log message for Depth Preview in TRACE mode:
Command Line
1[19443010513F4D1300] [0.1.2] [2.014] [MonoCamera(0)] [trace] Mono ISP took '0.866377' ms.
2[19443010513F4D1300] [0.1.2] [2.016] [MonoCamera(1)] [trace] Mono ISP took '1.272838' ms.
3[19443010513F4D1300] [0.1.2] [2.019] [StereoDepth(2)] [trace] Stereo rectification took '2.661958' ms.
4[19443010513F4D1300] [0.1.2] [2.027] [StereoDepth(2)] [trace] Stereo took '7.144515' ms.
5[19443010513F4D1300] [0.1.2] [2.028] [StereoDepth(2)] [trace] 'Median' pipeline took '0.772257' ms.
6[19443010513F4D1300] [0.1.2] [2.028] [StereoDepth(2)] [trace] Stereo post processing (total) took '0.810216' ms.
7[2024-05-16 14:27:51.294] [depthai] [trace] Received message from device (disparity) - parsing time: 11µs, data size: 256000
Linux/MacOS
Windows PowerShell
Windows CMD

Linux/MacOS

Command Line
1DEPTHAI_LEVEL=debug python3 script.py

CPU usage

When setting the Debug Level to debug (or lower), depthai will also print our CPU usage for LeonOS and LeonRT. CPU usage at 100% (or close to it) can cause many undesirable effects, such as higher frame latency, lower FPS, and in some cases even firmware crash.Compared to OAK USB cameras, OAK PoE cameras will have increased CPU consumption, as the networking stack is running on the LeonOS core. The easiest way to reduce CPU consumtpion is to reduce the pipeline complexity, or to reduce FPS of the cameras, as they are the main consumers of CPU (running 3A algorithms).Not having 100% CPU usage also drastically decreased frame latency, in the example for the script below it went from ~710 ms to ~110ms:

RAM usage

All RVC2-based OAK devices have 512 MiB (4 Gbit) on-board RAM, which is used for firmware (about 15MB), assets (a few KB up to 100MB, eg. NN models), and other resources, such as message pools where messages are stored.If you enable info (see Pipeline Debugging section), you will see how RAM is used:
Command Line
1[info] Memory Usage - DDR: 41.23 / 358.82 MiB, CMX: 2.17 / 2.50 MiB,
2LeonOS Heap: 20.70 / 78.63 MiB, LeonRT Heap: 3.51 / 23.84 MiB
As you can see, RAM is split between the two LEON (CPU) cores, CMX (used for image manipulation), and DDR (everything else). If DDR usage is close to the max (in this example, 358 MiB), you might get an error such as:
Command Line
1[error] Neural network executor '0' out of '2' error: OUT_OF_MEMORY
This means you should decrease RAM consumption, and we will take a look at a few ways on how to do this.

Decreasing RAM consumption

  • Pool sizes - some nodes (including ColorCamera, PointCloud, ImageManip, EdgeDetector, MonoCamera, StereoDepth, VideoEncoder, Warp) have configurable pool sizes (Pool docs here). This means that the user can configure how many messages will be stored in the pool (RAM). If you are hitting RAM usage limits, the easiest way to reduce it is by decreasing the pool size. API for configuring pool size is node.setNumFramesPool(num_frames_pool). For ColorCamera, user needs to specify 5 pool sizes colorCam.setNumFramesPool(raw, isp, preview, video, still). Note that decreasing pool size below 2 will cause pipeline issues.
  • Large frames If we change the resolution from 1080P to 4K in the RGB video example, DDR usage will increase from 41 MiB to 161 MiB. That's because 4K uses 4x more RAM compared to 1080P frame. An easy way to decrease RAM consumption is to use lower resolution / smaller frames.
  • VideoEncoder VideoEncoder nodes can consume a lot of RAM, especially at high resolutions. For example, RGB Encoding example consumes 259 MiB. If we change the resolution from 4K to 1080P, we decrease DDR consumption to only 65 MiB.
  • ImageManip Each ImageManip node will have its own (output) pool of 4 frames (by default), so having multiple ImageManips that are manipulating high resolution frames will consume a lot of DDR RAM. By default, each pool "spot" will consume 1 MiB, even if it's a small 300x300 RGB frame (which is 270kB). Specifying the output frame size can therefore decrease the RAM as well, eg. for a 300x300 RGB frame, you can set manip.setMaxOutputFrameSize(270000).
  • XLinkIn Just like ImageManip, each XLinkIn node has its own message pool as well. By default, each XLinkIn will consume 40 MiB, as each pool "spot" has 5 MiB reserved, and there are 8 "spots" in the pool. If you are sending 300x300 RGB frames from the host to the device, you can set xin.setMaxDataSize(270000), and also limit number of messages per pool xin.setNumFrames(4). This will decrease DDR RAM consumption from 40 MiB to about 1 MiB.
If you are just sending control/config from the host, you can set xin.setMaxDataSize(1), as CameraControl and ImageManipConfig only hold metadata (without any data, like NNData / ImgFrame/ Buffer).