Lost Frames when using NVIDIA® Jetson™ Boards

Symptoms

Lost frames are observed when using a camera together with an NVIDIA® Jetson™ Board running Linux. Jetson Orin™ boards are particularly affected.

Cause

Jetson™ Boards use a number of governors to throttle various clocks when hardware devices, CPUs, GPUs or memory are not being used. In particular, the EMC (memory) clock setting affects DMA transfers from camera devices into memory. In the case of NVIDIA Jetson™ Orin Boards another potential cause of problems has been identified and its effects substantially reduced since Impact Acquire v3.5.0. See Notes about PCIe credits and user-supplied buffers for more details on this.

Resolution

To avoid this situation there are several possible solutions:

• Ensure that the number of requests is high enough to buffer several hundred milliseconds at the framerate you are using. Modern desktop operating systems do not usually use real-time scheduling and may need a high number of buffers to ensure that there is always a buffer available for the next image.
• The BVS CA-BN cameras are capable of supplying images at several thousand frames per second if small areas of interest are used. However, the operating system may not be able thousands of interrupts per second. If you experience lost frames at very high frame rates and small areas of interest please consider using BlockScan mode instead. Refer to Using the BlockScan feature for more information.
• Use the jetson_clocks command, supplied by NVIDIA®, in order to turn off the governors and run the clocks at their maximum possible values. N.B. jetson_clocks will need to be called every time after starting the system.
• Install an autostart script which turns off the EMC governor and sets the memory clock to its maximum value every time you start the system.

An appropriate script could be written with these contents to a new file called, for example, "/usr/local/bin/set_emc_to_max.sh".

#/bin/sh
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
echo $(cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate) > /sys/kernel/debug/bpmp/debug/clk/emc/rate
exit 0

You could use 'systemd' to run this script once on start. Create the new file "/etc/systemd/system/emc_max.service" with these contents:

[Unit]
Description=set EMC clock to maximum service
After=systemd-modules-load.service

[Service]
Type=oneshot
ExecStart=/bin/sh /usr/local/bin/set_emc_to_max.sh

[Install]
WantedBy=multi-user.target

Activate the service ready for the next reboot like this:

sudo systemctl enable emc_max.service

You can start the service once, without rebooting, like this:

sudo systemctl start emc_max.service

You can disable the autostart at any time like this:

sudo systemctl disable emc_max.service