Multi-Camera multi pipeline IMU + Depth, Color setup fails with "Couldn't resolve request"

[brunovollmer]

---

Required Info
Camera Model	D430
Firmware Version	05.14.00.00
Operating System & Version	Linux, Ubuntu 22.04
Kernel Version (Linux Only)	5.10.66
Platform	Khadas Edge 2
SDK Version	2.50.0
Language	Python
Segment	Robotics

Issue Description

We are currently trying to get our multi-camera setup to work on our new board. The setup consists of two D430 and one D430I. We would like to pull 848x480 greyscale and depth images at 30 FPS and the IMU data at 200 FPS. We have a script in python that works on our development laptop running Ubuntu 22.04 with a 5.15 kernel.

The script has three pipelines for each camera and one additional one for the IMU data of the middle camera.

On the edge 2 board from khadas we have compiled librealsense with the following command:

cmake ../ -DBUILD_PYTHON_BINDINGS:bool=true -DFORCE_RSUSB_BACKEND:bool=true -DBUILD_WITH_CUDA:bool=false -DBUILD_GRAPHICAL_EXAMPLES:bool=false -DCMAKE_BUILD_TYPE=release

When running the script we receive the following error:

Traceback (most recent call last):
  File "scripts/record_image.py", line 443, in <module>
    main()
  File "scripts/record_image.py", line 411, in main
    position_to_camera_data = init_cameras(
  File "scripts/record_image.py", line 180, in init_cameras
    imu_pipe.start(imu_cfg)
RuntimeError: Couldn't resolve requests

We are suspecting it could be caused by the RSUSB backend but are unsure. We are currently unable to update the kernel of our board and are wondering if there is any other way to debug/fix this?

[MartyG-RealSense]

Hi @brunovollmer RuntimeError: Couldn't resolve requests indicates that the custom cfg configuration that was requested was not able to be provided by the camera. There are a few possible causes for this.

1. If the camera is on a USB 2.1 connection or is on USB 3 but mis-detected as USB 2.1, 30 FPS will not be supported for the 848x480 resolution. Only 6 FPS is supported on USB 2.1.

2. A speed of 200 for IMU is not accepted if 200 has been used for the Accel stream.

Acceptable IMU pairings on a D43x IMU-equipped camera are as follows:

Camera manuactured before early 2022
63 Accel, 200 Gyro

Camera manufactured after early 2022
100 Accel, 200 Gyro

So if 200 was requested for Accel, that could cause 'Couldn't resolve requests'.

You can check which speed is supported for Accel on your particular camera unit by launching the RealSense Viewer tool, looking in the Motion Module section of the Viewer's options side-panel and looking in the drop-down menu beside the Accel stream option.

3. D435 does not have an IMU component, so IMU streams cannot be provided by it. D430i should have an IMU, as indocated by the 'i' in its name.

[brunovollmer]

Hey @MartyG-RealSense,

thanks for the quick reply I will reply for each point individually:

1. I've checked and the cameras are detected as USB 3.2 devices.
2. I didn't specify it correctly. We are using 63/100 for Accel and 200 for Gyro. Additionally, it seems to work fine on the laptop we use. So I don't think the settings or script itself are the problem.
3. I've checked with rs-enumerate-devices and it has an IMU and the respective streams.

Any other ideas what to check?

[MartyG-RealSense]

As the cameras are all detected as USB 3.2 and the program works fine on a laptop, it does seem to suggest an issue other than the camera or the USB connection.

I note that the Khadas Edge 2 has a Rockchip chipset. For Rockchip (RK) based devices, building the librealsense from source code with the RSUSB backend is recommended, and you have already done so. Having said that, RSUSB is not ideal for multiple camera applications, and DFORCE_RSUSB_BACKEND=false (a 'native' SDK build) that has been kernel patched is best suited to multicam.

Does the librealsense build on the laptop use RSUSB or is it kernel-patched, please?

[brunovollmer]

I installed it via pip. I assume it's kernel-patched then? What would I need to do to kernel-patch the 5.10 kernel? Are there any hints what to look for or where to go?

[MartyG-RealSense]

Do you mean that you installed the Python wrapper with pip install pyrealsense2, please? If so then doing so does not apply a kernel patch to the librealsense SDK.

Kernel 5.10 is not supported by the SDK. It supports 5.11, 5.13 and 5.15. If you are unable to change the kernel on your computer then -DFORCE_RSUSB_BACKEND:bool=true is the appropriate configuration, as an RSUSB = true source-code build of the SDK bypasses the kernel (making it independent of the kernel) whilst still allowing non-RealSense applications to continue using the kernel.

[brunovollmer]

I mean that is what we did but there must be a way to use it with the 5.10 kernel or?

[brunovollmer]

How do these kernel patches work? Is it theoretically possible to write them for the 5.10 kernel and then compile it without RSUSB?

[MartyG-RealSense]

The D430 and D430i camera configurations do not have an RGB sensor and do not support streaming RGB from the left infrared sensor like the D405, D415 and D455 models do. As your issue title references trying to get a color stream but the D430 and D430i do not have RGB color, I was wondering how your script works on the laptop.

A RealSense user submitted user-created support for the 5.11 kernel in SDK 2.51.1 with a Pull Release (PR) at #9727 so self-creating support for patching of a particular kernel seems to be achievable. 8e7fd21 lists the code changes made (highlighted in green).

It should be noted that the SDK can work with unsupported kernels but there may be unpredictable consequences in regards to stability, which is why RSUSB = true to bypass the kernel is the better option if possible to use it (despite RSUSB not being ideal for multicam applications).

[MartyG-RealSense]

Hi @brunovollmer Do you require further assistance with this case, please? Thanks!

[brunovollmer]

I do unfortunately it is not fully solved yet. I've went through the available documentation of the provider of our board and was able to compile the kernel with the following two patches manually applied:

• https://github.com/IntelRealSense/librealsense/blob/master/scripts/realsense-hid-focal-master.patch
• https://github.com/IntelRealSense/librealsense/blob/master/scripts/realsense-camera-formats-focal-hwe-5.11.patch

I had to manually apply them as the automatic patching would not work (I assume due to slight changes in the kernel or something). I have to admit as this is all rather new to me it might also be some mistakes from my side. After installing the new kernel and checking the compilation time it, the errors unfortunately still continue. I still receive this error:

05/04 12:40:38,791 ERROR [547773912160] (librealsense-exception.h:52) Mipi device capability could not be grabbed Last Error: No such device
... (-> repeats the line above ~40 times)
05/04 12:40:38,796 ERROR [547773912160] (librealsense-exception.h:52) /dev/video24 is no video capture device Last Error: Invalid argument
05/04 12:40:38,796 ERROR [547773912160] (sensor.cpp:664) acquire_power failed: /dev/video24 is no video capture device Last Error: Invalid argument
Could not create device - /dev/video24 is no video capture device Last Error: Invalid argument . Check SDK logs for details

And when printing the kernel logs I see this:

[Thu Apr 6 14:41:29 2023] rockchip-csi2-dphy0: No link between dphy and sensor
[Thu Apr 6 14:41:29 2023] rkcif-mipi-lvds2: rkcif_update_sensor_info: stream[1] get remote terminal sensor failed!
[Thu Apr 6 14:41:29 2023] rkcif_scale_ch1: update sensor info failed -19
[Thu Apr 6 14:41:29 2023] rockchip-csi2-dphy0: No link between dphy and sensor
[Thu Apr 6 14:41:29 2023] rkcif-mipi-lvds2: rkcif_update_sensor_info: stream[2] get remote terminal sensor failed!
[Thu Apr 6 14:41:29 2023] rkcif_scale_ch2: update sensor info failed -19
[Thu Apr 6 14:41:29 2023] rockchip-csi2-dphy0: No link between dphy and sensor
[Thu Apr 6 14:41:29 2023] rkcif-mipi-lvds2: rkcif_update_sensor_info: stream[3] get remote terminal sensor failed!
[Thu Apr 6 14:41:29 2023] rkcif_scale_ch3: update sensor info failed -19
[Thu Apr 6 14:41:29 2023] rockchip-csi2-dphy0: No link between dphy and sensor
[Thu Apr 6 14:41:29 2023] rkcif-mipi-lvds: rkcif_update_sensor_info: stream[0] get remote terminal sensor failed!
[Thu Apr 6 14:41:29 2023] stream_cif_mipi_id0: update sensor info failed -19
[Thu Apr 6 14:41:29 2023] rockchip-csi2-dphy0: No link between dphy and sensor
[Thu Apr 6 14:41:29 2023] rkcif-mipi-lvds: rkcif_update_sensor_info: stream[1] get remote terminal sensor failed!
[Thu Apr 6 14:41:29 2023] stream_cif_mipi_id1: update sensor info failed -19
[Thu Apr 6 14:41:29 2023] rockchip-csi2-dphy0: No link between dphy and sensor
[Thu Apr 6 14:41:29 2023] rkcif-mipi-lvds: rkcif_update_sensor_info: stream[2] get remote terminal sensor failed!
[Thu Apr 6 14:41:29 2023] stream_cif_mipi_id2: update sensor info failed -19

I'm unsure if either my changes were wrong, not enough or not applied. Do you have any suggestions or ideas on how to check that?
The kernel compilation tools can be found here: https://github.com/khadas/fenix/tree/v1.4 and the linux kernel on the branch I'm using here: https://github.com/khadas/linux/blob/khadas-edges-5.10.y-release-v1.0.0/.
I'm super open to any suggestion what could cause such a problem and how to fix it or further debug it.

[MartyG-RealSense]

Someone with a Rockchip system that had the same set of errors as you did found that the solution in their particular case was to use Rockchip's U-Boot tool instead of booting the kernel manually.

https://www.linuxquestions.org/questions/linux-embedded-and-single-board-computer-78/booting-woes-with-hard-drive-based-rootfs-on-arm-sbc-board-4175716085/#post6378157

[brunovollmer]

After some more investigation it seems the problem is somewhere else. One weird thing I discovered is that when running rs-enumerate-devices it tries to access /dev/video24 and /dev/video32 which either don't exist or something else (this is my understanding of the output of v4l2-ctl --list-devices):

rkisp-statistics (platform: rkisp):
        /dev/video31
        /dev/video32

rkcif-mipi-lvds (platform:rkcif):
        /dev/media0
        /dev/media1
        /dev/media2

rkisp0 (platform:rkisp-unite):
        /dev/media3

rkisp_mainpath (platform:rkisp0-vir0):
        /dev/video24
        /dev/video25
        /dev/video26
        /dev/video27
        /dev/video28
        /dev/video29
        /dev/video30

Intel(R) RealSense(TM) Depth Ca (usb-xhci-hcd.7.auto-1.1):
        /dev/video33
        /dev/video34
        /dev/video35
        /dev/video36
        /dev/media4

Intel(R) RealSense(TM) Depth Ca (usb-xhci-hcd.7.auto-1.2):
        /dev/video37
        /dev/video38
        /dev/video39
        /dev/video40
        /dev/media5

Intel(R) RealSense(TM) Depth Ca (usb-xhci-hcd.7.auto-1.3):
        /dev/video41
        /dev/video42
        /dev/video43
        /dev/video44
        /dev/media6

Failed to open /dev/video0: No such device

The error message on running rs-enumerate-devices -s is:

 11/04 11:26:34,911 ERROR [548170473504] (librealsense-exception.h:52) Mipi device capability could not be grabbed Last Error: No such device
... Repetition of the previous line ...
 11/04 11:26:35,282 ERROR [548170473504] (librealsense-exception.h:52) Mipi device capability could not be grabbed Last Error: No such device
 11/04 11:26:35,287 ERROR [548170473504] (librealsense-exception.h:52) /dev/video32 is no video capture device Last Error: Invalid argument
 11/04 11:26:35,287 ERROR [548170473504] (sensor.cpp:664) acquire_power failed: /dev/video32 is no video capture device Last Error: Invalid argument
Could not create device - /dev/video32 is no video capture device Last Error: Invalid argument . Check SDK logs for details
 11/04 11:26:35,289 ERROR [548170473504] (librealsense-exception.h:52) Mipi device capability could not be grabbed Last Error: No such device
... Repetition of the previous line ...
 11/04 11:26:35,313 ERROR [548170473504] (librealsense-exception.h:52) Mipi device capability could not be grabbed Last Error: No such device
 11/04 11:26:35,318 ERROR [548170473504] (librealsense-exception.h:52) /dev/video24 is no video capture device Last Error: Invalid argument
 11/04 11:26:35,318 ERROR [548170473504] (sensor.cpp:664) acquire_power failed: /dev/video24 is no video capture device Last Error: Invalid argument
Could not create device - /dev/video24 is no video capture device Last Error: Invalid argument . Check SDK logs for details
Device Name                   Serial Number       Firmware Version    
Intel RealSense D430          153122079477        05.14.00.00         
Intel RealSense D430          207522076333        05.14.00.00 

Any idea what I could have misconfigured or how to fix that?

[MartyG-RealSense]

My own understanding is that /video addresses are assigned based on which devices are detected and the more devices that are detected, the more /video addresses there will be. The /video address numbers of a specific device may also change if it is unplugged from the USB port and re-inserted, making it difficult to access the device by always using the same /video number.

In a recent case at #11553 (comment) the subject of setting fixed addresses for devices with symbolic links via udev device-handling rules was discussed.

[MartyG-RealSense]

Hi @brunovollmer Do you require further assistance with this case, please? Thanks!

[brunovollmer]

After some back and forth with the intel support, we concluded that it might be a nonsolvable hardware issue for now. Thanks for the support

[MartyG-RealSense]

You are very welcome. Thanks very much for the update.

[igorehan]

@brunovollmer, Tell me, did you somehow get around this problem?
Judging by the following issue, you were able to move further