"Failed to set power state" when multicam on NVIDIA Jetson TX2

[rcroset]

Required Info
Camera Model	D400
Firmware Version	05.12.03.00
Operating System & Version	Ubuntu 18.04.4
Kernel Version (Linux Only)	4.9.140-tegra
Platform	NVIDIA Jetson TX2
SDK Version	2.34.1
Language	python
Segment	others

Issue Description

I am trying to connect 3 D435 cameras to an NVIDIA Jetson TX2 platform via a USB3 hub. If only 1 camera is connected, everything works fine, but when I connect the second one (and the third one) and try to launch the rs-multicam example (or some other code involving multiple cameras), I get a "Failed to set power state" error.

It doesn't seem to be related to the hub because it works fine on a Ubuntu 18.04 and 19.10 laptops. After lots of investigations, it also seems that this issue is not related to the hub nor the USB controller.

One easy way to reproduce is to connect 3 cameras and launch the realsense_device_manager.py script provided in the box dimensioner multicam python example.

Also, in realsense-viewer, if I try to open all Depth+RGB streams for the 3 cameras, some streams don't recieve any frames. This does not happen when using a laptop.

Any help would be appreciated !

[MartyG-RealSense]

There have been multicam problems reported recently with Jetson Xavier boards in regards to problems such as failure to set power state or cameras not being detected when more than one camera is attached. So this case with your TX2 may be related. An Xavier user recently offered details of a multicam workaround that he developed:

#5828 (comment)

[rcroset]

Thank you for your quick reply ! I had a look to this workaround and it doesn't seems to work unfortunately. I am working at high resolution (1280x720 for depth, 1920x1080 for RGB).
I managed to have two cameras with both Depth and RGB at 1280x720. When I raise the resolution of one of them, the program gets stuck when starting the second camera. When I add a third camera, I get the power state error

[MartyG-RealSense]

Is your hub a powered hub (drawing from the mains power socket) or an unpowered one (connected to the TX2's USB socket) please? Though I note the investigations you have already done that suggest it is not a hub-related issue.

[rcroset]

The hub is unpowered. It doesn't seems to be related to the hub because everything works fine (3 cameras with high resolutions) on a "regular" laptop.

[MartyG-RealSense]

It is not unusual for a camera that works fine on a desktop or laptop to have problems on a single-board computer such as Jetson, which are typically "low power" devices because of their use in applications such as mobile robotics where power efficiency may be a priority. So that does not necessarily rule out the hub as a problem, since if it is an unpowered hub then it is drawing its power from the low-power board's USB port instead of having its own independent strong power supply like it would if it were powered by the mains socket.

[rcroset]

I'm aware of those power limitations, but still, after doing some power measurments and calculations, it seems that the USB port has enough power to power the 3 cameras. Still we'll try with a powered hub and keep you posted.

[MartyG-RealSense]

Thanks very much, I look forward to your update. Good luck!

[rcroset]

In the meantime, do you have any idea of other things we might look at if the problem isn't the hub ? Because if we observe the power used by the hub (and the cameras) and the power of the USB port, we should have more than enough !

[MartyG-RealSense]

As well as the volume of power supplied, the stability of the power supplied to the USB port can be a factor. I once had a case where RealSense was used with an enterprise server with a 650W power supply and it still experienced disconnections. RealSense cameras are sensitive to the state of the port, and mains-powered hubs can help to provide a more reliable power supply to the port on the hub than a port on a computer or an unpowered hub can.

I see that you have already tried starting with one just camera and then inserting the others into the hub one at a time.

A couple of long-shots that you could try with the USB cable are:

1. Having the camera's USB cable plugged into the hub but unplugging and re-plugging the micro-end from the side of the camera. Or:

2. Turning the micro-end upside down and plugging it back into the side of the camera (the micro-size end fits both ways on USB-C cables).

Also, have you tried inserting each camera on its own with no other cameras connected, to check whether any of the individual cameras have a problem?

You could also try swapping the USB cables around (e.g put the USB cable from camera 1 on camera 2, and the cable from camera 2 on camera 1).

[rcroset]

Hi,
We've tried today with a powered USB hub and things didn't change at all. We also try the tricks you mentioned (turning the micro-end upside down). We notices some changes when we change the order in which the cameras are plugged into the hub (sometimes we can start only one pipeline, sometimes two, sometime three with very low resolutions), but we are still unable to make things work at high resolutions.
We checked cameras individually and with different cables to check if one of them has a problem, but it's not the case, everything works fine.

[MartyG-RealSense]

Thanks for trying everything. Did you build Librealsense for your TX2 from source code or with the debian packages, please?

I was just thinking that if it was from source then you may be able to reduce the processing load on the TX2's CPU by building Librealsense with CUDA support to offload some of the CPU's processing work onto the GPU.

If you installed Librealsense using the debian packages then they would already contain CUDA support.

[rcroset]

We've built Librealsense from source, but we already did it with CUDA support. This wasn't the issue.

However, we managed to solve our problem by deactivating the USB auto-suspend in the usbcore parameters. It seemed to have some issues with the uvcvideo driver. By being careful in the order the cameras are plugged in the hub and by applying the workaround you mentioned earlier, everything works !

FYI, in our case, the cameras need to be plugged in the hub in decreasing order regarding the serial number (highest SN in port 1, lowest SN in port 3). Then the cameras can be activated in any order. If we change the plugging order or we don't apply the workaround, we still get the failed to set power state error.

[MartyG-RealSense]

I'm very glad you found solutions for your problems! Thanks very much for posting your findings here so that others can benefit from them.

[rcroset]

You're welcome. Thank your for helping us through this.

[dongkaizhou]

Hello, I encountered the same problem in similar scenes.

Using USB-HUB to connect 3 cameras on TX2, USB-HUB has a separate power supply, an error occurs when using rs-multicam: ‘Failed to set power state’, SDK version is v2.32.1 (from the debian packages).

However, keeping the other hardware unchanged, use another TX2, which contains SDK version v2.29 (from source code), using rs-multicam can open 3 cameras.

The TX2 system is updated, I hope to use SDK v2.32.1 + (from the debian packages). Is there any suggestion?
Thanks

[MartyG-RealSense]

Hi @dongkaizhou The RealSense developers have created improvements to how multicam is handled that should be available in an upcoming SDK release. They may not fix a problem if it is related to a specific model of hub hardware but may help if the problem is related to rs2::pipeline.

#6467

In the meantime, a RealSense user has suggested a workaround.

#5828 (comment)

[dongkaizhou]

It turns out that many users have this kind of problem, look forward to the new SDK to solve this problem.

[MartyG-RealSense]

Hi @dongkaizhou Have you had an opportunity to try the new SDK version 2.35.0 yet please and find whether it solved the errors mentioned in this case?

[dongkaizhou]

Hi, I try to install the new SDK version 2.35.0, but installed on TX2, how to ensure that v2.35.0 is installed in the following way?
If no version number option is specified, will other versions be installed, such as v2.34.1

[MartyG-RealSense]

@dongkaizhou My understanding is that if the above instructions are used then the version installed should be whatever the latest Debian package version is. More information about DKMS package version numbers can be found in the link below.

#2518 (comment)

[dongkaizhou]

Thanks, I will pay attention to Debian package version, Where can I see the update time of all Debian package version?
Just now using camke\make to install SDK version 2.35.0 on my PC, and rs-mulicam can open two cameras without error.
But used SDK2.32.1 on tx2, opening multiple cameras failed. I just installed SDK2.32.1 on my PC. Using rs-mulicam to open the two cameras is also successful. Is it possible that sdk2.32.1 has also been partially modified recently? Or caused by the different compilation methods of tx2 and PC?

[MartyG-RealSense]

@dongkaizhou The link below explains that the DKMS packages are organised by Ubuntu distro instead of having a separate version for each individual SDK version. So there is a DKMS package for Bionic, another for Xenial, etc.

#5249 (comment)

So my understanding is that there is not a version history for the DKMS packages, and that earlier packages are overwritten with the newer version and so keep the same filename. The package for Xenial or for Bionic is installed by using the sudo add-apt-repository command provided in the installation instructions for Xenial (used with Ubuntu 16 LTS) or Bionic (used with Ubuntu 18 LTS).

I am also not aware of a formal information source for when a DKMS package has been updated.

You can upgrade only the librealsense2-dkms package specifically instead of upgrading all packages by using this instruction:

sudo apt-get --only-upgrade install librealsense2-utils librealsense2-dkms

Regarding multicam support, there are some known problems at the moment with camera detection in multicam setups on Jetson models. The RealSense developer team has created improved multicam support, and this was successfully merged into the development branch of the SDK on 1 June 2020. So it should be present in the next public SDK version.

The new multicam developments should help with multicam problems related to rs2::pipeline, though may not correct problems related to specific models of powered USB hub hardware.

#6467

In the meantime, a Jetson user has offered a workaround here:

#5828 (comment)

[dongkaizhou]

Thanks for your prompt reply, I just tried to update the SDK on Jetson TX2. Unfortunately, v2.32.1 is already the latest version, and it still fails when using rs-mulicam to open 2 cameras.
Can you help to update the DKMS packages as soon as possible? maybe many Jetson users have this problem.

[MartyG-RealSense]

@dongkaizhou 2.32.1 does not contain the multicam improvements as it is an older SDK. The SDK is now up to version 2.35.0 at the time of writing this. The Debian packages may not be in sync with what the current SDK version is though.

I am not on the Developer team (I am Support) and so cannot influence the timing of when the Debian packages are updated. The developer team are very aware of issues with multicam though, and the new improvements were created to try to resolve some of those problems.

Until the multicam improvements are available for users to test, the workaround link posted above remains the best available solution for Jetson users to try

[dongkaizhou]

OK, hoping that Debian packages will be updated soon and will continue to test this issue.
Thank you.

[dongkaizhou]

842ee1e is what?

[MartyG-RealSense]

IT looks as though at the time that David's message was written, it was suggesting to compile SDK version 2.33.1 from source.

[dongkaizhou]

So if you can install SDK from source, it is the best choice?

[MartyG-RealSense]

Ideally, if you are installing on Jetson Nano then I would recommend using JetsonHacks' most recent Nano guide, which uses Debian packages:

https://www.jetsonhacks.com/2019/12/22/install-realsense-camera-in-5-minutes-jetson-nano/

If you need to install the SDK on Nano from source though then Intel's own Jetson installation documentation recommends using the "backend" method of installation. It requires an internet connection to perform the installation but is not dependent on Linux versions or kernel versions and does not require patching to be performed.

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md

Regarding the above image saying to use the CMake flag -DFORCE_LIBUVC = TRUE - Intel introduced a new system a month after that guide was last updated that uses -DFORCE_RSUSB_BACKEND=true instead. So use -DFORCE_RSUSB_BACKEND=true (see the instructions link below).

Instructions for installing the 2.35.0 SDK source with the backend method are in the link below.

#6512 (comment)

[dongkaizhou]

Seeing this, Intel and Nvidia both recommend Debian installation.
Unfortunately, the SDK3.35.0 is compiled and installed on TX2, and it is still not possible to open the two cameras using rs-mulicam, but it is ok on the PC. It should be related to the Jetson tx2 environment.
Thanks

[MartyG-RealSense]

On the powered hub that I have, there is a USB connector on the side of the hub that is separate from the hub's main ports. This port is for a USB lead that goes from the hub port to a USB port on my computer. If I unplug this lead from the computer's USB port then the USB devices that are plugged into the hub no longer work.

If you have a similar cable on your powered USB hub, I wondered if you could make sure that it is plugged into a USB port on the Jetson TX2 that you are using, and not your other Jetson TX2, which you mention in this comment:

#6395 (comment)

I could see a possible situation where the cable connecting the hub to the computer might be plugged into the other TX2 board that is not powered up when you are using the TX2 that will not detect the computers. And so although the cameras are attached to the hub, the TX2 that returns the failed to set power state error does so because the hub's hub-to-computer USB cable is not plugged into it.

I have assembled together a rough diagram to illustrate what I am describing above. :)

[MartyG-RealSense]

@dongkaizhou SDK 2.35.2 has been published on the Releases page.

https://github.com/IntelRealSense/librealsense/releases/tag/v2.35.2

https://github.com/IntelRealSense/librealsense/wiki/Release-Notes#release-2352

I am aware that you need a Debian version though, so I provide this information in case you want to see what is added in 2.35.2.

[dongkaizhou]

Thank you very much for describing it with vivid pictures. I have tested this situation many times, and there are many attempts to connect the HUB to TX2. I have considered and tested the situation you described.

The tested situation is as follows:

1. The HUB has no external power supply and is connected to TX2 USB3.0.
2. HUB external power supply, connect with TX2 USB3.0.
3. HUBs of different manufacturers use the above method to connect TX2 USB3.0.

[dongkaizhou]

One interesting thing, when I plugged two cameras into the HUB, I connected the Hub to TX2. When trying to turn on the camera using rs-mulicam, the error ‘failed to set power state’ is reported; when using realsense-viewer to turn on one camera first, and then turn on the second camera, both cameras can be turned on.

[MartyG-RealSense]

Looking at the log information generated when using the RealSense Viewer, there are two interesting things:

• No data to return (-21) suggests that the IMU of one or both of the D435i needs to be calibrated.
  -. The errors about synthetic stream, control_transfer returned and out of frame resources all suggest that the USB is having problems communicating with the camera.

It may be a situation that improves in the new SDK 2.35.2 with the multicam improvements, if the fault is not with the particular model of USB hub.

Do you get less errors in the Viewer if you only have the Depth streams activated?

[dongkaizhou]

There is failed 'UNKNOWN in rs2 get option range... option: sharpness get data usb failed. error: RS2 USB STATUS IO'

[MartyG-RealSense]

@dongkaizhou Could you try updating your camera firmware to 5.12.5 please? You can do this by going to the 'More' option at the top of the RealSense Viewer's options side-panel and selecting 'Update to recommended firmware' from its menu.

[dongkaizhou]

Yes, update FW to 5.12.02, but the failed is exist.

[MartyG-RealSense]

The newest firmware is 5.12.5. You mentioned above that you updated to 5.12.02. Can you confirm please that you have tried 5.12.5

[dongkaizhou]

Hi, the main problem is the SDK-V2.32.1 version of TX2. I look forward to the Debian version update. If there is an update, please inform me. I am busy with other things recently.
Thaks.

[MartyG-RealSense]

Okay, thanks very much.

[MartyG-RealSense]

@dongkaizhou A new ROS wrapper version 2.2.14 has now been released that supports librealsense SDK 2.35.2.

https://github.com/IntelRealSense/realsense-ros/releases/tag/2.2.14

[MartyG-RealSense]

Case closed due to no further comments. @dongkaizhou please create a new GitHub issue if you still have problems when you return to RealSense work. Thanks!

[stino78]

I had a similar problem on Raspberry Pi OS, using pyrealsense2 and the D435i

I was opening two pipelines, one for the camera and one for the IMU. Starting the camera didn't give any problems, but when starting the IMU, right after, there was an error message "failed to set power state".

After may software and hardware attempts, I solved it in a trivial way... I added 500ms pause between the two startings. You might do a similar thing if you have more than one device.

Good luck!