Frequently fail to connect and/or stream from >3 cameras in realsense-viewer

[jfhamlin]

---

Required Info
Camera Model	D400
Firmware Version	(05.08.06.00)
Operating System & Version	Linux (Ubuntu 16)
Kernel Version (Linux Only)	4.13.0-36-generic
Platform	Intel NUC
SDK Version	2.10.1

Issue Description

I have 6 cameras connected over a USB 3.0 hub. I can stream three cameras at 480x270@30fps; when I enable the fourth, I get a static frame. When I enable any more cameras, I get no frames at all. Then I'll begin to see errors like:

 07/04 14:07:28,267 WARNING [140269571749632] (backend-v4l2.cpp:808) Empty frame has arrived.
 07/04 14:07:28,267 WARNING [140269571749632] (backend-v4l2.cpp:808) Empty frame has arrived.
 07/04 14:07:28,378 ERROR [140269596927744] (types.h:178) xioctl(VIDIOC_DQBUF) failed Last Error: No such device
 07/04 14:07:28,378 ERROR [140269596927744] (backend-v4l2.cpp:1185) xioctl(VIDIOC_DQBUF) failed Last Error: No such device
 07/04 14:07:28,379 ERROR [140269588535040] (types.h:178) xioctl(VIDIOC_DQBUF) failed Last Error: No such device
 07/04 14:07:28,379 ERROR [140269588535040] (backend-v4l2.cpp:1185) xioctl(VIDIOC_DQBUF) failed Last Error: No such device
 07/04 14:07:28,380 ERROR [140269580142336] (types.h:178) xioctl(VIDIOC_DQBUF) failed Last Error: No such device
 07/04 14:07:28,380 ERROR [140269580142336] (backend-v4l2.cpp:1185) xioctl(VIDIOC_DQBUF) failed Last Error: No such device
 07/04 14:07:28,735 ERROR [140269761644288] (types.h:178) get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
 07/04 14:07:28,735 ERROR [140269761644288] (error-handling.cpp:68) Error during polling error handler: get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device

When I quit, I'll find that most of the cameras are no longer visible as USB devices!

$ lsusb|grep Intel|wc -l
7
$ lsusb|grep Intel|wc -l
4
$ lsusb|grep Intel|wc -l
4
$ lsusb|grep Intel|wc -l
4
$ lsusb|grep Intel|wc -l
2
$ lsusb|grep Intel|wc -l
2
$ lsusb|grep Intel|wc -l
2

I'm forced to unplug the hub and then plug it back in to be able to see the cameras again. How can I stream from many devices reliably, especially without making the cameras inaccessible?

[jfhamlin]

This also happens when running a few cameras at 6fps. It will raise an exception saying a frame didn't arrive within 5000ms (sometimes the error is different), and the camera(s) will become inaccessible until I unplug it and plug it back in. This is a pretty catastrophic failure mode if the camera is deployed somewhere not easily accessible.

[jfhamlin]

Happens with D415s and D435s.

[josephduchesne]

I've noticed this with just 2 cameras. In 27 restarts, I've had this issue happen 4x.
The cameras are plugged into different controllers (the 4xUSB A is one controller, and 1xUSB C being another). So far the USB A port controller has been the only one to fail, but my sample size is still small. I have tried different USB A ports and swapping cables.

Symptom: lsusb | grep '8086:0b07' | wc -l returns 1 (expected 2)
No kernel messages reporting any activity on the bus dedicated to the errant camera. No other USB devices are connected to the NUC (just 2x D435).

Camera Model	D435
Firmware Version	(05.09.02.00)
Operating System & Version	Linux (Ubuntu 16)
Kernel Version (Linux Only)	4.8.0-58-generic
Platform	Intel NUC (nuc7i3syh)
SDK Version	2.10.1-0~realsense0.62

Note: This appears to be very similar to DSO-7976 @jfhamlin have you tried using the windows utility to update to the latest firmware? The DSO document claims to have fixed this issue (although my results say otherwise).

I automated a test. Configuration as above, it would reboot the PC if lsusb only showed one camera, and restart the program using the left or right camera individually if there was no depth data within 15 seconds of initialization. Green and purple pie slices are reboot 3x and reboot 4x.

[jfhamlin]

I haven’t updated the firmware, no. Thanks for the suggestion. I’ll update and report back.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@jfhamlin, what CPU does your NUC have? Are you streaming depth only and only running the veiwer?

Are the cameras connected directly to the NUC or through hubs?

With respect to @josephduchesne comment about DSO-7976, your issue is not on reboot correct? It is just that the cameras crash after adding the fourth camera, right? The DSO was related to cameras not enumerating on a cold boot.

[jfhamlin]

@RealSense-Customer-Engineering

@jfhamlin, what CPU does your NUC have?

CPU: i7-6770HQ

Are you streaming depth only and only running the veiwer?

yes, i'm only streaming depth, and while i've mostly been running my own software, i can reproduce in the viewer

your issue is not on reboot correct? It is just that the cameras crash after adding the fourth camera, right?

correct that it is not on reboot. but it's not a crash after adding the fourth camera, but after running multiple cameras for a time. i haven't found precise repro steps; it just happens after running the cameras for a bit, sometimes quickly and sometimes after a few minutes.

actually, this may be a duplicate of #1213 (which should be addressed urgently as it prevents many production uses)

[tedtman]

Hello @jfhamlin,

We're still reviewing this but I think you should be able to run four cameras. I have run eight at 424x240 15 fps although not for an extended time (maybe an hours). Also with respect to #1213, this is actually the cold boot issue that may have been resolved by DSO-7976 and f/w 5.9.0. @maciekd actually stated in his issue #1219 , that he was able to run 7 RS400 on an Intel NUC.

I will see if I can get a platform setup but I will need to dig up the D435's.

Regards,
Ted
Intel Customer Platform Engineering

[josephduchesne]

@RealSense-Customer-Engineering my issue occurs on reboot or cold boot, so it's more likely caused by whatever is behind #1213

[tedtman]

@josephduchesne can you open a new issue as this one is slightly different than @jfhamlin (not a reboot case). That being said, we are starting to look into a case with another customer that sounds the same as yours.

Regards,
Ted
Intel Customer Platform Engineering

[josephduchesne]

@tedtman I created #1615 to cover the issue(s) I've noticed

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@jfhamlin have you had any success yet? If not, do you have any new details you can provide that we can work with?

[poripsa]

@RealSense-Customer-Engineering I have noticed this issue as well specially when running on a different platform like ROS, however I havent updated the driver. I plan on doing that soon and reporting here. Also everytime a wrapper on the librealsense is changed the camera needs to be unplugged and replugged after it crashes. Any issue for that or should I create another one?

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@jfhamlin I ran six cameras at 480x270 30fps for two days without failure however after reviewing you initial information I realized my setup wasn't exactly the same as I only ran three of the cameras through a hub. That being said, it is difficult to predict performance using hubs,

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@poripsa, we can better serve you using your company portal for RealSense unless you prefer to include community feedback and if that is the case please open a new issue as your environment is a little different.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hello @jfhamlin, I will close this issue but if your problem still exists feel free to open a new issue.

[szx0112]

Is it possible to run the 3 or more cameras with both depth and color on full resolution (1280x720). Through realsense-viewer, I tested it works with 2 cameras, but will cause frame not arrived issue (either on color or depth frame) when adding the third camera. I also check the usb bandwidth, it takes significantly less than the upper limit of usb 3 (5G), so it seems that the bottleneck is not caused by the usb or usb hub (if stable). My PC with intel E5-2650, 64G DDR3 and Nvidia gf 1080i gpu.