Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

D435i timeouts Python #3460

Closed
tRosenflanz opened this issue Mar 11, 2019 · 42 comments
Closed

D435i timeouts Python #3460

tRosenflanz opened this issue Mar 11, 2019 · 42 comments
Assignees

Comments

@tRosenflanz
Copy link

tRosenflanz commented Mar 11, 2019


Required Info
Camera Model D435i
Firmware Version 05.10.13
Operating System & Version Linux (Ubuntu 18.04)
Kernel Version (Linux Only) 4.15.0-45-generic
Platform PC
SDK Version 2.19.1
Language python
Segment others

When multiple streams are enabled with gyroscope/acceloremeter, pipeline takes very long time to synchronize frames. If all 3 (depth/color/infrared) are enabled this didn't return for over 5 minutes for me
Both SDK and the apt packages updated.

import pyrealsense2 as rs
import numpy as np

def initialize_camera():
    #start the frames pipe
    p = rs.pipeline()
    conf = rs.config()
    CAM_WIDTH, CAM_HEIGHT, CAM_FPS = 848,480,15
#     conf.enable_stream(rs.stream.depth, CAM_WIDTH, CAM_HEIGHT, rs.format.z16, CAM_FPS)
    conf.enable_stream(rs.stream.color, CAM_WIDTH, CAM_HEIGHT, rs.format.rgb8, CAM_FPS)
    conf.enable_stream(rs.stream.infrared, CAM_WIDTH, CAM_HEIGHT, rs.format.y8, CAM_FPS)
    conf.enable_stream(rs.stream.accel,rs.format.motion_xyz32f,250)
    conf.enable_stream(rs.stream.gyro,rs.format.motion_xyz32f,200)
    prof = p.start(conf)
    return p
p = initialize_camera()

%%time
f = p.wait_for_frames(1000000) 

Returns in 30 seconds.

Related issue it appears: #3108
Works fine in realsense-viewer though

@dorodnic
Copy link
Contributor

Hi @tRosenflanz
Perhaps installed version of pyrealsense2 is different from the version of the SDK tools?

@tRosenflanz
Copy link
Author

%%bash
pip freeze |grep realse

pyrealsense2==2.19.1.580

Confirmed that it uses the correct path by doing pip uninstall and getting errors on import. Reinstall didn't fix the issue

@tRosenflanz
Copy link
Author

@dorodnic any possible updates? We can try to switch to C++ but I am wondering if the pip version of pyrealsense was not properly updated

@tRosenflanz
Copy link
Author

Some more information:

  • Confirmed that the same behavior happens on both Windows and Ubuntu machines.

  • Tried Firmwares 5.10.13 , 5.11.1.100 , and 5.11.4

  • Updated pip package to version 2.19.1.586

  • For some reason now the timeouts happen regardless of IMU sensor streams being activated. Even this timeouts:

import pyrealsense2 as rs
import numpy as np

def initialize_camera():
    #start the frames pipe
    p = rs.pipeline()
    conf = rs.config()
    CAM_WIDTH, CAM_HEIGHT, CAM_FPS = 848,480,15
    conf.enable_stream(rs.stream.depth, CAM_WIDTH, CAM_HEIGHT, rs.format.z16, CAM_FPS)
    conf.enable_stream(rs.stream.color, CAM_WIDTH, CAM_HEIGHT, rs.format.rgb8, CAM_FPS)
#    conf.enable_stream(rs.stream.infrared, CAM_WIDTH, CAM_HEIGHT, rs.format.y8, CAM_FPS)
#    conf.enable_stream(rs.stream.accel,rs.format.motion_xyz32f,250)
#    conf.enable_stream(rs.stream.gyro,rs.format.motion_xyz32f,200)
    prof = p.start(conf)
    return p
p = initialize_camera()
f = p.wait_for_frames(100) 

I was fairly sure that this worked fine before but maybe I missed something.

  • It works fine for D435 (Not i version) regardless of firmware or SDK version.

  • Infrared+Color takes a bit of time to get aligned frames but after that it works fine and returns within 66ms as expected. Depth+Color never get aligned.

  • Starting and stopping pipe doesn't help.

Can you reproduce the same bug on your side?

@beomsik-landingai
Copy link

Is there any update on this? Can you guys just try to reproduce it with d435i with python and enable all the streams?

@lramati
Copy link
Contributor

lramati commented Apr 7, 2019

I couldn't get all the streams to open, but I still haven't found the cause

@RealSenseCustomerSupport
Copy link
Collaborator


Hi tRosenflanz,

Sorry for coming to this issue late.
I tried both 2.19.0 and 2.19.1 off a Windows 10 system, and then modified a little bit of your code sample (to get it going with streaming).
By checking the elapsed time during init, I don't have a timeout issue.

Also I have no issue with Ubuntu 16.04 as well.
See attached screenshot off Windows test.

Thanks!

Attachment(s):
d435i_fw5.11.04_python_test.png - https://realsensesupport.intel.com/attachments/token/ctjcNhDtK7RD05wlYy6k3gC8X/?name=d435i_fw5.11.04_python_test.png

@beomsik-landingai
Copy link

@RealSenseCustomerSupport Could you share what you needed to modify the code?
And if possible, could you post the screenshot here, I couldn't access the attachment as I don't have an account in realsense support.

@RealSenseCustomerSupport
Copy link
Collaborator


Hi tRosenflanz,

You may try this script. And I only saw 1~3 seconds delay printed out.

import pyrealsense2 as rs
import numpy as np
import time
import cv2

def initialize_camera():
p = rs.pipeline()
conf = rs.config()
CAM_WIDTH, CAM_HEIGHT, CAM_FPS = 848,480,15
conf.enable_stream(rs.stream.depth, CAM_WIDTH, CAM_HEIGHT, rs.format.z16, CAM_FPS)
conf.enable_stream(rs.stream.color, CAM_WIDTH, CAM_HEIGHT, rs.format.rgb8, CAM_FPS)
conf.enable_stream(rs.stream.infrared, CAM_WIDTH, CAM_HEIGHT, rs.format.y8, CAM_FPS)
conf.enable_stream(rs.stream.accel,rs.format.motion_xyz32f,250)
conf.enable_stream(rs.stream.gyro,rs.format.motion_xyz32f,200)
prof = p.start(conf)
return p

t0 = time.time()
pipeline = initialize_camera()
print time.time() - t0, "seconds elapsed"

try:
while True:

    frames = pipeline.wait_for_frames()
    depth_frame = frames.get_depth_frame()
    color_frame = frames.get_color_frame()
    if not depth_frame or not color_frame:
        continue

    depth_image = np.asanyarray(depth_frame.get_data())
    color_image = np.asanyarray(color_frame.get_data())

    depth_colormap = cv2.applyColorMap(cv2.convertScaleAbs(depth_image, alpha=0.03), cv2.COLORMAP_JET)

    images = np.hstack((color_image, depth_colormap))

    cv2.namedWindow('RealSense', cv2.WINDOW_AUTOSIZE)
    cv2.imshow('RealSense', images)
    cv2.waitKey(1)

finally:

pipeline.stop()

@tRosenflanz
Copy link
Author

tRosenflanz commented Apr 10, 2019

The delay you print out is for pipeline starting - there is no issue there (it does take a couple seconds but not a real issue). The problem is in first call to pipeline.wait_for_frames which timeouts and from experiments takes more than even 5mins to get the first frame.

The actual code we use, streams the frames to the .bag file and when we try to open it via realsense viewer it errors on every frame (viewer quickly skips through the file showing white frames)

@RealSenseCustomerSupport
Copy link
Collaborator


Ok, here is what I have for streaming first few frames, with the same script but added time check around wait_for_frames().
For my live streaming, it can be seen that my init took ~1.5s, and first frame took 0.4s, and then rest were pretty fast.

1.51300416946
0.401124457214
0.00830704919095
0.0463328220528
0.046513210482
0.0484953228629
0.0510607271706
0.054194841107
0.0591895482728
0.0490699734278
0.0521916654065
0.0562293417443
0.0443453088207
0.0481685713668
0.0539350601655
0.0575914364104

Can you try it?
Or you can share your script and I will test.

Thanks!

@beomsik-landingai
Copy link

I tested the code posted here except updating print to print (..), and I got timeout.

0.49139833450317383 seconds elapsed
Traceback (most recent call last):
File "test.py", line 25, in
frames = pipeline.wait_for_frames()
RuntimeError: Frame didn't arrived within 5000

Given that you used print without parentheses, you are using python 2, and my python version is 3.6.5.
Could you test it with python 3 if it could make any difference? @RealSenseCustomerSupport

Thanks.

@beomsik-landingai
Copy link

just fyi, I've just tried it on python 2.7 as well, still got the same time out error.

@lramati
Copy link
Contributor

lramati commented Apr 11, 2019

After further testing, this appears to be an initialization issue, adding a call to time.sleep between p = initialize_camera() and f = p.wait_for_frames(100) fixed the problems for me on Python 3, even with all 5 streams enabled

@tRosenflanz
Copy link
Author

I will double check but I have tried p.wait_forframes(1000) before without any luck. Also in the final code we stream to .bag so it starts filling up with corrupted frames right away

@lramati
Copy link
Contributor

lramati commented Apr 11, 2019 via email

@tRosenflanz
Copy link
Author

We confirmed that putting a manual short sleep after pipeline start doesn't solve the issue for us. Did this consistently work on your side or just once?

@RealSenseCustomerSupport
Copy link
Collaborator


Just to reply to what you asked last time:

With my test code, off my windows 10, python3 has no issue. Neither for same test off Ubutnu16.04.

C:\rs_2.19.1\wrappers\python\examples>python3 d435i_test.py3
_frozen_importlib:219: FutureWarning: pybind11-bound class 'pyrealsense2.processing_block' is using an old-style placement-new 'init' which has been deprecated. See the upgrade guide in pybind11's docs. This message is only visible when compiled in debug mode.
_frozen_importlib:219: FutureWarning: pybind11-bound class 'pyrealsense2.filter' is using an old-style placement-new 'init' which has been deprecated. See the upgrade guide in pybind11's docs. This message is only visible when compiled in debug mode.
1.8543790101319968
0.39674976776339954
0.0015268206269309914
0.009456890405928231
0.05121627168441778
0.05396746527252727
0.05318272159692361

Thanks!

@tRosenflanz
Copy link
Author

We discovered that specifically 848x480 resolution doesn't work for us (neither 15 nor 30 FPS). It doesn't work in realsense-viewer either, Possibly a camera issue?

@RealSenseCustomerSupport
Copy link
Collaborator


Hi tRosenflanz,

Thanks for the update.
If you have issue with stream 848x480 15/30fps with RS Viewer, that seems to be abnormal.
And this is still with the D435i? What FW version and RS release you have with it?

BTW, do you still have the timeout issue with python?

Thanks!

@tRosenflanz
Copy link
Author

Yes this is D435i exclusively, with no issues on 2 other D435 we have. FW 5.1.14 and RS 2.20

Issues in both Python and RS with 848x480 depth channel exclusively. All other settings work fine and we have switched D435i to collect in 640x480 for now

@RealSenseCustomerSupport
Copy link
Collaborator


Hi tRosenflanz,

Can you run the built-in rs-enumerate-devices tool and share the output?
https://github.com/IntelRealSense/librealsense/tree/master/tools/enumerate-devices

Thanks!

@tRosenflanz
Copy link
Author

tRosenflanz commented Apr 18, 2019

Firmware is 5.1.4 (not 5.1.14 as I posted before)
Here is the output with one D435 and one D435i cameras attached
rs_enum_device.txt

As before: this is an issue on both Windows and Ubuntu machines (2 different machines, not even dual boot)

@RealSenseCustomerSupport
Copy link
Collaborator


Hi tRosenflanz,

Thanks for the update. From the dump, it looks like that there is nothing wrong, and 848x480 is there.

++++++++++++++++++
Device info:
Name : Intel RealSense D435I
Serial Number : 843112070583
Firmware Version : 05.11.04.00
255.255.255.255
Recommended Firmware Version : 05.10.03.00
Physical Port : /sys/devices/pci0000:00/0000:00:14.0/usb2/2-3/2-3:1.0/video4linux/video3
Debug Op Code : 15
Advanced Mode : YES
Product Id : 0B3A
Usb Type Descriptor : 3.2

Stream Profiles supported by Stereo Module
Supported modes:
stream resolution fps format
...
Infrared 1 848x480 @ 90Hz Y8
Infrared 2 848x480 @ 90Hz Y8
Infrared 1 848x480 @ 60Hz Y8
Infrared 2 848x480 @ 60Hz Y8
Infrared 2 848x480 @ 30Hz Y8
Infrared 1 848x480 @ 30Hz Y8
Infrared 2 848x480 @ 15Hz Y8
Infrared 1 848x480 @ 15Hz Y8
Infrared 1 848x480 @ 6Hz Y8
Infrared 2 848x480 @ 6Hz Y8
...

What about 848x480 with other frame rate?
Can you please test it off another Linux machine or a Windows 10 machine?

Thanks!

@RealSenseCustomerSupport
Copy link
Collaborator


Hi tRosenflanz,

Wonder if there is any update on this one?

Thanks!

@RealSenseCustomerSupport
Copy link
Collaborator


Hi tRosenflanz,

Any update on this one?
Or if there is nothing else needs help, this will be closed soon.

Thanks!

@tRosenflanz
Copy link
Author

Apologies for disappearing for a bit. It was hard to find a good 3rd machine to test on. I will update back soon

@r06631001
Copy link

r06631001 commented May 6, 2019

Hi:

I'm using C++ on windows 10. Below is my setting:

Camera Model | D435i
Firmware Version | 05.10.13
Operating System & Version | Windows 10
Platform | PC
SDK Version | 2.21.0
Language | C++ (with Qt creator)

And my realsense initialization is as below:
cfg.enable_stream(RS2_STREAM_GYRO, RS2_FORMAT_MOTION_XYZ32F, 200);
cfg.enable_stream(RS2_STREAM_ACCEL, RS2_FORMAT_MOTION_XYZ32F, 250);
cfg.enable_stream(RS2_STREAM_DEPTH, 848, 480, RS2_FORMAT_Z16, 15);
cfg.enable_stream(RS2_STREAM_COLOR, 848, 480, RS2_FORMAT_BGR8, 15);
cfg.enable_stream(RS2_STREAM_INFRARED, 848, 480, RS2_FORMAT_Y8, 15);

when i used the python code provided by RealSenseCustomerSupport, it works.
However, when i run it under C++ with Qt creator, the first call to pipeline.wait_for_frames timeouts.

Is their any solution to this?
Thanks a lot!

@RealSenseCustomerSupport
Copy link
Collaborator


Hi r06631001,

A quick test from me (by plugging in following lines to the built-in example (rs-capture) with Rs 2.21.0 release) shows that no issue to run it.
Can you please check with RS Viewer or try again?

rs2::config config;
int width = 848, height = 480, fps = 60;
config.enable_stream(RS2_STREAM_COLOR, width, height, RS2_FORMAT_BGR8, fps);
config.enable_stream(RS2_STREAM_DEPTH, width, height, RS2_FORMAT_Z16, fps);
config.enable_stream(RS2_STREAM_INFRARED, width, height, RS2_FORMAT_Y8, fps);
config.enable_stream(RS2_STREAM_GYRO, RS2_FORMAT_MOTION_XYZ32F, 200);
config.enable_stream(RS2_STREAM_ACCEL, RS2_FORMAT_MOTION_XYZ32F, 250);

Thanks!

@r06631001
Copy link

Hi,
I tried it on visual studio and it works. However, when I use the same code in Qt 5.10.0 or Qt 5.12.3 with visual studio compiler, it doesn't work with gyro and accelerator stream, only work with color, depth and infrared stream. Is there any conflict between Qt and the realsense library settings for IMU? Or any other solution to this issues?

Thanks a lot!

@RealSenseCustomerSupport
Copy link
Collaborator


Hi r06631001,

Thanks for the update and confirming the result off VisualStudio.
For issues off Qt, double check again, is it the case that built lib files for python wrapper do NOT work as built lib files for python wrapper? Or it is something else?
As default to be VisualStudio. if just for building lib files for python wrapper, would it be possible for you to switch to VisualStudio?

Thanks!

@RealSenseCustomerSupport
Copy link
Collaborator


Hi r06631001,

For issues off Qt, double check again, is it the case that built lib files for python wrapper do NOT work as built lib files for python wrapper off VisutalStudio? Or it is something else?

Thanks!

@RealSenseCustomerSupport
Copy link
Collaborator


Hi r06631001,

Any updates on this one? Or if you still need help?

Thanks!

@Ezward
Copy link

Ezward commented Jan 13, 2020

I found using python that if I put the imu (accel, gyro) streams in their own pipeline (leaving rgb and depth to share another pipeline) that there was usually no delay, but once in a while it would error on the first frame using the default 5 second timeout. See code in issue 5628

@Ezward
Copy link

Ezward commented Jan 13, 2020

I found when I cranked the frame wait time to 100ms for frames after the first frame that is worked well (if it started, which it mostly did), but if I moved the mouse there was a decent chance of it timing out. If it did not move the mouse, all was well. I'm asking for 30 frames per second for image and depth, 63 for accel and 200 for gyro; I'm getting good frame rates. The mouse driver might suck, but it's probably worth noting.

@Nimaro76
Copy link

Hello, I tried your code : import pyrealsense2 as rs
import numpy as np
import time
import cv2

def initialize_camera():
p = rs.pipeline()
conf = rs.config()
CAM_WIDTH, CAM_HEIGHT, CAM_FPS = 640,480,30
conf.enable_stream(rs.stream.depth, CAM_WIDTH, CAM_HEIGHT, rs.format.z16, CAM_FPS)
conf.enable_stream(rs.stream.color, CAM_WIDTH, CAM_HEIGHT, rs.format.rgb8, CAM_FPS)
conf.enable_stream(rs.stream.infrared, CAM_WIDTH, CAM_HEIGHT, rs.format.y8, CAM_FPS)
conf.enable_stream(rs.stream.accel,rs.format.motion_xyz32f,250)
conf.enable_stream(rs.stream.gyro,rs.format.motion_xyz32f,200)
prof = p.start(conf)
return p

pipeline = initialize_camera()

try:
while True:
frames = pipeline.wait_for_frames()
depth_frame = frames.get_depth_frame()
color_frame = frames.get_color_frame()
if not depth_frame or not color_frame:
continue

    depth_image = np.asanyarray(depth_frame.get_data())
    color_image = np.asanyarray(color_frame.get_data())

    depth_colormap = cv2.applyColorMap(cv2.convertScaleAbs(depth_image, alpha=0.03), cv2.COLORMAP_JET)

    images = np.hstack((color_image, depth_colormap))

    cv2.namedWindow('RealSense', cv2.WINDOW_AUTOSIZE)
    cv2.imshow('RealSense', images)
    cv2.waitKey(1)

finally:

pipeline.stop()

but I get this error: Traceback (most recent call last):

File ~\anaconda3\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
exec(code, globals, locals)

File c:\users\nimar\desktop\test.py:25
pipeline = initialize_camera()

File c:\users\nimar\desktop\test.py:21 in initialize_camera
prof = p.start(conf)

RuntimeError: Couldn't resolve requests
can you please help me and tell me what might be the issue? thank you

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Jul 15, 2023

Hi @Nimaro76 The error Couldn't resolve requests means that your 'conf' stream configuration instructions are requesting a configuration that the camera is unable to provide. An example of why this may occur is if the camera is being detected as being on a USB 2.1 connection instead of USB 3.2, as USB2 supports a more limited range of resolution and FPS combinations than a USB3 connection.

It is likely that it is the IMU frequencies that are causing the problem. The IMU component inside the camera was changed from around April 2022 onwards and the minimum supported Accel rate of that updated IMU is different to the original IMU component.

Old cameras: use 63 Accel, 200 gyro
Cameras manufactured after April 2022: use.100 Accel, 200 Gyro

If you need the script to be able to work with any D435i model regardless of its age then you could try removing the FPS definition and allow the camera to automatically use the default supported Accel and Gyro minimums of that particular camera.

conf.enable_stream(rs.stream.accel,rs.format.motion_xyz32f)
conf.enable_stream(rs.stream.gyro,rs.format.motion_xyz32f)

@Nimaro76
Copy link

I tried these options but I still have the same error:
File ~\anaconda3\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
exec(code, globals, locals)

File c:\users\nimar\desktop\untitled0.py:24
pipeline = initialize_camera()

File c:\users\nimar\desktop\untitled0.py:21 in initialize_camera
prof = p.start(conf)

RuntimeError: Couldn't resolve requests

@MartyG-RealSense
Copy link
Collaborator

And the same error occurs if you put the values directly into the conf instructions?

conf.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
conf.enable_stream(rs.stream.color, 640, 480, rs.format.rgb8, 30)
conf.enable_stream(rs.stream.infrared, 640, 480, rs.format.y8, 30)

You can test whether the conf instructions are the cause of the error by removing conf from the pipe start instruction so that the script ignores the conf lines and applies the camera's default configuration for the streams instead.

prof = p.start()

@Nimaro76
Copy link

my issue got resolved, thank you

@MartyG-RealSense
Copy link
Collaborator

It's great to hear that your issue was resolved, @Nimaro76 - thanks very much for the update!

@jrecasens
Copy link

@Nimaro76 How did you resolve the issue? I am facing the same problem, I cannot stream IMU with depth and color streams.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants