Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Running three D430 on Coral Dev Board #422

Closed
brunovollmer opened this issue Jul 19, 2021 · 12 comments
Closed

Running three D430 on Coral Dev Board #422

brunovollmer opened this issue Jul 19, 2021 · 12 comments
Assignees
@manoj7410
Labels
comp:thirdparty Thirdparty related issues Hardware:Dev Board Coral Dev Board issues type:support Support question or issue

Comments

@brunovollmer
Copy link

brunovollmer commented Jul 19, 2021 •
edited
Loading

Note: I've created the same Issue on the librealsense repo: IntelRealSense/librealsense#9443

Hey everybody,

I'm not sure if this is the right place but I thought maybe somebody had the same problem or some idea. I'm currently trying to connect three D430 + D4 vision boards to a coral dev board through one USB-C hub. Unfortunately I can't make it work as I either receive a segmentation fault (most of the times), a "Frame did not arrive within 5000" (sometimes) or it can't even find all cameras (disconnect solves that).

I've created a little test script to check if the cameras work:

import pyrealsense2 as rs
import numpy as np
import argparse
import cv2
import time

def parse_inputs():
    parser = argparse.ArgumentParser()
    parser.add_argument('--width',
                        default=848,
                        type=int,
                        help='width of stream')
    parser.add_argument('--height',
                        default=480,
                        type=int,
                        help='height of stream')
    parser.add_argument('--frame_rate',
                        default=30,
                        type=int,
                        help='frame rate of stream')
    parser.add_argument('--frames',
                        default=100,
                        type=int,
                        help='number of frames for the streaming test')

    return parser.parse_args()

def find_cameras(ctx):
    cameras = []
    devices = ctx.devices

    for dev in devices:
        name = dev.get_info(rs.camera_info.name)
        serial_number = dev.get_info(rs.camera_info.serial_number)

        cameras.append({'name': name, 'serial_number': serial_number})

    return cameras

def init_cameras(cameras, width, height, frame_rate):
    pipelines = {}
    
    for cam in cameras:
        serial_number = cam['serial_number']
        name = cam['name']

        pipe = rs.pipeline()
        cfg = rs.config()
        cfg.enable_device(serial_number)

        cfg.enable_stream(rs.stream.infrared,
                          width=width,
                          height=height,
                          format=rs.format.y8,
                          framerate=frame_rate)

        cfg.enable_stream(rs.stream.depth,
                          width=width,
                          height=height,
                          format=rs.format.z16,
                          framerate=frame_rate)

        profile = pipe.start(cfg)
        depth_scale = (
                profile.get_device().first_depth_sensor().get_depth_scale())

        cam['depth_scale'] = depth_scale

        pipelines[serial_number] = {'pipe': pipe, 'camera': cam}
    
    return pipelines

def test_stream(pipelines, nr_frames):

    colorizer = rs.colorizer()

    durations = []

    for i in range(nr_frames):            

        start_time = time.time()

        for serial_number, camera_data in pipelines.items():
            name = camera_data['camera']['name']
            pipe = camera_data['pipe']
            depth_scale = camera_data['camera']['depth_scale']

            frames = pipe.wait_for_frames()
            
            color_frame = frames.get_infrared_frame()

            depth_frame = frames.get_depth_frame()

            color = np.asanyarray(color_frame.get_data())
            color = cv2.cvtColor(color, cv2.COLOR_GRAY2RGB)

            depth_frame_color = np.asanyarray(
                colorizer.colorize(depth_frame).get_data())

            depth_frame = np.asanyarray(depth_frame.get_data())
            depth_frame = depth_frame * depth_scale
        
        end_time = time.time()

        durations.append(end_time - start_time)

    return 1/(sum(durations)/len(durations))

def main():
    args = parse_inputs()

    context = rs.context()

    cameras = find_cameras(context)
    print("Starting Intel Camera Test")
    print("-----------------------------------------------")

    print("Input Args")
    print(args)

    print("-----------------------------------------------")

    print(f"found {len(cameras)} camera(s)")
    print(f"camera details: {cameras}")

    pipelines = init_cameras(cameras, args.width, args.height, args.frame_rate)

    print(f"initialized {len(pipelines.keys())} camera(s)")

    print("-----------------------------------------------")

    print(f"Starting streaming test for {args.frames} frames")
    fps = test_stream(pipelines, args.frames)
    print("Finished streaming test")
    print(f"Average streaming FPS: {fps}")


if __name__ == "__main__":
    main()

The things that I've check so far:

  • Check USB-C hug: The USB-C Hub can handle the bandwidth and power as I can run the same script from my laptop without a problem.
  • Reduce resolution: I've tried to lower the resolution to 640x360 but that does not lead to any improvements unfortunately.
  • Lower frame-rate: Lowered it to 6 FPS but similar to reduced resolution.

I'm a bit lost now as I don't really know what to do to make it work. Unfortunately we have to run all cameras through one USB-C 3.0 port. Any ideas on what to try?
@manoj7410
Copy link

Can you share the output of :

v4l2-ctl --list-devices

v4l2-ctl --list-formats-ext --device /dev/video1

@manoj7410 manoj7410 self-assigned this Jul 20, 2021
@brunovollmer
Copy link
Author 

(env3.8) mendel@zippy-dog:~/soundmap$ v4l2-ctl --list-devices
i.MX6S_CSI (platform:30a90000.csi1_bridge):
	/dev/video0

Intel(R) RealSense(TM) Depth Ca (usb-xhci-hcd.0.auto-1.1):
	/dev/video1
	/dev/video2

Intel(R) RealSense(TM) Depth Ca (usb-xhci-hcd.0.auto-1.2):
	/dev/video3
	/dev/video4

Intel(R) RealSense(TM) Depth Ca (usb-xhci-hcd.0.auto-1.4):
	/dev/video5
	/dev/video6

(env3.8) mendel@zippy-dog:~/soundmap$ v4l2-ctl --list-formats-ext --device /dev/video1
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'Z16 ' (16-bit Depth)
		Size: Discrete 256x144
			Interval: Discrete 0.003s (300.000 fps)
			Interval: Discrete 0.011s (90.000 fps)
		Size: Discrete 424x240
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 480x270
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 848x100
			Interval: Discrete 0.003s (300.000 fps)
			Interval: Discrete 0.010s (100.000 fps)
		Size: Discrete 848x480
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
	[1]: '' (00000050-0000-0010-8000-00aa003)
(env3.8) mendel@zippy-dog:~/soundmap$

@brunovollmer
Copy link
Author

Another thing I realized is that only connecting three but not using all of them leads to problem. So if I connect all three but only initialize two it won't work. I get more and more the feeling the problem is not related to the actual bandwidth but some USB connectivity issue of the coral dev board.

@brunovollmer
Copy link
Author

@manoj7410 Do you think this problem could be related to the kernel and a newer version could fix it?

@brunovollmer
Copy link
Author

Any ideas on this?

@manoj7410
Copy link

I don't see 'YUYV' format supported in your camera. Can you please confirm if your camera supports 'YUYV' ?

@brunovollmer
Copy link
Author

It seems to only support Y8 as a format for the infrared image and Z16 as format for the depth image. We are using D430s:
image

@brunovollmer
Copy link
Author

If I compile librealsense without RUSB support it unfortunately does not work at all, as it only finds two of the three cameras and it can't initialize them. My question is if you have an idea of a possible patch to the native usb driver in the 4.14 kernel to make this work?

@brunovollmer
Copy link
Author

I checked some more and it appears anyways even if the document says otherwise:

ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: '' (00000032-0000-0010-8000-00aa003)
	[1]: 'UYVY' (UYVY 4:2:2)
		Size: Discrete 256x144
			Interval: Discrete 0.003s (300.000 fps)
			Interval: Discrete 0.011s (90.000 fps)
		Size: Discrete 424x240
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 480x270
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 848x100
			Interval: Discrete 0.003s (300.000 fps)
			Interval: Discrete 0.010s (100.000 fps)
		Size: Discrete 848x480
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
	[2]: 'GREY' (8-bit Greyscale)
		Size: Discrete 256x144
			Interval: Discrete 0.003s (300.000 fps)
			Interval: Discrete 0.011s (90.000 fps)
		Size: Discrete 424x240
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 480x270
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 848x100
			Interval: Discrete 0.003s (300.000 fps)
			Interval: Discrete 0.010s (100.000 fps)
		Size: Discrete 848x480
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
	[3]: 'Y8I ' (Interleaved 8-bit Greyscale)
		Size: Discrete 256x144
			Interval: Discrete 0.003s (300.000 fps)
			Interval: Discrete 0.011s (90.000 fps)
		Size: Discrete 424x240
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 480x270
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 848x100
			Interval: Discrete 0.003s (300.000 fps)
			Interval: Discrete 0.010s (100.000 fps)
		Size: Discrete 848x480
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.167s (6.000 fps)
		Size: Discrete 1280x800
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
	[4]: 'Y12I' (Interleaved 12-bit Greyscale)
		Size: Discrete 640x400
			Interval: Discrete 0.040s (25.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
		Size: Discrete 1280x800
			Interval: Discrete 0.040s (25.000 fps)
			Interval: Discrete 0.067s (15.000 fps)

@manoj7410
Copy link

I see YUY2 which is similar to YUYV. See this note at https://coral.ai/docs/dev-board/camera/#connect-a-usb-camera : Be sure that your list includes Pixel Format: 'YUYV'. Currently, YUYV is the only format supported. But the commands below refer to this format with the name YUY2, which is just a different name for the same thing.

Did you try to connect a single camera instead of multiple cameras ?

@brunovollmer
Copy link
Author

Single camera does not work if I don't enable RUSB support when I built the library.

@manoj7410 manoj7410 added comp:thirdparty Thirdparty related issues Hardware:Dev Board Coral Dev Board issues labels Aug 3, 2021
@hjonnala hjonnala added the type:support Support question or issue label Aug 4, 2021
@manoj7410
Copy link

Based on the conversation at IntelRealSense/librealsense#9443 marking this thread as closed for now. Feel free to reopen this issue whenever you need to.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@manoj7410 manoj7410

Labels
comp:thirdparty Thirdparty related issues Hardware:Dev Board Coral Dev Board issues type:support Support question or issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@manoj7410 @brunovollmer @hjonnala