Realsense2-net for Unity

[roeelie]

---

Required Info
Camera Model	D455/D435
Firmware Version	05.12.09.00
Operating System & Version	Win 10
Platform	PC/Raspberry Pi
SDK Version	v2.41.0
Language	unity (C#)
Segment	Network/Unity

Issue Description

I am trying to use a D455 camera with a raspberry pi 4 over ethernet using this guide: https://dev.intelrealsense.com/docs/open-source-ethernet-networking-for-intel-realsense-depth-cameras

I have succesfully build the source and added C# wrapper for rs2_device* rs2_create_net_device(int api_version, const char* address, rs2_error** error); similar to this post This works well for a regular c# program.

My question is: Is it possible to use realsense2-net devices in Unity. I've been looking into the c# wrapper code but can't figure out where to start. My endgoal is to use multiple camera's and use their pointclouds in Unity (over ethernet)

[MartyG-RealSense]

Hi @roeelie I will tag into this discussion a person called @ogoshen who has experience in questions about interfacing with RealSense wrappers from within Unity. An example of this is a project for connecting Unity to a RealSense camera and an Android-based Oculus Quest VR headset.

#4155

https://github.com/GeorgeAdamon/quest-realsense

There is also this case:

#2931

Hi @ogoshen Would it be possible for you to provide advice to @roeelie about the feasibility of the C# networking project proposed above, please? Thanks!

[ogoshen]

If everything works outside Unity and you're able to use the device with C# then this should be easy enough.

You'll probably want to create a custom class based on RsDevice, that creates the network device.
You should then be able to get the other components and samples working without modification.

[MartyG-RealSense]

Thanks so much for your time @ogoshen :)

[roeelie]

thanks @ogoshen @MartyG-RealSense :)

I got a bit further! I managed to get a stream visible in Unity. But there are some problems arising. Sometimes I just dont get a stream in Unity, although I do see the IR emitters going on. So I think i`m enabling the devices but not correctly starting a stream. Also if i change resolution to another (for example from 640x680 to 480x270 or vice versa) it doesn't work anymore. (My setup currently has 2 net devices - in the viewer application they work fine).

I think the problem has to do with that i pass a context to the pipeline. I created a custom class which is nearly the same as RsDevice like this.

`void OnEnable()
{
    context = new Context();
    dev = new net_device( ip );
    dev.add_to( context );

    m_pipeline = new Pipeline( context );

    cfg = new Config();
    //cfg.EnableDevice( DeviceConfiguration.RequestedSerialNumber );
    cfg.EnableStream( Profile.Stream, Profile.StreamIndex, Profile.Width, Profile.Height, Profile.Format, Profile.Framerate );
    
    ActiveProfile = m_pipeline.Start(cfg);
}`

In RsDevice.cs normally does cfg.EnableDevice(serial) which i can't do, so I guess thats one the issues that going wrong. I haven't figured out how to do this for a net device. Any suggestions?

[MartyG-RealSense]

Hi @roeelie This subject is outside of my programming knowIedge - @ogoshen will be better able to answer this one. I do though recall past efforts to use multicam in Unity, in which it was found that RsDevice was acting as a singleton. So a RealSense user called frenchtouchXR developed a method for enabling multicam in the RealSense Unity wrapper.

#2203
#2333
https://community.intel.com/t5/Items-with-no-label/Can-I-use-multiple-Intel-RealSense-cameras-in-a-single-Unity3d/m-p/684416

[ogoshen]

Glad to hear it works.
I'm sorry but I might not know enough about realsense2-net to help here, do have a few ideas though..

If the first run works, it might be that you aren't disposing the device or other resources, and any further run fails to stream.
That could also explain why switching resolutions would fail, as the device might still be streaming a different configuration.

Multi-cam in Unity should work fine, you can use multiple device components, each configured with a unique serial number.
Why can't you use a serial number? I assume the net-device supports it.. try to grab the numbers from the viewer and put them in Unity.

[MartyG-RealSense]

Hi again @roeelie As you are working on networking, I thought that I would highlight to you the upcoming LRS-Net 2.0 feature if you have not seen it already, as a possible future update for your project.

#8372

[MartyG-RealSense]

Hi @roeelie Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.