help with getting depth at a specific pixel unity

[80sVectorz]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D435
Firmware Version	05.12.07.100
Operating System & Version	Windows 10
Kernel Version (Linux Only)	does not apply then
Platform	PC
SDK Version	2
Language	unity
Segment	AR

Issue Description

<Describe your issue / question / feature request / etc..>

I'm having trouble finding how to read depth at a specified pixel can someone give me a code c# code snippet that reads the depth at a specific pixel or give some advice on how to use it or a link maybe? id really appreciate it.

[MartyG-RealSense]

Hi @krijnlol The link below has a RealSense C# tutorial for mapping a specified 2D point to 3D space, or a specified 3D point to 2D space. The author states that the tutorial can be used with Unity.

https://lightbuzz.com/intel-realsense-coordinate-mapping/

[80sVectorz]

@MartyG-RealSense the problem is that I don't know how to get ahold of a depth frame from the device
the link is still useful i just need to know how to get a depth frame

[MartyG-RealSense]

The SDK's C# wrapper has an introductory Hello World example for getting the depth frame and printing the distance in meters.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/csharp#hello-world

There is also a code snippet in the wrapper's C# Cookbook page of code examples for waiting for a coherent set of depth stream frames.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/csharp/Documentation/cookbook.md#wait-for-coherent-set-of-frames

Are these links of use to you, please?

[80sVectorz]

no not really i need as they are not for the unity wrapper i tried implementing them but they did not work

[MartyG-RealSense]

I can see your problem in regard to getting the depth of a specific coordinate as a numeric value in Unity, as the Unity wrapper applies camera data to a Unity material in order to render it onscreen. I had a case recently about this subject. It ended without a solution being found.

#7524

[80sVectorz]

yea i tried reading the material but i failed i could not find any good documentation on it

[MartyG-RealSense]

Possibly you could access the SDK's C# API from Unity and calculate a value that way, by doing the coordinate calculations outside of Unity as though it were a C# application.

[80sVectorz]

no that does not seem to work

[MartyG-RealSense]

The graphics-based nature of rendering camera data in Unity - including non-RealSense point cloud applications in the Unity Asset Store that lack coordinate displays - suggests that there is not a simple solution for accessing camera data in text or numeric form within Unity.

[mpottinger]

Not sure if this would help you or be what you're looking for but I have used IPC before to get depth frames from a camera that did not have a Unity SDK at all, and transferred the depth data to Unity that way

Any common IPC method would work. For example TCP/IP or UDP over localhost, or Shared Memory depending on the platform.

On Linux it's as simple as using a file in /dev/shm. I have passed data between Python, C++, and Unity that way.

Not pretty but it works.

[MartyG-RealSense]

Thanks so much for your advice @mpottinger

Other approaches for accessing camera data might be to use the OpenCV or ROS interfaces available in the Unity Asset Store, though they are not inexpensive and may provide too much complication to suit your project.

[80sVectorz]

i and what about reading the pixels from the DepthMat material?

[MartyG-RealSense]

I don't think that you could get meaningfully accurate values from a Unity material. It may be like trying to analyse a PNG depth image for depth - the precise depth information would likely be lost when converting the data to an image.

[80sVectorz]

i think it will be fine ass i just need to create a crude mesh from it

[MartyG-RealSense]

Hi @krijnlol Do you still require assistance with this case, please? Thanks!

[80sVectorz]

@MartyG-RealSense well I have been able to get the depth data but only in a delayed coroutine start function when put in an update function it says that the data is null

[MartyG-RealSense]

My research also indicates that coroutines are the correct way to go with this problem (perhaps a looping coroutine to keep the depth reading updating).

https://docs.unity3d.com/Manual/Coroutines.html

You can set a custom refresh rate for Update() such as 1/4 of its normal update speed but this is apparently an unreliable way to do it.

[wodndb]

Do you want to get depth value in depth frame?

I tried to get depth frame and access specific depth value in (x, y) coordinate of depth frame. But, I should convert depth value from byte to Single. It's too slow.

So I decided to get depth value in point cloud frame. Actually point cloud frame is calculated from depth frame. So z value of point cloud is same to depth frame. Also I can get z value without conversion of type.

---

You can find how to get pointcloud frame information in RealSense SDK for Unity (RsPointCloudRenderer.cs)

I recommend to customize this source code to get point cloud frame. You should copy streamed pointcloud frame to some of buffer. Streamed point cloud frame is used for polling, So You should not reference streamed point cloud frame directly.

In RsPointCloudRenderer.cs, You can see generating and visualizing mesh for point cloud. It's not important for you. In this case, LateUpdate method of RsPointCloudRenderer.cs is important part, Points instance copy raw point cloud values to Vector3[]. Let's see the codes.

    protected void LateUpdate()
    {
        if (q != null)
        {
            Points points;
            if (q.PollForFrame<Points>(out points))
                using (points)
                {
                    if (points.Count != mesh.vertexCount)
                    {
                        using (var p = points.GetProfile<VideoStreamProfile>())
                            ResetMesh(p.Width, p.Height);
                    }

                    if (points.TextureData != IntPtr.Zero)
                    {
                        uvmap.LoadRawTextureData(points.TextureData, points.Count * sizeof(float) * 2);
                        uvmap.Apply();
                    }

                    if (points.VertexData != IntPtr.Zero)
                    {
                        points.CopyVertices(vertices);  // <-- Here!

                        mesh.vertices = vertices;
                        mesh.UploadMeshData(false);
                    }
                }
        }
    }

Please copy vertices to another buffer. I used singletone class to save vertices and access from any source codes in Unity.

---

And, point cloud values is saved 1D Vector3 array. But image is 2D. How to access it?

if your target coordinate is (x, y), and image size (width, height) you can access depth value like below.

vertices[(y * width) + x].z

[MartyG-RealSense]

Thanks so much @wodndb for your assistance with this case! :)

[MartyG-RealSense]

Hi @krijnlol Do you require further assistance with this case, please? Thanks!

[80sVectorz]

@MartyG-RealSense no not really thanks to everyone for your help I hope you have a good day (:

[MartyG-RealSense]

Thanks very much @krijnlol for the update - have a great day too!

[ksjryan]

I am a beginner of unity who joined github after seeing the code of @wodndb . There are many things that I don't know, such as how to copy vertices to other buffer and what Singletone class is. Do you have a source code or unity package about it? If you have, can you upload it? Thank you.

I want to study by looking at your code or package. Please help me...

[wodndb]

@ksjryan I can't share source codes right now, sorry. If I have some times to prepare code, I will share it.

Q. How to copy vertices to other buffer?
Create singleton class or static class and create member field to save vertices. Type of vertices is Vector3[]. We can use this member field to buffer.
And replace a variable 'vertices' to the buffer which is member field of singletone class or static class.

Q. What is singletone class?
Singletone is design pattern about class which has only one instance.
https://wiki.unity3d.com/index.php/Singleton

[ksjryan]

@wodndb Thank you very much for your quick reply.
The part you answered was helpful, but I haven't solved it yet.
If you have time later, I really want to get the code. Thank you.

[Jordan5000]

Hi, I'm looking to achieve to solve the same issue. Is there any chance anyone has any code to show me how this is achieved. I also am new to Unity and don't completely understand vertices and singletons or how these would be implemented into the source code. @wodndb @krijnlol

Thanks for any help!

[wodndb]

@Jordan5000 Ok, Could you check this unity project?

https://github.com/wodndb/ExportPlyForRealSense

Maybe you can understand how to get point cloud and color image data and export to ply in this project.

[Jordan5000]

I'll have a look through that. Thanks!

[Jordan5000]

Thanks @wodndb I had a look through your code an managed to get it working. I'm not 100% sure if the measurements its outputting are scaled correctly. Do you know if the distance it is outputting is in meters or in feet?

For the image width would you just use the depth.Width defined in the same source code file RsPointCloudRenderer.cs as shown below. When I find this measurement it says the width is 640. I've also found that it outputs zero at the beginning when the width is set to 640 and only changes once the camera is moved which I find odd.

private void OnStartStreaming(PipelineProfile obj)
{
    q = new FrameQueue(1);

    using (var depth = obj.Streams.FirstOrDefault(s => s.Stream == Stream.Depth && s.Format == Format.Z16).As<VideoStreamProfile>())
        ResetMesh(depth.Width, depth.Height);
        

    Source.OnNewSample += OnNewSample;
}

[wodndb]

@Jordan5000 Maybe depth value from realsense camera is in meter.
https://dev.intelrealsense.com/docs/projection-in-intel-realsense-sdk-20