Further intermediate / advanced C# examples

[ThunderChild-001]

---

Required Info
Camera Model	D415 / D435
Firmware Version	(05.10.06)
Operating System & Version	Win 10
Platform	PC
SDK Version	2.16.5
Language	C#
Segment	{Robot/Smartphone/VR/AR/others }

Issue Description

I just started getting to know the D415 and the RealSense SDK a few days ago. Since my main programming language is C# I was interested in finding and adding more examples to the existing C# example base.

Specifically more advanced examples like the rs-measure and object tracking found in the C++ examples would be nice. Currently the C# examples are quite basic and having more intermediate and advanced examples would make getting into RealSense easier.

Having seen what was included in the old RealSense interms of object tracking it would be welcoming to see some of this in RealSense2 SDK as open freeware.

[dorodnic]

Hi @KarlAbson
That's a valid request.
These examples would still require 3rd-party components, such as OpenCV.
The goal of this project is to offer best access to Intel depth-sensing and motion-tracking hardware. Proprietary computer vision algorithms that used to be part of the SDK has been removed from our offering in favor of wider deployment and open code.

Anyway, I agree that rs-measure, rs-align and rs-dnn would benefit from being translated to .NET as well. We will try to add this in future releases. For now, if you have any questions or run into problems, feel free to open more issues.

[ThunderChild-001]

Hi @dorodnic
Sounds great! That would be really helpful. 3rd-party open source components would be no problem :-)

[ThunderChild-001]

Hi @dorodnic

I'm currently re-coding the rs-measure in C#. I'm happy to contribute this once it's finished.
I'm just having one or two problems however...I cannot seem to figure out how to do certain things in C#. In C++ it was very easy.

• Firstly setting up the filter options. In C++ it was as follows.
  decimate.set_option(RS2_OPTION_FILTER_MAGNITUDE, 1);
  spat.set_option(RS2_OPTION_HOLES_FILL, 5);
  spat.set_option(RS2_OPTION_FILTER_MAGNITUDE, 5);
  spat.set_option(RS2_OPTION_FILTER_SMOOTH_ALPHA, 1);
  spat.set_option(RS2_OPTION_FILTER_SMOOTH_DELTA, 50);

• Selecting the cameras pre-set. In c++:
  auto range = sensor.get_option_range(RS2_OPTION_VISUAL_PRESET);
  for (auto i = range.min; i < range.max; i += range.step)
  if (std::string(sensor.get_option_value_description(RS2_OPTION_VISUAL_PRESET, i)) == "High Density")
  sensor.set_option(RS2_OPTION_VISUAL_PRESET, i);

• Applying the path finding. Not sure if this is needed?
  pathfinding_queue.enqueue(data.get_depth_frame());

Would it be possible to get a little help with the C# equivalents for these.
After this it's just a case of building in the measuring tool. Currently thinking of the best solution for this.

Many thanks,
Karl

[ThunderChild-001]

I just ran a test of the current code. For some reason the distance I calculate is wrong and constantly changing. I'm taking the co-ordinates given in the C++ example manually to test so they should be the same. I must have missed a step either setting up the camera or in calculating....

The result I get from both depth_frame.GetDistance_3d(Start, Stop, depthintr);
and the class I wrote is the same... HelperClass.GetDistance_3d(depth_frame, Start, Stop, depthintr);

I was wondering if anyone could see anything?

Align align_to = new Align(Stream.Depth);
DecimationFilter decimate = new DecimationFilter();
SpatialFilter spatial = new SpatialFilter();
TemporalFilter temp = new TemporalFilter();
HoleFillingFilter holeFill = new HoleFillingFilter();
Colorizer colorizer = new Colorizer();

var cfg = new Config();
cfg.EnableStream(Stream.Depth, 1280, 720, Format.Z16, 30);
cfg.EnableStream(Stream.Color, 1280, 720, Format.Rgb8, 30);

Pipeline pipeline = new Pipeline();
var pp = pipeline.Start(cfg);

block = new CustomProcessingBlock((f, src) =>
{
using (var releaser = new FramesReleaser())
{
using (var frames = pipeline.WaitForFrames().DisposeWith(releaser))
{
var processedFrames = frames
.ApplyFilter(align_to).DisposeWith(releaser)
.ApplyFilter(decimate).DisposeWith(releaser)
.ApplyFilter(spatial).DisposeWith(releaser)
.ApplyFilter(temp).DisposeWith(releaser)
.ApplyFilter(holeFill).DisposeWith(releaser)
.ApplyFilter(colorizer).DisposeWith(releaser);

                        // Send it to the next processing stage
                        src.FramesReady(processedFrames);
                    }
                }
            });

// Register to results of processing via a callback:
block.Start(f =>
{
using (var releaser = new FramesReleaser())
{
// Align, colorize and upload frames for rendering
var frames = FrameSet.FromFrame(f, releaser);

                    //FrameSet.FromFrame(f.DisposeWith(releaser))

                    var color_frame = FramesReleaser.ScopedReturn(releaser, frames.ColorFrame);
                    var depthintr = (pp.GetStream(Stream.Depth) as VideoStreamProfile).GetIntrinsics();
                    var depth_frame = FramesReleaser.ScopedReturn(releaser, frames.DepthFrame);

                    PointF Start = new PointF(540, 264);
                    PointF Stop = new PointF(863, 246);

                    float k1 = depth_frame.GetDistance_3d(Start, Stop, depthintr);
                    float k2 = HelperClass.GetDistance_3d(depth_frame, Start, Stop, depthintr);
                    System.Diagnostics.Debug.WriteLine("k1: " + k1 + " k2: " + k2);

                    UploadImage(imgColor, color_frame);
                }
            });


    public static float GetDistance_3d(this DepthFrame frame, PointF from, PointF to, Intrinsics intr)
    {
        // Query the frame for distance
        // Note: this can be optimized
        // It is not recommended to issue an API call for each pixel
        // (since the compiler can't inline these)
        // However, in this example it is not one of the bottlenecks

        float udist = frame.GetDistance((int)@from.X, (int)@from.Y); //From
        float vdist = frame.GetDistance((int)to.X, (int)to.Y); //To

        // Deproject from pixel to point in 3D
        var upoint = DeprojectPixelToPoint(intr, from, udist);
        var vpoint = DeprojectPixelToPoint(intr, to, vdist);

        // Calculate euclidean distance between the two points
        return (float)Math.Sqrt(Math.Pow(upoint[0] - vpoint[0], 2) +
                                 Math.Pow(upoint[1] - vpoint[1], 2) +
                                 Math.Pow(upoint[2] - vpoint[2], 2));
    }

    static float[] DeprojectPixelToPoint(Intrinsics intrin, PointF pixel, float depth)
    {
        Debug.Assert(intrin.model != Distortion.ModifiedBrownConrady); // Cannot deproject from a forward-distorted image
        Debug.Assert(intrin.model != Distortion.Ftheta); // Cannot deproject to an ftheta image

        var ret = new float[3];
        float x = (pixel.X - intrin.ppx) / intrin.fx;
        float y = (pixel.Y - intrin.ppy) / intrin.fy;
        if (intrin.model == Distortion.InverseBrownConrady)
        {
            float r2 = x * x + y * y;
            float f = 1 + intrin.coeffs[0] * r2 + intrin.coeffs[1] * r2 * r2 + intrin.coeffs[4] * r2 * r2 * r2;
            float ux = x * f + 2 * intrin.coeffs[2] * x * y + intrin.coeffs[3] * (r2 + 2 * x * x);
            float uy = y * f + 2 * intrin.coeffs[3] * x * y + intrin.coeffs[2] * (r2 + 2 * y * y);
            x = ux;
            y = uy;
        }
        ret[0] = depth * x;
        ret[1] = depth * y;
        ret[2] = depth;
        return ret;
    }

[dorodnic]

Hi @KarlAbson

1. Setting options in C#:

var dec = new DecimationFilter();
dec.Options[Option.FilterMagnitude].Value = 3.0F;

2. I see now that the C# version of ValueDescription is a bit broken, so there is no way to do what C++ version is doing. You can however hard-code the preset option value.

3. pathfinding_queue.enqueue(data.get_depth_frame()); - this technically has little to do with RealSense. You just really don't want to perform heavy processing on your UI thread. FrameQueue object is a convenient way to solve this, but you can use any other tool, such as Task objects, or await keyword.

[ThunderChild-001]

Hi @dorodnic

Many thanks :-) Filters are now set up correctly and the correct measurements are being given.
How does one go about hard-coding the preset option? Is this using a Context instance and by QueryDevices etc?

[ogoshen]

1. I see now that the C# version of ValueDescription is a bit broken, so there is no way to do what C++ version is doing. You can however hard-code the preset option value.

See IOptionsContainer.OptionValueDescription and Sensor.CameraOption.GetValueDescription

string current_value_desc = dec.Options[Option.FilterMagnitude].ValueDescription;
string some_other_value_desc =  dec.Options[Option.FilterMagnitude].GetValueDescription(0f);
string from_options_container =  dec.Options.OptionValueDescription(Option.FilterMagnitude, 0f);

[ThunderChild-001]

Hi @dorodnic

Here is the little measuring sample project that I made. Please feel free to use it.
It gives accurate measurements but it currently does not use the High Density preset.

The code for loading that is commented out (lines 83 - 86).
For some reason including that leads to an eventual crash (access violation exception).
You can re-create it by either just waiting and eventually it will happen, or if you start moving the measuring buttons.

If you can think of any improvements let me know.

RS-DevProjectC#.zip

[JANGSOONMYUN]

Hi @KarlAbson ,

I'm trying to build DLL file of the realsense SDK, but I've failed.
Could you tell me how to make DLL file?

In your code, there are
Intel.Realsense.dll
realsense2.dll
And they work on x86 system. But I want to make it for x64 system.

[ThunderChild-001]

Hi @JANGSOONMYUN ,

I downloaded the master thread and built this using CMake as per the instructions online. I have not used the 64bit option but I don't believe this should be so hard to setup. I'm afraid I cannot instruct you on this however. If you have troubles you should be able to find both x64 and x86 dlls under C:\Program Files (x86)\Intel RealSense SDK 2.0\bin once you have installed the drivers / sdk. You can just use these and then you don't have any need to build them yourself.

[JANGSOONMYUN]

Hi, Kal Abson, Thank you for your answer! However, I've tried to use dll files on C:\Program Files (x86)\Intel RealSense SDK 2.0\bin , but it doesn't work... Error message is: System.DllNotFoundException: 'DLL 'realsense2' I've clicked References of the C# project and added [Intel.Realsense.dll]. But I didn't add [realsense2.dll] because it couldn't be added. This dll is not accessible. And then start to build it, but failed. And also I've tried to use CMake. I followed this site: https://cccommunications.com/intel-realsense-3d-depth-camera-sdk-setup-for-c-development/ After doing this, I've open and build \librealsense\build\INSTALL.vcxproj. Finally, I got [Intel.Realsense.dll] and [realsense2.dll] So, I also tested it but it didn't work... just same error as before. If you have any solution, please let me know. Best Regards, Jang Soonmyun 2019년 4월 9일 (화) 오후 8:10, KarlAbson <notifications@github.com>님이 작성:

…

Hi @JANGSOONMYUN <https://github.com/JANGSOONMYUN> , I downloaded the master thread and built this using CMake as per the instructions online. I have not used the 64bit option but I don't believe this should be so hard to setup. I'm afraid I cannot instruct you on this however. If you have troubles you should be able to find both x64 and x86 dlls under C:\Program Files (x86)\Intel RealSense SDK 2.0\bin once you have installed the drivers / sdk. You can just use these and then you don't have any need to build them yourself. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2809 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIc6OKNTnx-ZlcLTh-hiwCs-V-Rb1IsJks5vfIMxgaJpZM4Y74qu> .

[JANGSOONMYUN]

Hi, Kal Abson, I've solved the problem. I've placed dll files on wrong directory. Thank you! 2019년 4월 9일 (화) 오후 9:37, Soonmyun Jang <jsm890803.3@gmail.com>님이 작성:

…

Hi, Kal Abson, Thank you for your answer! However, I've tried to use dll files on C:\Program Files (x86)\Intel RealSense SDK 2.0\bin , but it doesn't work... Error message is: System.DllNotFoundException: 'DLL 'realsense2' I've clicked References of the C# project and added [Intel.Realsense.dll]. But I didn't add [realsense2.dll] because it couldn't be added. This dll is not accessible. And then start to build it, but failed. And also I've tried to use CMake. I followed this site: https://cccommunications.com/intel-realsense-3d-depth-camera-sdk-setup-for-c-development/ After doing this, I've open and build \librealsense\build\INSTALL.vcxproj. Finally, I got [Intel.Realsense.dll] and [realsense2.dll] So, I also tested it but it didn't work... just same error as before. If you have any solution, please let me know. Best Regards, Jang Soonmyun 2019년 4월 9일 (화) 오후 8:10, KarlAbson ***@***.***>님이 작성: > Hi @JANGSOONMYUN <https://github.com/JANGSOONMYUN> , > > I downloaded the master thread and built this using CMake as per the > instructions online. I have not used the 64bit option but I don't believe > this should be so hard to setup. I'm afraid I cannot instruct you on this > however. If you have troubles you should be able to find both x64 and x86 > dlls under C:\Program Files (x86)\Intel RealSense SDK 2.0\bin once you have > installed the drivers / sdk. You can just use these and then you don't have > any need to build them yourself. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#2809 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AIc6OKNTnx-ZlcLTh-hiwCs-V-Rb1IsJks5vfIMxgaJpZM4Y74qu> > . >