Getting error while runnig process of ProcessingBlock and PointCloud at same time

[omwen13]

Required Info
Camera Model	D435i
Firmware Version	05.13.00.50
Operating System & Version	Win (8.1/10)
Platform	PC
SDK Version	2.49.0.3474
Language	C#
Segment	Robot

Issue Description

I want to getting depth data from camera and then apply few filter the depth data and get point cloud using the depth data.
I wrote the code and get System.InvalidOperationException: error. The error is fixes when I disable getting point cloud and enable filter process or the exact opposite disable apply filter and enable getting point cloud.

        block = new CustomProcessingBlock((f, src) =>
        {
            System.Diagnostics.Debug.WriteLine("Start processing ");

            using (var releaser = new FramesReleaser())
            {
                foreach (ProcessingBlock p in blocks)
                    f = p.Process(f).DisposeWith(releaser);

                f = f.ApplyFilter(Decimation).DisposeWith(releaser);
                f = f.ApplyFilter(Spatial).DisposeWith(releaser);
                var frames = f.As<FrameSet>().DisposeWith(releaser);

                src.FrameReady(frames);
            }
        });

        block.Start(f =>
        {
            using (var frames = f.As<FrameSet>())
            {
                VideoFrame colorFrame = frames.ColorFrame.DisposeWith(frames);
                VideoFrame depthFrame = frames.DepthFrame.DisposeWith(frames);

                System.Diagnostics.Debug.WriteLine("Start point cloud ");

                PointCloud pc = new PointCloud();
                Points points = pc.Process(depthFrame).As<Points>();
            }
        });

[MartyG-RealSense]

Hi @omwen13 Does it make any difference if you move the post-processing filter lines to above the "Start point cloud" line?

f = f.ApplyFilter(Decimation).DisposeWith(releaser);
f = f.ApplyFilter(Spatial).DisposeWith(releaser);

PointCloud pc = new PointCloud();
Points points = pc.Process(depthFrame).As();

[omwen13]

Hi @MartyG-RealSense , I tried this code the error line is changed. I get the error in phase of ApplyFilter.

        block = new CustomProcessingBlock((f, src) =>
        {
            System.Diagnostics.Debug.WriteLine("Start processing   " + DateTime.Now);

            src.FrameReady(f);

        });

        block.Start(f =>
        {
            using (var frames = f.As<FrameSet>())
            {

                using (var releaser = new FramesReleaser())
                {
                    foreach (ProcessingBlock p in blocks)
                        f = p.Process(f).DisposeWith(releaser);

                    f = f.ApplyFilter(Decimation).DisposeWith(releaser);
                    f = f.ApplyFilter(Spatial).DisposeWith(releaser);

                    var frames1 = f.As<FrameSet>().DisposeWith(releaser);

                    VideoFrame colorFrame = frames1.ColorFrame.DisposeWith(frames);
                    VideoFrame depthFrame = frames1.DepthFrame.DisposeWith(frames);

                    depthFrame.ApplyFilter(Decimation);

                    System.Diagnostics.Debug.WriteLine("Start point cloud     " + DateTime.Now);

                    PointCloud pc = new PointCloud();
                    Points points = pc.Process(depthFrame).As<Points>();
                }


            }

The console log output

Started realsense 2.08.2022 16:02:39
received new frame 2.08.2022 16:02:40
Start processing 2.08.2022 16:02:40
Start point cloud 2.08.2022 16:02:40
received new frame 2.08.2022 16:02:40
Start processing 2.08.2022 16:02:40
Start point cloud 2.08.2022 16:02:40
received new frame 2.08.2022 16:02:40
Start processing 2.08.2022 16:02:40
Start point cloud 2.08.2022 16:02:40
received new frame 2.08.2022 16:02:40
Start processing 2.08.2022 16:02:40
Start point cloud 2.08.2022 16:02:40
received new frame 2.08.2022 16:02:40
Start processing 2.08.2022 16:02:40
Start point cloud 2.08.2022 16:02:40
received new frame 2.08.2022 16:02:40
Start processing 2.08.2022 16:02:40
Start point cloud 2.08.2022 16:02:40
received new frame 2.08.2022 16:02:40
Start processing 2.08.2022 16:02:40
Start point cloud 2.08.2022 16:02:40
received new frame 2.08.2022 16:02:40
Start processing 2.08.2022 16:02:40
Start point cloud 2.08.2022 16:02:40
received new frame 2.08.2022 16:02:40
Start processing 2.08.2022 16:02:40
Exception thrown: 'System.InvalidOperationException' in Intel.Realsense.dll
An unhandled exception of type 'System.InvalidOperationException' occurred in Intel.Realsense.dll
Error while running DecimationFilter processing block, check the log for info
'RealsenseTest.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\Remote Debugger\x64\Runtime\Microsoft.VisualStudio.Debugger.Runtime.NetCoreApp.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

[MartyG-RealSense]

Thanks very much for testing moving the filter code. Does the script still have errors if the decimation filter is commented out but the spatial filter is kept enabled?

[omwen13]

Previous error is back.

[MartyG-RealSense]

Is the project able to function well without using the filters at all?

May I ask why a custom processing block is being used instead of a standard pipeline like in the RealSense SDK's 'Generics' pointcloud example in the C# Cookbook page of examples, please?

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/csharp/Documentation/cookbook.md#generics-support

[omwen13]

Its normally working when disable the filter process lines as the following examle with standart pipeline.

        block = new CustomProcessingBlock((f, src) =>
        {
            using (var releaser = new FramesReleaser())
            {

                //System.Diagnostics.Debug.WriteLine("Start processing   " + DateTime.Now);
                //foreach (ProcessingBlock p in blocks)
                //    f = p.Process(f).DisposeWith(releaser);

                //f = f.ApplyFilter(Decimation).DisposeWith(releaser);
                //f = f.ApplyFilter(Spatial).DisposeWith(releaser);

                var frames1 = f.As<FrameSet>().DisposeWith(releaser);

                src.FrameReady(frames1);
            }


        });

        block.Start(f =>
        {
            using (var frames = f.As<FrameSet>())
            {

                VideoFrame colorFrame = frames.ColorFrame.DisposeWith(frames);
                VideoFrame depthFrame = frames.DepthFrame.DisposeWith(frames);

                System.Diagnostics.Debug.WriteLine("Start point cloud     " + DateTime.Now);

                PointCloud pc = new PointCloud();

                using (var points = pc.Process(depthFrame).As<Points>())
                {
                    var vertices = new float[points.Count * 3];
                    points.CopyVertices(vertices);
                }
            }
        });

When i uncomment out filter lines. I get same error.

        block = new CustomProcessingBlock((f, src) =>
        {
            using (var releaser = new FramesReleaser())
            {

                System.Diagnostics.Debug.WriteLine("Start processing   " + DateTime.Now);
                foreach (ProcessingBlock p in blocks)
                    f = p.Process(f).DisposeWith(releaser);

                f = f.ApplyFilter(Decimation).DisposeWith(releaser);
                f = f.ApplyFilter(Spatial).DisposeWith(releaser);

                var frames1 = f.As<FrameSet>().DisposeWith(releaser);

                src.FrameReady(frames1);
            }


        });

        block.Start(f =>
        {
            using (var frames = f.As<FrameSet>())
            {

                VideoFrame colorFrame = frames.ColorFrame.DisposeWith(frames);
                VideoFrame depthFrame = frames.DepthFrame.DisposeWith(frames);

                System.Diagnostics.Debug.WriteLine("Start point cloud     " + DateTime.Now);

                PointCloud pc = new PointCloud();

                using (var points = pc.Process(depthFrame).As<Points>())
                {
                    var vertices = new float[points.Count * 3];
                    points.CopyVertices(vertices);
                }
            }
        });

[MartyG-RealSense]

There was another C# user at #2809 (comment) who was trying to implement post-processing filters using CustomProcessingBlock. A RealSense team member gave advice at #2809 (comment) about implementing the filters in C#.

[omwen13]

The error fixed thanks @MartyG-RealSense for helping. The error reason is

            foreach (ProcessingBlock p in blocks)
                f = p.Process(f).DisposeWith(releaser);

above code parts (given the example of processing in csharp example repo). Using with process of point cloud and apply filter, causing the error. In addition above code parts apply all filter on depth image.

Working code parts:

        block = new CustomProcessingBlock((f, src) =>
        {
            using (var releaser = new FramesReleaser())
            {
                f = f.ApplyFilter(Decimation).DisposeWith(releaser);
                f = f.ApplyFilter(Spatial).DisposeWith(releaser);

                var frames1 = f.As<FrameSet>().DisposeWith(releaser);
                src.FrameReady(frames1);
            }
        });

        block.Start(f =>
        {
            using (var frames = f.As<FrameSet>())
            {
                VideoFrame colorFrame = frames.ColorFrame.DisposeWith(frames);
                VideoFrame depthFrame = frames.DepthFrame.DisposeWith(frames);

                PointCloud pc = new PointCloud();
                Points points = pc.Process(depthFrame).As<Points>();

            }
        });

[MartyG-RealSense]

It's great to hear that you achieved a solution, @omwen13 - thanks very much for the update and for sharing your C# solution with the RealSense community!