Stack empty exception

[cdrose]

---

Required Info
Camera Model	D415
Firmware Version	05.12.07.100
Operating System & Version	Win 10
Platform	PC
SDK Version	2.38.1.2223
Language	C#
Segment	Machine Vision

I am working on a multi-camera multi-threaded application using the C# wrapper and I get sporadic 'Stack empty' exceptions from librealsense. Sometimes on pipeline.WaitForFrames() but more often on depthFrame = fs.DepthFrame.DisposeWith(fs) or colorFrame = fs.ColorFrame.DisposeWith(fs) lines. The exceptions seem to be non terminal in that I can catch them and skip that frame and the application continues to capture frames and perform as expected.

I gather that the ObjectPool is related to how librealsense reuses memory to avoid garbage collection overhead. I also wonder if perhaps the way I am using the library is a bit unusual in that I am capturing frames, extracting several small regions of interest and then packing them back into Frames before running the depth filtering process. Benchmarking it shows a significant speed up over trying to process the full frame for my application but it probably means I am using more objects at any one time than the average realsense application. Could this be related?

The stack trace from Visual Studio looks like this:

   at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at System.Collections.Generic.Stack`1.Pop()
   at Intel.RealSense.ObjectPool.Get(Type t, IntPtr ptr)
   at Intel.RealSense.ObjectPool.Get[T](IntPtr ptr)
   at Intel.RealSense.Frame.Create[T](Frame other)
   at Intel.RealSense.Frame.Cast[T]()
   at Intel.RealSense.FrameSet.FirstOrDefault[T](Stream stream, Format format)
   at Intel.RealSense.FrameSet.get_ColorFrame()
   at RealSenseTrial.Orientation.RotationUnit.onFrameReceived(Object sender, FrameEventArgs args)

The realsense logs aren't much help:

 09/02 21:57:12,132 DEBUG [28076] (sync.cpp:526) fps 60 Color 7 8818 1612861032106.993164 
 09/02 21:57:12,132 DEBUG [28076] (sync.cpp:539) (TS: I Depth I Infrared I Infrared I Color )Color 7 8818 1612861032106.993164 fps 60 gap 16.666666 next_expected: 1612861032123.659912
 09/02 21:57:12,132 DEBUG [28076] (sync.cpp:526) fps 60 Color 7 8818 1612861032106.993164 
 09/02 21:57:12,132 DEBUG [28076] (sync.cpp:598) next expected of the missing stream didn't updated yet
 09/02 21:57:12,132 DEBUG [28076] (sync.cpp:343) (TS: I Depth I Infrared I Infrared I Color ) Color 7 8818 1612861032106.993164  Wait for missing stream: 4 next expected 1612861032106.664307
 09/02 21:57:12,132 DEBUG [28076] (frame-archive.h:148) CallbackFinished,Color,8818,DispatchedAt,1612861032132.675781
**Exception thrown: 'System.InvalidOperationException' in System.dll**
 09/02 21:57:12,140 DEBUG [20472] (global_timestamp_reader.cpp:123) librealsense::CLinearCoefficients::calc_value: 3715521.497000 -> 1612861032106.795166 with coefs:0.999895, -11.540764, 3567007.826000, 1612860883620.263184
 09/02 21:57:12,140 DEBUG [20472] (sensor.cpp:348) FrameAccepted,Depth,Counter,8793,Index,0,BackEndTS,1612861032121.000000,SystemTime,1612861032140.320313 ,diff_ts[Sys-BE],19.320313,TS,1612861032106.795166,TS_Domain,Global Time,last_frame_number,8792,last_timestamp,1612861032089.997559
 09/02 21:57:12,140 DEBUG [20472] (archive.cpp:293) CallbackStarted,Depth,8793,DispatchedAt,1612861032140.740723
 09/02 21:57:12,140 DEBUG [20472] (archive.cpp:293) CallbackStarted,Depth,8793,DispatchedAt,1612861032140.771484
 09/02 21:57:12,140 DEBUG [20472] (sync.cpp:147) DISPATCH (TS: )--> Depth 4 8793 1612861032106.795166 `

[MartyG-RealSense]

Hi @cdrose If you have a multicam C# application and suspect that your object pool may be filling up, a RealSense user in a case in the links below who was trying to optimize their application's speed had similar concerns.

https://support.intelrealsense.com/hc/en-us/community/posts/360049434593-Object-Pool-Size
#7201

The link below is also a useful C# reference about disposal.

#5369

It may be helpful to review the above links to see whether it provides useful insights for your own case and return here with any new questions that this research creates.

I would also add that if you have created a single application that handles multiple cameras (in the style of the rs-multicam example program) then it is recommended to use poll_for_frames() instead of wait_for_frames() for reasons described in this link:

#2422 (comment)

[cdrose]

Hi @MartyG-RealSense, thanks for the speedy reply. I had already read the first two issues you mention, and have now read the third also.

#7201 does sound like a similar issue although it is not mentioned what exception was actually thrown when the ObjectPool is empty, I am only guessing that is what has happened at this point. If that is the case then "Keep - bypass SDK frame pool to have more frames in memory" sounds like it may be a solution. I will try and work out how Keep should be used (I see there are a few references in the github issues).

#5369 seems unrelated unless I have missed something important in that thread.

#2422 I understand the discussion here but it is not relevant for my application. I can see how WaitForFrames() would be messy to use if one required multiple frames to be available before processing could continue, but in my case the cameras are used independently so I am free to process individual frames as they arrive with no need to syncronise the multi-camera setup.

So for me the main questions to answer are:
a) Am I correct that the objectpool has been exhausted and that is what is causing the Stack Empty exception?
b) Assuming the above is correct is Frame.Keep() the correct method to free up the object pool
c) If so how should Keep be used and is there any documentation or relevant examples

[MartyG-RealSense]

I researched the Stack Empty Exception thoroughly and can find no other cases of it in librealsense. The link below is the best information that I could find about dealing with it, and suggests that it may be happening due to the multi-threading.

https://stackoverflow.com/questions/14121799/stack-empty-exception

There is no other information relating to use of Keep() with C# to answer your questions about it and so I do not feel that I can recommend it, simply because there is insufficient information to conclude whether it is appropriate for your C# project.

I will say though that Keep() has the limitation that because the frames are stored in the computer's memory until the pipeline is closed and the frames processed, it is best suited for recording durations of up to 30 seconds. This is because the computer's memory capacity is consumed by the frames as the seconds pass.

I do not know of C# scripting examples for Keep() though, only C++ and Python ones. A C++ example of using Keep(), in case you wish to consider accessing C++ from C#, is in the link below,

#6865 (comment)

[cdrose]

The lack of other librealsense issues relating to the Stack Empty exception does imply that its something particular to my current project. A concurrency issue due to multi-threading would make sense, I will look into it further.

If the Stack Empty exception is not actually related to reaching the capacity of the ObjectPool then Keep is not the solution so I will see if I can solve my issue without using it. It would seem odd though that the frames are stored in memory until the pipeline is closed, surely one can still Dispose() of frames which have been stored with Keep() once they are no longer required.

[cdrose]

OK I'm not sure I'm totally qualified to fix this, but looking in ObjectPool.cs some things look a bit suspicious. The ObjectPool class is static and there is one PooledStack for each type. This means all Frames across all threads will be trying to use the same stack. The underlying stack is a System.Collections.Generic.Stack which is not inherently thread safe. Locks are used in the ObjectPool methods so it was clearly intended to work in a multithreaded environment.

The Stack Empty exception that I get comes from ObjectPool.Get() which in turn calls System.Collections.Generic.Stack.Pop(). The lock is acquired before checking the stack count, it is then released and reaquired before the object is popped off the stack. To me that looks like an opportunity for another thread to pop an object, leaving the original thread with an incorrect count and then attempting to pop an object from an empty stack. Releasing the lock in between checking the count and carrying out the Pop() looks like a bug to me.

private static object Get(Type t, IntPtr ptr)
        {
            var stack = GetPool(t);
            int count;
            lock ((stack as ICollection).SyncRoot)
            {
                count = stack.Count;
            }

            if (count > 0)
            {
                Base.PooledObject obj;
                lock ((stack as ICollection).SyncRoot)
                {
                    obj = stack.Pop();
                }

                obj.m_instance.Reset(ptr);
                obj.Initialize();
                return obj;
            }

            return CreateInstance(t, ptr);
        }

Edit:
Also the way the Factories are implemented, it looks like it is not possible to 'run out' of objects in the pool. As far as I can see if the pool is empty then a new object is created, which in turn is then pushed back into the stack when it is released. So it looks like in the C# wrapper at least, the pool grows to the size it needs to be to hold the largest number of objects that are in use at any one time. Very neat 👍

[MartyG-RealSense]

Given that the exception has been occurring sporadically for you (as reported in the opening comment of this discussion), the best way forward may be to continue catching the exceptions as you are already doing.

[cdrose]

Or, perhaps we should fix it?!

I have recompiled the C# wrapper with the following change. Will test and confirm if it works.

private static object Get(Type t, IntPtr ptr)
        {
            var stack = GetPool(t);
            int count;
            Base.PooledObject obj;
            lock ((stack as ICollection).SyncRoot)
            {
                count = stack.Count;
                if (count > 0)
                {
                    obj = stack.Pop();
                    obj.m_instance.Reset(ptr);
                    obj.Initialize();
                    return obj;
                }
            }
            return CreateInstance(t, ptr);
        }

[MartyG-RealSense]

Okay, thanks very much for the possible PR contribution!

[MartyG-RealSense]

Hi @cdrose Do you have an update that you can provide about your case, please? Thanks!

[cdrose]

@MartyG-RealSense sorry been busy with other things last week, have yet to fully test my changes. Should be back on it later this week.

[MartyG-RealSense]

Thanks very much for the update!

[cdrose]

I have tested my patched version of the C# wrapper and have not observed anything unexpected. I have also not seen a StackEmpty exception since making the change. As far as I can tell this change has fixed the issue.

[MartyG-RealSense]

Excellent news @cdrose - thanks!

[MartyG-RealSense]

Case closed due to successful outcome achieved and no further comments received.

[PorschefanRoel]

I had the same problem, it was a multi camera (4 camera's), multi treaded projected (Each couple of 2 camera's has it's own Tread with a Syncer object), resulting in the ObjectPool crash with the following stacktrace:

Exception Info: System.InvalidOperationException
at System.ThrowHelper.ThrowInvalidOperationException(System.ExceptionResource)
at System.Collections.Generic.Stack`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Pop()
at Intel.RealSense.ObjectPool.Get(System.Type, IntPtr)
at Intel.RealSense.ObjectPool.Get[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]
at Intel.RealSense.Sensor+<>c__DisplayClass21_0.b__0(IntPtr, IntPtr)

Recompiling the Intel.Realsense wrapper with the Get method as shown by @cdrose fixed it.

[MartyG-RealSense]

Thanks so much @PorschefanRoel for sharing your confirmation of successful use of the @cdrose method on this discussion!