Can not use keep() method and std::vector for post-processing color and depth frames saved in memory

[ghost]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D435
Firmware Version	05.09.11.00
Operating System & Version	Win 10
Platform	PC
SDK Version	2
Language	C++

Issue Description

I am trying to use the suggested "keep()" method to capture a large number of color and depth frames, and push them inside std::vector containers (have not been able yet to utilize rs2::frame_queue, perhaps this is better alternative to std::vector?). In the next step, I plan to filter depth frames, generate a point cloud from filtered depth and respective color frame, and push the point cloud inside another vector. I'm doing the post-processing step using a parallel for loop, utilizing std::thread. Capturing the raw data goes smoothly, and I can see that the time stamps are increasing with steps equal to 3-40 ms. Even the filtering inside the parallel for loop goes fine. However, the program always crashes if I try to push the point cloud inside the container. After lots of test, and hours of debugging, I have been able to partially
identify the source of the problem. Here is the summary:

std::vector<rs2::points> points; 
std::vector<rs2::frame> colors, depths;
// I push_back 1000 color and depth frames
// parts of post-processing below
depths[i]=decimation_filter.process( depths[i] ); // crashes
auto temp_depth=decimation_filter.process( depths[i] ); // but this is ok! interesting since no deep-copy happening
points.push_back(pointcloud.calculate(depths[i])); //crashes
pointcloud.map_to( colors[i] ); //crashes

I have all the safe guards in place to make sure frames are not empty. So what is happening here? Do you have a better suggestion to achieve my goal? Is there something wrong with using std::vector ?
Thanks,

[dorodnic]

Hi @eyewy
This is the full sample that worked for me:

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include <iostream>

int main(int argc, char * argv[]) try
{
    const int N = 100;
    rs2::decimation_filter decimation_filter;
    rs2::pointcloud pointcloud;
    
    std::vector<rs2::points> points;
    std::vector<rs2::frame> colors, depths;
    
    rs2::pipeline pipe;
    pipe.start();

    for (int i = 0; i < N; i++)
    {
        auto fs = pipe.wait_for_frames();
        
        auto depth = fs.get_depth_frame();
        auto color = fs.get_color_frame();
        
        depth.keep();
        color.keep();
        
        depths.push_back(depth);
        colors.push_back(color);
        
        pointcloud.map_to(colors[i]);
        depths[i] = decimation_filter.process(depths[i]);
        depths[i].keep();
        points.push_back(pointcloud.calculate(depths[i]));
        points[i].keep();
    }
    
    for (int i = 0; i < N; i++)
    {
        std::cout << depths[i].get_frame_number() << std::endl;
    }

    return EXIT_SUCCESS;
}
catch (const rs2::error & e)
{
    std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;
    return EXIT_FAILURE;
}
catch (const std::exception& e)
{
    std::cerr << e.what() << std::endl;
    return EXIT_FAILURE;
}

Let us know if it helps.

I must say that capturing 1000 frames can be challenging. Please make sure you are compiling an x64 application (x86 will run into OS allocation limit of about 1GB). If you are using Linux, please make sure a swap file is correctly configured.

[ghost]

Thanks, this helps. Three questions remaining:

1. How often should I call keep()? previously, I was calling it once, and only for frameset. What is the proper usage of keep? If I am doing a 4-step filtering, shall I call keep() after each step?
2. Let's say we have ample system memory available, and we are compiling x64 on Win 10. When you say 1000 frames is challenging, what is the limit? Is it something SDK? I am asking this, because I still have issues reaching the desired 1000 frames saved in memory as std::vector (400 looks to be ok).
3. What is the benefit of using rs2::frame_queue vs. the discussed approach, if any? Can you point me to an example that uses rs2::frame_queue for storage in memory?

Many thanks,

[dorodnic]

Hi @eyewy
Glad it helps!

How often should I call keep()? previously, I was calling it once, and only for frameset. What is the proper usage of keep? If I am doing a 4-step filtering, shall I call keep() after each step?

Every frame (or frameset) that goes into the vector. If you are doing 4 step processing but only saving the last one, I believe it would be sufficient to call keep once before saving it.

Let's say we have ample system memory available, and we are compiling x64 on Win 10. When you say 1000 frames is challenging, what is the limit? Is it something SDK? I am asking this, because I still have issues reaching the desired 1000 frames saved in memory as std::vector (400 looks to be ok).

I was referring exclusively to the RAM limitations. There should be no limit in the SDK, as far as I know. Please share more info about what happens when you go past 400.

What is the benefit of using rs2::frame_queue vs. the discussed approach, if any? Can you point me to an example that uses rs2::frame_queue for storage in memory?

Well, frame_queue is really not built for this. It is a synchronization primitive that is included to help you marshal frames from one thread to another, not so much for long term storage. vector should do fine.

[ghost]

Thanks, with your guidance I was able to save 1000 frames and then do post-processing. As you confirmed, with the correct implementation, the only limiting factor seems to be amount of RAM at disposal. This is critical for some applications, as post-processing and PC mapping in capture loop severely drops the frame rate. With this method, my frame time is always within 30-40ms. I suggest including your code sample in How-to page, as it was the topic of 2 other (closed) issues as well. I think we can close this issue.