Pointcloud : Texture coordinates are always {0.0, 0.0}

[tchapi]

Required Info
Camera Model	SR300
Firmware Version	3.21.0.0
Operating System & Version	Ubuntu 16.04.2 LTS
Kernel Version (Linux Only)	4.4.0-101-generic
SDK Version	latest master

When using the pointcloud example with a SR300, the texture coordinates are {0.0, 0.0} all the time, so I cannot retrieve the color information from the underlying color frame.

If I project the points myself, I manage to get the color information (though it's not aligned, etc)

Is this normal ? Do I miss a step ? Thanks a lot !

rs2::pointcloud pc;
rs2::points points;

rs2::pipeline pipe;
pipe.start();
        
for (auto i = 0; i < 30; ++i) pipe.wait_for_frames();
auto frames = pipe.wait_for_frames();

auto depth = frames.get_depth_frame();

points = pc.calculate(depth);
auto color = frames.get_color_frame();
pc.map_to(color);

auto vertices = points.get_vertices();              // get vertices
auto tex_coords = points.get_texture_coordinates(); // and texture coordinates
for (int i = 0; i < points.size(); i++)
{
    if (vertices[i].z)
    {
        // This is always 0.0 ....
        printf("%.4f, %.4f\n", tex_coords[i]);
    }
}

[dorodnic]

Hi @tchapi

Can you try:

auto color = frames.get_color_frame();
pc.map_to(color);
points = pc.calculate(depth);

Is the rs-pointcloud demo working properly on your system?

[tchapi]

Hi @dorodnic and thanks for your answer.
Unfortunately, it's exactly the same behaviour with your solution. The rs-pointcloud demo does not work (that is, I used the exact same process in my code and the color data does not show, as explained, the texture coordinates are always 0.0).

[dorodnic]

Ok, thanks clarifying. Do you see the option for texture mapping in the RealSense Viewer (3D tab)? What about a Windows machine with the same SR300 camera?

[tchapi]

I can open the RealSense Viewer on a Mac with the same camera and the pointcloud (3D tab) seems correct (with color, correctly aligned). I think the camera works well.

As well, when I use the "legacy" way of mapping, it works well (I get the color for each point) :

for (int dy = 0; dy < depth_intrin.height; ++dy)
    {
        for (int dx = 0; dx < depth_intrin.width; ++dx)
        {
            uint16_t depth_value = depth_image[dy * depth_intrin.width + dx];
            float depth_in_meters = depth_value * scale;

            if (depth_value == 0) continue;

            // Map from pixel coordinates in the depth image to pixel coordinates in the color image
            float depth_pixel[] = {(float)dx, (float)dy};
            float color_pixel[] = {0.0f, 0.0f};
            float color_point[] = {0.0f, 0.0f, 0.0f};
            float depth_point[] = {0.0f, 0.0f, 0.0f};
            rs2_deproject_pixel_to_point(depth_point, &depth_intrin, depth_pixel, depth_in_meters);
            rs2_transform_point_to_point(color_point, &depth_to_color, depth_point);
            rs2_project_point_to_pixel(color_pixel, &color_intrin, color_point);

            const int cx = (int)std::round(color_pixel[0]), cy = (int)std::round(color_pixel[1]);

                red = (uint8_t*) color_image + (cy * color_intrin.width + cx) * 3;
                green = red + 1;
                blue = green + 1;   
        }
    }

I'm thinking that something is missing in the rs-pointcloud example, and the texture coordinates are not updated like they should...

[tchapi]

Hi @dorodnic - do you think the PR from @AnnaRomanov just above correctly pinpoints the problem ? One of her commits seem to fix it but I'm not sure I understand the underlying problem and how it was effectively fixed

thanks !

[radikalliberal]

Hey, I might have an idea why this is happening. I just stumbled upon an error occurring when one calls pc.map_to(color). I'm using SR300
In a recent release, one line was added to map_to() in rs_processing.hpp Line 218

_block->set_option(RS2_OPTION_TEXTURE_SOURCE, float(mapped.get_profile().unique_id()));)

I'm getting the following error message:

"RealSense error calling rs2_set_option(options:000002592B3D2B10, option:35, value:5): invalid enum value for argument "option""

this error references the call in rs_processing.hpp and RS2_OPTION_TEXTURE_SOURCE
When the line is commented out the program works as anticipated.

[dorodnic]

Hi @tchapi
What we were able to reproduce is that the first call to calculate will always produce texture coordinates {0.0, 0.0} even after map_to. Unfortunately, we were not able to reproduce the exact issue you reported with texture coordinates being always {0.0, 0.0}.

The commit should be split into bug fix and renaming, but the key change is this - even when texture frame was already observed, new depth frame was not updating the extrinsics between the two streams.

@radikalliberal
The error message hint that there is a mismatch between the librealsense you have installed and the version you are compiling against. It is our fault really for not bumping minor version in last couple of releases.

Please try rebuilding the library from source and update if problem still occurs.

[tchapi]

Hi @dorodnic

Do you mean that doing :

auto color = frames.get_color_frame();
points = pc.calculate(depth);
pc.map_to(color);
points = pc.calculate(depth);

(two calls to calculate) should solve the problem ?

[ljc19800331]

@tchapi Hello, have you tried to map the color pixel to the 3D point cloud in python? I found difficulty in mapping the texture coordinates to the color image, and then back to the 3D vertice coordinates.

[tchapi]

No I haven't, I'm using the C/C++ lib.

For me, doing two calls to the calculate function did the trick :

points = pc.calculate(depth_frame);
points = pc.calculate(depth_frame);

hope it helps