Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove image-msg::frame-id; use timestamp #11751

Merged
merged 9 commits into from
May 1, 2023
Merged

Conversation

maloel
Copy link
Collaborator

@maloel maloel commented Apr 30, 2023

Follow-up to #11665, this introduces timestamp-based synchronization of metadata to images:

  • Changed frame-id in the MD to frame-number to be consistent, since that's what it's called in librs
  • It's now a number, not a string
  • Frames will now have a timestamp, even without metadata
  • Timestamp is communicated as nanoseconds - using doubles is asking for trouble
  • Removed frame_id field in image
  • Frame numbers are now still communicated, but are optional on the client since MD is the only way to communicate them
    • When not available, librs will number them sequentially - meaning we lose the ability to use the numbers for drop detection
  • The frame_id in the ROS2 image format is a string denoting the device & sensor - this needs more work

Along the way:

  • Fix behavior of dds_time with negatives, and added a dds-time unit-test that clarifies a lot of its (mis)behavior
  • Removed old image format
  • Refactoring of the message formats to all be similar: image_msg, flexible_msg, located under topics/
  • Improve rspy.test.remote.Error output
  • Separated publish_image from publish_motion

As before, I suggest looking at the commits one by one; they're fairly self-contained.

Next is adding the ros2imu format and getting that to work.

@maloel maloel requested a review from OhadMeir April 30, 2023 09:31
@@ -49,10 +52,9 @@ class dds_stream_server : public dds_stream_base
void close();

bool is_streaming() const override { return _streaming; }
void start_streaming( const image_header & header );
void start_streaming( const image_header & );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you remove the variable name?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because saying image_header header is redundant and unhelpful. If the type conveys the meaning, then everything else is extra. But that's in hindsight -- maybe I changed something and then undid it...?

raw_image.encoding() = _image_header.format.to_string();
raw_image.height() = _image_header.height;
raw_image.width() = _image_header.width;
raw_image.step() = uint32_t( image.raw_data.size() / _image_header.height );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check _image_header.height != 0 (can be in start_streaming)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try to address this when I do the IMU work, since an image_header is the wrong thing to use when the data isn't an image.

raw_image.height() = _image_header.height;
raw_image.width() = _image_header.width;
raw_image.step() = uint32_t( image.raw_data.size() / _image_header.height );
raw_image.is_bigendian() = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is true for Jetson?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's reflective of our images (whoever created the data) rather than the system. For now it should do.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I'm not entirely sure what it's indicative of. It might refer to the endian-ness of the message header, since the image format is pretty much set in stone...
I think we'll address this issue when we get there.

if( ! is_streaming() )
DDS_THROW( runtime_error, "stream '" + name() + "' cannot publish before start_streaming()" );

if( image.height != _image_header.height )
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need this? motion does not have height and width.
Same question for step

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a copy of publish_image until I get to the IMU.


DDS_API_CALL( _writer->get()->write( &raw_image ) );
}

} // namespace realdds
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check new line at EOF

server->publish( static_cast< const uint8_t * >( f.get_data() ), f.get_data_size(), f.get_frame_number() );
realdds::topics::image_msg image;
auto data = static_cast< const uint8_t * >( f.get_data() );
image.raw_data.assign( data, data + f.get_data_size() );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move the data?
Maybe construct a new vector on this range and move to raw_data

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Constructing a new vector (meaning a copy) and moving it is the same as assigning the data. We assigned the data inside publish before, too, so this is not a change.

if( res.rem < 0 )
{
--res.quot;
res.rem += 1000000000ull;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using this number a lot. Good opportunity to use constexpr uint64_t nanos_in_second = 1'000'000'000ull

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That might be a good idea, but I don't want to introduce this in a header at this point...

Copy link
Contributor

@OhadMeir OhadMeir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@maloel maloel merged commit 4074adb into IntelRealSense:dds May 1, 2023
@maloel maloel deleted the timestamp branch May 1, 2023 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants