rs.frame_metadata_value.backend_timestamp is always 0

[joshuafishman]

Required Info
Camera Model	D435
Firmware Version	2.49.0
Operating System & Version	Ubuntu 20.04
Kernel Version (Linux Only)	5.11.0-37-generic
Platform	PC
SDK Version	2.49.0
Language	Python
Segment	Robot

Issue Description

I'm trying to use the metadata timestamps to compensate for what appears to be an ~80ms delay (i.e., get_timestamp() reports a value which seems ~80ms late). However, rs.frame_metadata_value.backend_timestamp is always 0.

What might cause this? Could it be related to the unexplained delay?

Example script:

# First import the library
import pyrealsense2.pyrealsense2 as rs  # type: ignore
import time

def run():
    print(rs.__file__)
    start = time.time()

    # Create a context object. This object owns the handles to all connected realsense devices
    pipeline = rs.pipeline()
    rgb_dims = (480, 640)
    depth_dims = rgb_dims
    frame_rate = 15

    rs_config = rs.config()
    rs_config.enable_stream(
        rs.stream.depth, depth_dims[1], depth_dims[0], rs.format.z16, frame_rate
    )
    rs_config.enable_stream(
        rs.stream.color, rgb_dims[1], rgb_dims[0], rs.format.rgb8, frame_rate
    )
    profile = pipeline.start(rs_config)

    try:
        while True:
            # Create a pipeline object. This object configures the streaming camera and owns it's handle
            frames = pipeline.wait_for_frames()


            print(
                f"Sensor: {frames.get_frame_metadata(rs.frame_metadata_value.sensor_timestamp)/1000./1000.}"
                f" Frame: {frames.get_frame_metadata(rs.frame_metadata_value.frame_timestamp)/1000./1000.}"
                f" Backend: {frames.get_frame_metadata(rs.frame_metadata_value.backend_timestamp)}"
                f" Arrival: {frames.get_frame_metadata(rs.frame_metadata_value.time_of_arrival)/1000}"
                f" Global: {frames.get_timestamp()/1000}"
                f" Start: {start}"
            )

    finally:
        pipeline.stop()


if __name__ == "__main__":
    run()

prints e.g. Sensor: 888.8259390000001 Frame: 888.829451 Backend: 0 Arrival: 1635200628.696 Global: 1635200628.6944928 Start: 1635200623.3859117

[MartyG-RealSense]

Hi @joshuafishman In #7837 (comment) a RealSense team member explains that the timestamps of acquired frames will intentionally be older than the time of a call made by wait_for_frames().

In regard to a zero value for the backend timestamp, as mentioned in #7972 there have been past cases where this has been reported on devices with an Arm processor (Raspberry Pi, Nvidia Jetson, Android, etc). Are you using a computing device with an Arm processor, please?

Using a 5.11 version kernel could also potentially introduce unpredictable stability problems, as 5.11 is not yet officially supported in the librealsense SDK. Using a librealsense build based on the RSUSB backend should help to avoid this, as an RSUSB build of librealsense is not deoendent on Linux versions or kernel versions and does not require patching.

[joshuafishman]

Hi @MartyG-RealSense , thanks for your help! I built from source with the RSUSB backend and I'm using an AMD Ryzen 7 processor.
I'm a bit confused by your point about wait_for_frames. By comparison to other sensor streams, the timestamp returned by frames.get_timestamp() appears to be inaccurate (50-100 ms late) -- the most similar case I can find is #8003. Would wait_for_frames explain this?
It's possible that setting global_time_enabled=False would help us as well, but I don't fully understand that parameter. Unlike the issue I linked, we aren't using ROS.

[MartyG-RealSense]

The 400 Series cameras are designed to work with any Intel or Arm CPU. AMD processors are more of an uncertain area, though RealSense users have been able to use them with AMD Ryzen and Threadripper CPUs in the past but not always 100% problem-free.

As you are using an RSUSB build, it is less likely that your problem is related to your Linux version or kernel version.

My understanding (I am not involved in SDK development) is that when global time is True, device time and system time are aligned. Some RealSense users find that in their particular project, use of global time causes drift in their timestamp values that is resolved by disabling global time. #3909 and #4505 provide further explanation about global time.

[joshuafishman]

Hi @MartyG-RealSense , it seems that the backend timestamp is also 0 on an intel core i7 processor.

[joshuafishman]

Disabling the global timestamp with

    sensors = pipeline.get_active_profile().get_device().query_sensors()
    for sensor in sensors:
        sensor.set_option(rs.option.global_time_enabled, False)

seems to set get_timestamp to return the frame_timestamp, which we'd need the backend_timestamp to translate to host time.

[MartyG-RealSense]

If you need a timestamp based on the host clock then referring to the time_of_arrival timestamp (which your script already prints) may provide an alternative to the backend_timestamp. A RealSense team member describes the different types of timestamp in #2188 (comment)

[MartyG-RealSense]

Hi @joshuafishman Do you require further assistance with this case, please? Thanks!

[joshuafishman]

Hi @MartyG-RealSense , I haven't made any progress -- the time_of_arrival timestamp is later (ie more wrong) than the global timestamp, so that doesn't help us very much. Right now we just subtract 80ms from the global timestamp and results look reasonable, but that's not ideal and recovering the backend timestamp would be helpful.

[MartyG-RealSense]

I researched the case from the beginning again but did not find any new leads beyond what has already been listed in this case, unfortunately.

[joshuafishman]

OK -- I appreciate your efforts!

[MartyG-RealSense]

Case closed due to no further comments received.