topic Re: Realsense D435 timestamps in Items with no label

Realsense D435 timestamps

JCalo — Thu, 26 Jul 2018 15:26:12 GMT

Hi,

I have a question about the timestamps that are generated on the Realsense D435. How exactly are these timestamps generated? Is it when the depth camera starts recording, when the image is sent through the usb or at some different point?

I am combining the realsense D435 with another tracking device. I am using Nuitrack to get a skeleton body.

Re: Realsense D435 timestamps

idata — Thu, 26 Jul 2018 22:00:29 GMT

Hello Joost,

Thank you for your interest in the Intel® RealSense™ Depth Camera D435.

We are currently working on your request and will come with an answer.

Also, you can check the post in the link below: https://communities.intel.com/thread/120208 https://communities.intel.com/thread/120208

Regards,

George

Re: Realsense D435 timestamps

JCalo — Fri, 27 Jul 2018 08:53:59 GMT

Thanks, I already came by that post but it is about the synchronization of the RGB camera and depth camera. Currently I only obtain the depth frames which are then processed with Nuitrack to obtain a skeleton. When I plot the signals from Nuitrack and my other device, the data from the Realsense lags ~ 80 ms behind. I already validated the timing of the other device in a different way.

When I run the Intel RealSense viewer, the depth stream gives me a timestamp and also shows a red text "System Time" with the message:

"Hardware Timestamp unavailable! This is often an indication of improperly applied Kernel patch. Please refer to installation.md for more information"

Are the hardware timestamps available on the D435? If so, should I change something on my configuration? Furthermore, I can't seem to find the installation.md file.

Re: Realsense D435 timestamps

MartyG — Fri, 27 Jul 2018 09:47:26 GMT

System Time is the camera's software timestamp, and is used when hardware timestamps are unavailable.

The installation.md document can be found here.

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation.md librealsense/installation.md at master · IntelRealSense/librealsense · GitHub

As far as RGB and depth sync on D435 goes,Jesus G from Intel customer support has stated: "The RGB and depth sensors are not on the same PCB. The RGB is a standalone camera module, there is no HW sync connection between RGB sensor and depth sensor. They cannot be synced".

Whilst there is not a clear and definite statement on how that affects hardware timestamps, the fact that there is information about hardware timestamp on D415 but not on D435 probably answers the question.

Below is a link to the best reference I know of regarding hardware timestamps on D415, in case you are interested.

https://github.com/IntelRealSense/librealsense/issues/1294 Support of hardware timestamps · Issue # 1294 · IntelRealSense/librealsense · GitHub

Re: Realsense D435 timestamps

JCalo — Mon, 30 Jul 2018 06:55:57 GMT

Thank you MartyG, indeed I've seen that post but it's about the hardware sync between the RGB and depth sensor. Looking at this post:

https://github.com/IntelRealSense/librealsense/blob/master/doc/frame_metadata.md# os-support librealsense/frame_metadata.md at master · IntelRealSense/librealsense · GitHub

I got it able to work and now I'm indeed getting the right timestamps even in Nuitrack. Now my question is: is it possible to poll the device hardware time?

Re: Realsense D435 timestamps

MartyG — Mon, 30 Jul 2018 11:00:40 GMT

The discussion linked to below contains a short script that a user created to output depth and color frame info, including hardware timestamps, to a log.

https://github.com/IntelRealSense/librealsense/issues/1263 Sync frame and understanding frame rates, frame numbers and timestamps · Issue # 1263 · IntelRealSense/librealsense · Git…

Re: Realsense D435 timestamps

JCalo — Mon, 30 Jul 2018 14:11:06 GMT

Thanks MartyG, I've got my frames now with the metadata with which I'm able to obtain hardware timestamps and synchronize my applications. Only problem is that when I run the Nuitrack it is not possible to obtain this data so I'll have to find a workaround for this. One last question: I'm now obtaining this data by getting a frame but more ideally would be to just get the hardware time only, is this possible at the moment?

Or is it possible to synchronize the Intel RealSense D435 clock differently or make it run with the same speed as the PC clock?

Re: Realsense D435 timestamps

MartyG — Mon, 30 Jul 2018 14:52:34 GMT

On the SDK 2.0 documentation's API how-to page where they provide short scripts for different aspects of the camera (such as getting a stream or polling for frames), they list fetching the timestamp as a 'to do' item to add to the page at a later time.

https://github.com/IntelRealSense/librealsense/wiki/API-How-To API How To · IntelRealSense/librealsense Wiki · GitHub

Researching the subject myself, it looks like the instruction RS2_TIMESTAMP_DOMAIN_HARDWARE_CLOCK is responsible for the hardware timestamp.

The sample program Software Device includes some references to this hardware timestamp instruction.

https://github.com/IntelRealSense/librealsense/tree/master/examples/software-device librealsense/examples/software-device at master · IntelRealSense/librealsense · GitHub

Dorodnic, the RealSense SDK Manager, said about the Software Device sample on June 27 in an online seminar: "The sample program 'Software Device' can be used to create a fake RealSense device when it is run. The example will teach you how to inject synthetic frames into this fake device, and then you can use the regular point cloud, texture mapping, all the regular APIs to work with it. It can be useful to inject data from other sensors into our existing ecosystem. We're also thinking about giving people tools to map RealSense to other sorts of input".

Re: Realsense D435 timestamps

JCalo — Tue, 31 Jul 2018 09:28:02 GMT

Thanks, I've indeed come along the timestamp domain flag for some functions. Since Nuitrack is blocking me to having access to this extra metadata, is there any possibility to setup a second stream to the camera? I'm guessing not, since the applications also won't work with a second stream?

I can synchronize the timestamps before and after I use the Nuitrack but the clock of the D435 is drifting so much that even a linear interpolation of the time difference wouldn't give me enough accuracy...

Re: Realsense D435 timestamps

MartyG — Tue, 31 Jul 2018 09:55:46 GMT

Yes, the problem is that Nuitrack works with the 400 Series cameras but is its own SDK separate from RealSense SDK 2.0. So any multi-camera support in the SDK 2.0 software would not automatically carry over to the Nuitrack SDK. There was a discussion about this recently on the Nuitrack forum.

https://community.nuitrack.com/t/solved-is-it-possible-to-connect-to-multiple-cameras/435/7 https://community.nuitrack.com/t/solved-is-it-possible-to-connect-to-multiple-cameras/435/7

Re: Realsense D435 timestamps

JCalo — Thu, 02 Aug 2018 08:48:21 GMT

Alright, I managed to bypass Nuitrack somehow and change my timestamps. Now I got one more question, similar as my first one: when are the timetamps generated in the metadata?

sensor_timestamp: I'm guessing inside the camera since the device clock is used, is this before / after the sensor starts obtaining the image??

frame_timestamp: I'm guessing inside the camera since the device clock is used, is this when the frame starts getting send over the USB?

backend_timestamp: I'm guessing on the PC side since the system clock is used, is this the time when the frame starts arriving over the USB?

time_of_arrival: I'm guessing on the PC side since the system clock is used, is this the time when the frame is being constructed?

Again I'm using the Intel RealSense D435. When I looked at the difference between the frame_timestamp and sensor_timestamp it alternated between 32 and some other value (order of 1000's) with no other values, this made not much sense to me...

Re: Realsense D435 timestamps

MartyG — Thu, 02 Aug 2018 09:27:27 GMT

Here is the documentation page on frame metadata.

https://github.com/IntelRealSense/librealsense/blob/master/doc/frame_metadata.md librealsense/frame_metadata.md at master · IntelRealSense/librealsense · GitHub

Re: Realsense D435 timestamps

JCalo — Thu, 02 Aug 2018 09:38:15 GMT

Indeed, I know that page by heart now. But it doesn't answer my questions, I've also looked into parts of the code responsible for the timestamp generation but I couldn't yet find these answers there either.

Re: Realsense D435 timestamps

MartyG — Thu, 02 Aug 2018 09:42:08 GMT

A question of this technical depth is likely best posted on the RealSense GitHub help site where the RealSense developers and engineers are. You can post your question there by visiting the link below and using the 'New issue' button.

https://github.com/IntelRealSense/librealsense/issues Issues · IntelRealSense/librealsense · GitHub

Re: Realsense D435 timestamps

JCalo — Fri, 03 Aug 2018 06:44:08 GMT

Thanks, I've indeed got my answer over there: https://github.com/IntelRealSense/librealsense/issues/2188 Frame metadata timestamps · Issue # 2188 · IntelRealSense/librealsense · GitHub

Re: Realsense D435 timestamps

MartyG — Fri, 03 Aug 2018 06:55:04 GMT

Excellent, I'm very glad they were able to help you.