Hardware Sync of Color and Depth in D435?

[RSpri1]

Is there a hardware synchronization of color and depth cameras in the D435? When I run a pipeline with both streams and output the time stamps I see that the average lag of one sensor vs. the other is around 0, however the variance is quite high. In other words, the driver tries to give me color and depth frames of approximately matching time stamps but the mismatch can go as high as half the frame rate (see attached files).

Does the D435 have the ability to hardware-sync the frame capturing between color and depth, i.e. make sure the frames are really captured at the same time? If yes, do I need to turn on this capability somewhere in the driver?

Attached files show Color TS (us), Depth TS (us) and Color minus Depth

Thanks

[MartyG]

This information page on the high level and low level camera interfaces may provide useful insights.

https://github.com/IntelRealSense/librealsense/blob/development/doc/api_arch.md librealsense/api_arch.md at development · IntelRealSense/librealsense · GitHub

[RSpri1]

Thanks Marty. I have found this thread on github (https://github.com/IntelRealSense/librealsense/issues/774 RGB and depth sync · Issue # 774 · IntelRealSense/librealsense · GitHub ) and have responded there. I will also try to look into the syncer class although the documentation is a bit sparse.

[RSpri1]

I have tried grabbing frames with the rs2::syncer class but the time stamps aren't getting any better. So either the D435 is just not capable of HW sync (would be surprising) or it's turned off in the firmware (mine is runnin 05.08.05) or I need to enable it through the SDK but haven't found how to do it.

In any case it would be good to know if there's a way to resolve it.

[MartyG]

An SDK 2.0 documentation page last updated August 23 says: " Advanced camera features may still require kernel patches. Currently, getting hardware timestamps is dependent on a patch that has not yet been up-streamed. Without the patch applied you can still use the camera but you will receive a system-time instead of an optical timestamp."

https://github.com/IntelRealSense/librealsense/blob/165ae36b350ca950e4180dd6ca03ca6347bc6367/doc/rs400_support.md librealsense/rs400_support.md at 165ae36b350ca950e4180dd6ca03ca6347bc6367 · IntelRealSense/librealsense · GitHub

[RSpri1]

This is under the Linux section though. We're using Windows 10 (sorry, forgot to mention).

[MartyG]

SDK 2.0 is an advanced form of Librealsense that works with Windows, Linux and Mac (partial Mac support so far). Intel have switched their documentation system for the new cameras to Github instead of hosting the pages themselves, which is why the docs look totally different to the old SDK docs. The info on the linked page is equally applicable to Windows, Linux and Mac.

Edit: I see what you mean now about the Linux reference in the quoted heading. Sorry!

[MartyG]

I checked the release notes of past SDK 2.0 releases. The recommended firmware for the very first release was 5.08.07.0. This suggests that your 5.08.05.0 firmware is not even preferred by the earliest available version of 2.0, unfortunately.

[RSpri1]

Ok thanks. I'll try to get the FW updated then.

[RSpri1]

Unfortunately this issue remains unresolved after upgrading the FW.

[idata]

Hello rspring,

Your problem has been escalated to our development team. As soon as we have any update we will let youknow. Best Regards, Juan N.

[RSpri1]

I have just tried this on a D415 running firmware 05.08.09 and there is also no hardware synchronization between color and depth, same as on the D435.

Is RGB-depth hardware synchronization between color and depth a feature that is available at all, in at least some configurations? Or is hardware sync just something the D4x5 series doesn't support? It would be good to get a definite answer on this.

Thanks,

Rafael

[idata]

Hello rspring,

For HW sync between RGB and depth...

D415: the RGB and depth cameras are on the same PCB board. The HW sync connection between RGB and depth is in place, but the FW has not enabled it yet. It is planned for development with a release about 2 months from now.

D435: The RGB and depth sensors are not on the same PCB. The RGB is a standalone camera module, there is no HW sync connection between RGB sensor and depth sensor. They cannot be synced.

Regards,

Jesus

Intel Customer Support

[RSpri1]

Thank you for clearing this up. This is the information I was looking for.

[idata]

Jesus please tell me that isn't true.

The D400 cameras are supposed to have GPIO so they can be synced, or at least have a common time stamps so external devices can be correctly aligned. Without synchronization inside the device, what exactly is the point of GPIO?

This information should have been made explicitly clear so people knew to order the D415 instead of the D435.

[idata]

Hello Sam598,

We are aware of this need and are looking to provide a solution in an upcoming FW release.Regards,Jesus G.Intel Customer Support

[ESchl2]

WHAT! that's INSANE that the 435 can't support h/w synced depth/rgb frames due to a hardware design concern.

I'll reiterate Sam's note that this really should be made MUCH MORE CLEAR to purchasers. Jesus, the statement you made regarding the different capabilities renders these two cameras totally different. with the implication being that there are fairly large number of capture and mapping use cases for which the 435 is wholly unsuitable. S

It's so very disappointing that this was touted as a product FAMILY level feature, only to have it only supported on the lower level unit. I guess it's nice that the 415 it's less expensive, but there are quite a few of us out here that bought multiple 435s expecting to be able to (eventually) use sync'd depth/rgb frames.

To make it worse, the effect is so visibly obvious, even on a powerful machine (modern i7) ...

WRT to sam598 question, he's asking about the 435. so I'm not sure what FW update you're referring to, given that you just said this functionality will *never* in HW on the 435.

Can you confirm that the 435 will *never support h/w (time truth) accurate synced frames?

[idata]

Can we please have a clear statement the actual capability of the 400 series cameras regarding internal and external synchronization? Because

1) What's the point of having external genlock for a camera where hardware sync is not possible between the internal sensors? Does it make any sense at all? Would only one of the sensors be synced to the external signal, e.g., either the color or the depth, but from what you've said sync of both is not possible?

1.2) I'm going to go ahead and expect that your response will be something along the lines of "the onboard color/depth sync frequency will be considerably higher than the frame rate frequency?"

1.3) Can we please have a time-frame for a firmware update to address the present lack of strong sync in the 435?

2) Regarding the 415, what's the point of genlocking a camera with a rolling shutter? Presumably, this would have been a trickle-down feature from the 435. An external sync generator is used to accommodate fast-moving subjects. However, rolling shutter artifacts make depth reconstruction of moving objects virtually impossible. Presumably, the global shutter sensors in the 435 were meant to correct for the poor performance of the ZR300 series cameras when viewing objects undergoing motion of deformation.

Accurate information would be most appreciated - this shouldn't be a marketing forum. Thank you.

[ARyba]

I also would like to get a clarification on this.

-albertr

[ESchl2]

If you want hardware sync, get the 415 /sadface/

[ARyba]

Well, I already have a D415 with the latest 5.9.2.0 F/W , but I don't think it's working... It would be nice if Intel can clarify the situation.

-albertr