Hardware Sync of Color and Depth in D435?

[RSpri1]

Is there a hardware synchronization of color and depth cameras in the D435? When I run a pipeline with both streams and output the time stamps I see that the average lag of one sensor vs. the other is around 0, however the variance is quite high. In other words, the driver tries to give me color and depth frames of approximately matching time stamps but the mismatch can go as high as half the frame rate (see attached files).

Does the D435 have the ability to hardware-sync the frame capturing between color and depth, i.e. make sure the frames are really captured at the same time? If yes, do I need to turn on this capability somewhere in the driver?

Attached files show Color TS (us), Depth TS (us) and Color minus Depth

Thanks

[BChen59]

Is there any development on this front??

[MartyG]

The last estimate I saw from an Intel staff member for release of the hardware sync support was 'Q2 2018'. This puts it in a window of anytime between April and June 2018.

[SVens]

To our knowledge, this is the current state of things:

For HW Color/Depth Sync

1. This will never be possible with the D435 since the RGB and Depth Sensor are on different PCB

2. This is possible with D415 since the RGB and Depth sensors are on same PCB with the following:

a. FW 5.9.1 or later supports HW sync on D415

b. RGB manual exposure must be used

c. There is a known bug that prevents the RGB gain from being adjusted to achieve acceptable quality video

d. Until the gain is fixed (so that it can be set to levels that auto-exposure is achieved), HW sync for Color/Depth on D415 is not usable.

[YXia3]

@Intel Corporation, may I check could we do hardware synchronization for the color and depth data using D415. Your previous post suggests that you need a new firmware upgrade, may I check is this ready?

[MartyG]

Hardware sync for D415 and D435 is now available. D415 can do color and depth sync. D435 only has depth sync.

[YXia3]

Hi MartyG when you mentioned D435 only has depth synchronization, what does it really synced to?

Do you mean that the depth image and depth data are synced?

Thanks

Yingyu

[MartyG]

On the D435, the RGB sensor is not integrated on the camera's circuit board (it is integrated on the D415's board, but attached separately inside the D435's casing). This is why depth cannot sync with color on the D435.

The D435 was designed this way because Intel say that their user feedback indicated that customers wanted flexibility in the hardware configurations. So customers who need RGB can buy the D415, and others who do not need RGB can buy the D435 cased USB camera or one of the non-D415 camera circuit board modules (D400, 410, 420 or 430).

I only have a one-line description of the D435 sync that calls it a 'depth frame synchronization'. I take this to mean that the depth frames on the slave cameras align their timestamps with the depth frame timestamp on the master camera.

[ALow1]

I still cannot understand how a PCB difference make up the issue of the sync for D435?

The hardware architecture from the spec shows the RGB sensor is connected to the ISP before going to D4 then to the Host. Is this path different between D415 and D435?

All the conversation so far is suggesting me that the Hardware sync is possible only on D415 as it is designed on the same board. I cannot understand the difference between the 2 models apart from those stated from the spec.

what controls/signals are missing on D435 such that the HW sync cannot be achieved and has no issue on D415 ?

I try to analyse a bit further and think of the following possibility:

- HW sync is not possible on the dedicated RGB sensor -> regardless of D415 or D435

- HW sync is possible on D415 as the depth sensor (L/R imager) is as well a RGB sensor and the HW sync is done via this path. the D435 imager is B/W.

Can any intel staff explain further?

[MartyG]

Jesus G from Intel Support has described the D435 hardware sync in this way: "The RGB and depth sensors are not on the same PCB. The RGB is a standalone camera module, there is no HW sync connection between RGB sensor and depth sensor. They cannot be synced".

[ALow1]

Thanks for ya reply. Which is why I am asking if the support staff can provide more information. my earlier reply clearly puts out how the spec is describing the architecture. The problem is not so easily perceived as a PCB issue. Fundamentally, I think the RGB sensor reference between the 2 products are Different. Hoping to get a clearer picture of the problem.

[MartyG]

As far as I know, the way the D435 handles RGB is that when an RGB sensor is not attached, the firmware IDs the camera as a 'D430' (the RGB-less D430 Module Kit board that is inside the cased USB D435 camera). When an RGB sensor is attached on, the firmware assigns the ID 'D435' to the camera.

The D415 uses the OmniVision OV2740 sensor for IR and RGB. The data sheet for the 400 Series says that the D435 and D435i models use this sensor as the color sensor too.

[sshin9]

to MartyG,

In your mention "Hardware sync for D415 and D435 is now available", does "Hardware sync" mean "multi-camera hw sync"?

[MartyG]

Yes, it refers to multi camera hardware sync. The D415 can do depth and color hardware sync, whilst the D435 only has depth sync.

[sshin9]

I understood "hardware sync" all other people mentioned in this thread is depth frame and color frame synchronization in one camera.

Is there any way depth frame and color frame have exactly same timestamp(capturing at the same time)?

I tried to use "LEFT IR" camera as described in BKMs_Tuning_ReaSense_D4xx_Cam.pdf document. But I was desperate for the image quality.

[MartyG]

What syncing does it it syncs the frame timestamps of a stream on a "slave" camera with the timestamp of a stream on a "master" camera. So on the D435, you can only sync the depth frame timestamps of the slave cameras with the depth frame timestamp of the master camera.

This is because the D415 has its RGB sensor integrated on its camera module circuit board, whilst on the D435 the RGB sensor is not integrated and is attached separately inside the casing. So because the RGB sensor is not on the same circuit board, color sync cannot be done with the D435 model.

[rkash3]

Hi,

Did you manage to sync the RGB and Depth sensor of a single D415 (filming with the exact timestamp)? If so - what FW upgrade do you use?

Thanks!

[HChen92]

I have a D 415 camera, But cannot sync the color and depth sensor. The timestamp from the 2 sensors are different. Does anyone succeed the sync of D415?

[MartyG]

Are you the user who posted a similar message about sync on the GitHub at the same time under the name Desperad0, please?

https://github.com/IntelRealSense/librealsense/issues/774#issuecomment-453898417

If so, you should be able to receive a good answer from the GitHub. I would add though that if you are trying to do a hardware sync between more than one camera, please make sure that your cameras use an up to date firmware driver. Hardware sync requires two parts, an SDK that supports it (which for D415 is any SDK that is version 2.13.0 or newer) and also a firmware version that is 5.9.14 or newer.

Also please bear in mind that if you are doing hardware sync, a perfectly matched timestamp is not a sign that hardware sync is working correctly. With hardware sync, the synced timestamps will actually drift apart over time. When hardware sync is not active, the timestamps match perfectly, because each camera is processing its stream independently. The text below from Intel's multiple camera white paper describes the process.

********************

Now to the somewhat counter intuitive aspect of time stamps. If NO HW sync is enabled, the time stamps will now surprisingly appear to be perfectly aligned. This is because each individual ASIC is counting the exact same number of cycles between frames, and then sending them off. So according to their own timelines they are sending frames at say, exactly 33.333ms intervals, for a 30fps mode.

By contrast, if HW Sync is enabled, the time stamps will actually drift over time. You might expect this drift to be on the order of less than 1m s /minute. Knowing this, you can actually validate that your sync is working by looking at the drift in the difference of the time stamps. If you see NO DRIFT, then there is NO HW sync. If you see DRIFT, then the units are actually HW synced. (Yes, we know this looks like a typo!). 

[HChen92]

Thank you for the reply, MartyG. Yeah. That is me posting it there.

I used the lastest SDK(I think it is 2.16.x) and the firmware of the D415 camera is 05.11.01.00.

The following is the output of the 3 D415 cameras when I followed the github thread: https://github.com/IntelRealSense/librealsense/issues/2637. The last columns is the depth timestamp and the color timestamp seperately. It seems the timestamps from depth sensor are the same.

2,frames from ,839112061702:,Drift:,3.326904296875e+01,System Time ,TS:,1.547177302739e+12(118),System Time ,TS:,1.547177302739e+12(118),System Time ,TS:,1.547177302727e+12(118),

2,frames from ,839112062372:,Drift:,3.332202148438e+01,System Time ,TS:,1.547177302738e+12(108),System Time ,TS:,1.547177302738e+12(108),System Time ,TS:,1.547177302735e+12(108),

2,frames from ,748512060635:,Drift:,3.334106445312e+01,System Time ,TS:,1.547177302739e+12(122),System Time ,TS:,1.547177302739e+12(122),System Time ,TS:,1.547177302740e+12(120),

2,frames from ,839112061702:,Drift:,3.330102539062e+01,System Time ,TS:,1.547177302772e+12(119),System Time ,TS:,1.547177302772e+12(119),System Time ,TS:,1.547177302761e+12(119),

2,frames from ,839112062372:,Drift:,3.332275390625e+01,System Time ,TS:,1.547177302771e+12(109),System Time ,TS:,1.547177302771e+12(109),System Time ,TS:,1.547177302769e+12(109),

2,frames from ,748512060635:,Drift:,3.330859375000e+01,System Time ,TS:,1.547177302772e+12(123),System Time ,TS:,1.547177302772e+12(123),System Time ,TS:,1.547177302773e+12(121),

2,frames from ,839112061702:,Drift:,3.318017578125e+01,System Time ,TS:,1.547177302805e+12(120),System Time ,TS:,1.547177302805e+12(120),System Time ,TS:,1.547177302794e+12(120),

2,frames from ,839112062372:,Drift:,3.329370117188e+01,System Time ,TS:,1.547177302805e+12(110),System Time ,TS:,1.547177302805e+12(110),System Time ,TS:,1.547177302802e+12(110),

2,frames from ,748512060635:,Drift:,3.288647460938e+01,System Time ,TS:,1.547177302805e+12(124),System Time ,TS:,1.547177302805e+12(124),System Time ,TS:,1.547177302807e+12(122),

2,frames from ,839112061702:,Drift:,3.284350585938e+01,System Time ,TS:,1.547177302838e+12(121),System Time ,TS:,1.547177302838e+12(121),System Time ,TS:,1.547177302828e+12(121),

2,frames from ,839112062372:,Drift:,3.332934570312e+01,System Time ,TS:,1.547177302838e+12(111),System Time ,TS:,1.547177302838e+12(111),System Time ,TS:,1.547177302836e+12(111),

2,frames from ,748512060635:,Drift:,3.329711914062e+01,System Time ,TS:,1.547177302838e+12(125),System Time ,TS:,1.547177302838e+12(125),System Time ,TS:,1.547177302840e+12(123),

2,frames from ,839112061702:,Drift:,3.361938476562e+01,System Time ,TS:,1.547177302872e+12(122),System Time ,TS:,1.547177302872e+12(122),System Time ,TS:,1.547177302862e+12(122),

2,frames from ,839112062372:,Drift:,3.365258789062e+01,System Time ,TS:,1.547177302872e+12(112),System Time ,TS:,1.547177302872e+12(112),System Time ,TS:,1.547177302870e+12(112),

2,frames from ,748512060635:,Drift:,3.333032226562e+01,System Time ,TS:,1.547177302871e+12(126),System Time ,TS:,1.547177302871e+12(126),System Time ,TS:,1.547177302874e+12(124),

2,frames from ,839112061702:,Drift:,3.273999023438e+01,System Time ,TS:,1.547177302905e+12(123),System Time ,TS:,1.547177302905e+12(123),System Time ,TS:,1.547177302895e+12(123),

2,frames from ,839112062372:,Drift:,3.299511718750e+01,System Time ,TS:,1.547177302905e+12(113),System Time ,TS:,1.547177302905e+12(113),System Time ,TS:,1.547177302903e+12(113),

2,frames from ,748512060635:,Drift:,3.327880859375e+01,System Time ,TS:,1.547177302905e+12(127),System Time ,TS:,1.547177302905e+12(127),System Time ,TS:,1.547177302908e+12(125),

2,frames from ,839112061702:,Drift:,3.349755859375e+01,System Time ,TS:,1.547177302938e+12(124),System Time ,TS:,1.547177302938e+12(124),System Time ,TS:,1.547177302929e+12(124),

2,frames from ,839112062372:,Drift:,3.328540039062e+01,System Time ,TS:,1.547177302938e+12(114),System Time ,TS:,1.547177302938e+12(114),System Time ,TS:,1.547177302937e+12(114),

2,frames from ,748512060635:,Drift:,3.331958007812e+01,System Time ,TS:,1.547177302938e+12(128),System Time ,TS:,1.547177302938e+12(128),System Time ,TS:,1.547177302941e+12(126),

2,frames from ,839112061702:,Drift:,3.321484375000e+01,System Time ,TS:,1.547177302971e+12(125),System Time ,TS:,1.547177302971e+12(125),System Time ,TS:,1.547177302962e+12(125),

2,frames from ,839112062372:,Drift:,3.333105468750e+01,System Time ,TS:,1.547177302971e+12(115),System Time ,TS:,1.547177302971e+12(115),System Time ,TS:,1.547177302970e+12(115),

2,frames from ,748512060635:,Drift:,3.334765625000e+01,System Time ,TS:,1.547177302971e+12(129),System Time ,TS:,1.547177302971e+12(129),System Time ,TS:,1.547177302975e+12(127),

2,frames from ,839112061702:,Drift:,3.369140625000e+01,System Time ,TS:,1.547177303005e+12(126),System Time ,TS:,1.547177303005e+12(126),System Time ,TS:,1.547177302996e+12(126),

2,frames from ,839112062372:,Drift:,3.331591796875e+01,System Time ,TS:,1.547177303005e+12(116),System Time ,TS:,1.547177303005e+12(116),System Time ,TS:,1.547177303004e+12(116),

2,frames from ,748512060635:,Drift:,3.414550781250e+01,System Time ,TS:,1.547177303005e+12(130),System Time ,TS:,1.547177303005e+12(130),System Time ,TS:,1.547177303008e+12(128),

2,frames from ,839112061702:,Drift:,3.287768554688e+01,System Time ,TS:,1.547177303038e+12(127),System Time ,TS:,1.547177303038e+12(127),System Time ,TS:,1.547177303030e+12(127),

2,frames from ,839112062372:,Drift:,3.332983398438e+01,System Time ,TS:,1.547177303038e+12(117),System Time ,TS:,1.547177303038e+12(117),System Time ,TS:,1.547177303038e+12(117),

2,frames from ,748512060635:,Drift:,3.246777343750e+01,System Time ,TS:,1.547177303038e+12(131),System Time ,TS:,1.547177303038e+12(131),System Time ,TS:,1.547177303042e+12(129),

2,frames from ,839112061702:,Drift:,3.388940429688e+01,System Time ,TS:,1.547177303072e+12(128),System Time ,TS:,1.547177303072e+12(128),System Time ,TS:,1.547177303063e+12(128),

2,frames from ,839112062372:,Drift:,3.341406250000e+01,System Time ,TS:,1.547177303071e+12(118),System Time ,TS:,1.547177303071e+12(118),System Time ,TS:,1.547177303072e+12(118),

2,frames from ,748512060635:,Drift:,3.398901367188e+01,System Time ,TS:,1.547177303072e+12(132),System Time ,TS:,1.547177303072e+12(132),System Time ,TS:,1.547177303076e+12(130),

2,frames from ,839112061702:,Drift:,3.291577148438e+01,System Time ,TS:,1.547177303105e+12(129),System Time ,TS:,1.547177303105e+12(129),System Time ,TS:,1.547177303097e+12(129),

2,frames from ,839112062372:,Drift:,3.327685546875e+01,System Time ,TS:,1.547177303105e+12(119),System Time ,TS:,1.547177303105e+12(119),System Time ,TS:,1.547177303105e+12(119),

2,frames from ,748512060635:,Drift:,3.271044921875e+01,System Time ,TS:,1.547177303105e+12(133),System Time ,TS:,1.547177303105e+12(133),System Time ,TS:,1.547177303109e+12(131),

2,frames from ,839112061702:,Drift:,3.313452148438e+01,System Time ,TS:,1.547177303138e+12(130),System Time ,TS:,1.547177303138e+12(130),System Time ,TS:,1.547177303130e+12(130

[chung15]

hi, I also face the same issue, which the frames are not synced when I tried to use wait_for_frames.

I also saw your replies on Githun and wondering if you found the answer to get the two frames synced.

I tried rs.syncer in python and cant get it work, it would be great if you can provide your example for using the syncer.

my frame data is below:

every 10 second/press space take one frame. and the result frame is actually matched if i accept +-20 ms. But when I use wait_for_frames I can#t get the right match.