topic Re: D435 VSYNC Pin in Items with no label

D435 VSYNC Pin

JWare — Thu, 02 Aug 2018 22:52:38 GMT

Is it true that external synchronization mode is only supported by the D415? We noticed that there is a 1.8V pulse that appears on the VSYNC pin at the camera framerate. Can we rely on this as a synchronization mechanism for external devices for frames captured on the D435?

If the pulse on the VSYNC pin is tied to the image capture in a reliable and repeatable way, when does it occur relative to image capture? For example, is it at the beginning of the frame capture, at the end, or something else?

Does changing the RS2_OPTION_INTER_CAM_SYNC_MODE option affect the behavior of the VSYNC pin in default vs master mode?

Regards,

-Jake

Re: D435 VSYNC Pin

MartyG — Fri, 03 Aug 2018 06:37:48 GMT

Here's the long version, if you want to get into the technicalities of the subject ...

Here are the pin-out details from the 400 Series data sheet document for the external sync pins

.I guess you are referring to pin 5, the ZVSYNC?

Earlier this year before the sync function was supported, another user who tried to get useful output from the pins said: "We tried to read out pin 1 from the external sensors sync connector (GVSYNC0) while enabling the "Output Trigger" option in the RealSense viewer but we could only measure a very noisy signal containing shutter noise, and not a clear trigger signal (e.g. 5V TTL)". They were told by an Intel support representative that support in the firmware was required for it to work.

The experiences of the above user suggest that you might be able to get some kind of noisy raw output from a pin but it may not be usable without the firmware support for sync on D435.

The multi-camera sync white paper says that whilst you can have a master and slave setup with the cameras, "Alternatively an external signal generator can also be used as the master trigger with all cameras set to slave mode. When applying an external sync pulse, the HW SYNC input requires a 100 microsecond positive pulse at the nominal camera frame rate, 33.33 ms for a 30Hz frame rate for example".

"Inputs are high impedance, 1.8V CMOS voltage levels. However, it is important to make sure to use a high resolution signal generator. The frequency of the signal generator needs to exactly match the sensor frame rate. For example, if the sensor is set up as 30 fps, the real frame rate may be 30.015 fps. You may need to use an oscilloscope to measure the real frame and configure the signal generator to the same frequency. For this reason, it may be better to just using one additional camera as master sync signal generator"

At the start of 2018, Intel did a demo of four D435s linked together with hardware sync cable to produce 'volumetric capture'. They initiated the start and stop of capture using a commercial grade flash.

https://realsense.intel.com/intel-realsense-volumetric-capture/ Volumetric Capture @ Sundance using Intel RealSense Depth Cameras

The principle of using the flash as a trigger was probably similar to this:

http://www.dslreports.com/forum/r31259855-Syncing-a-camera-and-a-pulsed-flash-light-using-a-TTL-signal http://www.dslreports.com/forum/r31259855-Syncing-a-camera-and-a-pulsed-flash-light-using-a-TTL-signal

Re: D435 VSYNC Pin

JWare — Fri, 03 Aug 2018 15:10:30 GMT

Thanks, Marty. You are right that I am referring to pin 5 of the External Sync Sensor Connector and that I want to get technical. Thanks for providing the additional details and all of your posts on this and other topics related to the realsense cameras. I have read and benefited from many of them. Let me add some more information to clarify.

I'm relatively certain the "other user" you are referring to created https://github.com/IntelRealSense/librealsense/issues/1212 this github issue. Similar to what you said, they claimed that pin 1 was noisy and unusable, but pin 5 seemed to have a usable pulse on it. We confirmed this and have setup a synchronization scheme for another sensor based on it. Note that this is true despite the fact that external synchronization isn't technically supported on the D435. For example, our experience is that checking sensors[DEPTH].supports(rs2_option::RS2_OPTION_INTER_CAM_SYNC_MODE) in the ROS driver returns false. I'm guessing the same would be true of setting it with the method outlined in the https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/RealSense_Multiple_Camera_WhitePaper.pdf multiple camera synchronization white paper you referred to. However, the fact remains that there is a 1.8V pulse on pin 5 of the External Sync Sensor Connector that occurs every frame. If nothing else, we just want to know if that pulse is actually tied to a specific event associated with frame capture. Any info on this would be much appreciated.

As an aside, we are using the D435 on a multi-rotor drone, so we would prefer to stick with its global shutter cameras as opposed to switching to the rolling shutter cameras of the D415. We will probably pick up a D415 just to investigate this situation, though.

Thanks again,

-Jake

Re: D435 VSYNC Pin

MartyG — Fri, 03 Aug 2018 15:34:31 GMT

Thanks for your kind words.

Whilst I could not find information of a relationship between pin 5 and frame generation, I wonder if such a pulse would manifest somehow in the frame metadata that is generated every frame.

https://github.com/IntelRealSense/librealsense/blob/master/doc/frame_metadata.md librealsense/frame_metadata.md at master · IntelRealSense/librealsense · GitHub

Re: D435 VSYNC Pin

JWare — Fri, 03 Aug 2018 16:05:27 GMT

Marty,

You are right that the frame metadata has useful information in it. We modified the ROS driver to extract and publish it so we could use the realsense's internal timestamps, exposure time, and frame counter. However, it isn't clear how we might use any of that information to figure out if the sync pulse is related to a particular event during frame capture.

Regards,

-Jake

Re: D435 VSYNC Pin

MartyG — Fri, 03 Aug 2018 16:20:46 GMT

I believe that a slave camera checks for sync, and it produces unsynced frames if it does not detect a sync signal. so there is apparently checking going on.

RealSense Group CTO Anders said recently: "When you set a camera to slave it will still acquire frames at the right frame rate even if there is no sync input signal. It tries to sync but if there is no signal that comes in at the right time, it gives up and triggers itself."

Re: D435 VSYNC Pin

idata — Sun, 05 Aug 2018 18:52:15 GMT

Do we have any indication for when this may be officially supported for D435? Thanks!

Re: D435 VSYNC Pin

MartyG — Sun, 05 Aug 2018 19:57:07 GMT

I have no knowledge of when D435 will be able to initiate sync in the same way D415 does, though apparently D435s can respond to an externally generated sync pulse.

In the absence of an official sync solution for D435, the next easiest approach may be to use a D415 as master sync signal generator and have all the slaves be D435s.

Re: D435 VSYNC Pin

idata — Tue, 07 Aug 2018 01:30:42 GMT

Marty, what version of the Product Data Sheet did you reference above? I'm looking at July 2018 and they're a little bit different. Is there a newer version?

Looks like you were looking at January 2018, actually (update).

Thanks!

Re: D435 VSYNC Pin

MartyG — Tue, 07 Aug 2018 06:04:38 GMT

Yes, there were some changes across editions. In the March 2018 edition onward, for example, the section relating to adding a tracking module component to the camera module was removed because the tracking module was discontinued by Intel, and they decided not to enable use of the 50 pin tracking module connector for custom components.

There were also separate editions published, one at the camera's launch that covered only the D415 and D435 USB cameras, and a larger one that covered all the modules in the 400 Series family. The pin-out details quoted above can be found on page 46 of the most recent July 2018 edition of the larger publication that covers all modules.

https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/Intel-RealSense-D400-Series-Datasheet.pdf https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/Intel-RealSen…

However, I checked the layout of the information and you were correct - the images were taken from page 56 of the January 2018 edition. I could confirm this because the details are split across 2 pages in the January edition, and all on one page in the July one, and I remember copy-and-pasting from the two pages.

There are some descriptive differences in the most recent July 2018 pin-out. The main difference is that the chart no longer refers to the pins providing external sensor interrupts for the timestamp.

Here's January's one:

And here's the updated July one:

Re: D435 VSYNC Pin

idata — Thu, 09 Aug 2018 18:12:22 GMT

Marty, with a sync cable, is it possible to trigger the synchronization to be ON in the Realsense viewer? Or is custom software needed?

Re: D435 VSYNC Pin

MartyG — Thu, 09 Aug 2018 18:43:22 GMT

I do not have two 400 Series cameras so I cannot test sync myself. I would speculate though that once you add more than one camera to the RealSense Viewer via the Add Source option then the Sync button would un-ghost and be able to be used to trigger sync, with whatever of the cable-linked cameras that is set to Master being the one that provides the sync pulse. The master camera should be a D415. My understanding is that a D435 could be a slave but cannot currently act as the master signal generator, and that an all-D435 setup needs an external trigger signal

You should also be using at least SDK 2.13.0 (when sync was added to the SDK) and firmware 5.9.14 (when sync was added to firmware). You need sync support in both the SDK and firmware for it to work.

Of course, you should also have the sync cabling connected up before initiating sync.

I believe that when hardware sync is working correctly, you should no longer get the red System Time message (which is software sync) when clicking the info button on a stream's toolbar.

Re: D435 VSYNC Pin

MartyG — Sun, 12 Aug 2018 16:08:56 GMT

You may also be interested in a couple of multi-camera scripts that have been made available.

Re: D435 VSYNC Pin

Sahira_Intel — Wed, 15 Aug 2018 18:03:34 GMT

Hi jakeware,

Were you able to get your questions answered, or did you need further assistance?

Please let us know!

Regards,

Intel Customer Support

Re: D435 VSYNC Pin

JWare — Thu, 16 Aug 2018 04:45:34 GMT

srizvi, thanks for following up on this thread. My questions has not yet been answered. For clarity, I will restate the the most important and relevant subset of my questions below.

When you enable the output trigger on the D435, there is a 1.8V pulse that appears at the camera framerate on pin 5 (VSYNC) of the External Sensor Sync Connector pin. Is this pulse on the VSYNC pin tied to image capture in a reliable and repeatable way? If so, when does it occur relative to image capture? For example, is it at the beginning of the frame capture, at the end, or something else?

Thanks again for your help,

-Jake

Re: D435 VSYNC Pin

idata — Tue, 21 Aug 2018 16:13:19 GMT

Hi Jake,

Thank you for being patient with us while we researched your question.

The output sync pulse can be recognized as the VSYNC pulse of the depth sensor. It outputs 7 lines after the end of exposure. The latest development firmware, v 5.10.3 with support for multi-camera sync is now available for download here!

https://downloadcenter.intel.com/download/28076/Latest-Firmware-for-Intel-RealSense-D400-Product-Family?v=t

Regards,

Sahira

Intel Customer Support

Re: D435 VSYNC Pin

MartyG — Tue, 21 Aug 2018 16:19:10 GMT

Thanks a lot for the heads-up about the firmware update, Sahira - I'll highlight it in a forum message and update it with the release details when the firmware documentation is updated.

Re: D435 VSYNC Pin

JWare — Tue, 21 Aug 2018 20:47:11 GMT

srizvi, thanks again for your feedback and no worries on the delay.

Can you clarify what you mean by "7 lines after the end of exposure"? Are you referring to lines of code or processed lines of the image? Is it a relatively constant amount of time? Do you have an estimate on how much time passes during those "7 lines"?

Also, my understanding was that 5.9.14 already supported multi-camera synchronization for the D415. Does this new version support it on the D435 explicitly? Can you just say a little bit more about how this new firmware version addresses the questions raised in this thread and the D435 specifically?

Thanks again,

-Jake

Re: D435 VSYNC Pin

idata — Tue, 21 Aug 2018 22:14:25 GMT

Hi jake,

7 lines of exposure time is equal to about 20 microseconds.

The 5.10.3 firmware adds multi camera sync support to the D435 and maintains multi camera sync support for the D415.

More information about multi-camera sync is available on the whitepapers here:

https://simplecore.intel.com/realsensehub/wp-content/uploads/sites/63/Multiple_Camera_WhitePaper04.pdf

Regards,

Sahira

Intel Customer Support

Re: D435 VSYNC Pin

JWare — Mon, 27 Aug 2018 20:36:01 GMT

srizvi, that new firmware is exactly what we needed and seems to have done the trick for us. We tested the new realsense firmware and our external IMU with https://github.com/ethz-asl/kalibr kalibr, and got sub-ms synchronization once we accounted for half the exposure time reported in the metadata. Thanks to you and everyone on the RealSense team for working on that and being responsive to threads like this.

Having done all that, we have one more question. We noticed that there is some latency associated with the realsense outputs. At 30 Hz, it seems to be about a framerate. That number also seems to change with the camera framerate. Can you guys speak to how much latency we should expect relative to the strobe on the VSYNC pin and how it varies with respect to framerate? Is it always about 1 frame capture period?

Many thanks,

-Jake