Use custom Framerates

[JSmit77]

Is it possible to have the pipeline return images at the maximum possible frame rate consistent with exposure?

Currently if I vary the exposure on the IR cameras in linear increments, the pipeline will return frames at what I believe to be the closest maximum rate from a predefined set of rates (specifically at 30, 25, 15, or 6).

For instance if I set the exposure levels as shown below, these are the corresponding frame rates I receive frames at (1920 x 1080 IR @ max 25 FPS, from pipeline) .

Exposure (microseconds) Frame rate (FPS) 50002510,0002515,0002520,0002525,0002530,0002535,0002540,0001545,0001550,0001555,0001560,0001565,0001570,000675,000680,000685,000690,0006

What I would like is for a smoother drop in frame rates as the exposure increases. I understand there is some underlying processing which limits how fast frames can be returned, but I would assume that if the pipeline can maintain 25 FPS at 35,000 exposure, then It should be able to maintain something slightly less than 25, say 20-24 FPS for a 40,000 exposure.

[idata]

Hello harmonic,

In the https://software.intel.com/sites/default/files/Intel_RealSense_Depth_Cam_D400_Series_Datasheet.pdf datasheet for Intel® RealSense™ Depth Camera D400 series page 52, you will see the supported framerates for each resolution. They match the frame rates that you are seeing. Frame rates are not continuous. Only certain frame rates are supported.I hope you find this information helpful.Please, let us know if further assistance is required or if we can close this case.Best regards,Josh B.Intel Customer Support

[JSmit77]

Hello Josh,

Thank you for the information.

I was just curious to see if it was possible to have custom rates. Currently my application operates at a constant rate anyway so operating at one of the predefined rates is not an issue.

Thanks,

Harmonic

[idata]

Hello harmonic,

Thank you for your reply. I am glad to hear that you found the information we shared with you useful. Please, let us know if further assistance is required or if we can close this case. Best regards, Josh B.Intel Customer Support

[idata]

Hi Josh,

Is it possible to have realsense cameras run at 24 and 25 frames per second? If not what is the possibility that these frame rates could be added?

They are extremely common frame rates for video production, and it's surprising to me that intel still doesn't support them after all of these years.

[MartyG]

Apologies for entering into Josh's case.

You can gain access to 25 FPS streaming in the RealSense Viewer software in IR mode if you set the resolution to the maximum 1920x1080, disable Depth and set the IR format to Y8 or Y16. If those conditions are met, the 'Off' button un-greys and becomes activatable.

[idata]

Thanks Marty, that's interesting. Can depth then be enabled at 25 fps after setting color and ir?

25 fps is definitely useful for European and other PAL countries. Now if it could just go one frame slower...

[MartyG]

It's slightly complicated. If you try to enable depth after successfully enabling 25 FPS then the off button re-disables. Also, 25 FPS only seems to be available for the D415 camera model, and when the camera is in Calibration mode, or using RY8_LY8 image format (Dynamic Calibration mode). Though somehow the RealSense Viewer allows that mode to be used without being in calibration mode.

[idata]

Hmm, a depth camera without depth doesn't seem particularly useful, but I'm sure it has it's use cases, and it's a good sign that additional industry standard formats are possible.

Thanks for your help, and I look forward to hearing from Josh.

[idata]

Hello harmonic,

Thank you for your reply. We will be sending your feedback to the RealSense engineering and development team; we will be updating this thread when we get a response. Thank you for your patience and understanding. Best regards, Josh B.Intel Customer Support

[idata]

Hello harmonic,

Thank you for your patience. We understand that 24 and 25 fps are common frame rates in video production. However, RealSense cameras are depth cameras and, as such, are targeted and optimized for depth usages such as in robotics, VR, drones, etc. I hope you find this information helpful. Please, let us know if further assistance is required or if we can close this case. Best regards, Josh B.Intel Customer Support.

[idata]

Hi Josh,

Harmonic was not asking about 24 and 25 fps, I was.

I think it's worth pointing out that there are several drones that capture footage at 24 and 25 fps, and augmented and mixed reality applications greatly benefit from industry standard video formats.

It would be a shame to limit the functionality of the device and prohibit additional use cases. If it's possible to provide support for these frame rates (that have been standards for decades) I would strongly encourage it.

Is it at all possible on a hardware level to provide support for this, and if so what would it take to get it done?

Thanks again for your quick replies, it's greatly appreciated!

[idata]

It's also worth considering that right now, there is no such thing as a "standard" for depth camera framerates. There have only been a handful of commercial or near commercial depth cameras including microsoft kinect, dual camera smartphones, and realsense cameras. Also the dual camera iphone cameras capture depth at 24 fps, so it's not unheard of.

My point is; the "standard" for depth cameras are still being defined, in large part by Intel. I believe these formats need to be a part of that standard.

[idata]

Hello sam598,

Thank you for your reply, As mentioned before in the datasheet for Intel® RealSense™ Depth Camera D400 series page 52, you will see the supported frame rates for each resolution. Frame rates are not continuous. Only certain frame rates are supported. We will take your feedback into consideration for further development. Thank you for your patience and understanding. Please, let us know if further assistance is required or if we can close this case. Best regards, Josh B.Intel Customer Support

[idata]

Thanks Josh,

I am aware of the current frame rate limitations. My question (and I believe Harmonic's as well) was more asking if it's even possible for the hardware to support custom frame rates (is it a hardware limitation, or a firmware one?), and if so what would it take/when could developers expect support?

Since the feature is not implemented yet, and it's likely other developers will come to the forum to inquire about or request this feature, I'm not sure why you would want to close the case at this time?

Thanks again for your help and willingness to participate with the developer community.

[idata]

Hello Sam598 and Harmonic,

We need more information in order to help our engineering team property prioritize this feature request.Please clarify whether you are inquiring about supporting custom frame rates in Color or Depth, or both. Harmonic's original question above is for IR cameras.We understand that 24 and 25 fps are common rates for COLOR video cameras, but, what is the use case for these rates in DEPTH? Please explain how having these frame rates in DEPTH will be beneficial. What would 24 or 25 fps in DEPTH enable you to do that you cannot do with the rates provided?Keep in mind that the usages you mentioned are all for color cameras, not depth.Regards,Jesus G.Intel Customer Support

[idata]

Hello @Sam598 and @Harmonic,

I was able to find more information regarding shooting video at 24 fps vs 30 or 60 fps. The excerpt below is copied from: https://www.androidcentral.com/shooting-video-24fps-vs-30fps-vs-60fps?

Shoot everything in 30FPS or 60FPS all of the time.

This is because only very expensive cameras can truly shoot video at 24FPS, and the processing programs we have now are designed to convert higher speed videos into 24FPS "cinematic" mode. If you plan to do any serious editing and your camera can handle 60FPS for the full length of the video you'll be taking, use it. If not, use 30FPS. If you need that cinematic look of 24 frames (actually 24.9 or 25FPS in countries that use the PAL format for video), you do it in your video processing program.

Video shot at 30 or 60FPS and then converted will look better than video shot in a 24FPS mode. Only use 24FPS for special effects once you learn what the raw video looks like from your camera.In addition please refer to https://www.theverge.com/2017/7/26/16026238/smartphone-video-editing-apps-how-to-tips-iphone-android https://www.theverge.com/2017/7/26/16026238/smartphone-video-editing-apps-how-to-tips-iphone-android.Please let us know if you find this information useful.Regards,Jesus G.Intel Customer Support

[idata]

Hi Jesus,

Thank you for continuing to look into this, I really appreciate it.

There are a few points I would like to make.

1) The article's assertion that "only very expensive cameras can truly shoot at 24FPS" is flat out wrong. The article even contradicts itself. Most modern smartphones have the capability, or can be configured in the camera capture API to provide footage at 24 fps. Android phone's like the LG V20, V30 and the iPhone since it first allowed video capture have all had this feature. Additionally most consumer video cameras, including GoPro, DSLR and DJI Drone cameras all capture 24 fps. It is an incredibly common video format, that all of these other companies recognize as desirable to both professionals and consumers.

2) Unfortunately converting 30 or 60 to 24 fps never works. Dropping frames creates a stuttering effect since there is not a constant delta time between frames. There are technique using optical flow to create interpolated frames, but it is processor intensive, not realtime, and introduces undesirable motion artifacts. The only way to really capture 24 fps is to have the hardware capture it natively, which is what the companies and products above do.

3) To answer your earlier question, if you had the color camera running at 24 fps, it would follow that you would the depth camera to run at the same frame rate. There may be a few use cases, but I would imagine the vast majority of developers would want PAIRED color and depth frames.

In fact I'm having a hard time thinking of a use case where a developer would not want synchronized color and depth pairs. Having the color and depth frames out of sync with each other greatly complicates computer vision and scanning applications. Especially when you are dealing with a moving subject or camera.

On a somewhat related note, this is also why it's distressing for other developers to hear that the D435 will not support synchronized color and depth frames as advertised.

4) Again, my question was actually if it's even possible for the hardware to support custom frame rates (is it a hardware limitation, or a firmware one?), and if so what would it take/when could developers expect support?

If it is possible to allow custom frame rates for the color and depth cameras, but the team is worried about having the change the firmware or API for every requested frame rate, maybe allowing the API to set the frame rate with a single integer value would be the simplest solution?

Again, thank you for your time and responses, it is greatly appreciated and hopefully my points are coming across clear.

[idata]

Hello Sam598,

Thank you for your detailed response. I will find out from the engineering team whether the limitation is in the hardware of the firmware. I'm pretty sure this is a firmware limitation but I just want to make sure.Regarding the use case. RealSense cameras are primarily depth cameras. Depth usages are their primary targets. Requirements are driven by depth first, not color. Please let us know why you would need Depth at custom frame rates of 24 or 25 fps. We know why it would be useful in color, but why Depth? Saying that we should support it in depth because it is good for color usages is not a sufficient reason.Regards,JesusIntel Customer Support

[idata]

Hello Sam598,

I confirmed that this would require a firmware change; it is not a hardware limitation. Making this change has not been scoped so I cannot answer the rest of your question regarding what would it take to make this happen.Regards,JesusIntel Customer Support