Panorama image stitching

[NSkar]

I am interested in combining m RealSense camera video frames into stitched panoramic images: RGB, Depth, and combined RGB-D.

The task is pretty common, but I failed to find implementation, and developing it from scratch is possible, but time consuming... As opposed to regular panoramic image stitching, the parallax and feature points located on different planes are needed to be taken into account.

I have seen the SLAM with D435i, which sound similar, but it uses IMU. The image stitching can be done without IMU (most of your phones does it nicely).

I'd appreciate if anyone could direct me to a relevant code. Ideally open source, but software packages would also do. I would like it to stitch RGB-D, but stitching RGB only using/not using depth information will also be good.

Thanks!

[MartyG]

Back in 2013, Intel published an article on using two cameras for stitching of panoramic RGB images and provided C++ source code for use with Microsoft DirectShow.

​https://software.intel.com/en-us/articles/dual-camera-360-panorama-application

[NSkar]

Hi Marty.

This is not exactly what I was hoping for, but indeed relevant, so I will try to use it.

Thank you very much!

Best wishes,

Nikolay

[MartyG]

Glad I could help!

I did another research check and found a 2017 paper on panoramic stitching involving RealSense. It got overlooked in the original search because they titled the paper as 'panoramic estimation'. They created their images using a 'RealSense Snapshot', an early form of RealSense camera built into some tablets and laptops in 2015.

https://www.ingentaconnect.com/contentone/ist/ei/2017/00002017/00000015/art00003?crawler=true&mimetype=application/pdf

Here's a YouTube video about the Snapshot camera.

https://www.youtube.com/watch?v=gDgzfAolLs8

[NSkar]

Yep, this looks good, thanks!

Is there a way to get their code? I was not able to find in in their GitHub repositories, and I positive RealSense users will appreciate this code added to demos section.

Thanks!

[MartyG · Accepted solution]

I could not find source code for this particular paper. I did locate one of the papers about panoramic stitching listed in their reference sources though, which may provide further useful insights. This paper provides some formulas. The 2017 paper's approach was a hybrid approach of this paper and another method, because they found that the method in this paper did not cope well with motion and left artifacts. It may be suitable for you if your images do not involve moving objects though.

https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Zhang_Casual_Stereoscopic_Panorama_2015_CVPR_paper.pdf

I did further research about using OpenCV and Python to do panoramic stitching, with a focus on articles that had accompanying source code, and found this:

https://www.pyimagesearch.com/2016/01/11/opencv-panorama-stitching/

[NSkar]

Thanks!