Converting RealSense D415 bag file to PCD

[AMahm3]

I've received my RealSense D415 10 days ago (I'm not a developer) and I was trying to use it for room scanning. I've recorded a *.bag file through the viewer and the Unity wrapper and I'm trying to convert the .bag file to a point cloud; I've tried using ROS:

rosrun pcl_ros bag_to_pcd

But the output directory is always empty, so is there a way that I can use to get a point cloud (then create a mesh) from the D415?

BTW: I've tried the *.PLY in the viewer and that's not a 3D scan it's simply a depth snap/shot of what is directly in-front of the camera not a real 3D scan. RecFusion can be used but it has limitations on the size of the object you want to scan.

Thanks in advance.

[idata]

Hello GamesMechanic,

I have replicated your issue and I did not encounter the same errors as you. Here are my steps:

1. Downloaded the latest Librealsense build 2.16 (Source Code (zip) ) from: https://github.com/IntelRealSense/librealsense/releases
2. Unziped it on C:/git/
3. Opened CMAKE GUI and used C:/git/librealsense-2.16 as "where is the source code" and C:/git/librealsense-2.16/build as "where to build the binaries"
4. Pressed Configure and selected the compiler
5. Made sure that "BUILD_SHARED_LIBS" is unchecked and "BUILD_EXAMPLES" is checked.
6. Pressed Generate and after that Open Project

Please let me know if this works with you as well! Best regards,Eliza

[AMahm3]

Hello Eliza,

Thanks for your detailed reply, I've tried it and it produced the some errors seen below in visual studio:

After some research I've found that the error is from the fact that the altstr.h isn't installed with visual studio and I've to install it as per the screenshot below:

The build was then completed, after that when I try to run the executable file generated i get a black screen splash and an error txt file is generated that is empty as seen below (the first two txt files):

I would appreciate your help to solve this problem. Thanks

[GRedd]

Could we please get a compiled version of this convertor added to the sdk like it is for all the other tools? I think this would be very helpful to a lot of people.

[GRedd]

Hi Eliza,

I am also following along here and have managed to follow your instructions and open the solution in visual studio. I can build and run the viewer from source, seems to work fine. But when I try to run the convertor in visual studio I get the following errors,

'rs-convert.exe' (Win32): Loaded 'C:\git\librealsense-2.16.0\build\Debug\rs-convert.exe'. Symbols loaded.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\ntdll.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\kernel32.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\KernelBase.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\user32.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\win32u.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\gdi32.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\gdi32full.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\msvcp_win.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\ucrtbase.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\ole32.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\combase.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\rpcrt4.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\bcryptprimitives.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\sechost.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\oleaut32.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\advapi32.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\msvcrt.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\shlwapi.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\cfgmgr32.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\setupapi.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\mf.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\mfplat.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\winusb.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\mfreadwrite.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\kernel.appcore.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\mfcore.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\crypt32.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\msasn1.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\powrprof.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\ksuser.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\bcrypt.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\cryptbase.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\imm32.dll'. Cannot find or open the PDB file.

'rs-convert.exe' (Win32): Loaded 'C:\Windows\System32\RTWorkQ.dll'. Cannot find or open the PDB file.

Exception thrown at 0x00007FFFDBC4A388 in rs-convert.exe: Microsoft C++ exception: TCLAP::CmdLineParseException at memory location 0x00000081C55ADE20.

Exception thrown at 0x00007FFFDBC4A388 in rs-convert.exe: Microsoft C++ exception: TCLAP::ExitException at memory location 0x00000081C55AB660.

The thread 0x58ac has exited with code 1 (0x1).

The thread 0x4298 has exited with code 1 (0x1).

The thread 0x3bdc has exited with code 1 (0x1).

The thread 0x351c has exited with code 1 (0x1).

The program '[21656] rs-convert.exe' has exited with code 1 (0x1).

[AMahm3]

Yep, that would be great but does this tool work on an entire bag file (15/30 fps) or frame by frame that's important to know.

[idata]

Hello GamesMechanics and gpred,

Thank you for the information provided! I managed to get the convert tool working by doing the following:

• Opened RealsenseToolsConvert from C:\git\librealsense-2.15.0\build\tools\convert (this is my view)
• Build the entire solution. On my case, I had all the files on "Solution Configuration" Debug and "Solution Platform" Win32 (This is my case, yours can be different as long as every single solution from "Solution Explorer" has the same configuration)
• After I got Build Successful I have opened rs-convert.exe from C:\git\librealsense-2.15.0\build\Debug\rs-convert.exe (the file path can be found by: right-click on rs-convert from Visual Studio -> Properties -> Linker -> Command Line -> right next to "OUT" )
• A terminal has opened and I used the command from https://github.com/IntelRealSense/librealsense/tree/master/tools/convert

I hope this will help you! Best regards,Eliza

[GRedd]

Thanks Eliza. I managed to get it to build by following your instructions. Unfortunately when I run rs-convert.exe it just exits immediately and produces an empty text document as a debug log.

[idata]

Hello gpred,

My apologizes, the last step mentioned by me is not necessary.Could you please open a terminal in the location of the rs-convert.exe and use the command from https://github.com/IntelRealSense/librealsense/tree/master/tools/convert?Thank you! Eliza

[AMahm3]

Hello Eliza,

Thanks for your reply, I'm still facing the same problem I've reported before and gpred is reporting below, when I start the rs-convert it just closes instantaneously and an empty text file is created. I've tried building everything and most of the tools and intel viewer work except the converter. I've also tried creating the binaries again from scratch using CMake and that didn't change anything.

I've been trying/research the ROS nodelet to convert the bag file and couldn't get my head around it and it would be great if intel would create a simple tutorial regarding that.

Appreciate your feedback, thanks.

[idata]

Hello GamesMechanic,

Could you pleas open a Command Prompt windows in the location of the rs-convert.exe (for me it is C:\git\librealsense-2.15.0\build\Debug\) and run the command from https://github.com/IntelRealSense/librealsense/tree/master/tools/convert ? Thank you!Eliza

[GRedd]

Hi Eliza, I have now opened the terminal in the location of rs-convert.exe and used the command from the link provided

Running,

C:\git\librealsense-2.16.0\build\Debug>rs-convert -i 20180713_120746.bag -p C:\plyout

Produces the following output,

99%

PNG converter

134 Infrared frame(s) processed

134 Depth frame(s) processed

And 134 depth frames and infrared frames are output to C:\ and appended with the filenames plyout as expected

Running,

C:\git\librealsense-2.16.0\build\Debug>rs-convert -i 20180713_120746.bag -l C:\plyout

Produces the output,

99%

PLY converter

But unfortunately no ply files are produced and it is these that I require.

[AMahm3]

Hello Eliza,

Thanks for your reply, you meant running it from CMD, that was my problem. Well the tool ran and generated many many many ply files (one for each frame) which isn't useful as the ply is a mesh not a point cloud so I can't construct a model from it. I though the tool would generate/construct a model from the point cloud captured.

The tool is of little/no use as it's very similar to the capture tool in the viewer. Would you please provide a small tutorial/guide on how to use nodelets (ROS) to convert the bag file to PCD as that would be more useful for applications like whole room scanning.

Thanks in advance.

[GRedd]

Great that you have been able to get it working to get ply files GamesMechanic......I wish it was working for me.

I think you ply files can still be used to get a point cloud depending on how they are specified. If they are just a series of vertices, which the viewer ones appear to be I think, then this is essentially a point cloud.

For example, I can convert a single exported frame from the viewer from a ply to a pcd in Matlab using the code pasted below. If I had many ply files (like you now do) I could simply run over them all in a loop to do the conversion.

ptCloud = pcread('C:\testply.ply');

pcshow(ptCloud);

pcwrite(ptCloud,'C:\testpcd.pcd','Encoding','ascii');

[AMahm3]

Thanks for the information gpred, it's really very helpful, I'll give it ago, though I still wish that I was able to scan and generate a mesh for large objects/rooms automatically like RecFusion (but it's limited to 5m x 5m x 5m).

Wish that you would solve your problem with the ply converter soon. I've found this code for converting bag files (generated by cameras like D400) to PCD using ROS nodelets:

https://gist.github.com/bhaskara/2400165 Example ROS launch file that uses depth_image_proc to convert an RGB-depth image pair into a point cloud · GitHub

Don't know how to use it yet, but it might be helpful for others, or someone in the community might be able to inform us. Thanks again Eliza and gpred.

[MartyG]

Intel was doing room scanning with RealSense back in 2015, using a RealSense Smartphone Development Kit (an Android phone with a ZR300 camera in that had some test units made for developers but never got released on public sale).

https://www.youtube.com/watch?v=XU54hK1UpTA Intel combines RealSense 3D technology with Google Project Tango @ IDF 2015 - YouTube

With the modern-day 400 Series cameras, Intel did "volumetric capture" of humans by capturing their data as a point cloud.

https://realsense.intel.com/intel-realsense-volumetric-capture/ Volumetric Capture @ Sundance using Intel RealSense Depth Cameras

The FAQ about the project suggests that rendering as a point cloud rather than a mesh was a technical choice, and not because it couldn't be done. Indeed, 'surface reconstruction' (turning a point cloud into a mesh) can be done by a variety of techniques. If you want to do the conversion within the RealSense SDK, the best option may be to combine RealSense SDK 2.0 with the OpenCV computer vision software (which SDK 2.0 is fully compatible with) so you can access the range of conversion techniques. Google for 'opencv surface reconstruction' for more details.

If you would prefer a pre-made solution, OpenCV has a module called OpenSfM for converting image files to 3D reconstructions.

OpenSfm:

https://github.com/mapillary/OpenSfM GitHub - mapillary/OpenSfM: Open source Structure from Motion pipeline

Documentation:

https://media.readthedocs.org/pdf/opensfm/latest/opensfm.pdf https://media.readthedocs.org/pdf/opensfm/latest/opensfm.pdf

Another approach would be to use Monocular SLAM to render a point cloud into a mesh in real-time:

https://www.doc.ic.ac.uk/~gj414/monocular_slam/monofusion.html https://www.doc.ic.ac.uk/~gj414/monocular_slam/monofusion.html

[GRedd]

No worries GamesMechanic. Glad I could help.

I only wish Intel would be a bit more transparent and helpful. There are so many applications for capturing point clouds with a timestamp at high frequency. But for reasons unknown Intel have made this very difficult and seem to be only interested in providing incremental half baked solutions. Today I modified the code of the realsense viewer to display milliseconds. The code to display milliseconds was commented out, for reasons which are beyond me. This is exactly what we need for our and I am sure many others applications, point clouds with a high resolution time stamp. The fact that the code for this was in the viewer but removed, gives me pause for concern. Why?

We captured very good point cloud data at around 20 fps with the SR300 for a research project. Based on the hype around the 400 series being even better we ordered 12 of them with plans to order many more if things went well. So far we have not managed to get the same data from the 400 series due to the poor documentation and lack of examples for SDK 2. What we have managed to get (at lower sample rates due to the previously mentioned difficulties) suggests that the point cloud accuracy of the 400 series is comparatively poor under the same environmental settings (laboratory with controlled lighting). I can only conclude that Intel is making it difficult to get point cloud data as they know the performance of these cameras are comparatively poor. I certainly hope that's not the case, but have little evidence to affirm the contrary at this point.

I will keep trying as we have invested a fair bit in this as a solution. But to anybody starting out I would say consider the SR300 if you can get your hands on enough of them. Failing that, even the kinetic may be more robust/reliable. At the very least watch this space before making a decent investment. We aim to publish some work on the comparative performance of these cameras in a peer reviewed academic journal, and that will hopefully be of value to people looking to use this technology.

[MartyG]

Hi gpred,

The SR300 and the 400 Series use different depth scales by default, in order for the 400 Series to be able to 'see' a lot further than the SR300. This means though that at close range, the 400 Series is not as accurate as the SR300. You can though change the 400 Series' depth scale to match the SR300's if you wish. The link below provides explanation.

[GRedd]

Thanks Marty. Will have a go.

[AMahm3]

Hello MartyG,

Thanks so much for the information, of course I'll try that, the results you've attached is very interesting, as with gpard we've ordered two of the D400 cameras and but we were aiming at using it for full room scanning and large machinery scanning. Thanks again.

[AMahm3]

Hello gpred,

Yes, totally agree with you on that, I think they're eyeing only very specific applications.

[MartyG]

Intel's goal is actually to have the 400 Series cameras and its SDK on as many hardware and software platforms as possible. The difficulty is though that this means there are so many different bases to cover that, for very specialized applications, the developer community can respond faster with a solution than the RealSense team can.

An example of this is the Nvidia Jetson TX boards. Until recently, the main source of help for getting RealSense up and running on it was websites such as JetsonHacks or RealSense community members who had already done it and could offer advice to others. When Intel gets the opportunity, they are sometimes able to add official support for the more popular niche applications, such as the addition in SDK 2.13.0 of CUDA-optimized implementation for Jetson-TX.

RealSense SDK is open-source so that developers can contribute code for integration into the SDK, via a 'pull'.