RealSense Capabilities with Aero RTF Drone

[JLy]

Hey Guys

Working on getting my drone up and running and am very interested in combining the technologies shown in this video (https://www.youtube.com/watch?v=8lh2EiwQjJ8 IDF Shenzhen 2015 Intel RealSense Technology Partnership w/JD.com - YouTube)

with the Aero drone kit.

Some of the things I would like to be able to do:

Hover the drone a preset distance from a wall or box

Ability to measure and detect boxes - similar to how the JD.com demo

Direct the drone to fly to a identified location autonomously

Not completely sure where to start. I have read forums that discuss how to do this with just the Realsense, but I don't see much combining it with the Aero. I am looking for any suggestions, resources, examples, etc.

Thanks!

[MartyG]

The 'Intel Aero Platform for UAVs' forum is the place where you can get specialist support for drone building and using Aero with RealSense.

Although I am a RealSense specialist, my knowledge of RealSense + drones is less developed. I will do my best to assist you though.

For the autopilot part of your project, I would recommend relying on a Flight Controller component. Intel's own "Ready To Fly" drone kit supports an open-source controller called DroneCode PX4, so I would suggest that this may be a very reliable choice for partnering with an Aero board.

http://px4.io/ Open Source for Drones - PX4 Pro Open Source Autopilot

This page describes the autopilot function of DroneCode:

https://docs.px4.io/en/getting_started/px4_basic_concepts.html https://docs.px4.io/en/getting_started/px4_basic_concepts.html

You can find a repository of other software components for Aero on this page:

https://github.com/intel-aero Intel® Aero Platform for UAVs · GitHub

The Aero board prefers to use a flavor of Linux called Yocto Project, rather than the more popular Ubuntu. You can develop an app in Ubuntu on a full PC and port your code to the Aero board though. An application can also be wirelessly streamed from an Ubuntu machine on the ground to the drone via wi-fi, though this option may not be practical if your drone is going to travel a long distance.

One of the components in the previously mentioned repository of software components is the ability to add support for Librealsense SDK (and thus the ability to use RealSense cameras with Aero) to Yocto Project Linux.

https://libraries.io/github/IntelRealSense/meta-intel-librealsense https://libraries.io/github/IntelRealSense/meta-intel-librealsense

Once you have a Librealsense installation running on the drone in Yocto, that ought to make it possible to use traditional RealSense applications such as scanning box sizes. Speaking of that box-scanning application, there is a tutorial available for it.

Regarding box measurement with RealSense, you may find this research paper useful:

https://www.researchgate.net/figure/304897148_fig6_Figure-6-Real-time-and-automatic-measurement-of-box-dimensions-using-RealSense-R200 Real-time and automatic measurement of box dimensions using... | Figure 6 of 6

[JLy]

Marty - thanks so much for the support. Many useful links you have provided!

The Aero board prefers to use a flavor of Linux called Yocto Project, rather than the more popular Ubuntu. You can develop an app in Ubuntu on a full PC and port your code to the Aero board though.

Most of my prior development has been done using Windows and i am not too familiar with Linux - is this still practical for developing on the Aero?

The part I am a bit confused on is exactly how to transfer apps to the Aero.. I do not think I have the librealsense SDK installed. I want to get started learning with some of the demos that are on the following links but I don't completely understand how to transfer/compile them.

https://github.com/intel-aero/sample-apps GitHub - intel-aero/sample-apps: This repository will contain source code for sample applications

https://software.intel.com/en-us/articles/intel-aero-realsense-streaming-rgb-depth-data# _Toc467619798 Code Sample: Broadcast RGB and Depth Data from an Intel® Aero Compute Board | Intel® Software

https://github.com/intel-aero/meta-intel-aero/wiki/06-Computer-Vision-3D-AI 06 Computer Vision 3D AI · intel-aero/meta-intel-aero Wiki · GitHub

Thanks again!

[MartyG]

Linux comes in a number of different varieties with different names (Ubuntu, Debian, Yocto, Arch Linux, etc). RealSense focuses its support on Ubuntu for the RealSense camera models. So Ubuntu = Linux.

This guide on setting up an Aero development environment may be of use to you.

https://software.intel.com/en-us/articles/intel-aero-compute-board-developer-guide-for-setting-up-a-simulation-environment Set Up a Simulation Environment with the Intel® Aero Compute Board Developer Guide | Intel® Software

See the 'porting simulation to the Aero Compute Board' link in this article's contents list for beginner setup resources.

If you are using the Ready To Fly drone kit then the ideal RealSense partner for the Aero board is the Vision Accessory Kit, supplied with the RTF.

[JLy]

I think that link is exactly what i need to get started. Let me read it this weekend and I'll see if I run into any issues.

[MartyG]

Best of luck with your study. I look forward to hearing your report on how it goes for you. Have a great weekend!

[JLy]

Thanks again - you as well, I will check in next week

[JLy]

Hi Marty -

Hope you had a good weekend. I made some progress with the Aero, but I am still left a bit confused.

Here is what I accomplished -

1) Determined it would be easier to develop on Ubuntu so I set up a VM with Eclipse as well as the realsense and gstreamer libraries

2) Attempted to port one the sample applications on the Aero (https://software.intel.com/en-us/articles/intel-aero-realsense-streaming-rgb-depth-data# _Toc467619823 Code Sample: Broadcast RGB and Depth Data from an Intel® Aero Compute Board | Intel® Software

- Opened in Eclipse and was able to build

- Followed steps to transfer to the Aero

- Building on the Aero resulted in the following errors:

I am starting to get a better understanding of how all of this fits together, but I don't know where to go from here. I was under the impression that the libraries were installed on the Aero?

Thanks

[MartyG]

I don't believe Librealsense is preloaded on Aero by default, because it is not needed by Aero unless the RealSense Vision Accessory Kit is attached to the board. So I think you should add this layer which enables Librealsense to work with Yocto if you have not done so already.

https://github.com/intel-aero/meta-intel-aero-librealsense GitHub - intel-aero/meta-intel-aero-librealsense: Yocto layer for librealsense

[JLy]

Marty, excuse my lack of knowledge but I am stuck at the first step. I really do appreciate your help with this.

"Add the meta-intel-realsense layer to conf/bblayers.conf in your build directory"

How do I edit bblayers.conf?

[MartyG]

There is no need to apologize - I didn't know the answer to this question either. I researched a guide to adding layers in Yocto though, and this is my best guess (not being a drone expert) at what the procedure should be.

conf/bblayers.conf is the name of a file in your build directory that contains a list of layers so that the build system is aware of the existence of those layers. The BBLAYERS += instruction adds the Librealsense layer to the layer list so that the build system can find it. In this case, it is telling Yocto how to find a layer called 'meta-intel-realsense'.that has alredy been created.

So I would guess that you would input the following list of three commands into the Yocto terminal window.

**************

BBLAYERS += "path/to/meta-intel-realsense"

BBLAYERS += "path/to/meta-openembedded/meta-oe"

require include/intel-librealsense.inc

*************

The first of these lines sets up the layer in the conf/bblayers file.

The second line adds dependency layers to the conf/bblayers file.

The third and final line creates a new conf/auto.conf file in your build directory.

[JLy]

Hi Marty

I tried the command through SSH and get the error "command not found"

Also, I am unsure of the path to the libraries on the Aero

[MartyG]

I've checked through this issue again, and the documentation page I linked to does seem to be the correct one for installing Librealsense on Yocto. So I went through past references on the Intel drone forum. Intel support staff member Andres said that on the Ready To Fly drone that you have, "Librealsense, as well the corresponding examples are already installed in the RTF drone." Andres pointed that user to this page:

https://github.com/intel-aero/meta-intel-aero/wiki Home · intel-aero/meta-intel-aero Wiki · GitHub

I apologize for not knowing this, if this is the case - it is not mentioned in any documentation that I have seen.

Apparently, the path that the pre-installed Librealsense examples are at is:

/usr/share/librealsense/examples/

[JLy]

Documentation for the Aero is very difficult to comprehend and sift through. This is frustrating for me.

I have navigated to that directory and do not see the "librealsense" folder.

I do however see gstreamer and a handful of others. Do you know how to install the examples?

[MartyG]

Andres said in a May 2017 posting that the examples have been moved to this directory in the latest version of the OS

/usr/bin/

[JLy]

I have checked the directory and again no "librealsense"

Is there a way we can tag Andres on this to get his input?

[MartyG]

It would be difficult to tag in Andres specifically, as Intel support agents use a single shared account and post their name manually at the bottom of their messages. The best hope is that Andres reads the message or another agent lets him know about this case so he can contribute.

If I read Andres' message about the change correctly though, there is no 'librealsense' folder in the new usr/bin location. The sample files are placed inside it by project name. For example, the Librealsense sample cpp-capture's address is in usr/bin/cpp-capture

[JLy]

We are making progress...I think

You are right, the examples are in there, but when I execute them I get the following errors:

[MartyG]

The UVC_SET_CUR Error 5 error is rather difficult to diagnose, unfortunately.

I would recommend trying another sample in the usr/bin folder first if there are other samples and seeing if that works.

I would also suggest checking if the three cameras are properly attached to the Aero board.

[JLy]

I have tried other examples with the same pattern, first the 'window != NULL' failed" and then the input/output error

[MartyG]

Your command-line name jon@jon-VirtualBox could provide a clue. Someone who had this kind of error was once told by one of the leaders of Librealsense, ddiakopoulost:

"I've tested in the past on Virtual Box and the USB 3.0 software translation layer is really buggy. It was a long time ago, but I believe I ran into the same issue. VMWare's hypervisor does at least work, but we still don't recommend it because of streaming timeouts after 30-60 seconds. Best bet is to try on the native hardware!"

Librealsense's front page on its website also says it doesn't like VMs.