Community
cancel
Showing results for 
Search instead for 
Did you mean: 
JFlip1
Beginner
2,100 Views

GStreamer depth data

Hi

I'm trying to capture depth data from the intel realsense on the rtf drone in order to do some processing on a workstation. First I setup a SSH connection and then I'm able to set up a stream but for the moment it's pretty useless. On the workstation I execute following command:

gst-launch-1.0 -e udpsrc port=5013 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, a-framerate=(string)30.000000, a-framesize=(string)640-480" ! rtph264depay ! avdec_h264 ! autovideosink

And on the drone:

gst-launch-1.0 v4l2src device=/dev/video12 do-timestamp=true ! autovideoconvert ! vaapih264enc ! rtph264pay ! udpsink host=192.168.1.3 port=5013

With these commands a window will open on the workstation and a distorted green capture will be displayed. See image below.

I was wondering how to adjust the command in order to have proper dimensions and not have multiple layers above each other. Second I'm actually only interested in the integer depth values so I'd like to know how to write to a file (I assume filesink) and what file format to use.

Kind regards,

Jeroen Flipts

0 Kudos
12 Replies
idata
Community Manager
481 Views

Hello Jeroen,

Thank you for your interest in the Intel® Aero Compute Board.

I saw that you have been participating in this other thread /thread/113443 https://communities.intel.com/thread/113443, as community member lstrano suggested, you should update the drone (https://github.com/intel-aero/meta-intel-aero/wiki/02-Initial-Setup https://github.com/intel-aero/meta-intel-aero/wiki/02-Initial-Setup) before we can attempt any troubleshooting, since the issue you are experiencing may be related to a previous image version.

 

I'll be waiting for your response.

Have a nice day.

Regards,

 

Andres V.
idata
Community Manager
481 Views

Hello Jerorn,

 

 

I saw in this thread https://communities.intel.com/thread/113443 https://communities.intel.com/thread/113443 that you ordered a USB OTG cable.

 

 

Please get back to us if you keep experiencing the GStreamer issue after flashing the board with the latest image.

 

 

If you have any other question or update, don't hesitate to contact us.

 

 

Regards,

 

Andres V.
JFlip1
Beginner
481 Views

Dear Andres

Over the weekend I updated my drone to the latest version as well as the flight controller. If I execute the commands given above I still get the same green stream. Is it possible to look further into this. Or do you know which commands should work?

Kind regards

Jeroen Flipts

idata
Community Manager
481 Views

Hello Jeroen,

 

 

I'm glad to hear that you could update your drone.

 

 

Regarding the GStreamer issue you are experiencing, I would like to try and reproduce it, could you please provide the specifications of your host PC/laptop (particularly the OS you are using)?

 

 

I assume you are following this guide https://github.com/intel-aero/meta-intel-aero/wiki/WiFi-Streaming, have you tried WiFi streaming using QGroundControl?

 

 

I'll be waiting for your response.

 

 

Have a nice day.

 

 

Regards,

 

Andres V.
JFlip1
Beginner
481 Views

Hi Andres

Thanks for the reply and the support I'm getting on the forums.

I'm working on Ubuntu 16.04.

For this question I started from the GStreamer commands stated on this page of the wiki: https://github.com/intel-aero/meta-intel-aero/wiki/WiFi-Streaming WiFi Streaming · intel-aero/meta-intel-aero Wiki · GitHub . I adapted the commands for wifi streaming until I got a working stream. My best guess for whats going wrong is that I'm missing the video format and the width an height parameters from the outgoing stream on the yocto.

On this page of the wiki: https://github.com/intel-aero/meta-intel-aero/wiki/RealSense RealSense · intel-aero/meta-intel-aero Wiki · GitHub there is a camera preview command for the depth data. When I try to execute this command on the yocto, I get the following output:

root@intel-aero:~# gst-launch-1.0 v4l2src device=/dev/video11 ! xvimagesink

Setting pipeline to PAUSED ...

ERROR: Pipeline doesn't want to pause.

ERROR: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Could not initialise Xv output

Additional debug info:

../../../gst-plugins-base-1.8.2/sys/xvimage/xvimagesink.c(1760): gst_xv_image_sink_open (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

Could not open display (null)

Setting pipeline to NULL ...

Freeing pipeline ...

Kind regards

Jeroen Flipts

idata
Community Manager
481 Views

Hello Jeroen,

 

 

Thank you for the information provided.

 

 

I'll check the guide and try to reproduce the issue.

 

 

As soon as I have relevant information I'll contact you through this post.

 

 

In meantime, if you have any update regarding the issue you are experience, don't hesitate to share it.

 

 

Have a nice day.

 

 

Regards,

 

Andres V.
JFlip1
Beginner
481 Views

Hi

I looked a bit further into it and came to the following assumptions. When I run the command from in the picture I get the supported formats from the depth camera device.

Gstreamer will default to the first format, but I was wondering why this has size 314x938. In my eyes this doesnt make sense, but it could explain why the green Image from the original question looks like it has multiple layers on top of eachother. I'd guess 4 because this would get an aspect ratio of 4/3. So basically the captured image is split divided into 4 bands an all are placed on top of eachother. This is only a theory though, but if true would there be a way to split the layers of the image so the original could be created.

A second thing I can deduct form the command image above is that the format Z16 is supported by the camera. I have no idea how i can possibly get the stream into Z16 format because I haven't found any reference that it is supported by Gstreamer. The only documentation I find about the Z16 format is that it is an unsigned 16-bit integer per pixel mapped linearly to depth. Actually this is exactly what I'd like to pull from the camera. I assumed this would be a basic feature because what would otherwise be the point of putting a depth camera on the drone. If you buy the standalone r200 you can use the sdk to acces the point cloud but with the aero on yocto I haven't found a way.

Any suggestions would be appreciated.

Kind regards

Jeroen Flipts

idata
Community Manager
481 Views

Hello Jeroen,

 

 

I have an update regarding this topic.

 

 

I ran the following commands:

 

sh-4.3# gst-launch-1.0 v4l2src device=/dev/video11 ! xvimagesink

 

 

And didn't get any error message, the depth camera output was as expected (green stream display, 320x962)

 

 

Could you please provide the output to the following command?

 

aero-get-version.py

 

 

I'll be waiting for your response.

 

 

Have a nice day.

 

 

Regards,

 

Andres V.
JFlip1
Beginner
481 Views

Hi Anders

So i ran the following command gst-launch-1.0 v4l2src device=/dev/video11 ! xvimagesink and got this error:

root@intel-aero:~# gst-launch-1.0 v4l2src device=/dev/video11 ! xvimagesink Setting pipeline to PAUSED ... ERROR: Pipeline doesn't want to pause. ERROR: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Could not initialise Xv output Additional debug info: ../../../gst-plugins-base-1.8.2/sys/xvimage/xvimagesink.c(1760): gst_xv_image_sink_open (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Could not open display (null) Setting pipeline to NULL ... Freeing pipeline ...

When I run aero-get-version.py I get this:

root@intel-aero:~# aero-get-version.py BIOS_VERSION = Aero-01.00.12_Prod OS_VERSION = Poky Aero (Intel Aero Linux Distro) 1.3.1 (krogoth)" AIRMAP_VERSION = 1.8 FPGA_VERSION = 0xff

idata
Community Manager
481 Views

Hello Jeroen,

 

 

Thank you for the information provided.

 

 

I'll proceed to analyze the error message that you got.

 

 

As soon as I find useful information I'll post it here.

 

 

Thank you for your patience.

 

 

Regards,

 

Andres V.
idata
Community Manager
481 Views

Hello Jeroen,

I have been trying to replicate the issue you experienced, but the output I get is the same as the one I mentioned in the previous post. So I informed about this to the RealSense engineering team, to see if they know the reason behind this, to which they mentioned that GStreamer is not supported by RealSense.

With the previous in mind, the alternative is to use librealsense to access the camera. I also found an example that creates a GStreamer node that gets data from librealsense, I haven't tested it, but if you are interested in giving it a try, please follow this link: https://github.com/01org/camera-streaming-daemon/blob/master/samples/stream_realsense.cpp https://github.com/01org/camera-streaming-daemon/blob/master/samples/stream_realsense.cpp.

Have a nice day.

Regards,

 

Andres V.
JFlip1
Beginner
481 Views

Hello Anders

Our project with the Intel Aero drone has been terminated. Though we were unable to succesfully stream depth data from the drone to a workstation, I'd like to thank you for your help and time. You can mark the thread as completed if you wish to.

Kind regards

Jeroen Flipts

Reply