Publishing images from bottom camera to ROS topics

[trix]

Hi,

Like the title says, I'm trying to publish images from the bottom camera of the RTF drone using ROS. Initially, I modified the existing http://wiki.ros.org/usb_cam usb_cam ROS package and this allowed me to communicate with the camera, but I've been struggling with trying to then calibrate my camera since the video stream that I get seems to just be a noisy image (I attached an image of it). To clarify, I'm still able to get some kind of stream but it seems to splice together 5 of the same streams into one, which can be seen with the 5 different columns.

I've considered that the problem is simply a matter of decoding the yuv420 pixel format of the bottom camera into RGB, but I am still fairly new to C++ so I have been struggling with this problem. Am I going about this in the right direction or is something else causing this noisy stream?

Any help would be appreciated.

[trix · Accepted solution]

Actually, nevermind. My problem wasn't about the pixel format but when I was initializing the device.

[VBluk]

Would you mind sharing what you did to get it to work? I'm trying to do the same thing and I've no clue where to begin. Thanks!

[trix]

Apologies @valts, I didn't see your message sooner. What I did was modify the usb_cam ROS package like what I said in my original post. I had to change the camera driver to be able to specify the cameraId since the VGA and 8MP cameras both use the /dev/video2 device node. The function I changed was the init_device method in usb_cam.cpp and I ended up with this:

void UsbCam::init_device(int image_width, int image_height, int framerate)

{

struct v4l2_capability cap;

struct v4l2_cropcap cropcap;

struct v4l2_crop crop;

struct v4l2_format fmt;

unsigned int min;

struct v4l2_streamparm stream_params;

if (-1 == xioctl(fd_, VIDIOC_S_INPUT, &cameraId))

errno_exit("VIDIOC_S_INPUT");

memset(&stream_params, 0, sizeof(stream_params));

stream_params.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;

stream_params.parm.capture.capturemode = 0x8000;

if (-1 == xioctl(fd_, VIDIOC_S_PARM, &stream_params))

<a name="usb_cam.cpp-863" style="colo...

[CRamé]

Thanks Trixie! Your code really helped me out. For anyone else trying to do the same thing the pixel format you need is 'grey' and set int cameraId=1.

[SAhma5]

Thank you Trixie and Coline . This discussion really helped me. I have been trying to do the similar thing but somehow I am having problems while setting the camera frame rate . I am getting

$ roslaunch usb_cam usb_cam.launch

... logging to /home/aero/.ros/log/15cc9aa0-48a1-11e8-8bc5-72689d3f106b/roslaunch-aero-8057.log

Checking log directory for disk usage. This may take awhile.

Press Ctrl-C to interrupt

Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://aero:33698/

SUMMARY

========

PARAMETERS

* /rosdistro: kinetic

* /rosversion: 1.12.13

* /usb_cam/camera_frame_id: usb_cam

* /usb_cam/image_height: 480

* /usb_cam/image_width: 640

* /usb_cam/io_method: mmap

* /usb_cam/pixel_format: grey

* /usb_cam/video_device: /dev/video2

NODES

/

usb_cam (usb_cam/usb_cam_node)

ROS_MASTER_URI= http://localhost:11311

process[usb_cam-1]: started with pid [8074]

[ INFO] [1524673020.178612270]: using default calibration URL

[ INFO] [1524673020.178778798]: camera calibration URL: file:///home/aero/.ros/camera_info/head_camera.yaml

[ INFO] [1524673020.178932426]: Unable to open camera calibration file [/home/aero/.ros/camera_info/head_camera.yaml]

[ WARN] [1524673020.178994965]: Camera calibration file /home/aero/.ros/camera_info/head_camera.yaml not found.

[ INFO] [1524673020.179062853]: Starting 'head_camera' (/dev/video2) at 640x480 via mmap (grey) at 30 FPS

[ WARN] [1524673020.299492649]: Couldn't set camera framerate

[ WARN] [1524673020.311671785]: Failed to open /dev/video2: Device or resource busy

[ WARN] [1524673020.319405389]: Failed to open /dev/video2: Device or resource busy

Can anyone help with this or I am missing something completely? Thank you

[CRamé]

Hi shakeeb,

I get similar warnings but I can still stream with image_view

Here is the output I get for reference

SUMMARY

========

PARAMETERS

* /image_view/autosize: True

* /rosdistro: kinetic

* /rosversion: 1.12.13

* /usb_cam/camera_frame_id: usb_cam

* /usb_cam/framerate: 30

* /usb_cam/image_height: 480

* /usb_cam/image_width: 640

* /usb_cam/io_method: userptr

* /usb_cam/pixel_format: grey

* /usb_cam/video_device: /dev/video2

NODES

/

image_view (image_view/image_view)

usb_cam (usb_cam/usb_cam_node)

auto-starting new master

process[master]: started with pid [2439]

ROS_MASTER_URI= http://localhost:11311

setting /run_id to dc8070c0-4245-11e8-aa35-3200479c7b50

process[rosout-1]: started with pid [2452]

started core service [/rosout]

process[usb_cam-2]: started with pid [2455]

process[image_view-3]: started with pid [2456]

[ INFO] [1523972846.936222314]: using default calibration URL

[ INFO] [1523972846.936394332]: camera calibration URL: file:///home/coline/.ros/camera_info/head_camera.yaml

[ INFO] [1523972846.936559812]: Unable to open camera calibration file [/home/coline/.ros/camera_info/head_camera.yaml]

[ WARN] [1523972846.936612114]: Camera calibration file /home/coline/.ros/camera_info/head_camera.yaml not found.

[ INFO] [1523972846.936672041]: Starting 'head_camera' (/dev/video2) at 640x480 via userptr (grey) at 30 FPS

[ WARN] [1523972847.058261827]: Couldn't set camera framerate

[ INFO] [1523972847.065940069]: Using transport "raw"

[ WARN] [1523972847.081353066]: Failed to open /dev/video2: Device or resource busy

[ WARN] [1523972847.093548857]: Failed to open /dev/video2: Device or resource busy

[BSzeb]

Thank you Trixie and Coline for your contribution. Helped me out! shakeeb you need to change the io_method to userptr

For those struggling with the same problem here is a quick guide to have the bottom VGA camera as a ROS topic:

cd ~/catkin_ws/src # cd into the src folder of your catkin workspace

git clone https://github.com/szebedy/usb_cam.git https://github.com/szebedy/usb_cam.git

cd ..

catkin_make

source ./devel/setup.bash

roslaunch usb_cam usb_cam.launch /

rosrun image_view image_view image:=/usb_cam/image_raw

[栩张]

Hi chillax

Thanks for your sharing.It's really useful.

I have some question.

Actually I got the same situation like Coline

I want know if there is any way I can get the video or picture for calibration the camera?

Thanks

[BSzeb]

Hi tuantuan91,

I'm not sure if I understood your question but if you followed my instructions you should have an image stream from the bottom camera available under the /usb_cam/image_raw ROS topic. You can use the images on this topic to calibrate your camera.

[CRamé]

Hi tuantuan91,

I calibrated the camera using the camera calibration tool from http://wiki.ros.org/camera_calibration camera_calibration - ROS Wiki (it was already included in my ROS desktop full install).

It's pretty straightforward if you follow http://wiki.ros.org/camera_calibration/Tutorials/MonocularCalibration camera_calibration/Tutorials/MonocularCalibration - ROS Wiki

Hope that helps

[栩张]

Hi chillax

I just do as your instructions above.

The topic is available but there is no images output.

I can get the message (grey value between 0-255) through the command: rostopic echo.

Is there any procedure I missed?

Thanks

[BSzeb]

Hi tuantuan91

rosrun image_view image_view image:=/usb_cam/image_raw is the answer you seek.

Have fun

[栩张]

Hi chillax

It works well!

Thanks for your help.

I'm wondering if you are using the RTF to do the SLAM/VO research?

[栩张]

Hi chillax

Sorry to disturb you again.

Did you calibrate the R200 camera use the same way？

Thanks

[BSzeb]

Hi tuantuan91

I am using the bottom camera for VIO. Most VIO algorithms need more computational power than what the Intel Aero can offer, therefore I've decided to use http://rpg.ifi.uzh.ch/svo2.html SVO 2.0. The R200 camera comes from the factory calibrated. All you need to do is install the Intel Realsense node as discussed https://github.com/intel-aero/meta-intel-aero/wiki/05-Autonomous-drone-programming-with-ROS here. I recommend going through all the tutorials on this wiki page. You will find the calibration of the R200 under the /camera/rgb/camera_info ROS topic.

[栩张]

Hi chillax

Thanks for your help and sharing.I find the intrinsic parameters yesterday.

As you say you are doing research in VIO.

How do your get the extrinsic parameters between camera and imu?

I know the SVO2.0 but I think they did not publish the source code for the algorithm.It's just binaries files.

How could you develop by these things?

Thanks

[栩张]

Hi chillax

sorry for disturbing you again.

I tried to install SVO2.0 but failed.

So I decided to install the SVO.

but after I installed the algorithm I found there still have some issue.

CLEAR PARAMETERS

• /svo/

PARAMETERS

• /rosdistro: kinetic
• /rosversion: 1.12.13
• /svo/cam_cx: 309.936417
• /svo/cam_cy: 242.427118
• /svo/cam_d0: -0.022232
• /svo/cam_d1: -0.034436
• /svo/cam_d2: 0.003345
• /svo/cam_d3: 8.5e-05
• /svo/cam_fx: 414.756208
• /svo/cam_fy: 414.983046
• /svo/cam_height: 480
• /svo/cam_model: Pinhole
• /svo/cam_topic: /usb_cam/image_raw
• /svo/cam_width: 640
• /svo/grid_size: 30
• /svo/init_rx: 3.14
• /svo/init_ry: 0.0
• /svo/init_rz: 0.0
• /svo/loba_num_iter: 0
• /svo/max_n_kfs: 10

NODES

/svo (svo_ros/vo)

ROS_MASTER_URI=http://localhost:11311/ http://localhost:11311

process[svo-1]: started with pid [3429]

create vo_node[ WARN] [1528234776.616960704]: Cannot find value for parameter: svo/publish_img_pyr_level, assigning default: 0[ WARN] [1528234776.618874699]: Cannot find value for parameter: svo/publish_every_nth_img, assigning default: 1[ WARN] [1528234776.620884213]: Cannot find value for parameter: svo/publish_every_nth_dense_input, assigning default: 1[ WARN] [1528234776.625195170]: Cannot find value for parameter: svo/publish_world_in_cam_frame, assigning default: 1[ WARN] [1528234776.627013647]: Cannot find value for parameter: svo/publish_map_every_frame, assigning default: 0[ WARN] [1528234776.628746680]: Cannot find value for parameter: svo/publish_point_display_time, assigning default: 0[ WARN] [1528234776.832704321]: Cannot find value for parameter: svo/publish_markers, assigning default: 1[ WARN] [1528234776.834422304]: Cannot find value for parameter: svo/publish_dense_input, assigning default: 0[ WARN] [1528234776.836315998]: Cannot find value for parameter: svo/accept_console_user_input, assigning default: 1[ INFO] [1528234776.838576352]: Found parameter: svo/cam_model, value: Pinhole[ INFO] [1528234776.840500385]: Found parameter: svo/cam_width, value: 640[ INFO] [1528234776.842313561]: Found parameter: svo/cam_height, value: 480[ INFO] [1528234776.844522799]: Found parameter: svo/cam_fx, value: 414.756[ INFO] [1528234776.846400992]: Found parameter: svo/cam_fy, value: 414.983[ INFO] [1528234776.848601592]: Found parameter: svo/cam_cx, value: 309.936[ INFO] [1528234776.850640783]: Found parameter: svo/cam_cy, value: 242.427[ INFO] [1528234776.852500837]: Found parameter: svo/cam_d0, value: -0.022232[ INFO] [1528234776.854717613]: Found parameter: svo/cam_d1, value: -0.034436[ INFO] [1528234776.856453597]: Found parameter: svo/cam_d2, value: 0.003345[ INFO] [1528234776.858919014]: Found parameter: svo/cam_d3, value: 8.5e-05[ WARN] [1528234776.913663131]: Cannot find value for parameter: svo/init_tz, assigning default: 0[ WARN] [1528234776.916072582]: Cannot find value for parameter: svo/init_ty, assigning default: 0[ WARN] [1528234776.917951262]: Cannot find value for parameter: svo/init_tx, assigning default: 0[ INFO] [1528234776.919817542]: Found parameter: svo/init_rz, value: 0[ INFO] [1528234776.921600428]: Found parameter: svo/init_ry, value: 0[ INFO] [1528234776.923323124]: Found parameter: svo/init_rx, value: 3.14[ WARN] [1528234776.926382753]: Cannot find value for parameter: svo/trace_name, assigning default: svo[ WARN] [1528234776.928838619]: Cannot find value for parameter: svo/trace_dir, assigning default: /tmp[ WARN] [1528234776.930731638]: Cannot find value for parameter: svo/n_pyr_levels, assigning default: 3[ WARN] [1528234776.932477822]: Cannot find value for parameter: svo/use_imu, assigning default: 0[ WARN] [1528234776.934625432]: Cannot find value for parameter: svo/core_n_kfs, assigning default: 3[ WARN] [1528234776.936946752]: Cannot find value for parameter: svo/map_scale, assigning default: 1[ INFO] [1528234776.938851797]: Found parameter: svo/grid_size, value: 30[ WARN] [1528234776.940821370]: Cannot find value for parameter: svo/init_min_disparity, assigning default: 50[ WARN] [1528234776.942595281]: Cannot find value for parameter: svo/init_min_tracked, assigning default: 50[ WARN] [1528234776.944481799]: Cannot find value for parameter: svo/init_min_inliers, assigning default: 40[ WARN] [1528234776.946676124]: Cannot find value for parameter: svo/klt_max_level, assigning default: 4[ WARN] [1528234776.948418484]: Cannot find value for parameter: svo/klt_min_level, assigning default: 2[ WARN] [1528234776.950450036]: Cannot find value for parameter: svo/reproj_thresh, assigning default: 2[ WARN] [1528234776.952328266]: Cannot find value for parameter: svo/poseoptim_thresh, assigning default: 2[ WARN] [1528234776.954134592]: Cannot find value for parameter: svo/poseoptim_num_iter, assigning default: 10[ WARN] [1528234776.956149368]: Cannot find value for parameter: svo/structureoptim_max_pts, assigning default: 20[ WARN] [1528234776.957996159]: Cannot find value for parameter: svo/structureoptim_num_iter, assigning default: 5[ WARN] [1528234776.959730930]: Cannot find value for parameter: svo/loba_thresh, assigning default: 2[ WARN] [1528234776.961582308]: Cannot find value for parameter: svo/loba_robust_huber_width, assigning default: 1[ INFO] [1528234776.963426524]: Found parameter: svo/loba_num_iter, value: 0[ WARN] [1528234776.965105654]: Cannot find value for parameter: svo/kfselect_mindist, assigning default: 0.12[ WARN] [1528234776.967127081]: Cannot find value for parameter: svo/triang_min_corner_score, assigning default: 20[ WARN] [1528234776.969089779]: Cannot find value for par...

[BSzeb]

I haven't tried SVO, only SVO 2.0. If you download and extract the binaries you should have an installation guide PDF. Just follow the instructions there. I've disabled the IMU because it made my results a lot worse.

[栩张]

Hi

chillax

I have done the svo test but the performance was bad.

I read the guide PDF but still can't install the SVO2.0

anyway thanks for your sharing and help.

[DMarc8]

Hi Chillax,

I followed your instructions to publish images from the bottom camera on ros topic. I'm able to see the images using the ros node image_view but I got some problems near the bottom and left edges of the image, as you can see in the attached sample.

Have you experienced this problem too ?

Thank you in advance,

Daniele