topic Re: question about the rs-measure example in Items with no label

question about the rs-measure example

zting1 — Tue, 05 Mar 2019 16:45:23 GMT

I have some question about the example of rs-measure. mainly in the code below：

std::thread video_processing_thread([&]() {
        // In order to generate new composite frames, we have to wrap the processing
        // code in a lambda
        rs2::processing_block frame_processor(
            [&](rs2::frameset data, // Input frameset (from the pipeline)
                rs2::frame_source& source) // Frame pool that can allocate new frames
        {
            // First make the frames spatially aligned
            data = data.apply_filter(align_to);
 
            // Decimation will reduce the resultion of the depth image,
            // closing small holes and speeding-up the algorithm
            data = data.apply_filter(dec);
 
            // To make sure far-away objects are filtered proportionally
            // we try to switch to disparity domain
            data = data.apply_filter(depth2disparity);
 
            // Apply spatial filtering
            data = data.apply_filter(spat);
 
            // Apply temporal filtering
            data = data.apply_filter(temp);
 
            // If we are in disparity domain, switch back to depth
            data = data.apply_filter(disparity2depth);
 
            // Send the post-processed depth for path-finding
            pathfinding_queue.enqueue(data.get_depth_frame());
 
            //Apply color map for visualization of depth
            data = data.apply_filter(color_map);
 
            // Send the composite frame for rendering
            source.frame_ready(data);
        });
        // Indicate that we want the results of frame_processor
        // to be pushed into postprocessed_frames queue
        frame_processor >> postprocessed_frames;
 
        while (alive)
        {
            // Fetch frames from the pipeline and send them for processing
            rs2::frameset fs;
            if (pipe.poll_for_frames(&fs)) frame_processor.invoke(fs);
        }
    });

I can understand the central idea is that:

1.when receive a frame , frame processor invoke the processing function,that is the lambda function:

[&](rs2::frameset data, // Input frameset (from the pipeline)
                rs2::frame_source& source) // Frame pool that can allocate new frames
        {
               ……
         }

confusion 1. if fs is the first parameter(rs2::frameset data

) of the lambda function, what is the second parameter(rs2::frame_source& source

), I can't find any variable Incoming the lambda function as the second parameter.

confusio2. frame_processor >> postprocessed_frames; the operator >> is a callback function to notice the frame_ready. Am I right? Can I see it as frame_processor.operator>>(postprocessed_frames)? if so, I can't find any operation that can push back the frame processed to postprocessed_frames. Or what does the operator >> push back to the postprocessed_frames?

confusion 3: if the postprocessed_frames is the parameter of on_frame, and the operator >> also return on_frame, which variable receive the on_frame?

template<class S>
        S& operator>>(S& on_frame)
        {
            start(on_frame);
            return on_frame;
        }

Can you trouble explain? Thanks a lot!

Re: question about the rs-measure example

Eliza_D_Intel — Wed, 06 Mar 2019 14:07:06 GMT

Hello zting1, Thank you for your interest in the Intel RealSense Technology. Here are the answers to your questions: Confusion 1: The "source" is not an input. It is an output. This code in rs_measure.cpp: rs2::processing_block frame_processor( [&](rs2::frameset data, // Input frameset (from the pipeline) rs2::frame_source& source) // Frame pool that can allocate new frames Calls this template below, where "processing function" = (rs2::frameset data, rs2::frame_source& source). You can see this if you debug the rs-measure.cpp code with a breakpoint at the above function call and step into the call. template<class S> processing_block(S processing_function) { rs2_error* e = nullptr; _block = std::shared_ptr<rs2_processing_block>( rs2_create_processing_block(new frame_processor_callback<S>(processing_function), &e), rs2_delete_processing_block); options::operator=(_block); error::handle(e); } The "frame_processor_callback" from the above function, calls this template below. The "on_frame_function" returns the output - src, which is a reference to the input - frm. class frame_processor_callback : public rs2_frame_processor_callback { T on_frame_function; public: explicit frame_processor_callback(T on_frame) : on_frame_function(on_frame) {} void on_frame(rs2_frame* f, rs2_source * source) override { frame_source src(source); frame frm(f); on_frame_function(std::move(frm), src); } Confusion 2 and 3: This command frame_processor >> postprocessed_frames; "frame_processor" is the input, and "postprocessed_frames" is the output. The "on_frame" returns to "postprocessed_frames." The description in bold tells you what it does * Does the same thing as "start" function * * \param[in] on_frame address of callback function for noticing the frame to be processed is ready. * \return address of callback function. */ template<class S> S& operator>>(S& on_frame) { start(on_frame); return on_frame; } Thank you and best regards, Eliza

Re: question about the rs-measure example

zting1 — Wed, 06 Mar 2019 23:04:07 GMT

Hi, ElizaD_Intel, Thanks for your explanation.

I'm confused about the "on_frame" returns to "postprocessed_frames."

"on_frame" is a function, and "postprocessed_frames" is a queue of frame. How can the "on_frame" return to "postprocessed_frames"?

I'm looking forward to your further reply.

Re: question about the rs-measure example

Eliza_D_Intel — Thu, 07 Mar 2019 00:21:11 GMT

Hello zting1, According to this comment in rs-measure.cpp, the results of the frame_processor get pushed into postprocessed_frames. Somehow all the function templates, callbacks, and references make it all work. // Indicate that we want the results of frame_processor // to be pushed into postprocessed_frames queue frame_processor >> postprocessed_frames; Thank you and best regards, Eliza

Re: question about the rs-measure example

zting1 — Thu, 07 Mar 2019 08:58:41 GMT

Hello ElizaD,

I understood the comment. But I just encounter some problem when use the code block, so I want to know this code block better.

Thanks for your reply!

Re: question about the rs-measure example

Eliza_D_Intel — Thu, 07 Mar 2019 16:55:19 GMT

Hello zting1, Could you please let us know what are the problems you have encountered? Thank you and best regards, Eliza

Re: question about the rs-measure example

zting1 — Thu, 07 Mar 2019 18:17:43 GMT

Hello ElizaD

I wrote a simple program to get distance from the depth frame based on the example of rs-measure, but the code crash when it run to

auto udist = depth.get_distance(u, v);

and there is no useful message output. I don't know how to fix it. Can you give me some advice. Thanks a lot!

The complete code is below, it is just a process thread and a main thread to visit the depth.

// License: Apache 2.0. See LICENSE file in root directory.
// Copyright(c) 2015-2017 Intel Corporation. All Rights Reserved.
 
#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include <librealsense2/rsutil.h>
#include "example.hpp"          // Include short list of convenience functions for rendering
// This example will require several standard data-structures and algorithms:
#define _USE_MATH_DEFINES
#include <math.h>
#include <queue>
#include <algorithm>            // std::min, std::max
#include <thread>
#include <atomic>
#include <opencv2/opencv.hpp>   // Include OpenCV API
#include <mutex>
//PCL
#include <pcl/common/common_headers.h>
#include <pcl/visualization/pcl_visualizer.h>
#include <pcl/io/pcd_io.h>
#include <pcl/filters/passthrough.h>
typedef pcl::PointXYZRGB RGB_Cloud;
typedef pcl::PointCloud<RGB_Cloud> point_cloud;
typedef point_cloud::Ptr cloud_pointer;
typedef point_cloud::Ptr prevCloud;
using pixel = std::pair<int, int>;
//Global variable
int i = 1; // Index for incremental file name
int width = 1280;
int height = 720;
using namespace std;
 
std::shared_ptr<pcl::visualization::PCLVisualizer> createRGBVisualizer(pcl::PointCloud<pcl::PointXYZRGB>::ConstPtr cloud);
cloud_pointer PCL_Conversion_rect(const rs2::depth_frame& frame, const rs2::video_frame &color, const vector<cv::Point_<u_int32_t>> &vertex,const rs2_intrinsics intr_);
 
int main(int argc, char * argv[]) try
{
    rs2::align align_to(RS2_STREAM_COLOR);
    // Decimation filter reduces the amount of data (while preserving best samples)
    rs2::decimation_filter dec;
    // If the demo is too slow, make sure you run in Release (-DCMAKE_BUILD_TYPE=Release)
    // but you can also increase the following parameter to decimate depth more (reducing quality)
    dec.set_option(RS2_OPTION_FILTER_MAGNITUDE, 2);
    // Define transformations from and to Disparity domain
    rs2::disparity_transform depth2disparity;
    rs2::disparity_transform disparity2depth(false);
    // Define spatial filter (edge-preserving)
    rs2::spatial_filter spat;
    // Enable hole-filling
    // Hole filling is an agressive heuristic and it gets the depth wrong many times
    spat.set_option(RS2_OPTION_HOLES_FILL, 5); // 5 = fill all the zero pixels
    // Define temporal filter
    rs2::temporal_filter temp;
 
    // After initial post-processing, frames will flow into this queue:
    rs2::frame_queue postprocessed_frames;
 
    // Declare RealSense pipeline, encapsulating the actual device and sensors
    rs2::pipeline pipe;
 
    rs2::config cfg;
    cfg.enable_stream(RS2_STREAM_DEPTH,width,height,RS2_FORMAT_Z16,30);
    // For the color stream, set format to RGBA
    // To allow blending of the color frame on top of the depth frame
    cfg.enable_stream(RS2_STREAM_COLOR, width,height,RS2_FORMAT_BGR8,30);
    // Start streaming with default recommended configuration
    auto profile = pipe.start(cfg);
    auto stream = profile.get_stream(RS2_STREAM_COLOR).as<rs2::video_stream_profile>();
    auto intr =stream.get_intrinsics();
 
    cv::Point_<u_int32_t> left_up(540,240);
    cv::Point_<u_int32_t> right_up(740,240);
    cv::Point_<u_int32_t> right_down(740,420);
    cv::Point_<u_int32_t> left_down(540,420);
 
    // Alive boolean will signal the worker threads to finish-up
    std::atomic_bool alive{ true };
    // Video-processing thread will fetch frames from the camera,
    // apply post-processing and send the result to the main thread for rendering
    // It recieves synchronized (but not spatially aligned) pairs
    // and outputs synchronized and aligned pairs
    std::thread video_processing_thread([&]() {
        // In order to generate new composite frames, we have to wrap the processing
        // code in a lambda
        rs2::processing_block frame_processor(
                [&](rs2::frameset data, // Input frameset (from the pipeline)
                    rs2::frame_source& source) // Frame pool that can allocate new frames
                {
                    // First make the frames spatially aligned
                    data = data.apply_filter(align_to);
 
                    // Decimation will reduce the resultion of the depth image,
                    // closing small holes and speeding-up the algorithm
                    data = data.apply_filter(dec);
 
                    // To make sure far-away objects are filtered proportionally
                    // we try to switch to disparity domain
                    data = data.apply_filter(depth2disparity);
 
                    // Apply spatial filtering
                    data = data.apply_filter(spat);
 
                    // Apply temporal filtering
                    data = data.apply_filter(temp);
 
                    // If we are in disparity domain, switch back to depth
                    data = data.apply_filter(disparity2depth);
 
                    // Send the composite frame for rendering
                    source.frame_ready(data);
                });
        // Indicate that we want the results of frame_processor
        // to be pushed into postprocessed_frames queue
        frame_processor >> postprocessed_frames;
 
        while (alive)
        {
            // Fetch frames from the pipeline and send them for processing
            rs2::frameset fs;
            if (pipe.poll_for_frames(&fs)) frame_processor.invoke(fs);
        }
    });
 
//main thread
    while (alive) {
        static rs2::frameset current_frameset;
        postprocessed_frames.poll_for_frame(&current_frameset);
        if(current_frameset)
        {
            auto depth = current_frameset.get_depth_frame();
            auto color = current_frameset.get_color_frame();
            int cloud_width  = right_up.x - left_up.x;
            int cloud_height = left_down.y - left_up.y;
            for (int i = 0; i < cloud_height; i++)
            {
                for (int j = 0; j < cloud_width; ++j) {
                    int index_pcl = i * cloud_width +j;
                    int v = left_up.y + i  ;
                    int u = left_up.x + j;
                    cout<<"u="<<u<<",v="<<v<<endl;
                    auto udist = depth.get_distance(u, v);
                    cout<<"u="<<u<<",v="<<v<<endl;
    }
    alive = false;
    video_processing_thread.join();
    return EXIT_SUCCESS;
}
catch (const rs2::error & e)
{
    std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;
    return EXIT_FAILURE;
}
catch (const std::exception & e)
{
    std::cerr << e.what() << std::endl;
    return EXIT_FAILURE;
}

Re: question about the rs-measure example

Eliza_D_Intel — Thu, 07 Mar 2019 22:30:07 GMT

Hello zting1, Another RealSense user posted a similar issue on GitHub, and by debugging the code he found that the issue was not related to the function. Here is the link: https://github.com/IntelRealSense/librealsense/issues/3365 Thank you, Eliza

Re: question about the rs-measure example

zting1 — Thu, 07 Mar 2019 22:56:00 GMT

Hello ElizaD,

Oh no, The user is me. There was another problem that time. But this time I simplify the code as above, I really don't know what cause the crash. Can you help me?

Re: question about the rs-measure example

Eliza_D_Intel — Fri, 08 Mar 2019 00:10:24 GMT

Hello zting1, Could you please let us know what is the error you are receiving when running this code? what is the message received by your compiler? Thank you, Eliza

Re: question about the rs-measure example

zting1 — Fri, 08 Mar 2019 12:30:16 GMT

Hello Eliza,

The message is :Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

I just can't get anything from this message.

Re: question about the rs-measure example

zting1 — Fri, 08 Mar 2019 12:39:08 GMT

I point out which line of code cause the exiting, The code in this line is below:

                    // Decimation will reduce the resultion of the depth image,
                    // closing small holes and speeding-up the algorithm
                   data = data.apply_filter(dec);

I'm trying to figure out why could this line cause the exiting. If you have some idea, please let me know. Thanks a lot!

Re: question about the rs-measure example

zting1 — Fri, 08 Mar 2019 17:00:04 GMT

Hello Eliza

The problem has been solved in github, Thanks for your patient answer.