Re: How can we change the temporal filter and sensor options?

XSali · ‎04-17-2019

Hi everyone,

We are trying to use the temporal filter and the high accuracy visual preset by code with the D435 RealSense camera but we can not modify this options.

Using the realsense-viewer both the visual present and the temporal filter options are modified correctly. The problem appears when modifing them by code, the value is modified but the difference is not appreciated in the depth output.

This is the code that we are using:

#include <librealsense2/rs.hpp>
#include <opencv2/opencv.hpp>
 
int main(int argc, char const *argv[])
{
 
	rs2::temporal_filter filter;
	filter.set_option(RS2_OPTION_FRAMES_QUEUE_SIZE, 16);
	filter.set_option(RS2_OPTION_FILTER_SMOOTH_ALPHA, 0.07);
	filter.set_option(RS2_OPTION_FILTER_SMOOTH_DELTA, 88);
	filter.set_option(RS2_OPTION_HOLES_FILL, 8);
 
	std::vector<rs2_option> supported = filter.get_supported_options();
	for (rs2_option opt : supported){
		std::cout << filter.get_option_name(opt) << " (" << filter.get_option_description(opt) << "): " << filter.get_option(opt) << std::endl;
		rs2::option_range rangeF = filter.get_option_range(opt);
		for (int i = rangeF.min; i <= rangeF.max; i = i + rangeF.step){
			auto des = filter.get_option_value_description(opt, i);
			if(NULL != des){
				std::cout << "  " << i << ": " << des << std::endl;
			}else{
				break;
			}
		}
	}
 
    //Create a context
    rs2::context ctx;
    auto list = ctx.query_devices();
	rs2::device device = list.front(); // Get a snapshot of currently connected devices
	std::vector<rs2::sensor> sensors = device.query_sensors();
	rs2::sensor sensor = sensors[0]; //0 : Stereo Module
 
	// To set an option to a different value, we can call set_option with a new value
	try
	{
		sensor.set_option(RS2_OPTION_VISUAL_PRESET, 3); //3: High Accuracy
	}
	catch (const rs2::error& e)
	{
		// Some options can only be set while the camera is streaming,
		// and generally the hardware might fail so it is good practice to catch exceptions from set_option
		std::cerr << "Failed to set option " << RS2_OPTION_VISUAL_PRESET << ". (" << e.what() << ")" << std::endl;
	}
 
    rs2::config cfg;
    cfg.enable_device(device.get_info(RS2_CAMERA_INFO_SERIAL_NUMBER));
    rs2::pipeline pipe;
    rs2::pipeline_profile profile = pipe.start(cfg);
    std::cout << "Pipeline started" << std::endl;
 
	rs2::frameset frames;
	rs2::frame depth;
	rs2::colorizer color_map;
 
	bool is_capturing = true;
 
	while(is_capturing){
		try{
			frames = pipe.wait_for_frames();
 
			//use frames here
			depth = frames.get_depth_frame();
 
			int width = depth.as<rs2::video_frame>().get_width();
			int height = depth.as<rs2::video_frame>().get_height();
 
			//Create OpenCV matrix of size (w,h) from the colorized depth data
			rs2::frame colorized_depth = color_map.process(depth);
			cv::Mat color_image(cv::Size(width, height), CV_8UC3, (void*)colorized_depth.get_data(), cv::Mat::AUTO_STEP);
			cv::cvtColor(color_image, color_image, CV_BGR2RGB);
			cv::imshow("Depth", color_image);
 
			rs2::frame filtered = depth;
			filtered = filter.process(filtered);
			rs2::frame colorized_filtered = color_map.process(filtered);
 
			width = filtered.as<rs2::video_frame>().get_width();
			height = filtered.as<rs2::video_frame>().get_height();
 
			cv::Mat filter_image(cv::Size(width, height), CV_8UC3, (void*)colorized_filtered.get_data(), cv::Mat::AUTO_STEP);
			cv::cvtColor(filter_image, filter_image, CV_BGR2RGB);
			cv::imshow("Filtered depth", filter_image);
			cv::waitKey(1);
 
		}catch(rs2::error e){
			is_capturing = false;
			std::cout << "Stop capturing" << std::endl;
		}
	}
 
    pipe.stop();
 
    return 0;
}

How can we modify the options by code?

Thank you

Eliza_D_Intel · ‎04-17-2019

Hello XSali, Thank you for your interest in the Intel RealSense D435 camera. You may find this issue on GitHub useful for your project: https://github.com/IntelRealSense/librealsense/issues/2073 Thank you, Eliza

XSali · ‎04-18-2019

Hello Eliza,

I have applied the modifications specified in the issue that you comment but the problem persists. The temporal filter is applied, there is a difference between the original depth and the filtered one. The problem is that even though I modify the filter options (smooth alpha, smooth delta and temporal persistence) the filtered result is always the same.

Original depth frame (without post-processing):

RealSense Viewer:

Temporal filter with smooth delta = 18 (left) and Temporal filter with smooth delta = 88 (right)

Own code:

Temporal filter with smooth delta = 18 (left) and Temporal filter with smooth delta = 88 (right)

The same happens with the visual preset, even though I modify the sensor value, the depth frame is exactly the same.

RealSense Viewer:

Default preset (left) and High accuracy (right)

Own code:

Default preset (left) and High accuracy (right)

What can I do to solve the problem?

Thank you,

Xènia

Eliza_D_Intel · ‎04-18-2019

Hello XSali, Thank you very much for the pictures and details! For the High Accuracy preset issue, is it possible to use RS2_RS400_VISUAL_PRESET_HIGH_ACCURACY instead of "3" ? on sensor.set_option(RS2_OPTION_VISUAL_PRESET, 3); //3: High Accuracy Maybe you can use instead of a "try" block an "if" like this: https://github.com/IntelRealSense/librealsense/blob/master/examples/measure/rs-measure.cpp#L145 Please let us know if you notice any difference! Thank you, Eliza

XSali · ‎04-18-2019

Hello Eliza,

There is no difference with this new code, the depth stills unchanged.

Thank you,

Xènia

XSali · ‎04-25-2019

Hello Eliza,

Do you have any idea about the problem that we have? Can we try any other modification to solve the problem?

Thank you,

Xènia

JesusG_Intel · ‎04-29-2019

Hello Xenia, we have escalated this case to engineering. We will update you as soon as we have a response. Regards, Jesus Intel Customer Support

XSali · ‎04-30-2019

Hello Jesus,

I have been able to solve the problem related with the high accuracy preset. The solution was to use the following code instead of the set_option function:

if(device.is<rs400::advanced_mode>()){
	auto advanced_mode_dev = device.as<rs400::advanced_mode>();
	if(!advanced_mode_dev.is_enabled()){
		advanced_mode_dev.toggle_advanced_mode(true);
	}
	std::ifstream t("highAccuracy.json");
	std::string str((std::istreambuf_iterator<char>(t)), std::istreambuf_iterator<char>());
	advanced_mode_dev.load_json(str);
}

Where the highAccuracy.json file is obtained using the RealSense Viewer.

The problem with the temporal filter is still unsolved.

Thank you,

Xènia

JesusG_Intel · ‎05-02-2019

Hello Xenia, thanks for the update. I was going so suggest you use that method of loading the json file. We are still looking into the temporal filter issue. Regards, Jesus

JesusG_Intel · ‎05-08-2019

Hello Xenia, I was able to get a filtered image using your code. See the attached files. You can see in the filtered depth image that there are fewer holes. The effect can be subtle. Is this not what you are expecting?

XSali · ‎05-15-2019

Hello Jesus,

This is what I am expecting. The problem is that even if I change the temporal filter options, the filtered result is always the same.

Regards,

Xènia

JesusG_Intel · ‎05-15-2019

Hello Xenia, please provide the settings that you are testing and changing with screenshots of the effects so that we can reproduce the problem. Regards, Jesus G. Intel Customer Support