Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

OpenVINO benchmark samples seem to be inconsistent

Spencer_K_Intel
Employee
480 Views

I've been doing some test recently with the OpenVINO samples. I've been working with "benchmark_app" & "perfcheck." I've noticed that these benchmark tools can be inconsistent. I've been using the landmarks-regression-retail-0009 model. From my input, I'm just using some generic .bmp file. 

From my testing, I have found that 20-25% of the results (looking at average FPS) are outliers in the data. I was using the "landmarks-regression-retail-0009" model. To me this seems like a very large number of outliers for running the same test on repeat. In my testing, I do wait about 10 seconds before I call the same command again to run the benchmark.

I could be doing the test wrong. 

 

If you need some more information from me, let me know.

Looking for some guidance on this. Thanks.
 

0 Kudos
9 Replies
Shubha_R_Intel
Employee
480 Views

Dearest Kase Rohlfing, Spencer,

I noticed that you are an intel employee. Send me an email and we can pursue this.

Thanks,

Shubha

0 Kudos
Spencer_K_Intel
Employee
480 Views

Hi Shubha,

Can you get in contact with me? I thought I emailed the correct person, but I didn't. 

Thanks

0 Kudos
Shubha_R_Intel
Employee
480 Views

Dear Kase Rohlfing, Spencer,

On second thought I felt it would benefit the community to discuss it on the forum itself.  Please provide specific results here regarding your experiments. If  there is demonstrated inconsistency I will be happy to reproduce and file a bug. You mentioned  I've been working with "benchmark_app" & "perfcheck." as well as  the landmarks-regression-retail-0009 model. Can you give me the specific command-lines you are using for benchmark_app and perfcheck ? Also you didn't mention the device(s) you were running these tests on. Finally, please make sure you're using the latest version of OpenVino 2019R1.1.

Thanks,

Shubha

 

0 Kudos
Spencer_K_Intel
Employee
480 Views

Hi Shubha, 

Currently on 2019R1.1, ran tests on a 9600k & N4200. I've included both results from both CPUs. Inside there will be a .csv for the numbers recorded and .png for a screenshot of the boxplot.

Commands for benchmark_app:

exec /home/test/inference_engine_samples_build/intel64/Release/benchmark_app - m /home/test/intel_models/vehicle-attributes-recognition-barrier-0039/FP32/vehicle-attributes-recognition-barrier-0039.xml -i /home/test/image.bmp -niter 1000 -report_type no_counters -report_folder /home/test -api async -d CPU 

Note: I did not include the results from the "perfcheck" sample. If you would like me to, I can upload those as well. But I got similar results with the "perfcheck" sample. 

0 Kudos
Shubha_R_Intel
Employee
480 Views

Dearest Kase Rohlfing, Spencer,

I really appreciate your doing this. Thanks ! Your efforts will not go ignored. Also yes, please upload results for the perfcheck too so that I can take them together and investigate.

Thanks !

Shubha

0 Kudos
Spencer_K_Intel
Employee
480 Views

Shubha,

Here are the files from the perfcheck benchmark. I ran this on the N4200 cpu. I also got similar results on the 9600k, but I didn't capture the results. The command I was using was: 

exec /home/test/inference_engine_samples_build/intel64/Release/perfcheck -m /home/test/intel_models/landmarks-regression-retail-0009/FP32/landmarks-regression-retail-0009.xml -input_dir /home/test/image.bmp -num_iterations 1000 -d CPU

Let me know if you need any additional information.

0 Kudos
Shubha_R_Intel
Employee
480 Views

Dear Kase Rohlfing, Spencer,

Fantastic ! We will definitely investigate this.

Thanks,

Shubha

0 Kudos
Spencer_K_Intel
Employee
480 Views

Hello Shubha,

I was wondering if there were any updates on this issue. 

Thanks,

Spencer

0 Kudos
Shubha_R_Intel
Employee
480 Views

Dearest Kase Rohlfing, 

No updates yet. But I promise to look at it next week at the latest.

Thanks for your patience,

Shubha

0 Kudos
Reply