I've been doing some test recently with the OpenVINO samples. I've been working with "benchmark_app" & "perfcheck." I've noticed that these benchmark tools can be inconsistent. I've been using the landmarks-regression-retail-0009 model. From my input, I'm just using some generic .bmp file.
From my testing, I have found that 20-25% of the results (looking at average FPS) are outliers in the data. I was using the "landmarks-regression-retail-0009" model. To me this seems like a very large number of outliers for running the same test on repeat. In my testing, I do wait about 10 seconds before I call the same command again to run the benchmark.
I could be doing the test wrong.
If you need some more information from me, let me know.
Looking for some guidance on this. Thanks.
Dear Kase Rohlfing, Spencer,
On second thought I felt it would benefit the community to discuss it on the forum itself. Please provide specific results here regarding your experiments. If there is demonstrated inconsistency I will be happy to reproduce and file a bug. You mentioned I've been working with "benchmark_app" & "perfcheck." as well as the landmarks-regression-retail-0009 model. Can you give me the specific command-lines you are using for benchmark_app and perfcheck ? Also you didn't mention the device(s) you were running these tests on. Finally, please make sure you're using the latest version of OpenVino 2019R1.1.
Currently on 2019R1.1, ran tests on a 9600k & N4200. I've included both results from both CPUs. Inside there will be a .csv for the numbers recorded and .png for a screenshot of the boxplot.
Commands for benchmark_app:
exec /home/test/inference_engine_samples_build/intel64/Release/benchmark_app - m /home/test/intel_models/vehicle-attributes-recognition-barrier-0039/FP32/vehicle-attributes-recognition-barrier-0039.xml -i /home/test/image.bmp -niter 1000 -report_type no_counters -report_folder /home/test -api async -d CPU
Note: I did not include the results from the "perfcheck" sample. If you would like me to, I can upload those as well. But I got similar results with the "perfcheck" sample.
Dearest Kase Rohlfing, Spencer,
I really appreciate your doing this. Thanks ! Your efforts will not go ignored. Also yes, please upload results for the perfcheck too so that I can take them together and investigate.
Here are the files from the perfcheck benchmark. I ran this on the N4200 cpu. I also got similar results on the 9600k, but I didn't capture the results. The command I was using was:
exec /home/test/inference_engine_samples_build/intel64/Release/perfcheck -m /home/test/intel_models/landmarks-regression-retail-0009/FP32/landmarks-regression-retail-0009.xml -input_dir /home/test/image.bmp -num_iterations 1000 -d CPU
Let me know if you need any additional information.