- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I've been doing some test recently with the OpenVINO samples. I've been working with "benchmark_app" & "perfcheck." I've noticed that these benchmark tools can be inconsistent. I've been using the landmarks-regression-retail-0009 model. From my input, I'm just using some generic .bmp file.
From my testing, I have found that 20-25% of the results (looking at average FPS) are outliers in the data. I was using the "landmarks-regression-retail-0009" model. To me this seems like a very large number of outliers for running the same test on repeat. In my testing, I do wait about 10 seconds before I call the same command again to run the benchmark.
I could be doing the test wrong.
If you need some more information from me, let me know.
Looking for some guidance on this. Thanks.
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dearest Kase Rohlfing, Spencer,
I noticed that you are an intel employee. Send me an email and we can pursue this.
Thanks,
Shubha
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Shubha,
Can you get in contact with me? I thought I emailed the correct person, but I didn't.
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear Kase Rohlfing, Spencer,
On second thought I felt it would benefit the community to discuss it on the forum itself. Please provide specific results here regarding your experiments. If there is demonstrated inconsistency I will be happy to reproduce and file a bug. You mentioned I've been working with "benchmark_app" & "perfcheck." as well as the landmarks-regression-retail-0009 model. Can you give me the specific command-lines you are using for benchmark_app and perfcheck ? Also you didn't mention the device(s) you were running these tests on. Finally, please make sure you're using the latest version of OpenVino 2019R1.1.
Thanks,
Shubha
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Shubha,
Currently on 2019R1.1, ran tests on a 9600k & N4200. I've included both results from both CPUs. Inside there will be a .csv for the numbers recorded and .png for a screenshot of the boxplot.
Commands for benchmark_app:
exec /home/test/inference_engine_samples_build/intel64/Release/benchmark_app - m /home/test/intel_models/vehicle-attributes-recognition-barrier-0039/FP32/vehicle-attributes-recognition-barrier-0039.xml -i /home/test/image.bmp -niter 1000 -report_type no_counters -report_folder /home/test -api async -d CPU
Note: I did not include the results from the "perfcheck" sample. If you would like me to, I can upload those as well. But I got similar results with the "perfcheck" sample.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dearest Kase Rohlfing, Spencer,
I really appreciate your doing this. Thanks ! Your efforts will not go ignored. Also yes, please upload results for the perfcheck too so that I can take them together and investigate.
Thanks !
Shubha
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Shubha,
Here are the files from the perfcheck benchmark. I ran this on the N4200 cpu. I also got similar results on the 9600k, but I didn't capture the results. The command I was using was:
exec /home/test/inference_engine_samples_build/intel64/Release/perfcheck -m /home/test/intel_models/landmarks-regression-retail-0009/FP32/landmarks-regression-retail-0009.xml -input_dir /home/test/image.bmp -num_iterations 1000 -d CPU
Let me know if you need any additional information.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear Kase Rohlfing, Spencer,
Fantastic ! We will definitely investigate this.
Thanks,
Shubha
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Shubha,
I was wondering if there were any updates on this issue.
Thanks,
Spencer
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dearest Kase Rohlfing,
No updates yet. But I promise to look at it next week at the latest.
Thanks for your patience,
Shubha
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page