topic Dear Shubha in Intel® Distribution of OpenVINO™ Toolkit

Accuracy Checker: running sample

gomes__antonio — Fri, 14 Jun 2019 11:08:17 GMT

Hi,

I'm trying to test the Accuracy Checker tool on YOLO but first I want run the sample on https://docs.openvinotoolkit.org/latest/_tools_accuracy_checker_sample_README.html to check that everthing is okay.

I already install all requirements asked in C:\Program Files (x86)\IntelSWTools\openvino\python\python3.6\requirements.txt but I don´t get it how to download the model and dataset files.

On documentation it asks to execute the command git lfs pull but this command don't work because it does not have a repository.

What am I missing? Thanks

I'm using Windows 10

Dear gomes, antonio,

Shubha_R_Intel — Fri, 14 Jun 2019 16:11:44 GMT

Dear gomes, antonio,

I've written a very detailed post on how to use calibration tools here in dldt issue 171.

Please clone the dldt github https://github.com/opencv/dldt.git .

The github (open source) version of OpenVino is not exactly the same as the one you download in the release package. Yes of course the code is similar but the packaging is quite different.

Hope it helps,

Thanks,

Shubha

Dear Shubba,

gomes__antonio — Sat, 22 Jun 2019 16:37:24 GMT

Dear Shubba,

Thanks for reply but it did not help much. It's a lot for my programming skills. I thought it was simpler to use the Accuracy Checker tool.

The documentation https://docs.openvinotoolkit.org/latest/_inference_engine_tools_accuracy_checker_tool_README.html is very confuse and it's not possible run the sample to try understand how the tool works.

I want to calculate the precision and the recall for Yolov3 in the COCO dataset but I can not even run the accuracy_check.py, give the error: ModuleNotFoundError: No module named 'openvino'

is there any simpler tutorial?

Dear gomes, antonio,

Shubha_R_Intel — Mon, 24 Jun 2019 22:14:00 GMT

Dear gomes, antonio,

Unfortunately there isn't a simpler tutorial. But this error:

ModuleNotFoundError: No module named 'openvino'

Is easily solved. Simply set your environment variables - setupvars.bat (*.sh), located under bin. From the shell that you've run this script (setting environment variables), try again and run the Python program.

Thanks,

Shubha

Dear Shubha

Atsunori__Sumi — Wed, 16 Oct 2019 00:36:03 GMT

Dear Shubha

I have recently discovered and used this tool, but I'm getting an error and can't measure yolo-v3 mAP.

I would like to know the mAP results measured by Intel. In addition, please tell me how to use this tool.

Thanks,

I'm sorry to continue The log

Atsunori__Sumi — Wed, 16 Oct 2019 02:35:01 GMT

I'm sorry to continue The log is shown for a problem that is currently not working.

$python3 /opt/intel/openvino/deployment_tools/tools/accuracy_checker_tool/accuracy_check.py -c ./test.cfg -m path/to -s /path/to/coco/val2017/ -a /path/to/coco/annotations_pascalformat/

Processing info:
model: yolo_v3
launcher: dlsdk
device: CPU
dataset: coco
OpenCV version: 4.1.2-openvino
Traceback (most recent call last):
File "/opt/intel/openvino/deployment_tools/tools/accuracy_checker_tool/accuracy_check.py", line 19, in <module>
main()
File "/opt/intel/openvino_2019.3.334/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/main.py", line 202, in main
model_evaluation_mode(config, progress_reporter, args)
File "/opt/intel/openvino_2019.3.334/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/main.py", line 218, in model_evaluation_mode
model_evaluator = ModelEvaluator.from_configs(launcher_config, dataset_config)
File "/opt/intel/openvino_2019.3.334/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/evaluators/model_evaluator.py", line 57, in from_configs
dataset = Dataset(dataset_config)
File "/opt/intel/openvino_2019.3.334/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/dataset.py", line 66, in __init__
raise ConfigError('path to converted annotation or data for conversion should be specified')
accuracy_checker.config.config_validator.ConfigError: path to converted annotation or data for conversion should be specified

----
$cat test.cfg:
models:
- name: yolo_v3
launchers:
- framework: dlsdk
device: CPU
model: ./path/to/yolo_v3.xml
weights: ./path/to/yolo_v3.bin
adapter: yolo_v3
cpu_extensions: cpu_extentions_avx512.so
datasets:
- name: coco
annotation: ~/path/to/coco/annotations_pascalformat/
data_source: ~/path/to/coco/val2017/

Had you have a chance to read

Vladimir_Dudnik — Wed, 16 Oct 2019 07:22:44 GMT

Had you have a chance to read through accuracy checker documentation first? I think that a chapter on annotation conversion may help to see that accuracy checker require conversion of original dataset annotation. You may do that as separate preliminary step by call to convert_annotation command and generate .pickle/.json files with converted annotation or you may modify accuracy checker configuration file of your model to call annotation conversion from accuracy checker.

We would appreciate if you can point us to specific places in tool documentation that you may find unclear, so we will have chance to improve that.

Regards,
Vladimir

Dear vladimir-dudnik

Atsunori__Sumi — Thu, 17 Oct 2019 02:30:43 GMT

Dear vladimir-dudnik

Thank you for your reply.

As you said, I used convert_annotation to generate json / pickle file. After that, I changed the annotation path in the cfg file and re-run accuracy_check.

But the error message doesn't seem to change. In the error log, it looks like path specification.

Is there anything wrong with my operation?

thanks.

----detail

$cat test.cfg
models:
- name: yolo_v3
launchers:
- framework: dlsdk
device: CPU
model: tmp_19r3/yolo_v3.xml
weights: tmp_19r3/yolo_v3.bin
adapter: yolo_v3
cpu_extensions: cpu_extentions_avx512.so
datasets:
- name: coco
annotation: /path/to/dir_conv/ ★change
data_source: /path/to/coco/val2017/

---

$convert_annotation mscoco_detection -o dir_conv --annotation_file /path/to/coco/annotations_2014/instances_val2014.json

.....

$accuracy_check -c ./test.cfg -m ./path/to/model -s /path/to/coco/val2017/ -a /path/to/dir_conv/

Processing info:
model: yolo_v3
launcher: dlsdk
device: CPU
dataset: coco
OpenCV version: 4.1.2-openvino
Traceback (most recent call last):
File "/usr/local/bin/accuracy_check", line 9, in <module>
load_entry_point('accuracy-checker==0.7.3', 'console_scripts', 'accuracy_check')()
File "/usr/local/lib/python3.5/dist-packages/accuracy_checker-0.7.3-py3.5.egg/accuracy_checker/main.py", line 202, in main
model_evaluation_mode(config, progress_reporter, args)
File "/usr/local/lib/python3.5/dist-packages/accuracy_checker-0.7.3-py3.5.egg/accuracy_checker/main.py", line 218, in model_evaluation_mode
model_evaluator = ModelEvaluator.from_configs(launcher_config, dataset_config)
File "/usr/local/lib/python3.5/dist-packages/accuracy_checker-0.7.3-py3.5.egg/accuracy_checker/evaluators/model_evaluator.py", line 57, in from_configs
dataset = Dataset(dataset_config)
File "/usr/local/lib/python3.5/dist-packages/accuracy_checker-0.7.3-py3.5.egg/accuracy_checker/dataset.py", line 66, in __init__
raise ConfigError('path to converted annotation or data for conversion should be specified')
accuracy_checker.config.config_validator.ConfigError: path to converted annotation or data for conversion should be specified

Dear Atsunori, Sumi,

Ekaterina_A_Intel1 — Thu, 17 Oct 2019 07:33:12 GMT

Dear Atsunori, Sumi,

What value does content your `annotation` field of config value?

As I can see, you use argument -a, which specify prefix for your annotation path. So, your configuration file should contain relative path to generated via convert_annotation mscoco_detection.pickle (It means, that you need to specify not absolute path, but only file name which stored in your dir_conv directory, if your -a value include this dir.

Also for correct execution detection model, you need to specify `dataset_meta` field with path to generated mscoco_detection.json (also generated by convert_annotation).

Please make sure, that path to your generated annotation pickle is correct.

Thank you,

Ekaterina