Accuracy Checker: running sample

[gomes__antonio]

Hi,

I'm trying to test the Accuracy Checker tool on YOLO but first I want run the sample on https://docs.openvinotoolkit.org/latest/_tools_accuracy_checker_sample_README.html to check that everthing is okay.

I already install all requirements asked in C:\Program Files (x86)\IntelSWTools\openvino\python\python3.6\requirements.txt but I don´t get it how to download the model and dataset files.

On documentation it asks to execute the command git lfs pull but this command don't work because it does not have a repository.

What am I missing?  Thanks

I'm using Windows 10

[Shubha_R_Intel]

Dear gomes, antonio,

I've written a very detailed post on how to use calibration tools here in dldt issue 171.

Please clone the dldt github https://github.com/opencv/dldt.git .

The github (open source) version of OpenVino is not exactly the same as the one you download in the release package. Yes of course the code is similar but the packaging is quite different.

Hope it helps,

Thanks,

Shubha

[gomes__antonio]

Dear Shubba,

Thanks for reply but it did not help much.  It's a lot for my programming skills. I thought it was simpler to use the Accuracy Checker tool.

The documentation https://docs.openvinotoolkit.org/latest/_inference_engine_tools_accuracy_checker_tool_README.html is very confuse and it's not possible run the sample to try understand how the tool works.

I want to calculate the precision and the recall for Yolov3 in the COCO dataset but I can not even run the accuracy_check.py, give the error: ModuleNotFoundError: No module named 'openvino'

is there any simpler tutorial?

[Shubha_R_Intel]

Dear gomes, antonio,

Unfortunately there isn't a simpler tutorial.  But this error:

ModuleNotFoundError: No module named 'openvino'

Is easily solved. Simply set your environment variables - setupvars.bat (*.sh), located under bin. From the shell that you've run this script (setting environment variables), try again and run the Python program.

Thanks,

Shubha

[Atsunori__Sumi]

Dear Shubha

I have recently discovered and used this tool, but I'm getting an error and can't measure yolo-v3 mAP.

I would like to know the mAP results measured by Intel. In addition, please tell me how to use this tool.

Thanks,

[Atsunori__Sumi]

I'm sorry to continue The log is shown for a problem that is currently not working.

$python3 /opt/intel/openvino/deployment_tools/tools/accuracy_checker_tool/accuracy_check.py -c ./test.cfg -m path/to -s /path/to/coco/val2017/ -a /path/to/coco/annotations_pascalformat/

Processing info:
model: yolo_v3
launcher: dlsdk
device: CPU
dataset: coco
OpenCV version: 4.1.2-openvino
Traceback (most recent call last):
  File "/opt/intel/openvino/deployment_tools/tools/accuracy_checker_tool/accuracy_check.py", line 19, in <module>
    main()
  File "/opt/intel/openvino_2019.3.334/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/main.py", line 202, in main
    model_evaluation_mode(config, progress_reporter, args)
  File "/opt/intel/openvino_2019.3.334/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/main.py", line 218, in model_evaluation_mode
    model_evaluator = ModelEvaluator.from_configs(launcher_config, dataset_config)
  File "/opt/intel/openvino_2019.3.334/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/evaluators/model_evaluator.py", line 57, in from_configs
    dataset = Dataset(dataset_config)
  File "/opt/intel/openvino_2019.3.334/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/dataset.py", line 66, in __init__
    raise ConfigError('path to converted annotation or data for conversion should be specified')
accuracy_checker.config.config_validator.ConfigError: path to converted annotation or data for conversion should be specified

----
$cat test.cfg:
models:
- name: yolo_v3
  launchers:
     - framework: dlsdk
       device: CPU
       model: ./path/to/yolo_v3.xml
       weights: ./path/to/yolo_v3.bin
       adapter: yolo_v3
       cpu_extensions: cpu_extentions_avx512.so
  datasets:
     - name: coco
       annotation: ~/path/to/coco/annotations_pascalformat/
       data_source: ~/path/to/coco/val2017/

[Vladimir_Dudnik]

Had you have a chance to read through accuracy checker documentation first? I think that a chapter on annotation conversion may help to see that accuracy checker require conversion of original dataset annotation. You may do that as separate preliminary step by call to convert_annotation command and generate .pickle/.json files with converted annotation or you may modify accuracy checker configuration file of your model to call annotation conversion from accuracy checker.

We would appreciate if you can point us to specific places in tool documentation that you may find unclear, so we will have chance to improve that. 

Regards,
   Vladimir

[Atsunori__Sumi]

Dear vladimir-dudnik

Thank you for your reply.

As you said, I used convert_annotation to generate json / pickle file. After that, I changed the annotation path in the cfg file and re-run accuracy_check.

But the error message doesn't seem to change. In the error log, it looks like  path specification.

Is there anything wrong with my operation?

thanks.

----detail

$cat test.cfg 
models:
- name: yolo_v3
  launchers:
     - framework: dlsdk
       device: CPU
       model: tmp_19r3/yolo_v3.xml
       weights: tmp_19r3/yolo_v3.bin
       adapter: yolo_v3
       cpu_extensions: cpu_extentions_avx512.so
  datasets:
     - name: coco
       annotation: /path/to/dir_conv/ ★change
       data_source: /path/to/coco/val2017/

---

$convert_annotation mscoco_detection -o dir_conv --annotation_file /path/to/coco/annotations_2014/instances_val2014.json

.....

$accuracy_check -c ./test.cfg -m ./path/to/model -s /path/to/coco/val2017/ -a /path/to/dir_conv/

Processing info:
model: yolo_v3
launcher: dlsdk
device: CPU
dataset: coco
OpenCV version: 4.1.2-openvino
Traceback (most recent call last):
  File "/usr/local/bin/accuracy_check", line 9, in <module>
    load_entry_point('accuracy-checker==0.7.3', 'console_scripts', 'accuracy_check')()
  File "/usr/local/lib/python3.5/dist-packages/accuracy_checker-0.7.3-py3.5.egg/accuracy_checker/main.py", line 202, in main
    model_evaluation_mode(config, progress_reporter, args)
  File "/usr/local/lib/python3.5/dist-packages/accuracy_checker-0.7.3-py3.5.egg/accuracy_checker/main.py", line 218, in model_evaluation_mode
    model_evaluator = ModelEvaluator.from_configs(launcher_config, dataset_config)
  File "/usr/local/lib/python3.5/dist-packages/accuracy_checker-0.7.3-py3.5.egg/accuracy_checker/evaluators/model_evaluator.py", line 57, in from_configs
    dataset = Dataset(dataset_config)
  File "/usr/local/lib/python3.5/dist-packages/accuracy_checker-0.7.3-py3.5.egg/accuracy_checker/dataset.py", line 66, in __init__
    raise ConfigError('path to converted annotation or data for conversion should be specified')
accuracy_checker.config.config_validator.ConfigError: path to converted annotation or data for conversion should be specified
 

[Ekaterina_A_Intel1]

Dear Atsunori, Sumi,

What value does content your `annotation` field of config value?

As I can see, you use argument -a, which specify prefix for your annotation path. So, your configuration file should contain relative path to generated via convert_annotation mscoco_detection.pickle (It means, that you need to specify not absolute path, but only file name which stored in your dir_conv directory, if your -a value include this dir.

Also for correct execution detection model, you need to specify `dataset_meta` field with path to generated mscoco_detection.json (also generated by convert_annotation). 

Please make sure, that path to your generated annotation pickle is correct.

Thank you,

Ekaterina