problem with using accuracy checker

[whung6]

So I'd successfully convert my model into INT8 format, after this, I'm trying to use the accuracy checker to see whether my model will perform.

I ran the following command line

$ python3 accuracy_check.py \

--config /opt/intel/openvino_2019.3.376/deployment_tools/open_model_zoo/tools/accuracy_checker/configs/resnet-50.yml \

--models /opt/intel/openvino_2019.3.376/deployment_tools/open_model_zoo/tools/downloader/public/resnet-50/ \

--source /home/ubuntu/imagenet/val/ \

-a /home/ubuntu/imagenet/annotations/imagenet/ \

-td CPU \

-e /opt/intel/openvino_2019.3.376/deployment_tools/inference_engine/lib/intel64/libcpu_extension_avx2.so \

-M /opt/intel/openvino_2019.3.376/deployment_tools/model_optimizer/ \

--definitions /opt/intel/openvino_2019.3.376/deployment_tools/tools/calibration_tool/configs/definitions.yml

however the result is

Result:

15:25:29 accuracy_checker WARNING: /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a sy

nonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

 _np_qint8 = np.dtype([("qint8", np.int8, 1)])

15:25:29 accuracy_checker WARNING: /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a sy

nonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

 _np_quint8 = np.dtype([("quint8", np.uint8, 1)])

15:25:29 accuracy_checker WARNING: /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a sy

nonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

 _np_qint16 = np.dtype([("qint16", np.int16, 1)])

15:25:29 accuracy_checker WARNING: /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a sy

nonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

 _np_quint16 = np.dtype([("quint16", np.uint16, 1)])

15:25:29 accuracy_checker WARNING: /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a sy

nonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

 _np_qint32 = np.dtype([("qint32", np.int32, 1)])

15:25:29 accuracy_checker WARNING: /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a sy

nonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

 np_resource = np.dtype([("resource", np.ubyte, 1)])

15:25:29 accuracy_checker WARNING: /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type'

as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

 _np_qint8 = np.dtype([("qint8", np.int8, 1)])

15:25:29 accuracy_checker WARNING: /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type'

as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

 _np_quint8 = np.dtype([("quint8", np.uint8, 1)])

15:25:29 accuracy_checker WARNING: /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type'

as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

 _np_qint16 = np.dtype([("qint16", np.int16, 1)])

15:25:29 accuracy_checker WARNING: /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type'

as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

 _np_quint16 = np.dtype([("quint16", np.uint16, 1)])

15:25:29 accuracy_checker WARNING: /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type'

as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

 _np_qint32 = np.dtype([("qint32", np.int32, 1)])

15:25:29 accuracy_checker WARNING: /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type'

as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.

 np_resource = np.dtype([("resource", np.ubyte, 1)])

Processing info:

model: resnet-50-cf

launcher: caffe

device: CPU

dataset: imagenet_1000_classes

OpenCV version: 4.1.2-openvino

Traceback (most recent call last):

 File "accuracy_check.py", line 19, in <module>

  main()

 File "/opt/intel/openvino_2019.3.376/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/main.py", line 202, in main

  model_evaluation_mode(config, progress_reporter, args)

 File "/opt/intel/openvino_2019.3.376/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/main.py", line 218, in model_evaluation_mode

  model_evaluator = ModelEvaluator.from_configs(launcher_config, dataset_config)

 File "/opt/intel/openvino_2019.3.376/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/evaluators/model_evaluator.py", line 57, in from_configs

  dataset = Dataset(dataset_config)

 File "/opt/intel/openvino_2019.3.376/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/dataset.py", line 66, in __init__

  raise ConfigError('path to converted annotation or data for conversion should be specified')

accuracy_checker.config.config_validator.ConfigError: path to converted annotation or data for conversion should be specified

-------------------------------------------------------​----------------------

Clearly I'm missing something, but i can't figure out what is missing.

I also provide the file context below to make it easier to know my situation.​

resnet-50.yml：

models:

 - name: resnet-50-cf

  launchers:

   - framework: caffe

    model:  public/resnet-50/resnet-50.prototxt

    weights: public/resnet-50/resnet-50.caffemodel

    adapter: classification

  datasets:

   - name: imagenet_1000_classes

    preprocessing:

     - type: resize

      size: 256

      aspect_ratio_scale: greater

     - type: crop

      size: 224

     - type: normalization

      mean: 104, 117, 123

 - name: resnet-50

  launchers:

   - framework: dlsdk

    tags:

     - FP32

    model:  public/resnet-50/FP32/resnet-50.xml

    weights: public/resnet-50/FP32/resnet-50.bin

    adapter: classification

   - framework: dlsdk

    tags:

     - FP16

    model:  public/resnet-50/FP16/resnet-50.xml

    weights: public/resnet-50/FP16/resnet-50.bin

    adapter: classification

  datasets:

   - name: imagenet_1000_classes

    preprocessing:

     - type: resize

      size: 256

      aspect_ratio_scale: greater

     - type: crop

      size: 224

The definitions.yml file is from https://software.intel.com/en-us/forums/computer-vision/topic/807243

Thanks

[David_C_Intel]

Hi whung6,

Thank you for contacting Intel Customer Support.

I am currently working on your inquiry, I will reply you back as soon as possible.

If you have any other questions, feel free to ask.

Regards,

David C.

Intel Customer Support Technician

A Contingent Worker at Intel

[whung6]

Sure! Thank you!

[David_C_Intel]

Hi whung6,

I apologize for the delay in our response, looking further into the error message. It looks like you may need to update the definitions.yml to ensure data_source is pointing to the path to the directory with input data and the annotation is pointing to the pickle file.

Let us know if you need further assistance.

Best Regards,

David C.

Intel Customer Support Technician

A Contingent Worker at Intel

[whung6]

Thanks for the solution, I did mispoint to the wrong directory, after fixing the problem I can now work well.