Very low accuracy on FP32

[whung6]

I'm recently trying on the accuracy test for mobilenet-v1-1.0-224. After I downloaded the model from the model_downloader to create .prototxt and .caffemodel file. I used the model_optimizer to create .xml and .bin for FP32 format, however, when I ran the accuracy_check.py code, the accuracy is very low. Is there anything I did wrong, FP32 should have a high accuracy instead.

For model_downloader:

$ python3 downloader.py --name mobilenet-v1-1.0-224 --output_dir ~/models

For model_optimizer:

$ python3 mo.py --input_model ~/models/public/mobilenet-v1-1.0-224/mobilenet-v1-1.0-224.caffemodel --input_proto ~/models/public/mobilenet-v1-1.0-224/mobilenet-v1-1.0-224.prototxt --data_type FP32 --output_dir ~/models/public/mobilenet-v1-1.0-224/ --input_shape [1,3,224,224] --input data --mean_values data[103.94,116.78,123.68] --scale_values data[58.8235294117647] --output prob --framework caffe --reverse_input_channels

For accuracy_check:

 python3 accuracy_check.py --config /opt/intel/openvino_2019.3.376/deployment_tools/open_model_zoo/tools/accuracy_checker/configs/mobilenet-v1-1.0-224.yml -m /home/ubuntu/models/ -s /home/ubuntu/imagenet/val/ -a /home/ubuntu/imagenet/annotations/imagenet -td CPU -e /opt/intel/openvino_2019.3.376/deployment_tools/inference_engine/lib/intel64/ -M /opt/intel/openvino_2019.3.376/deployment_tools/model_optimizer/ -tf dlsdk -tt I8 -d /opt/intel/openvino_2019.3.376/deployment_tools/open_model_zoo/tools/accuracy_checker/dataset_definitions.yml

My mobilenet-v1-1.0-224.yml:

models:

 - name: mobilenet-v1-1.0-224-cf

  launchers:

   - framework: caffe

    model:  public/mobilenet-v1-1.0-224/mobilenet-v1-1.0-224.prototxt

    weights: public/mobilenet-v1-1.0-224/mobilenet-v1-1.0-224.caffemodel

    adapter: classification

  datasets:

   - name: imagenet_1000_classes

    preprocessing:

     - type: resize

      size: 256

     - type: crop

      size: 224

     - type: normalization

      mean: 103.94, 116.78, 123.68

      std: 58.8235294

 - name: mobilenet-v1-1.0-224

  launchers:

   - framework: dlsdk

    tags:

     - FP32

    model:  public/mobilenet-v1-1.0-224/mobilenet-v1-1.0-224.xml

    weights: public/mobilenet-v1-1.0-224/mobilenet-v1-1.0-224.bin

    adapter: classification

   - framework: dlsdk

    tags:

     - FP16

    model:  public/mobilenet-v1-1.0-224/FP16/mobilenet-v1-1.0-224.xml

    weights: public/mobilenet-v1-1.0-224/FP16/mobilenet-v1-1.0-224.bin

    adapter: classification

   - framework: dlsdk

    tags:

     - I8sm

    model:  public/mobilenet-v1-1.0-224/I8sm/mobilenet-v1-1.0-224_i8.xml

    weights: public/mobilenet-v1-1.0-224/I8sm/mobilenet-v1-1.0-224_i8.bin

    adapter: classification

   - framework: dlsdk

    tags:

     - I8

    model:  public/mobilenet-v1-1.0-224/I8/mobilenet_v1_1.0_224_frozen_i8.xml

    weights: public/mobilenet-v1-1.0-224/I8/mobilenet_v1_1.0_224_frozen_i8.bin

    adapter: classification

  datasets:

   - name: imagenet_1000_classes

    data_source: /home/ubuntu/imagenet/val/ILSVRC2012_img_val

    annotation:  /home/ubuntu/imagenet/annotations/imagenet.pickle

    dataset_meta: /home/ubuntu/imagenet/annotations/imagenet.json

    annotation_conversion:

     converter: imagenet

     annotation_file: /home/ubuntu/imagenet/val.txt

     labels_file:   /home/ubuntu/imagenet/synset_words.txt

     has_background: True

    subsample_size: 2000

    #has_background: True

    preprocessing:

     - type: bgr_to_rgb

     - type: resize

      size: 256

     - type: crop

      size: 224

     - type: normalization

      mean: 103.94, 116.78, 123.68

      std: 58.8235294

    metrics:

     - name: accuracy@top1

      type: accuracy

      top_k: 1

     - name: accuracy@top5

      type: accuracy

      top_k: 5

Result:

Processing info:

model: mobilenet-v1-1.0-224

launcher: dlsdk

launcher tags: FP32

device: CPU

dataset: imagenet_1000_classes

OpenCV version: 4.1.2-openvino

IE version: 2.1.custom_releases/2019/R3_ac8584cb714a697a12f1f30b7a3b78a5b9ac5e05

Loaded CPU plugin version: 2.1.32974

2000 objects processed in 19.012 seconds

accuracy@top1: 0.15%

accuracy@top5: 0.80%

[David_C_Intel]

Hi whung6,

Thank you for reaching out, we are still working on this issue.

We will contact you as soon as possible. 

Best regards,

David C.

Intel Customer Support Technician

A Contingent Worker at Intel

[David_C_Intel]

Hi whung6,

I apologize for the delay in our response. The development team has made changes to the Accuracy Checker Tool and introduced a Post-Training Optimization tool in the latest OpenVINO™ toolkit 2020.1 release.

Could you try using the two tools with the new release and let us know if you continue to see the same behavior?

You can download the OpenVINO™ toolkit 2020.1 release from the download page.

David C.

Intel Customer Support Technician

A Contingent Worker at Intel