validation_app openvino R5 produces 0% accuracy for FP32

[torazurey__vadim]

Hello!

I tried to calculate FP32 accuracy using Openvino R5 "validation_app" but it produced the weird results:
[ INFO ] Average infer time (ms): 6.36 (157.17 images per second with batch size = 1)
Top1 accuracy: 0.00% (0 of 10 images were detected correctly, top class is correct)
Top5 accuracy: 0.00% (0 of 10 images were detected correctly, top five classes contain required class)
where the correct output should be like this:
Top1 accuracy: 70.00% (7 of 10 images were detected correctly, top class is correct)
Top5 accuracy: 80.00% (8 of 10 images were detected correctly, top five classes contain required class)

Could you please point me what I am doing wrong? Why did I receive 0 of 10 images instead 8 of 10 images or similar?

Here is the my run condition:
OS: 

• Ubuntu 16.04 

App version:

• Openvino R5

Trained Model:

• ResNet50
• I also tried to use this trained model: https://github.com/BVLC/caffe/tree/master/models/bvlc_reference_caffenet but it produces the same result - "0 of 10 images".

Validation Dataset:

• 10 Imagenet jpeg pictures with class ID 449 were downloaded from http://169.44.201.108:7002/imagenet/val/ and placed to /dataset folder along with labels.txt

Here is the contents of the my labels.txt file:
   ILSVRC2012_val_00000293.JPEG 449
   ILSVRC2012_val_00002138.JPEG 449
   ILSVRC2012_val_00003014.JPEG 449
   ILSVRC2012_val_00006697.JPEG 449
   ILSVRC2012_val_00007197.JPEG 449
   ILSVRC2012_val_00009111.JPEG 449
   ILSVRC2012_val_00009191.JPEG 449
   ILSVRC2012_val_00009346.JPEG 449
   ILSVRC2012_val_00009379.JPEG 449
   ILSVRC2012_val_00009396.JPEG 449

Here are the steps which I used to run "validation_app":

1. python3 mo.py --input_model /ResNet50/ResNet50.caffemodel --input_proto /ResNet50/ResNet50.prototxt --output_dir /ResNet50
2. validation_app -m /ResNet50/ResNet50.xml -d CPU -t C -i /dataset/labels.txt

Here is the full output:
- Path to the Input Model: /ResNet50/ResNet50.caffemodel
- Path for generated IR: /ResNet50
- IR output name: ResNet50
- Log level: ERROR
- Batch: Not specified, inherited from the model
- Input layers: Not specified, inherited from the model
- Output layers: Not specified, inherited from the model
- Input shapes: Not specified, inherited from the model
- Mean values: Not specified
- Scale values: Not specified
- Scale factor: Not specified
- Precision of IR: FP32
- Enable fusing: True
- Enable grouped convolutions fusing: True
- Move mean values to preprocess section: False
- Reverse input channels: False
Caffe specific parameters:
- Enable resnet optimization: True
- Path to the Input prototxt: /ResNet50/ResNet50.prototxt
- Path to CustomLayersMapping.xml: Default
- Path to a mean file: Not specified
- Offsets for a mean file: Not specified
Model Optimizer version: 1.5.12.49d067a0

[ SUCCESS ] Generated IR model.
[ SUCCESS ] XML file: /ResNet50/ResNet50.xml
[ SUCCESS ] BIN file: /ResNet50/ResNet50.bin
[ SUCCESS ] Total execution time: 7.31 seconds.

[ INFO ] InferenceEngine:
     API version ............ 1.4
     Build .................. 19154
[ INFO ] Parsing input parameters
[ INFO ] Loading plugin

     API version ............ 1.5
     Build .................. lnx_20181004
     Description ....... MKLDNNPlugin
[ INFO ] Loading network files
[ INFO ] Preparing input blobs
[ INFO ] Batch size is 1
[ INFO ] Device: CPU
[ INFO ] Collecting labels
[ INFO ] Starting inference
Progress: [....................] 100.00% done
[ INFO ] Inference report:
     Network load time: 58.59ms
     Model: /ResNet50/ResNet50.xml
     Model Precision: FP32
     Batch size: 1
     Validation dataset: /dataset/labels.txt
     Validation approach: Classification network
[ INFO ] Average infer time (ms): 6.36 (157.17 images per second with batch size = 1)
Top1 accuracy: 0.00% (0 of 10 images were detected correctly, top class is correct)
Top5 accuracy: 0.00% (0 of 10 images were detected correctly, top five classes contain required class)

[Shubha_R_Intel]

Dearest Vadim, sorry you are having troubles with the validation app. Here are online instructions for the validation app:

https://docs.openvinotoolkit.org/R5/_samples_validation_app_README.html

And here is your command:

validation_app -m /ResNet50/ResNet50.xml -d CPU -t C -i /dataset/labels.txt

I see that you are passing in your labels file to -i, which is not correct. For -i you should pass in your path to images. Use -lbl for the labels file path.

Hope it helps. Thanks for using OpenVino !

Shubha

[torazurey__vadim]

Hello Shubha!
Thank you for your help and for the fast answer!

I already tried earlier in my trys to run validation_app to use -lbl option but did not achieve any results, but I still tried again as you advised and received a message: "[ ERROR ] Input problem: Images list is not specified (missing -i option)"

According validation_app documentation -i is not only the path to images, for -i you can also pass the path to .txt file with a list of images - that's what I did (you can see it in my post in the "Validation Dataset" section). I created dataset according instruction from section "Dataset Format for Classification: List of Images (ImageNet*-like)" from here: https://software.intel.com/en-us/articles/OpenVINO-IE-Samples#preparing-dataset-validation so I put images along with .txt file in the same folder. As you see I did everything according to the documentation. validation_app received the commands I passed and did not generate any errors about missing images. At the moment I do not know what to do next to make validation_app produce non zero accuracy. 
Could you please provide for me a complete correct launch line?

[Shubha_R_Intel]

Dearest Vadim. First of all, thank you for your patience. I reproduced your exact problem actually, using your images and  https://github.com/BVLC/caffe/tree/master/models/bvlc_reference_caffenet

validation_app -m /ResNet50/ResNet50.xml -d CPU -t C -i /dataset/labels.txt

And your above command seems valid, so sorry for doubting you earlier. 

By class 449, you mean "tench" right ? As obtained from https://gist.github.com/aaronpolhamus/964a4411c0906315deb9f4a3723aac57 where tench is 

a European freshwater fish of the minnow family, popular with anglers and widely introduced elsewhere, including several US states.

This definitely may be a bug. Let me investigate and I will post the findings here.

Thanks for using OpenVino !

Shubha

[torazurey__vadim]

>>By class 449, you mean "tench" right ?

yes

>>Let me investigate and I will post the findings here.

Thank you, please keep me aware of your further findings!