mobilenet openvino ncs2/CPU wrong classification

[Nunez-Yanez__Jose]

Hello,

I successfully downloaded and converted googlenet v2 from the model zoo. I used this command to get the NCS2 model.

mo.py --input_model C:\Users\eejlny\projects\models\model_zoo2\open_model_zoo\model_downloader\classification\googlenet\v2\caffe\googlenet-v2.caffemodel --output_dir C:\Users\eejlny\projects\models\model_zoo2\open_model_zoo\model_downloader\classification\googlenetfp16\v2\caffe --data_type FP16

I also got a FP32 for the CPU and it also works.

For my work I need to use mobilenet v2 and I have followed the same steps but although the conversion works fine the mobilenet model does not have any accuracy. I upgraded to the new openvino release and downloaded the mobilenet v2 from the model zoo again but the result is the same.

I get the expected accuracy with imagenet using googlenet-v2 but mobilenet-v2 (or other variants of mobilenet) do not work.

The run on the CPU looks particular wrong since the classification output is stuck to a single class with mobilenet. The output is almost always class 834. 

Any other steps I need to take to work with mobilenet-v2 ?

Thanks,

[Nunez-Yanez__Jose]

Hello,

I have tried to do the same thing in a Linux system and I am having the same problems. I do not understand what I am doing wrong. Below is an example with inception V2 that I got from from https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html and the supported fronzen quantized topologies. I am using the latest openvino that works OK with googlenet or squezzenet.

I got the frozen pb and optimized as below:

 python3 /hdd2/programs/openvino/openvino_2019.1.133/deployment_tools/model_optimizer/mo.py --input_model /hdd1/eejlny/projects/openvino/models/inception/FP32/classification/inception/v2/tf/inception-v2.frozen.pb /hdd1/eejlny/projects/op                         envino/models/inception/ir/FP32//classification/inception/v2/tf --data_type FP32

Then I tried to use the xml with imagenet but it does not classify correctly as you can see below for some examples. I am just using the CPU for this this. I am assuming this has been trained with imagenet but the web page does not seem to say anything ? 

Image ./1000_images//ILSVRC2012_val_00000690.jpg Predicted 905 905: 'window shade', Expected 127: 'white stork, Ciconia ciconia',
 Image ./1000_images//ILSVRC2012_val_00000691.jpg Predicted 905 905: 'window shade', Expected 955: 'jackfruit, jak, jack',
 Image ./1000_images//ILSVRC2012_val_00000692.jpg Predicted 905 905: 'window shade', Expected 306: 'rhinoceros beetle',
 Image ./1000_images//ILSVRC2012_val_00000693.jpg Predicted 905 905: 'window shade', Expected 425: 'barn',
 Image ./1000_images//ILSVRC2012_val_00000694.jpg Predicted 905 905: 'window shade', Expected 190: 'Sealyham terrier, Sealyham',
 Image ./1000_images//ILSVRC2012_val_00000695.jpg Predicted 905 905: 'window shade', Expected 370: 'guenon, guenon monkey',
 

I have attached a zip with the PB and the generated XML/bin. Can someone give me a hint of where is my mistake, please ? I know my setup is not totally wrong because I can get googlenet and squeezenet to work but many others such as mobilenet and inception do not get any correct outputs. Thanks

Thanks,
 

[Tsin__Ross]

As far as I know, the activation component of OpenVINO does not generate correct results. It took me a long time to investigate and bypass it.

 Do not use logistic activation in OpenVINO

[Nunez-Yanez__Jose]

Thanks for the reply.

I was just using the examples provided in the Intel model zoo and trying to use mobilenet v2 for Myriad. So if these are broken maybe they should be investigated further since they are just the examples. 

[Shubha_R_Intel]

Dear Nunez-Yanez, Jose,

Thank you for your *.zip file. I will certainly investigate this.

 You say

Then I tried to use the xml with imagenet 

How did you do this ? Did you run one of our (Python or C++) classification samples ?

Please give me more details. Thanks !

Shubha

[Nunez-Yanez__Jose]

Thanks,

Yes, I was using code based on your cpp classification_sample_async that I modified to do ImageNet.

I guess an explanation is that my code is broken but as I said classification accuracy is OK for googlenet and squeezenet so the code seems OK.

Classification seems to be all over the place for inception and mobilenet models though for both CPU and Myriad targets.

As a reference I have attached my labels and main.cpp  from classification_sample_async.

[Nunez-Yanez__Jose]

Hello,

To further clarify the problem I am having I am uploading a zip file with converted googlenet (that  works and achieves 56% T1 accuracy on my test) and mobilenet (that does not work and the outputs are mostly stuck to a single class).

Both are based on examples from model zoo and are just targeting the CPU (FP32). Example run:

./async_jose -i ./50000_images/ -d CPU -m /hdd1/eejlny/projects/openvino/models/mobilenet-v2/ir/FP32/classification/mobilenet/v2/caffe/mobilenet-v2.xml

[Shubha_R_Intel]

Dearest Nunez-Yanez, Jose,

Thank you for all your supporting files. I will certainly look into it soon. Again, thanks for your patience !

Shubha

[Shubha_R_Intel]

Dear Nunez-Yanez, Jose,

I did some research and came upon the conclusion that the problems you're having with mobilenet_v2 may have nothing to do with OpenVino. The below is a Stack Overflow discussion regarding the lack of accuracy of this model:

https://stackoverflow.com/questions/55001855/loaded-mobilenet-model-giving-wrong-predictions

Have you tried mobilenet inference outside of OpenVino first ? (for instance, by using Tensorflow ).

Thanks,

Shubha

[Nunez-Yanez__Jose]

Thanks for the reply. I have just been doing more tests to understand the problem.

The models work correctly using openvino with python and the predictions are correct. This confirms that the xml and bin are correct.

Yet when I tried to use them with your c++ demo examples the predictions are wrong.  For example, the attached frozen_model.xml and bin corresponds to mobilenet_v2 that works correctly with openvino and python for the attached images with predictions 648 (medical chest) and 386 (elephant).

The outputs for the classification_sample demo application are incorrect as shown below. Any ideas what could be wrong with the cpp demo provided by you ? This is the original code and I have not modified anything.

Throughput: 310.3042678 FPS

[ INFO ] Execution successful
eejlny@it034655:/hdd1/eejlny/projects/openvino$ Top 10 results:

Image ./1_image//elephant.jpg

classid probability
------- -----------
794     0.8827393
549     0.0231773
904     0.0081506
619     0.0076212
916     0.0045124
721     0.0036566
443     0.0033500
760     0.0029196
556     0.0020896
782     0.0018409

Image ./1_image//object.jpg

classid probability
------- -----------
721     0.4615425
794     0.2896880
549     0.0367076
591     0.0312646
893     0.0242253
443     0.0178238
411     0.0107063
750     0.0060976
904     0.0043285
999     0.0041631
 

[Shubha_R_Intel]

Dear Nunez-Yanez, Jose

 What do you mean by this statement ? Can you kindly clarify ?

The models work correctly using openvino with python and the predictions are correct. This confirms that the xml and bin are correct.

Are you saying that classification_sample.py produced correct results while the C++ classfication_sample.exe didn't ? If this is the case, then this is clearly a bug.

Please let me know.

Thanks,

Shubha

[Nunez-Yanez__Jose · Accepted solution]

OK, I finally found out the problem and it is not an openvino problem, sorry for not realising earlier.

The problem is that this mobilenet network and similar need an additional normalization step to normalize the inputs to the network to the range 0 to 1. Something like this has to be added to the classify.py and C++ code. I did not know about these details.

 processedImg = (np.array(processedImg)) / 255.0

Now it works as expected.

[Shubha_R_Intel]

Dear Nunez-Yanez, Jose,

Fantastic debugging and troubleshooting. And thank you so much for informing the OpenVino community !

Shubha

[Cagampan__John_Berna]

Nunez-Yanez, Jose wrote:

OK, I finally found out the problem and it is not an openvino problem, sorry for not realising earlier.

The problem is that this mobilenet network and similar need an additional normalization step to normalize the inputs to the network to the range 0 to 1. Something like this has to be added to the classify.py and C++ code. I did not know about these details.

 processedImg = (np.array(processedImg)) / 255.0

Now it works as expected.

Hi there, we are also encountering the same issues using mobilenet_1.0_224 topology. Can we ask for the detailed steps on how you did the normalization.

Thanks,

Bernard :)