Inference Result Discrepancy between Keras and OpenVino

[Brewer__Frank]

Hi,

I have a trained MobileNetV2 model on Keras but I am getting discrepancy in results (20 mismatches out of 2500 images) in between inference directly done on Keras side vs .h5 file converted to protocol buffer (.pb) and then to OpenVino IR files. Do you have any idea regarding where it might be originating from or any solution suggestions?

For inference on the Keras side, I'm using my GPU Nvidia GTX 1070 (during training no specific floating point precision is defined)

This is how I load the model, and do the inference on the Keras side:

import os
import sys
from utils import *
from keras.models import Model
from keras.layers import Flatten, Dense, Reshape, Dropout, GlobalAveragePooling2D
from keras.preprocessing.image import ImageDataGenerator
from keras.preprocessing import image
from keras.applications import mobilenet_v2


h5File = 'trainedModel.h5'
imgDir = '../images'

baseModel = mobilenet_v2.MobileNetV2(input_shape=(224, 224, 3), include_top=False, weights='imagenet')
x = baseModel.output
x = GlobalAveragePooling2D()(x)
predictions = Dense(units=len(classes), activation='softmax')(x)

model = Model(inputs=baseModel .input, outputs=predictions )

model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])


for imname in os.listdir(imgDir):


    imgL= image.load_img(imgFullName, target_size=(224,224))

    img = image.img_to_array(imgL)
    img = img.reshape(-1,224,224,3)
    out = model.predict(img / 255.)

    predictedClass = np.argmax(np.mean(out, axis=0))

OpenVino side:

First I convert saved .h5 file to .pb with the following script:

import tensorflow as tf
from tensorflow.python.framework import graph_io
from tensorflow.keras.models import load_model


# Clear any previous session.
tf.keras.backend.clear_session()

save_pb_dir = './model'
model_fname = './model/trainedModel.h5'

def freeze_graph(graph, session, output, save_pb_dir='.', save_pb_name='trainedModel.pb', save_pb_as_text=False):
    with graph.as_default():
        graphdef_inf = tf.graph_util.remove_training_nodes(graph.as_graph_def())
        graphdef_frozen = tf.graph_util.convert_variables_to_constants(session, graphdef_inf, output)
        graph_io.write_graph(graphdef_frozen, save_pb_dir, save_pb_name, as_text=save_pb_as_text)
        return graphdef_frozen

# This line must be executed before loading Keras model.
tf.keras.backend.set_learning_phase(0)

model = load_model(model_fname)

session = tf.keras.backend.get_session()

INPUT_NODE = [t.op.name for t in model.inputs]
OUTPUT_NODE = [t.op.name for t in model.outputs]
print(INPUT_NODE, OUTPUT_NODE)
frozen_graph = freeze_graph(session.graph, session, [out.op.name for out in model.outputs], save_pb_dir=save_pb_dir)

Then with modelOptimizer I convert it to IR files:

python mo_tf.py --input_model trainedModel.pb --model_name trainedModel --input_shape [1,224,224,3] --scale 255 

And then I use Image Classification C++ Sample Async example to load the IR files, and do the inference. There are 20 mismatches out of 2500 images and whereas on the Keras side correct class probability for those mismatches are > 0.9, on OpenVino side highest class scores are in between 0.5 and 0.6.

Oh and in case it is required, here is the model optimizer's log: (Warning regarding cudart64_100.dll is not important I guess,right? [Even though I have a GPU on my system])

C:\Program Files (x86)\IntelSWTools\openvino\deployment_tools\model_optimizer>python mo_tf.py --input_model trainedModel.pb --model_name trainedModel --input_shape [1,224,224,3] --scale 255
Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      C:\Program Files (x86)\IntelSWTools\openvino\deployment_tools\model_optimizer\trainedModel.pb
        - Path for generated IR:        C:\Program Files (x86)\IntelSWTools\openvino\deployment_tools\model_optimizer\.
        - IR output name:       trainedModel
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         Not specified, inherited from the model
        - Output layers:        Not specified, inherited from the model
        - Input shapes:         [1,224,224,3]
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         255.0
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
TensorFlow specific parameters:
        - Input model in text protobuf format:  False
        - Path to model dump for TensorBoard:   None
        - List of shared libraries with TensorFlow custom layers implementation:        None
        - Update the configuration file with input/output node names:   None
        - Use configuration file used to generate the model with Object Detection API:  None
        - Operations to offload:        None
        - Patterns to offload:  None
        - Use the config file:  None
Model Optimizer version:        2019.3.0-408-gac8584cb7
2019-11-15 11:14:59.099136: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
2019-11-15 11:14:59.102118: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

[ SUCCESS ] Generated IR model.
[ SUCCESS ] XML file: C:\Program Files (x86)\IntelSWTools\openvino\deployment_tools\model_optimizer\.\trainedModel.xml
[ SUCCESS ] BIN file: C:\Program Files (x86)\IntelSWTools\openvino\deployment_tools\model_optimizer\.\trainedModel.bin
[ SUCCESS ] Total execution time: 11.17 seconds.

[Luis_at_Intel · Accepted solution]

Thanks for reaching out, I can see from the info that you are using one of the samples in the OpenVINO toolkit. The Inference Engine samples load input images in BGR channels order, if your model is trained on images loaded with the RGB channels you may need to reverse the order (visit this link for more info). If that is the case include the flag --reverse_input_channels when converting your model to IR format.

python mo_tf.py --input_model trainedModel.pb --model_name trainedModel --input_shape [1,224,224,3] --scale 255 --reverse_input_channels  

Please give this a try and let us know if this helps.

Regards,

Luis

[Brewer__Frank]

Luis_at_Intel (Intel) wrote:

Thanks for reaching out, I can see from the info that you are using one of the samples in the OpenVINO toolkit. The Inference Engine samples load input images in BGR channels order, if your model is trained on images loaded with the RGB channels you may need to reverse the order (visit this link for more info). If that is the case include the flag --reverse_input_channels when converting your model to IR format.

python mo_tf.py --input_model trainedModel.pb --model_name trainedModel --input_shape [1,224,224,3] --scale 255 --reverse_input_channels  

Please give this a try and let us know if this helps.

Regards,

Luis

Hey Luis,

Thanks for your feedback. Yes that was exactly the issue I had. I logged back into forum in order to write the solution and noticed that you have already replied. Thank you :)

Best regards

[Kamma__Vijayakumar]

Hi All,

I have a same issue could you please help me out on this. Have tried all the options are like --reverse_input_channels.

I have same kind of h5 file which is trained on mobilenet_v2 and converted to pb and then using mo.py to convert the file. There is a accuracy drop drastically.

C:\Program Files (x86)\IntelSWTools\openvino_2020.2.117\deployment_tools\model_optimizer>python mo_tf.py --input_model C:\Users\VijayakumarKamma\Desktop\SFTYOV\mask_classifier20.h5.pb --input_shape [1,224,224,3] --output_dir C:\Users\VijayakumarKamma\Desktop\FMOV\exp\class20\FP16 --scale 255 --reverse_input_channels

Thanks you!!

[Kamma__Vijayakumar]

Could you please help me on this ASAP ?

[Luis_at_Intel]

Hi @Kamma, Vijayakumar,

Please elaborate more as to how you are getting such accuracy drop, are you using any of the samples/demos to test your model? If so let us know which sample was used, share sample input image, which plugin tested (CPU, GPU, MYRIAD), and your TF version used to train model.

Note there is a new release of OpenVINO (2020.3), if possible please give this a try and let us know the results.

Regards,

Luis

[Kamma__Vijayakumar]

Hi Thank you for reply.

i was running a video as source and running parallely with the both models like openvino and h5 models. Where openvino showing no face_mask but h5 was showing face_mask.

* You can try on any face mask wearing videos. 

I was using GPU, and tried with FP16,FP32 .

when i try with out this (reverse_input_channels) while conversion the accuracy is nothing that is 0.

after converting the model with reverse_input_channels option i have got some 70 % accuracy but not at all working like H5 model.

i was  using tensorflow 1.15.0  to convert H5 to PB file,  and for training have used TF 2.0 (some publicly available model.)

i am using opnvino 2020.2.

Would you please suggest me ASAP !!

Thank you!!

[Kamma__Vijayakumar]

Hi,

This time i have trained model on TF 1.15 gpu and converted with same TF 1.15.0

Provided all the resource as well along with running ipynb files to run both h5 and OPENVINO models. 

[Kamma__Vijayakumar]

please refer to these OpenVino files.

[Kamma__Vijayakumar]

@Luis_at_Intel (Intel)

 Please help me on this !!

[Max_L_Intel]

Hi Vijayakumar.

I think you might need to use TensorFlow conversion specific parameters from this guide https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_tf_specific_Convert_Object_Detection_API_Models.html

Command example:

<INSTALL_DIR>/deployment_tools/model_optimizer/mo_tf.py --input_model=/tmp/ssd_inception_v2_coco_2018_01_28/frozen_inference_graph.pb --transformations_config <INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/tf/ssd_v2_support.json --tensorflow_object_detection_api_pipeline_config /tmp/ssd_inception_v2_coco_2018_01_28/pipeline.config --reverse_input_channels

Since you have trained your model with TF version 1.15, you need to pick up the following json file: 

ssd_support_api_v.1.15.json — for frozen SSD topologies trained manually using the TensorFlow* Object Detection API version 1.15 or higher.

Support for TensorFlow version 1.15 has been just recently added, so please make sure you are using the latest available OpenVINO toolkit build.

Hope this helps.
Best regards, Max.

[Kamma__Vijayakumar]

Hi Max,

I have trained classification algorithm  not detection algorithm. I have not used any SSd pipeline.

This is on MobileNetV2 pre-trained model which classifies the faces into mask or no_mask.

https://www.pyimagesearch.com/2020/05/04/covid-19-face-mask-detector-with-opencv-keras-tensorflow-and-deep-learning/

Which i have used as my training process.

I am not sure why i have to use the pipeline config and ssd related things. Could you please explain me ?

[Kamma__Vijayakumar]

Would you please help me on this asap ?