[OpenVINO 2018.R4] After converted inception_v3, openvino result is different with tensorflow

Stevenys_L_ · ‎11-27-2018

Hi,

I follow this issue(https://software.intel.com/en-us/forums/computer-vision/topic/799700)
and Intel tutorial(https://software.intel.com/en-us/articles/OpenVINO-Using-TensorFlow)
to convert my own model(inception_v3) to .xml & .bin,

but the output result top_1~top_5 is different with the original tensorflow
(reference: "result/classify_flower_openvino_r4.csv", "result/classify_flower_tf.csv")

Here is my steps (Train the flower datasets from tensorflow-slim):

- Tensorflow slim steps:

1. Using TensorFlow-Slim to fine-tuning a model from an existing checkpoint

   export DATASET_DIR=/<MY_PATH>/slim_flower/flowers
   export TRAIN_DIR=/<MY_PATH>/slim_flower/inception_v3
   export CHECKPOINT_PATH=/<MY_PATH>/slim_flower/inception_v3.ckpt

   python train_image_classifier.py \
        --train_dir=${TRAIN_DIR} \
        --dataset_dir=${DATASET_DIR} \
        --dataset_name=flowers \
        --dataset_split_name=train \
        --model_name=inception_v3 \
        --checkpoint_path=${CHECKPOINT_PATH} \
        --checkpoint_exclude_scopes=InceptionV3/Logits,InceptionV3/AuxLogits \
        --trainable_scopes=InceptionV3/Logits,InceptionV3/AuxLogits

2. Then it will generate .ckpt and .pbtxt (in openvino_issue/inception_v3_flower):
- model.ckpt-1535333.data-00000-of-00001
- model.ckpt-1535333.index
- model.ckpt-1535333.meta
- graph.pbtxt

3. Dwonload tensorflow_models, and put the annex code in slim project

$ git clone https://github.com/tensorflow/models.git tensorflow_models
$ cp -r openvino_issue/flower_photos /<PATH>/tensorflow_models/research/slim/
$ cp -r openvino_issue/inception_v3_flower /<PATH>/tensorflow_models/research/slim/
$ cp openvino_issue/classify_flower_tf_slim.py /<PATH>/tensorflow_models/research/slim/

4. Start inference

$ python3 classify_flower_tf_slim.py \
      --checkpoint_path=openvino_issue/inception_v3_flower/model.ckpt-1535333 \
      --test_list=./flower_list.txt \
      --test_dir=openvino_issue/flower_photos \
      --batch_size=1 \
      --num_classes=5 \
      --model_name=inception_v3

- OpenVINO R4 steps:

1. Cause of .pbtxt has no input placeholder, so I use openvino_issue/model_freeze_graph.py to generate it

  $ cd /<PATH>/tensorflow_models/research/slim/
  $ cp openvino_issue/model_freeze_graph.py /<PATH>/tensorflow_models/research/slim/
  $ python3 model_freeze_graph.py

Then it will generate (in openvino_issue/openvino_step_1 folder) :
- flower_inception_v3.ckpt.data-00000-of-00001
- flower_inception_v3.ckpt.index
- flower_inception_v3.ckpt.meta
- new_graph.pbtxt

2. freeze model

  $ cd /opt/intel/computer_vision_sdk_2018.4.420/deployment_tools/model_optimizer
  $ mkdir -p openvino_issue/inception_v3_flower

  ## copy flower_inception_v3.ckpt.data-00000-of-00001, flower_inception_v3.ckpt.index,  flower_inception_v3.ckpt.meta, new_graph.pbtxt into openvino_issue/inception_v3_flower ##

  $ python3 /usr/local/lib/python3.5/dist-packages/tensorflow/python/tools/freeze_graph.py \
      --input_graph openvino_issue/inception_v3_flower/new_graph.pbtxt \
      --input_checkpoint openvino_issue/inception_v3_flower/flower_inception_v3.ckpt \
      --output_node_names InceptionV3/Predictions/Reshape_1 \
      --output_graph flower_frozen_inception_v3.pb

3. check topology

  $ python3 /opt/intel/computer_vision_sdk_2018.4.420/deployment_tools/model_optimizer/mo/utils/summarize_graph.py --input_model ./flower_frozen_inception_v3.pb
       
  1 input(s) detected:
  Name: input, type: float32, shape: (-1,299,299,3)
  1 output(s) detected:
  InceptionV3/Predictions/Reshape_1

4. convert .xml & .bin

$ python3 /opt/intel/computer_vision_sdk_2018.4.420/deployment_tools/model_optimizer/mo_tf.py \
       --input_model /opt/intel/computer_vision_sdk_2018.4.420/deployment_tools/model_optimizer/flower_frozen_inception_v3.pb \
       --input_shape [1,299,299,3] \
       --mean_value [127.5,127.5,127.5] \
       --scale 127.5 \
       --input input \
       --model_name flower_inception_v3_r4 \
       --output_dir /volume1/home/admin/stevenys/classify_flower

5. The classifiation of openvino code : classify_flower_openvino_r4.py
command: $ python3 classify_flower_openvino_r4.py (in openvino_issue.zip)

<p.s.> If csv of result cannot open by excel, you can execute code again
1. $ python3 test_image_classifier.py ......
2. $ python3 classify_flower_openvino_r4.py

The results are not the same, do you have any suggest ?
Or in OpenVINO step 1 have something wrong?

Thank you for your help.

Here is my file link(276MB):
https://drive.google.com/file/d/1VOUb9mJGZj7dyQzuDgcHaJVHADjpFpGk/view?usp=sharing

kumar__sumit · ‎03-07-2019

I am trying to do inference on open vino after retraining the inceptionV3 model. I was following the steps you mentioned above but in OpenVINO R4 steps 1, I couldn't find openvino_issue/model_freeze_graph.py file. Could you please tell me where can i find this file or the way to add input placeholder layer in the .pbtxt file generated earlier.

Thanks,

sumit

Shubha_R_Intel · ‎03-08-2019

freeze_graph.py is a Tensorflow product and you can find it here (see below link). If you installed Tensorflow on your computer though, there should be a local version on your machine.

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py

smith__jet · ‎04-20-2019

hello, i tried to calculate fp32 accuracy using Openvino R5 "validation_app" but it produced this:
[ INFO ] Average infer time (ms): 64.45 (15.52 images per second with batch size = 1) │classification_sample_async InferenceEngineUnitTests speech_sample
Top1 accuracy: 1.02% (51 of 5000 images were detected correctly, top class is correct)
Top5 accuracy: 16.94% (847 of 5000 images were detected correctly, top five classes contain required

Well , i think the results should be like this:
top1 accuracy: 80.2%(it is tested using tensorflow).

Here is my run condition:
**OS**: Ubuntu 16.04
**project version**: Openvino R5(2019 branch)
**pretrained model**: **inception_v4 tensorflow model** [https://github.com/tensorflow/models/tree/master/research/slim](url)
**validation dataset** : imagenet 2012(i use one tenth of total images and 5 images each class)

Here are my experimental steps:

**1. Converting a TensorFlow Model**
under the dldt/model-optimizer, run mo_tf.py, the following is my sh script:

> python3 mo_tf.py \
> --input_model ./inception_v4.pb \
> --model_name Inception_v4 \
> --output_dir inception_v4 \
> --input_shape [1,299,299,3] \
> --data_type FP32 \
> --mean_value [127.5,127.5,127.5] \
> --scale 127.5 \
> --reverse_input_channels \
> --input input_img \
> --output InceptionV4/Logits/Logits/BiasAdd

output:
Model Optimizer arguments:
Common parameters:
- Path to the Input Model: ****inception_v4.pb
- Path for generated IR: ****/inception_v4
- IR output name: Inception_v4
- Log level: ERROR
- Batch: Not specified, inherited from the model
- Input layers: input_img
- Output layers: InceptionV4/Logits/Logits/BiasAdd
- Input shapes: [1,299,299,3]
- Mean values: [127.5,127.5,127.5]
- Scale values: Not specified
- Scale factor: 127.5
- Precision of IR: FP32
- Enable fusing: True
- Enable grouped convolutions fusing: True
- Move mean values to preprocess section: False
- Reverse input channels: True
TensorFlow specific parameters:
- Input model in text protobuf format: False
- Path to model dump for TensorBoard: None
- List of shared libraries with TensorFlow custom layers implementation: None
- Update the configuration file with input/output node names: None
- Use configuration file used to generate the model with Object Detection API: None
- Operations to offload: None
- Patterns to offload: None
- Use the config file: None
Model Optimizer version: unknown versio

it generate three files:
Inception_v4.bin Inception_v4.mapping Inception_v4.xml

**2. run validation_app**
first build the project under dldt/inference-engine/, then validation_app binaries could be found in inference-engine/bin/intel64/Release, run the shell like this:
```
./validation_app -t C \
-i ***imagenet_val/5000_image/img_class.txt \
-m ****/dldt/model-optimizer/inception_v4/Inception_v4.xml \
-d CPU \
-ppType ResizeCrop \
-ppSize 299
```

my image list formats as:
ILSVRC2012_val_00005901.JPEG 97
ILSVRC2012_val_00006460.JPEG 97
ILSVRC2012_val_00001602.JPEG 406
ILSVRC2012_val_00002523.JPEG 406
ILSVRC2012_val_00002876.JPEG 406

well, 5000 images are placed under imagenet_val/5000_image.
i have checked all my steps and confused about why the top1 acc is only **1.02%**!!!!!!

Shubha_R_Intel · ‎04-23-2019

Dear smith, jet and kumar, sumit,

First if you haven't already done so, please upgrade to 2019 R1 ! Next, reading your steps above, I don't think you used model optimizer properly to create IR for an Inception model. We have very specific steps to do that in the below link:

https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_tf_specific_Convert_Slim_Library_Models.html

Thanks for using OpenVino !

Shubha