Error in converting custom ssd model using Tensorflow2 Object detection API

[Denko]

Hi, I am trying to convert a custom SSD MobileNet V2 FPNLite 320x320 from TensorFlow2 model zoo to Openvino Intermediate Representation (IR) to leverage it on Intel movidius compute stick 2 and Raspberry Pi 4. 

System information (version)

• OpenVINO = 2021.3.0-2787-60059f2c755-releases/2021/3
• Operating System / Platform => Windows 10 10.0.19042
• Problem classification: Model Conversion
• Framework: TensorFlow
• Model name:  SSD MobileNet V2 FPNLite 320x320

Detailed description

The model was trained in google colab, then I' ve got the frozen graph obtained with following command:

python Tensorflow/models/research/object_detection/exporter_main_v2.py --input_type=image_tensor --pipeline_config_path=Tensorflow/workspace/models/my_ssd_mobnet_data_v2/pipeline.config --trained_checkpoint_dir=Tensorflow/workspace/models/my_ssd_mobnet_data_v2 --output_directory=Tensorflow/workspace/models/my_ssd_mobnet_data_v2/export

the export folder with saved_model and pipeline.config is in attachments : export.tar

I have tried two versions of conversion the following link to the documentation and Error bellow:

Converting TensorFlow* Object Detection API Models

python mo_tf.py --saved_model_dir export\saved_model --transformations_config extensions\front\tf\ssd_support_api_v2.0.json --tensorflow_object_detection_api_pipeline_config export\pipeline.config

[ ERROR ] -------------------------------------------------
[ ERROR ] ----------------- INTERNAL ERROR ----------------
[ ERROR ] Unexpected exception happened.
[ ERROR ] Please contact Model Optimizer developers and forward the following information:
[ ERROR ] Exception occurred during running replacer "ObjectDetectionAPIOutputReplacement (<class 'extensions.front.tf.ObjectDetectionAPI.ObjectDetectionAPIOutputReplacement'>)": 'inputs'

 extended bug report is in attachment: bag_report_Object_Detection_API_Models.txt

Convert TensorFlow* 2 Models 

python mo_tf.py --saved_model_dir export\saved_model

[ ERROR ] Cannot infer shapes or values for node "StatefulPartitionedCall/map/TensorArrayV2_2".
[ ERROR ] Tensorflow type 21 not convertible to numpy dtype.
[ ERROR ]
[ ERROR ] It can happen due to bug in custom shape infer function <function tf_native_tf_node_infer at 0x000002763D08A0D0>.
[ ERROR ] Or because the node inputs have incorrect values/shapes.
[ ERROR ] Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ] Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ] Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "StatefulPartitionedCall/map/TensorArrayV2_2" node.
For more information please refer to Model Optimizer FAQ, question #38. (https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=38#question-38)

extended bug report is in attachment: bag_report_Convert_TensorFlow2_Models.txt

[Iffa_Intel]

Hi,

Most of the time these kinds of issues persist when an unsupported model is used.

However, looking at this GitHub thread [Enable TF 2.0 Object Detection API models by lazarevevgeny · Pull Request #3556 · openvinotoolkit/openvino (github.com)], it seems that SSD MobileNet V2 FPNLite is supported for both 320x320 & 640x640.

Plus, if you re-trained the model, the biggest chance is there may be an issue with the configuration file faster_rcnn_support_api_v2.0.json, since some part of the model may also has changed, causing the json to be no further compatible.

We are currently investigating this and will get back to you shortly.

Sincerely,

Iffa

[Denko]

Thank you for your reply 

Finally I found thous GitHub threads

[Question about converting Tensorflow Object Detection 2.3 to Model Optimizer ]

[Question about converting Tensorflow Object Detection 2.4 to Model Optimizer ]

I build openVino from Master Branch using official documentation 

version: custom_master_66b3efbb6cc3f5c08f482e43dd370a38684f185a

command: python mo_tf.py 

--saved_model_dir export\saved_model 

--transformations_config extensions\front\tf\ssd_support_api_v2.4.json

--tensorflow_object_detection_api_pipeline_config export\pipeline.config --reverse_input_channels

Script returns saved_model_tmp.bin, but not .xml file  

and an error :

The Preprocessor block has been removed. Only nodes performing mean value subtraction and scaling (if applicable) are kept.

[ ERROR ]  Error while emitting attributes for layer StatefulPartitionedCall/ssd_mobile_net_v2fpn_keras_feature_extractor/FeatureMaps/top_down/projection_3/BiasAdd/Add/Transpose (id = 282). It usually means that there is unsupported pattern around this node or unsupported combination of attributes.

I may have built the project incorrectly, but I have not received the corresponding errors

Thank you in advance

Sincerely,

Denis 

[Iffa_Intel]

Have you freeze your model?

I'm not sure whether You know about this but You'll need to freeze your native model in order to convert them to IR using Model Optimizer.

You may refer to the details here: https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html#loading-nonfrozen-models

Sincerely,

Iffa

[Denko]

The model was exported from google colab with following command:

python Tensorflow/models/research/object_detection/exporter_main_v2.py --input_type=image_tensor --pipeline_config_path=Tensorflow/workspace/models/my_ssd_mobnet_data_v2/pipeline.config --trained_checkpoint_dir=Tensorflow/workspace/models/my_ssd_mobnet_data_v2 --output_directory=Tensorflow/workspace/models/my_ssd_mobnet_data_v2/export

Are you sure that this link fits my problem? 

Because for object detection API there is another documentation Converting TensorFlow* Object Detection API Models 

I obtained frozen graph from my ‘saved_model’ dir according to your last link, the command.txt is in attachments

I have changed the original saved_model.pd to this output fozen_graph.pb, but got an error 

python mo_tf.py

--saved_model_dir export\saved_model

--transformations_config extensions\front\tf\ssd_support_api_v2.4.json

--tensorflow_object_detection_api_pipeline_config export\pipeline.config --reverse_input_channels

[ FRAMEWORK ERROR ]  Cannot load input model: SavedModel format load failure: 'utf-8' codec can't decode byte 0xff in position 24: 'utf-8' codec can't decode byte 0xff in position 24: invalid start byte in field: tensorflow.MetaGraphDef.MetaInfoDef.tensorflow_version

and another command that does no require .json :

mo_tf.py --input_model export\saved_model\saved_model.pb --input_shape [1,320,320,3]

Thank you anyway ! 

Since the latest version of OpenVino supported my model from model zoo I came to the conclusion that the problem may be in my custom model. 

When I find the problem I’ll post the solution in this post 

Sincerely,

Denis

[Diego_Bima]

I also have the same problem and I have been looking for the solution for several days.
I can convert the model loaded from the Model Zoo, but when I train it with my custom data I can't convert it anymore.
I tried to freeze the original model and then couldn't convert it to IR. This is why I think the problem is in the freeze process.

[Zulkifli_Intel]

Hello Diego,

We recommend you to open a new case, in order for us to serve you better.

Regards,

Zulkifli

[Iffa_Intel · Accepted solution]

Hi,

After verifying the latest change in the model optimizer, the Developer fixed the model optimized for TF2.

The issue that you encounter is due to the .json file version and the model optimized for OpenVINO version 2021.3 does not have the fix for TF2.

Here the fix;

1. git clone the latest OpenVINO open-source version from GitHub.
2. compile the latest OpenVINO version based on documentation (change the python version)
3. after successful compile, go to model-optimizer and run the following command;
4. python mo.py --saved_model_dir export\saved_model --transformations_config \openvino\model-optimizer\extensions\front\tf\ssd_support_api_v2.4.json" --tensorflow_object_detection_api_pipeline_config export\pipeline.config --reverse_input_channels --scale 127.5 --mean_values [127.5,127.5,127.5]
5.  please use ssd_support_api_v2.4.json

Another option is, since this change will be available on OpenVINO Distribution 2021.4, you can wait for next available release for the fix.

Hope this helps!

Sincerely,

Iffa

[Denko]

This solution works for me
Thank you for your time !

[Iffa_Intel]

Greetings,

Intel will no longer monitor this thread since this issue has been resolved. If you need any additional information from Intel, please submit a new question. 

Sincerely,

Iffa