FasterRCNN Resnet50 COCO trained model not working with Model Optimzer

[Ammar]

Hi, i am trying to convert my own trained Faster RCNN model to IR, but hitting an error.

Openvino version: 2021.3

model: TF2 Faster RCNN Resnet50 coco

Here's my CLI input

mo_tf.py --input_model ./Downloads/frozen_inference_graph.pb --transformations_config /opt/intel/openvino_2021/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support_api_v2.0.json --tensorflow_object_detection_api_pipeline_config ./Downloads/faster_rcnn_resnet50_coco.config --input_shape [1,300,300,3] --reverse_input_channels --data_type=FP16

And the log output:

Model Optimizer arguments:
Common parameters:
- Path to the Input Model: /home/ammar/./Downloads/frozen_inference_graph.pb
- Path for generated IR: /home/ammar/.
- IR output name: frozen_inference_graph
- Log level: ERROR
- Batch: Not specified, inherited from the model
- Input layers: Not specified, inherited from the model
- Output layers: Not specified, inherited from the model
- Input shapes: [1,300,300,3]
- Mean values: Not specified
- Scale values: Not specified
- Scale factor: Not specified
- Precision of IR: FP16
- Enable fusing: True
- Enable grouped convolutions fusing: True
- Move mean values to preprocess section: None
- Reverse input channels: True
TensorFlow specific parameters:
- Input model in text protobuf format: False
- Path to model dump for TensorBoard: None
- List of shared libraries with TensorFlow custom layers implementation: None
- Update the configuration file with input/output node names: None
- Use configuration file used to generate the model with Object Detection API: /home/ammar/./Downloads/faster_rcnn_resnet50_coco.config
- Use the config file: None
- Inference Engine found in: /opt/intel/openvino_2021/python/python3.8/openvino
Inference Engine version: 2.1.2021.3.0-2787-60059f2c755-releases/2021/3
Model Optimizer version: 2021.3.0-2787-60059f2c755-releases/2021/3
2021-05-07 10:52:28.593284: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
[ WARNING ] Model Optimizer removes pre-processing block of the model which resizes image keeping aspect ratio. The Inference Engine does not support dynamic image size so the Intermediate Representation file is generated with the input image size of a fixed size.
[ WARNING ] The model resizes the input image keeping aspect ratio with min dimension 600, max dimension 1024. The provided input height 300, width 300 is transformed to height 600, width 600.
[ ERROR ] -------------------------------------------------
[ ERROR ] ----------------- INTERNAL ERROR ----------------
[ ERROR ] Unexpected exception happened.
[ ERROR ] Please contact Model Optimizer developers and forward the following information:
[ ERROR ] Exception occurred during running replacer "ObjectDetectionAPIPreprocessor2Replacement (<class 'extensions.front.tf.ObjectDetectionAPI.ObjectDetectionAPIPreprocessor2Replacement'>)":
[ ERROR ] Traceback (most recent call last):
File "/opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 290, in apply_transform
for_graph_and_each_sub_graph_recursively(graph, replacer.find_and_replace_pattern)
File "/opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo/middle/pattern_match.py", line 59, in for_graph_and_each_sub_graph_recursively
func(graph)
File "/opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo/front/tf/replacement.py", line 48, in find_and_replace_pattern
self.transform_graph(graph, desc._replacement_desc['custom_attributes'])
File "/opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/extensions/front/tf/ObjectDetectionAPI.py", line 604, in transform_graph
assert start_nodes[0] in graph.nodes
AssertionError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo/main.py", line 345, in main
ret_code = driver(argv)
File "/opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo/main.py", line 309, in driver
ret_res = emit_ir(prepare_ir(argv), argv)
File "/opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo/main.py", line 252, in prepare_ir
graph = unified_pipeline(argv)
File "/opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo/pipeline/unified.py", line 25, in unified_pipeline
class_registration.apply_replacements(graph, [
File "/opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 340, in apply_replacements
apply_replacements_list(graph, replacers_order)
File "/opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 326, in apply_replacements_list
apply_transform(
File "/opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo/utils/logger.py", line 124, in wrapper
function(*args, **kwargs)
File "/opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 314, in apply_transform
raise Exception('Exception occurred during running replacer "{} ({})": {}'.format(
Exception: Exception occurred during running replacer "ObjectDetectionAPIPreprocessor2Replacement (<class 'extensions.front.tf.ObjectDetectionAPI.ObjectDetectionAPIPreprocessor2Replacement'>)":

[ ERROR ] ---------------- END OF BUG REPORT --------------

[Ammar]

Apologies. I made a mistake. The model was trained on TF 1.14 and Openvino 2020.4

The command

mo_tf.py --input_model ./Downloads/frozen_inference_graph.pb --transformations_config /opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support_api_v1.14.json --tensorflow_object_detection_api_pipeline_config ./Downloads/faster_rcnn_resnet50_coco.config  --data_type=FP16

The log output

[ WARNING ] Use of deprecated cli option --tensorflow_use_custom_operations_config detected. Option use in the following releases will be fatal. Please use --transformations_config cli option instead
Model Optimizer arguments:
Common parameters:
- Path to the Input Model: /home/ia4/keras_openvino/model/training_robot/frozen_inference_graph.pb
- Path for generated IR: /home/ia4/keras_openvino/converted_training_robot
- IR output name: frozen_inference_graph
- Log level: ERROR
- Batch: Not specified, inherited from the model
- Input layers: Not specified, inherited from the model
- Output layers: Not specified, inherited from the model
- Input shapes: [1,1024,600,3]
- Mean values: Not specified
- Scale values: Not specified
- Scale factor: Not specified
- Precision of IR: FP16
- Enable fusing: True
- Enable grouped convolutions fusing: True
- Move mean values to preprocess section: False
- Reverse input channels: True
TensorFlow specific parameters:
- Input model in text protobuf format: False
- Path to model dump for TensorBoard: None
- List of shared libraries with TensorFlow custom layers implementation: None
- Update the configuration file with input/output node names: None
- Use configuration file used to generate the model with Object Detection API: /home/ia4/keras_openvino/model/training_robot/pipeline.config
- Use the config file: /opt/intel/openvino_2020.3.355/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support_api_v1.14.json
Model Optimizer version:
[ WARNING ] Model Optimizer removes pre-processing block of the model which resizes image keeping aspect ratio. The Inference Engine does not support dynamic image size so the Intermediate Representation file is generated with the input image size of a fixed size.
The Preprocessor block has been removed. Only nodes performing mean value subtraction and scaling (if applicable) are kept.
[ ERROR ] Failed to match nodes from custom replacement description with id 'ObjectDetectionAPIProposalReplacement':
It means model and custom replacement description are incompatible.
Try to correct custom replacement description according to documentation with respect to model node names
[ ANALYSIS INFO ] Your model looks like TensorFlow Object Detection API Model.
Check if all parameters are specified:
--tensorflow_use_custom_operations_config
--tensorflow_object_detection_api_pipeline_config
--input_shape (optional)
--reverse_input_channels (if you convert a model to use with the Inference Engine sample applications)
Detailed information about conversion of this model can be found at
https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_tf_specific_Convert_Object_Detection_API_Models.html
[ ERROR ] Exception occurred during running replacer "ObjectDetectionAPIDetectionOutputReplacement" (<class 'extensions.front.tf.ObjectDetectionAPI.ObjectDetectionAPIDetectionOutputReplacement'>): Found the following nodes '[]' with name 'crop_proposals' but there should be exactly 1. Looks like ObjectDetectionAPIProposalReplacement replacement didn't work.

[IntelSupport]

Hi Ammar,

Thanks for reaching out.

Firstly, I would like to clarify if your faster_rcnn_resnet50_coco model is a custom model or downloaded the model from the OpenVINO Model Downloader? Also, are you executing all the workaround in OpenVINO 2020.4 environment?

Meanwhile, I have tested the same model that I downloaded from Model Downloader and generated the .xml and .bin file using the converter.py script. I would suggest you use converter.py to generate the IR file of OpenVINO.

Refer to this documentation for more information regarding the converter.

https://docs.openvinotoolkit.org/latest/omz_models_model_faster_rcnn_resnet50_coco.html

Regards,

Aznie

[Ammar]

Hi Aznie,

it's a custom model that we've trained with additional classes.

Can you clarify what you mean by "Also, are you executing all the workaround in OpenVINO 2020.4 environment?"

 "I would suggest you use converter.py to generate the IR file of OpenVINO." is there anything in particular about this python script that i should pay attention to vs mo.py?

[IntelSupport]

Hi Ammar,

As you mentioned, you are using OpenVINO 2020.4 but from your execution " Use the config file: /opt/intel/openvino_2020.3.355/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support_api_v1.14.json " the .json file detected is from 2020.3. Please make sure you are running and using all the files from the same OpenVINO version which is 2020.4. I guess you have many versions of OpenVINO installed in your machine. Please make sure you are using the correct OpenVINO environment with your model.

Meanwhile, the converter.py (model converter) converts the models that are not in the Inference Engine IR format into that format using Model Optimizer. During the conversion of the model into Intermediate Representation (IR) using the Model Optimizer, you need to specify the argument/flag in the command according to the supported topologies. However, converter.py will automatically find and match the topologies that satisfy the model.

Since your model is a custom model, you have to make sure that you are using the correct configuration file that matches your model. The documentation below gives detailed rules about which *.json you can use with which Tensorflow API version.

https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_convert_model_tf_specific_Convert_Object_Detection_API_Models.html

Regards,

Aznie

[Ammar]

Hi Aznie,

I've decided to use openvino 2021.3.394

I ran the following command:

python3 /opt/intel/openvino_2021/deployment_tools/model_optimizer/mo_tf.py --input_model Downloads/frozen_inference_graph.pb --tensorflow_object_detection_api_pipeline_config Downloads/faster_rcnn_resnet50_coco.config --transformations_config /opt/intel/openvino_2021/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support_api_v1.14.json

The Output:

/opt/intel/openvino_2021.3.394/d- Path to the Input Model: /home/i/Downloads/frozen_inference_graph.pbeployment_tools/model_optimizer/mo/main.py:89: SyntaxWarning: "is" with a literal. Did you mean "=="?
if op is 'k':
Model Optimizer arguments:
Common parameters:
- Path to the Input Model: /home/i/Downloads/frozen_inference_graph.pb
- Path for generated IR: /home/i/.
- IR output name: frozen_inference_graph
- Log level: ERROR
- Batch: Not specified, inherited from the model
- Input layers: Not specified, inherited from the model
- Output layers: Not specified, inherited from the model
- Input shapes: Not specified, inherited from the model
- Mean values: Not specified
- Scale values: Not specified
- Scale factor: Not specified
- Precision of IR: FP32
- Enable fusing: True
- Enable grouped convolutions fusing: True
- Move mean values to preprocess section: None
- Reverse input channels: False
TensorFlow specific parameters:
- Input model in text protobuf format: False
- Path to model dump for TensorBoard: None
- List of shared libraries with TensorFlow custom layers implementation: None
- Update the configuration file with input/output node names: None
- Use configuration file used to generate the model with Object Detection API: /home/i/Downloads/faster_rcnn_resnet50_coco.config
- Use the config file: None
- Inference Engine found in: /opt/intel/openvino_2021/python/python3.8/openvino
Inference Engine version: 2.1.2021.3.0-2787-60059f2c755-releases/2021/3
Model Optimizer version: 2021.3.0-2787-60059f2c755-releases/2021/3
2021-05-12 16:10:31.183079: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/mediasdk/lib/:/opt/intel/openvino_2021/data_processing/dl_streamer/lib:/opt/intel/openvino_2021/data_processing/gstreamer/lib:/opt/intel/openvino_2021/opencv/lib:/opt/intel/openvino_2021/deployment_tools/ngraph/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/external/tbb/lib::/opt/intel/openvino_2021/deployment_tools/inference_engine/external/hddl/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/external/omp/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/external/gna/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/external/mkltiny_lnx/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/lib/intel64:/usr/lib/x86_64-linux-gnu:/usr/lib
2021-05-12 16:10:31.183101: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
[ WARNING ] Model Optimizer removes pre-processing block of the model which resizes image keeping aspect ratio. The Inference Engine does not support dynamic image size so the Intermediate Representation file is generated with the input image size of a fixed size.
Specify the "--input_shape" command line parameter to override the default shape which is equal to (600, 600).
The Preprocessor block has been removed. Only nodes performing mean value subtraction and scaling (if applicable) are kept.
[ ERROR ] Failed to match nodes from custom replacement description with id 'ObjectDetectionAPIProposalReplacement':
It means model and custom replacement description are incompatible.
Try to correct custom replacement description according to documentation with respect to model node names
[ ANALYSIS INFO ] Your model looks like TensorFlow Object Detection API Model.
Check if all parameters are specified:
--tensorflow_use_custom_operations_config
--tensorflow_object_detection_api_pipeline_config
--input_shape (optional)
--reverse_input_channels (if you convert a model to use with the Inference Engine sample applications)
Detailed information about conversion of this model can be found at
https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_tf_specific_Convert_Object_Detection_API_Models.html
[ ERROR ] Exception occurred during running replacer "ObjectDetectionAPIDetectionOutputReplacement" (<class 'extensions.front.tf.ObjectDetectionAPI.ObjectDetectionAPIDetectionOutputReplacement'>): Found the following nodes '[]' with name 'crop_proposals' but there should be exactly 1. Looks like ObjectDetectionAPIProposalReplacement replacement didn't work.

My comment

I didn't use converter.py and decided to use mo_tf.py instead

Can you explain to me what the errors below mean? What I did was train the model with an additional class. Will that change the parameter?

[ ERROR ] Exception occurred during running replacer "ObjectDetectionAPIDetectionOutputReplacement" (<class 'extensions.front.tf.ObjectDetectionAPI.ObjectDetectionAPIDetectionOutputReplacement'>): Found the following nodes '[]' with name 'crop_proposals' but there should be exactly 1. Looks like ObjectDetectionAPIProposalReplacement replacement didn't work.

I'm attaching my files so you can have a go. I don't think it's openvino version related.

LINK: https://bit.ly/3eBPhtc

[IntelSupport]

Hi Ammar,

Thank you for providing the model. The error might be related to the compatibility of the version of TensorFlow with the .json file with your model. We are currently investigating this and will get back to you with the information at the earliest

Regards,

Aznie

[IntelSupport]

Hi Ammar,

Just to make sure, are you using Tensorflow 1.14 version? Also, could you try this command line and see the output.

python mo_tf.py --input_model "Downloads\frozen_inference_graph.pb" --reverse_input_channels -- data_type=FP32 --input_shape=[1,600,1024,3] --input=image_tensor --transformations_config "<INSTALL_DIR>\deployment_tools\model_optimizer\extensions\front\tf\faster_rcnn_support.json" --tensorflow_object_detection_api_pipeline_config "\Downloads\faster_rcnn_resnet50_coco.config"

Regards,

Aznie

[Ammar]

Hi Aznie,

Yes, we confirm using TF1.14

Just to be safe we have tried all the faster_rcnn_support json reference. They all failed with the same error.

[IntelSupport]

Hi Ammar,

Before this, you have mentioned that you trained the model with TensorFlow 1.14 and OpenVINO 2020.4 so the best to use with tf v1.14 version is using faster_rcnn_support_api_v1.14.json.

What can I suggest you, try to run the model optimizer on 2020.4 and using the command line that was shared before. Since your model is a custom model,there might be some issue with the configuration file (.json). Some parts of the model may change and causing the .json file to be incompatible.

Meanwhile, for your custom model, you can try to use faster_rcnn_inception_v2_coco as a reference. You can download the model using Model Downloader.

Plus, for your information, OpenVINO 2021.3 supports tf v1.15 and above only.

Regards,

Aznie

[Ammar]

Hi Aznie,

Sure, we can give that a go. On your suggestion to use  faster_rcnn_inception_v2_coco, are you suggesting we train our model using this instead of current one? Would you have any points what would be the difference between the two?

Cheers,

[IntelSupport]

Hi Ammar,

Yes, you could use the faster_rcnn_inception_v2_coco as a reference for your custom model. The structure, topologies, and layers of the model can be a reference to you. This model is from open model zoo and can successfully be converted into IR OpenVINO.

Another suggestion is, you could try to freeze your model with TensorFlow version 1.15 and then run the model optimized with faster_rcnn_support_api_v1.15.json.

Regards,

Aznie

[IntelSupport]

Hi Ammar,

Some update for you, we have tested your model with faster_rcnn_support_api_v1.13.json and finally can be succesfully optimized.Here is the command that I used:

Python mo.py --input_model "frozen_inference_graph.pb" --reverse_input_channels --input_shape=[1,600,1024,3] --input=image_tensor --transformations_config "<INSTALL_DIR>\openvino_2021.3.394\deployment_tools\model_optimizer\extensions\front\tf\faster_rcnn_support_api_v1.13.json" --tensorflow_object_detection_api_pipeline_config "faster_rcnn_resnet50_coco (1).config"

Regards,

Aznie

[IntelSupport]

Hi Ammar,

This thread will no longer be monitored since we have provided a solution. If you need any additional information from Intel, please submit a new question.

Regards,

Aznie