Problems converting to IR files the TF custom Faster RCNN with openvino

[CBell1]

Hello,

I have some problems converting to IR files the TF custom Faster RCNN openvino using TF 1.14.0 openvino_2019.1.144 python 3.5.2

Command:
$ sudo python3 deployment_tools/model_optimizer/mo_tf.py --input_model=/home/cosma/Development/venvov/dataset/train/fruits_faster_rcnn_inception_inference_graph/frozen_inference_graph.pb --tensorflow_use_custom_operations_config=/opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support_api_v1.7.json --tensorflow_object_detection_api_pipeline_config=/home/cosma/Development/venvov/dataset/train/fruits_faster_rcnn_inception_inference_graph/pipeline.config --reverse_input_channels

With or without --reverse_input_channels, I have the following error:

[ WARNING ] Model Optimizer removes pre-processing block of the model which resizes image keeping aspect ratio. The Inference Engine does not support dynamic image size so the Intermediate Representation file is generated with the input image size of a fixed size.
Specify the "--input_shape" command line parameter to override the default shape which is equal to (600, 600).
The Preprocessor block has been removed. Only nodes performing mean value subtraction and scaling (if applicable) are kept.
WARNING: Logging before flag parsing goes to stderr.
E1103 13:47:32.123020 140314342913792 main.py:317] Exception occurred during running replacer "ObjectDetectionAPIDetectionOutputReplacement" (<class 'extensions.front.tf.ObjectDetectionAPI.ObjectDetectionAPIDetectionOutputReplacement'>): Found the following nodes '[]' with name 'crop_proposals' but there should be exactly 1. Looks like ObjectDetectionAPIProposalReplacement replacement didn't work.

I Have the same issue using:
faster_rcnn_support_api_v1.7.json
faster_rcnn_support_api_v1.10.json
faster_rcnn_support_api_v1.13.json

Anyhow, I suppose the problem is related to the expected input shape.
So, it could be interesting to know that using --input_shape [600,600] at the end of the command (wathever number I use), the error change as below:

E1103 16:37:00.925748 139676718106368 main.py:323] -------------------------------------------------
E1103 16:37:00.925927 139676718106368 main.py:324] ----------------- INTERNAL ERROR ----------------
E1103 16:37:00.925991 139676718106368 main.py:325] Unexpected exception happened.
E1103 16:37:00.926042 139676718106368 main.py:326] Please contact Model Optimizer developers and forward the following information:
E1103 16:37:00.926090 139676718106368 main.py:327] Exception occurred during running replacer "ObjectDetectionAPIPreprocessorReplacement (<class 'extensions.front.tf.ObjectDetectionAPI.ObjectDetectionAPIPreprocessorReplacement'>)": index 2 is out of bounds for axis 0 with size 2
E1103 16:37:00.926984 139676718106368 main.py:328] Traceback (most recent call last):
  File "/opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 167, in apply_replacements
    replacer.find_and_replace_pattern(graph)
  File "/opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/mo/front/tf/replacement.py", line 89, in find_and_replace_pattern
    self.replace_sub_graph(graph, match)
  File "/opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/mo/front/common/replacement.py", line 131, in replace_sub_graph
    new_sub_graph = self.generate_sub_graph(graph, match)  # pylint: disable=assignment-from-no-return
  File "/opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/extensions/front/tf/ObjectDetectionAPI.py", line 430, in generate_sub_graph
    height, width = calculate_placeholder_spatial_shape(graph, match, pipeline_config)
  File "/opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/extensions/front/tf/ObjectDetectionAPI.py", line 332, in calculate_placeholder_spatial_shape
    user_defined_width = user_defined_shape[2]
IndexError: index 2 is out of bounds for axis 0 with size 2

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/mo/main.py", line 312, in main
    return driver(argv)
  File "/opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/mo/main.py", line 263, in driver
    is_binary=not argv.input_model_is_text)
  File "/opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/mo/pipeline/tf.py", line 127, in tf2nx
    class_registration.apply_replacements(graph, class_registration.ClassType.FRONT_REPLACER)
  File "/opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 190, in apply_replacements
    )) from err
Exception: Exception occurred during running replacer "ObjectDetectionAPIPreprocessorReplacement (<class 'extensions.front.tf.ObjectDetectionAPI.ObjectDetectionAPIPreprocessorReplacement'>)": index 2 is out of bounds for axis 0 with size 2

E1103 16:37:00.927108 139676718106368 main.py:329] ---------------- END OF BUG REPORT --------------
E1103 16:37:00.927165 139676718106368 main.py:330] ------------------------------------------------

Again, it is confirmed that the weak point of OpenVino are still the procesesses to generate the IR files !

Do you have any suggestions from model optimizer developers?

Should I have to downgrade OV to 2019.1.133 ?
If yes, where I can find the link to download exactly this version?

Thank you

[CBell1]

Hello,

any suggestions here?

I used also TF 1.15, same issue.

It seems there is a parameter not correct using model_optimizer/mo_tf.py to generate the IR files from a custom .pb file generated from Faster RCNN.

Thank you

[JesusE_Intel]

Hi Cosma,

The OpenVINO toolkit 2019 R3.1 includes a newer version of the configuration file for TF API 1.14. Could you try upgrading your OpenVINO installation and using the included faster_rcnn_support_api_v1.14.json config file?

Regards,

Jesus

[CBell1]

Hi Jesus,

I installed the l_openvino_toolkit_p_2019.3.376

Unfortunately, it doesn't solve the problem, below all commands with the same error:

With or without the --reverse_input_channels --data_type FP16

sudo python3 mo_tf.py --input_model "/home/cosma/Development/venvov/dataset/train/train_2/fruits_faster_rcnn_inception_inference_graph/frozen_inference_graph.pb" --tensorflow_use_custom_operations_config "/opt/intel/openvino_2019.3.376/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support_api_v1.14.json" --tensorflow_object_detection_api_pipeline_config "/home/cosma/Development/venvov/dataset/train/train_2/fruits_faster_rcnn_inception_inference_graph/pipeline.config" --reverse_input_channels --data_type FP16

[ WARNING ] Model Optimizer removes pre-processing block of the model which resizes image keeping aspect ratio. The Inference Engine does not support dynamic image size so the Intermediate Representation file is generated with the input image size of a fixed size.
Specify the "--input_shape" command line parameter to override the default shape which is equal to (600, 600).
The Preprocessor block has been removed. Only nodes performing mean value subtraction and scaling (if applicable) are kept.
[ ERROR ]  Failed to match nodes from custom replacement description with id 'ObjectDetectionAPIProposalReplacement':
It means model and custom replacement description are incompatible.
Try to correct custom replacement description according to documentation with respect to model node names
[ ERROR ]  Found the following nodes '[]' with name 'crop_proposals' but there should be exactly 1. Looks like ObjectDetectionAPIProposalReplacement replacement didn't work.
Exception occurred during running replacer "ObjectDetectionAPIDetectionOutputReplacement" (<class 'extensions.front.tf.ObjectDetectionAPI.ObjectDetectionAPIDetectionOutputReplacement'>): Found the following nodes '[]' with name 'crop_proposals' but there should be exactly 1. Looks like ObjectDetectionAPIProposalReplacement replacement didn't work.

FYI:

I trained the model using the TensorFlow API object_detection from https://github.com/TensorFlow/models

I generated the .pb file using the following tool:

python3 export_inference_graph.py --input_type image_tensor --pipeline_config_path /home/cosma/Development/venvov/pretrained_model/faster_rcnn_inception_v2_coco_CUST.config --trained_checkpoint_prefix /home/cosma/Development/venvov/dataset/train/train_2/model.ckpt-5424 --output_directory /home/cosma/Development/venvov/dataset/train/train_2/fruits_faster_rcnn_inception_inference_graph

Please, let me know if you need more information

Thank you

[Ramawi_Putra__Jhon]

Hi Cosma,

I also have a Faster RCNN Inception V2 model (custom dataset) and successfully converted to IR using this command:

python mo_tf.py --input_model inceptionv2model.pb --output_dir ir_model\FP16 --reverse_input_channels --tensorflow_object_detection_api_pipeline_config pipeline.config --tensorflow_use_custom_operations_config extensions\front\tf\faster_rcnn_support_api_v1.7.json --input_shape [1,600,1024,3]

I have openvino 2019.3.379 and tensorflow 1.15.0 installed. But the next issue is the model took about 5 minutes tobe loaded to NCS2. That's unacceptable to me.

exec_net = plugin.load(network=net) #it takes > 5 minutes to complete

Regards,

Jhon R Putra 

[CBell1]

Hi Jhon,

Thank you for your replay.

I didn't solve the problem.

Could it be related to Training part ?

If yes, I used the TensorFlow API Object Detection tool to Train my custom data with Faster RCNN:

https://github.com/tensorflow/models

research/object_detection

I customized the faster_rcnn_inception_v2_coco_CUST.config

and I trained the RCNN using the following command:

python3 train.py --logtostderr --train_dir=/home/cosma/Development/venvov/dataset/train/train_2 --pipeline_config_path=/home/cosma/Development/venvov/dataset/train/faster_rcnn_inception_v2_coco_CUST.config

The Training .ckpt results are OK.

Conversion from model.ckpt to .pb file: OK

Unfortunately, as mentioned, I have some troubles converting the .pb to IR (see error messages in my previous posts above) !

Do you have suggestions?

Thank you in advance

Regards

[CBell1]

Hi Jesus,

please, do you have suggestions?

thank you

[JesusE_Intel]

Hi Cosma,

Could you share the model for me to try on my end? I will send you a private message to share the model privately.

Regards,

Jesus

[whatthisismyname]

same problem i cant load the model i just hangs in there at least you have to wait 5min mine just sits there waiting or loading.. any solution is appreciated.