object detection zoo resnet v50 fpn coco failing after transfer learning

[Connolly__John]

I downloaded the ssd_resnet_v50 fpn network from tensorflow,, and then ran the mo_tf.py script that comes packaged with openvino 

with the following input args

python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py \
--saved_model_dir ~//Downloads/resnet/ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03/saved_model \
--tensorflow_object_detection_api_pipeline_config ~//Downloads/resnet/ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03/pipeline.config \
--tensorflow_use_custom_operations_config /opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/ssd_v2_support.json \
--reverse_input_channels

and everything worked like a charm.

I then used the object detection api to do transfer learning on this stock model. I converted my stock model wtih the script at models/research/object_detection/export_inference_graph.py

Then I tried again to convert to openvino using the following command: 

python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py \

--saved_model_dir ~/old_models/v21/saved_model/ \

--tensorflow_object_detection_api_pipeline_config ~/old_models/v21/pipeline.config \

--tensorflow_use_custom_operations_config ~/old_models/v21/ssd_v2_support.json \

--reverse_input_channels

and now I get the following error output:

[ ERROR ]  Failed to match nodes from custom replacement description with id 'ObjectDetectionAPIPreprocessorReplacement':
It means model and custom replacement description are incompatible.
Try to correct custom replacement description according to documentation with respect to model node names
[ ERROR ]  Failed to match nodes from custom replacement description with id 'ObjectDetectionAPISSDPostprocessorReplacement':
It means model and custom replacement description are incompatible.
Try to correct custom replacement description according to documentation with respect to model node names
[ ERROR ]  Cannot infer shapes or values for node "Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Select_4".
[ ERROR ]  
[ ERROR ]  
[ ERROR ]  It can happen due to bug in custom shape infer function <function Select.infer at 0x7fa50799fd08>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Select_4" node. 
 For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #38. 

I have tried with the ssd_v2_support.json and ssd_support_api_v1.14.json, neither succeed. 

I am using pytthon3.6 with tensorflow 1.14 

[Shubha_R_Intel]

Dear Connolly, John,

Can you try adding:

"Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Select_4"

right after   "Postprocessor/ToFloat" in your ssd_v2_support.json ?

 "start_points": [
                "Postprocessor/Shape",
                "Postprocessor/scale_logits",
                "Postprocessor/Tile",
                "Postprocessor/Reshape_1",
                "Postprocessor/ToFloat",             
"Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Select_4"
            ]

You didn't attach the full debug log so I'm not sure exactly where it fits in. You'd be able to tell by perusing the log with the following switch --log_level DEBUG.

Let me know if this works for you.

Shubha

[Connolly__John]

Thank you for the prompt feedback. Unfortunately that did not work for me - I have attached the debug output from this attempt. 

_{You didn't attach the full debug log so I'm not sure exactly where it fits in. You'd be able to tell by perusing the log with the following switch --log_level DEBUG.}

I'm having trouble understanding what you mean here, could you explain what should be fitting where?

[Shubha_R_Intel]

Dear Connolly, John,

In fact, the file you attached is the DEBUG log print obtained by passing in --log_level DEBUG. Thank you for that. I really think this may be a Model Optimizer bug, based on what you have told me. Before training, no problems, MO worked like a charm. After training, it broke. Yep, I've seen this before. And the reason it happens is because the Tensorflow model changes underneath us. I have PM'd you so that you can privately send me your trained model. 

We'll definitely get back to you with an answer and hopefully a solution also.

Sorry for the inconvenience and thanks for using OpenVino !

Shubha

[Shubha_R_Intel]

Dear Connolly, John,

I have reproduced your issue and filed a bug. I will definitely update you once a resolution has been found. 

Thanks,

Shubha

[Shubha_R_Intel]

Dear John and everyone, it looks like John was able to succeed with the attached revised *.json file. If anyone else runs into the same issue as John, feel free to use this json file for the -tensorflow_use_custom_operations_config argument.

Thanks !

Shubha