Generating IR files for custom trained mask rcnn model

[Kannan_K_Intel]

Hi,

I am trying to generate IR files for custom trained Mask RCNN model on tensorflow.

Model: Mask RCNN Inception V2

Tensorflow version: 1.9

Openvino: 2019 R2

Command used for IR generation:

python mo_tf.py --output_dir C:\Users\kkanna2x\Desktop\mask_rcnn\car\ir  --input_model C:\Users\kkanna2x\Desktop\mask_rcnn\main\frozen_inference_graph.pb  --tensorflow_use_custom_operations_config extensions\front\tf\mask_rcnn_support_api_v1.14.json  --tensorflow_object_detection_api_pipeline_config C:\Users\kkanna2x\Desktop\mask_rcnn\main\pipeline.config  --reverse_input_channels

and I am getting the following error:

E0927 15:11:51.982321  9468 replacement.py:89] Failed to match nodes from custom replacement description with id 'ObjectDetectionAPIProposalReplacement':
It means model and custom replacement description are incompatible.
Try to correct custom replacement description according to documentation with respect to model node names
E0927 15:11:56.294863  9468 replacement.py:89] Failed to match nodes from custom replacement description with id 'ObjectDetectionAPIDetectionOutputReplacement':
It means model and custom replacement description are incompatible.
Try to correct custom replacement description according to documentation with respect to model node names
E0927 15:11:58.087721  9468 main.py:307] Exception occurred during running replacer "ObjectDetectionAPIMaskRCNNROIPoolingSecondReplacement" (<class 'extensions.front.tf.ObjectDetectionAPI.ObjectDetectionAPIMaskRCNNROIPoolingSecondReplacement'>): End node "Reshape_10" is not reachable from start nodes: ['CropAndResize_1/CropAndResize'].

I tried with 2019 R1 and 2018 R5 versions of openvino and also tried with training the model using tensorflow 1.13. And I have attached the error log.

Hope someone can help.

[Shubha_R_Intel]

Dear K, Kannan,

Did you use the same Model Optimizer command after custom-training your model ? I assume before custom-training the MO worked perfectly fine and generated IR ?

If so, then I suspect it's a bug. Fortunately you are an Intel employee. Please email me directly and I will file a bug ticket on your behalf. Also I will need you to send me the custom trained *.pb file. Likely this *.pb is far too large to email so you will have to use Syncplicity to send that to me.

Sorry for the inconvenience,

Thanks

Shubha

[Kannan_K_Intel]

Hi Shubha,

Attaching the tensorflow graphs and error logs.

Thanks and Regards

Kannan K

[Shubha_R_Intel]

Dear K, Kannan,

As discussed this appears to be a bug and I've filed a bug on your behalf.

Thanks for your patience,

Shubha

[Tech__Xplorazzi]

Hi Kanan,

Did your problem get solved with the bug fixed?  The command you listed was "... --input_model C:\Users\kkanna2x\Desktop\mask_rcnn\main\frozen_inference_graph.pb ...." and mentioned that it was a custom-trained mask rcnn model. I was wondering the .pb file you provided was it from the openvinotoolkit page (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html) or generated by you? In our mask_rcnn custom model training we generate a .h5 model, how was that converted .pb? We are just getting started with OpenVino so questions may look basic. Please let me know.

thanks

Dash (Xplorazzi) 

[Kannan_K_Intel · Accepted solution]

Tech, Xplorazzi wrote:

Hi Kanan,

Did your problem get solved with the bug fixed?  The command you listed was "... --input_model C:\Users\kkanna2x\Desktop\mask_rcnn\main\frozen_inference_graph.pb ...." and mentioned that it was a custom-trained mask rcnn model. I was wondering the .pb file you provided was it from the openvinotoolkit page (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_conver...) or generated by you? In our mask_rcnn custom model training we generate a .h5 model, how was that converted .pb? We are just getting started with OpenVino so questions may look basic. Please let me know.

thanks

Dash (Xplorazzi) 

Hey,

I trained the model using tensorflow object detection api, which will give you  pb file after you freeze the model at a convenient checkpoint.

And yes my issue got resolved with the .json file provided by the openvino team.

I have tried with Matterport Mask RCNN, which is a keras based implementation. There we will be getting a .h5 model file. There are some scripts available to convert a keras h5 file to pb. But I didnt find anything useful in my case, because the inference was wrong after the model being converted to a .pb file. 

My suggestion would be, use tensorflow object detection api for your mask rcnn training (possibly transfer learning). 

repo: https://github.com/tensorflow/models/tree/master/research/object_detection

you can follow this medium article for a quick intro to mask rcnn in object detection api:

https://medium.com/@vijendra1125/custom-mask-rcnn-using-tensorflow-object-detection-api-101149ce0765

[DanielSerna]

Hi @Kannan_K_Intel,

I'm actually working with custom Matterport Mask RCNN, and have tried many things... even in another thread I received support from intel, but haven't succeded...

If I understood correctly, you could train with tensorflow object detection api, and that way you succeeded in converting to IR ? if so... if you succeeded why would you want to try Matterport Mask RCNN ?

It would be really helpfull to get a feedback from you, since you been working on the same issues I been having... also if it's not a lot of trouble, maybe you could check the thread I created you can look it up with this title: 'Converting Maskrcnn to IR error' 

I'm instered in the option you said about training with tensorflow object detection api, to see if I have to change my approach and retrain my model with the api, but only if you succeded all the way to doing inference with openvino...

Thanks a lot for your time !! and looking forward to hearing from you.

[Kannan_K_Intel]

Hey @DanielSerna 

To first to answer your question on why I used Matterport MaskRCNN, with a limited set of training data the Matterport gave more accuracy compared to tf object detection api.

But when it came to optimizing in OpenVINO, the Matterport isn't directly supported by OpenVINO.

And yes, I would strongly suggest to retrain your model on tf object detection api, if optimizing in OpenVINO is important for you. 

If you can provide the link to your  thread, I can take a look.

[DanielSerna]

Hi @Kannan_K_Intel ,

thanks a lot for your response, I have made some progress on trying to convert Matterport MaskRCNN to openvino, here's the link: https://community.intel.com/t5/Intel-Distribution-of-OpenVINO/Converting-Maskrcnn-to-IR-error/m-p/1200839#M20300 , please let me know what you think about it, I think that with a few fixes maybe I could get it converted to openvino but I'll have to see if I can...

About the tf object detection api I will give it a try, so just to be clear, you didn't have any problems when converting to openvino with the tf object detection api ?

Thanks