Convert torchvision mask rcnn model

[Matzeox]

Hello,

after the new version of openvino (2021) was release with the support of onnx opset 11, i wanted to try to convert the torch vision mask rcnn model to IR. I used the mask rcnn model from torchvision tutorial (https://github.com/pytorch/vision/tree/master/references/detection).

I have successfully converted the torchvision model to onnx with opset11. Unfortunately, the conversion from onnx to IR does not work.

I run the following command for converting:

python3 mo.py --input_model ~/maskrcnn.onnx --output_dir ~/ --input_shape [1,627,940,3] --log_level DEBUG
Model Optimizer arguments:
Common parameters:
- Path to the Input Model: /home/xxx/maskrcnn.onnx
- Path for generated IR: /home/xxx/
- IR output name: maskrcnn
- Log level: DEBUG
- Batch: Not specified, inherited from the model
- Input layers: Not specified, inherited from the model
- Output layers: Not specified, inherited from the model
- Input shapes: [1,627,940,3]
- Mean values: Not specified
- Scale values: Not specified
- Scale factor: Not specified
- Precision of IR: FP32
- Enable fusing: True
- Enable grouped convolutions fusing: True
- Move mean values to preprocess section: None
- Reverse input channels: False
ONNX specific parameters:
Model Optimizer version: 2021.1.0-1237-bece22ac675-releases/2021/1

Error message:

[ 2020-10-16 09:19:17,569 ] [ DEBUG ] [ infer:130 ] Partial infer for Sub_11
[ 2020-10-16 09:19:17,569 ] [ DEBUG ] [ infer:131 ] Op: Sub
[ 2020-10-16 09:19:17,569 ] [ DEBUG ] [ infer:132 ] Inputs:
[ 2020-10-16 09:19:17,569 ] [ DEBUG ] [ infer:32 ] input[0]: shape = [627 940 3], value = <UNKNOWN>
[ 2020-10-16 09:19:17,569 ] [ DEBUG ] [ infer:32 ] input[1]: shape = [3 1 1], value = [[[0.485]]

[[0.456]]

[[0.406]]]
[ 2020-10-16 09:19:17,570 ] [ DEBUG ] [ infer:145 ] Outputs:
[ 2020-10-16 09:19:17,570 ] [ DEBUG ] [ infer:32 ] output[0]: shape = [ -1 940 3], value = <UNKNOWN>
[ ERROR ] Shape [ -1 940 3] is not fully defined for output 0 of "Sub_11". Use --input_shape with positive integers to override model input shapes.
[ ERROR ] Cannot infer shapes or values for node "Sub_11".
[ ERROR ] Not all output shapes were inferred or fully defined for node "Sub_11".
For more information please refer to Model Optimizer FAQ, question #40. (https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=40#question-40)
[ ERROR ]
[ ERROR ] It can happen due to bug in custom shape infer function <function Elementwise.__init__.<locals>.<lambda> at 0x7fbf85488f28>.
[ ERROR ] Or because the node inputs have incorrect values/shapes.
[ ERROR ] Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ 2020-10-16 09:19:17,572 ] [ DEBUG ] [ infer:197 ] Node "Sub_11" attributes: {'pb': input: "317"
input: "318"
output: "319"
name: "Sub_11"
op_type: "Sub"
, 'kind': 'op', '_in_ports': {0: {'control_flow': False}, 1: {'control_flow': False}}, '_out_ports': {0: {'control_flow': False}}, 'name': 'Sub_11', 'op': 'Sub', 'type': 'Subtract', 'version': 'opset1', 'infer': <function Elementwise.__init__.<locals>.<lambda> at 0x7fbf85488f28>, 'type_infer': <function Elementwise.type_infer at 0x7fbf903b8488>, 'can_be_bias': True, 'can_be_fused': True, 'in_ports_count': 2, 'out_ports_count': 1, 'is_eltwise': True, 'stop_value_propagation': False, 'axis': None, 'dim_attrs': ['spatial_dims', 'batch_dims', 'axis', 'channel_dims'], 'shape_attrs': ['shape', 'output_shape', 'pad', 'window', 'stride'], 'IE': [('layer', [('id', <function Op.substitute_ie_attrs.<locals>.<lambda> at 0x7fbf8f903158>), 'name', 'type', 'version'], [('data', [], []), '@ports', '@consts'])], 'is_output_reachable': True, 'is_undead': False, 'is_const_producer': False, 'is_partial_inferred': False}
[ ERROR ] Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "Sub_11" node.
For more information please refer to Model Optimizer FAQ, question #38. (https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=38#question-38)
[ 2020-10-16 09:19:17,573 ] [ DEBUG ] [ main:311 ] Traceback (most recent call last):
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/mo/middle/passes/infer.py", line 169, in partial_infer
node_name)
mo.utils.error.Error: Not all output shapes were inferred or fully defined for node "Sub_11".
For more information please refer to Model Optimizer FAQ, question #40. (https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=40#question-40)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 288, in apply_transform
for_graph_and_each_sub_graph_recursively(graph, replacer.find_and_replace_pattern)
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/mo/middle/pattern_match.py", line 58, in for_graph_and_each_sub_graph_recursively
func(graph)
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/extensions/middle/PartialInfer.py", line 32, in find_and_replace_pattern
partial_infer(graph)
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/mo/middle/passes/infer.py", line 199, in partial_infer
refer_to_faq_msg(38)) from err
mo.utils.error.Error: Stopped shape/value propagation at "Sub_11" node.
For more information please refer to Model Optimizer FAQ, question #38. (https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=38#question-38)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/mo/main.py", line 298, in main
ret_code = driver(argv)
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/mo/main.py", line 265, in driver
ret_res = emit_ir(prepare_ir(argv), argv)
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/mo/main.py", line 234, in prepare_ir
graph = unified_pipeline(argv)
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/mo/pipeline/unified.py", line 29, in unified_pipeline
class_registration.ClassType.BACK_REPLACER
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 334, in apply_replacements
apply_replacements_list(graph, replacers_order)
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 324, in apply_replacements_list
num_transforms=len(replacers_order))
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/mo/utils/logger.py", line 124, in wrapper
function(*args, **kwargs)
File "/opt/intel/openvino_2021.1.110/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 304, in apply_transform
)) from err
mo.utils.error.Error: Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "Sub_11" node.
For more information please refer to Model Optimizer FAQ, question #38. (https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=38#question-38)

Could someone help me?

Thanks!

[Rizal_Intel]

Hi Matzeox,

Your error is probably caused by an incorrect input shape.

I tried replicating your issue by converting the pretrained model example ( I used the maskrcnn_resnet50_fpn export example, I did not retrain it but just convert it to test compatibility).

I used a dummy input of torch.rand(3,400,400).

I used OpenVINO 2021.1 and when I executed model optimizer using input shape [1,400,400,3] I encountered similar error to yours.

When I used input shape [1,3,400,400] (following my export dummy input), I did not encounter the same error.

However, I encountered another error as follows:

[ ERROR ] There is no registered "infer" function for node "Min_28" with op = "Min". Please implement this function in the extensions.

This would mean that there are still unsupported layers in this model.

You can check the list of ONNX supported layers.

The Mask RCNN is not among ONNX Supported Topologies but there is a specific project that seems compatible.

In summary, it seems that currently it is not possible to implement torchvision Mask RCNN in OpenVINO.

Regards,

Rizal

[Rizal_Intel]

Hi Matzeox,

Intel will no longer monitor this thread since we have provided a solution.

If you need any additional information from Intel, please submit a new question. 

Regards,

Rizal