Model optimizer / ONNX resize node issue

[Vt__Emiliana]

Hi,

I have a heatmap regression model I trained in PyTorch and converted to ONNX format for inference. Now I want to try using OpenVINO to speed up inference, but I have trouble running it through the model optimizer. From what I read, support for the Resize node has been added with the 2020 release. I can see that there is a resize.py in the extensions\ops folder too, but I get the following error:

C:\Program Files (x86)\IntelSWTools\openvino_2020.1.033\deployment_tools\model_optimizer>python mo.py --input_model openvino.onnx --input_shape (1,3,224,224)
Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      C:\Program Files (x86)\IntelSWTools\openvino_2020.1.033\deployment_tools\model_optimizer\openvino.onnx
        - Path for generated IR:        C:\Program Files (x86)\IntelSWTools\openvino_2020.1.033\deployment_tools\model_optimizer\.
        - IR output name:       openvino
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         Not specified, inherited from the model
        - Output layers:        Not specified, inherited from the model
        - Input shapes:         (1,3,224,224)
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
ONNX specific parameters:
Model Optimizer version:        2020.1.0-61-gd349c3ba4a
[ ERROR ]  Cannot infer shapes or values for node "721".
[ ERROR ]  There is no registered "infer" function for node "721" with op = "Resize". Please implement this function in the extensions. For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #37.
[ ERROR ]
[ ERROR ]  It can happen due to bug in custom shape infer function <UNKNOWN>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "721" node. For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #38.

I have tried with both models using a scale_factor and a size argument for the Resize node. Is there a way to solve this issue?

One of the models I used for testing can be found here.

[SIRIGIRI_V_Intel]

Hi Emiliana,

What is the opset version that you are using?

The latest OpenVINO version (2020.1) supports resize operation for opset-10 version.

For a work around, can you try converting the pytorch model to tensorflow(.pb) and optimize with OpenVINO.

Regards,

Ram prasad

[Vt__Emiliana]

Hi and thank you for your response!

The issue is probably the opset version then. If I try to export the model to opset-10, PyTorch gives an error that align_corners=True is only supported with opset-11 there.

I will look into converting the model to tensorflow like you suggested, but at least using onnx-tf also seems like it will not work due to being limited to opset-10.

[SIRIGIRI_V_Intel]

Please try to use align_corners=True or similar operation using opset-10

Regards,

Ram prasad

[Vt__Emiliana]

After changing my Upsample layer from bilinear to nearest, I was able to export with an opset version below 11 and got the model to work with the model optimizer.

Thank you!

[JAIVIN_J_Intel]

Hi Emiliana,

Thanks for sharing the information with OpenVINO Community!

Regards,

Jaivin

[LIU__Billy]

I've met the same problems, and I replace bilinear with nearest. However, it doesn't work. When I was trying to convert pytorch to onnx with opset 9, I can still get the warning message:

ONNX's Upsample/Resize operator did not match Pytorch's Interpolation until opset 11. Attributes to determine how to transform the input were added in onnx:Resize in opset 11 to support Pytorch's behavior (like coordinate_transformation_mode and nearest_mode).

When I try to ignore it and convert onnx model with model optimizer:

[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.UpsampleToResample.UpsampleToResample'>): After partial shape inference were found shape collision for node Interpolate/Upsample_229 (old shape: [  1.           2.         256.         305.99999571], new shape: [  1   2 256 305])

When I try to use opset10 or 11, the model optimizer:

ONNX specific parameters:
Model Optimizer version:     2020.1.0-61-gd349c3ba4a
[ ERROR ]  Cannot infer shapes or values for node "Resize_213".
[ ERROR ]  There is no registered "infer" function for node "Resize_213" with op = "Resize". Please implement this function in the extensions. 
 For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #37. 
[ ERROR ]  
[ ERROR ]  It can happen due to bug in custom shape infer function <UNKNOWN>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "Resize_213" node. 
 For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #38. 
 

[Jiang__Lele]

LIU, Billy wrote:

I've met the same problems, and I replace bilinear with nearest. However, it doesn't work. When I was trying to convert pytorch to onnx with opset 9, I can still get the warning message:

ONNX's Upsample/Resize operator did not match Pytorch's Interpolation until opset 11. Attributes to determine how to transform the input were added in onnx:Resize in opset 11 to support Pytorch's behavior (like coordinate_transformation_mode and nearest_mode).

When I try to ignore it and convert onnx model with model optimizer:

[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.UpsampleToResample.UpsampleToResample'>): After partial shape inference were found shape collision for node Interpolate/Upsample_229 (old shape: [  1.           2.         256.         305.99999571], new shape: [  1   2 256 305])

When I try to use opset10 or 11, the model optimizer:

ONNX specific parameters:
Model Optimizer version:     2020.1.0-61-gd349c3ba4a
[ ERROR ]  Cannot infer shapes or values for node "Resize_213".
[ ERROR ]  There is no registered "infer" function for node "Resize_213" with op = "Resize". Please implement this function in the extensions. 
 For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_...), question #37. 
[ ERROR ]  
[ ERROR ]  It can happen due to bug in custom shape infer function <UNKNOWN>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "Resize_213" node. 
 For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_...), question #38. 
 

Hi LIU, I also met the same problem. Here might be a overridden solution. 

From your error log I can see that, you use ONNX's Resize operator to upsample the input. However, ONNX's op will use build-in *Scales* in FP32 format to calculate the output shape. This behaviour will make the output shape form int(1, 2, 256, 305) to float(1, 2, 256, 305.99999571). So far the error occured. 

So, if you override the 'old shape' with the 'new shape', the error would no longer exist. It works for me. Hope this helps.

[章__章之星]

Jiang, Lele wrote:

@staticmethod def resize_infer(node: Node): layout = node.graph.graph['layout'] assert len(layout) == 4 input_shape = node.in_node(0).shape if input_shape is None:

return scale_value = node.in_node(1).value

node.out_node().shape = int64_array(input_shape * scale_value) 

this is the src of resize in \openvino_2020.1.033\deployment_tools\model_optimizer\extensions\ops。When I was trying to convert pytorch to onnx with opset 11, I can still get the warning message. and i do not understand what you say that you override the 'old shape' with the 'new shape'. when i look for src of resize, int64_array function can avoid  This behaviour will make the output shape form int(1, 2, 256, 305) to float(1, 2, 256, 305.99999571).  

Quote:

LIU, Billy wrote:

I've met the same problems, and I replace bilinear with nearest. However, it doesn't work. When I was trying to convert pytorch to onnx with opset 9, I can still get the warning message:

ONNX's Upsample/Resize operator did not match Pytorch's Interpolation until opset 11. Attributes to determine how to transform the input were added in onnx:Resize in opset 11 to support Pytorch's behavior (like coordinate_transformation_mode and nearest_mode).

When I try to ignore it and convert onnx model with model optimizer:

[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.UpsampleToResample.UpsampleToResample'>): After partial shape inference were found shape collision for node Interpolate/Upsample_229 (old shape: [  1.           2.         256.         305.99999571], new shape: [  1   2 256 305])

When I try to use opset10 or 11, the model optimizer:

ONNX specific parameters:
Model Optimizer version:     2020.1.0-61-gd349c3ba4a
[ ERROR ]  Cannot infer shapes or values for node "Resize_213".
[ ERROR ]  There is no registered "infer" function for node "Resize_213" with op = "Resize". Please implement this function in the extensions. 
 For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_...), question #37. 
[ ERROR ]  
[ ERROR ]  It can happen due to bug in custom shape infer function <UNKNOWN>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "Resize_213" node. 
 For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_...), question #38. 
 

Hi LIU, I also met the same problem. Here might be a overridden solution. 

From your error log I can see that, you use ONNX's Resize operator to upsample the input. However, ONNX's op will use build-in *Scales* in FP32 format to calculate the output shape. This behaviour will make the output shape form int(1, 2, 256, 305) to float(1, 2, 256, 305.99999571). So far the error occured. 

So, if you override the 'old shape' with the 'new shape', the error would no longer exist. It works for me. Hope this helps.