Error when doing inference with "HETERO:GPU,CPU"

[MAbom]

Hello,

openvino: 2021.4

I trained a mask rcnn model using tf OD API 2. I was able to use the openvino model optimizer to get the IR model.

I checked the model then using the benchmark_tool in python with both devices "CPU" and "GPU" and both worked fine.

But when trying the "Hetero" plugin I get the following exception:

[Step 1/11] Parsing and validating input arguments
[ WARNING ]  -nstreams default value is determined automatically for a device. Although the automatic selection usually provides a reasonable performance, but it still may be non-optimal for some cases, for more information look at README.
[Step 2/11] Loading Inference Engine
[ INFO ] InferenceEngine:
         API version............. 2021.4.0-3839-cd81789d294-releases/2021/4
[ INFO ] Device info
         CPU
         MKLDNNPlugin............ version 2.1
         Build................... 2021.4.0-3839-cd81789d294-releases/2021/4
         GPU
         clDNNPlugin............. version 2.1
         Build................... 2021.4.0-3839-cd81789d294-releases/2021/4
         HETERO
         heteroPlugin............ version 2.1
         Build................... 2021.4.0-3839-cd81789d294-releases/2021/4

[Step 3/11] Setting device configuration
[Step 4/11] Reading network files
[ INFO ] Read network took 75.03 ms
[Step 5/11] Resizing network to match image sizes and given batch
[ INFO ] Network batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Network input 'image_info' precision FP32, dimensions (NC): 1 3
[ INFO ] Network input 'input_tensor' precision U8, dimensions (NCHW): 1 3 572 768
[ INFO ] Network output 'masks' precision FP32, dimensions (NCHW): 2 1 33 33
[ INFO ] Network output 'reshape_do_2d' precision FP32, dimensions (NC): 2 7
[Step 7/11] Loading the model to the device
[ ERROR ] Function contains several inputs and outputs with one friendly name!
Traceback (most recent call last):
  File "C:\Program Files (x86)\Intel\openvino_2021.4.582\python\python3.6\openvino\tools\benchmark\main.py", line 255, in run
    exe_network = benchmark.load_network(ie_network)
  File "C:\Program Files (x86)\Intel\openvino_2021.4.582\python\python3.6\openvino\tools\benchmark\benchmark.py", line 63, in load_network
    num_requests=1 if self.api_type == 'sync' else self.nireq or 0)
  File "ie_api.pyx", line 367, in openvino.inference_engine.ie_api.IECore.load_network
  File "ie_api.pyx", line 379, in openvino.inference_engine.ie_api.IECore.load_network
RuntimeError: Function contains several inputs and outputs with one friendly name!

I ran the following command:

(tensorflow2) C:\Program Files (x86)\Intel\openvino_2021.4.582\deployment_tools\tools\benchmark_tool>python benchmark_app.py -m "C:\Users\Mouhammad\Documents\Intel\OpenVINO\openvino_models\ir\public\saved_model_smaller_FP16\saved_model.xml" -d HETERO:CPU,GPU -b 1 -api sync

Any idea what I am doing wrong here?

Thanks!

Regards,

Mouhammad

[Zulkifli_Intel]

Hello Mouhammad Abomoharam,

Greetings to you.

We're able to run the -d HETERO:CPU,GPU on benchmark_app using the Intel Open Model Zoo (OMZ) model. Please give it a try on the OMZ model and see whether the same error occurs or not. Share with us your model, so we can take a look at it.

Sincerely,

Zulkifli

[MAbom]

Hello Zulkifli,

thanks for your reply.

I have downloaded and converted one of the mask rcnn models from OMZ:

https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/mask_rcnn_resnet50_atrous_coco

I got the same error as before when using the HETERO plugin:

[Step 1/11] Parsing and validating input arguments
[ WARNING ]  -nstreams default value is determined automatically for a device. Although the automatic selection usually provides a reasonable performance, but it still may be non-optimal for some cases, for more information look at README.
[Step 2/11] Loading Inference Engine
[ INFO ] InferenceEngine:
         API version............. 2021.4.0-3839-cd81789d294-releases/2021/4
[ INFO ] Device info
         CPU
         MKLDNNPlugin............ version 2.1
         Build................... 2021.4.0-3839-cd81789d294-releases/2021/4
         GPU
         clDNNPlugin............. version 2.1
         Build................... 2021.4.0-3839-cd81789d294-releases/2021/4
         HETERO
         heteroPlugin............ version 2.1
         Build................... 2021.4.0-3839-cd81789d294-releases/2021/4

[Step 3/11] Setting device configuration
[Step 4/11] Reading network files
[ INFO ] Read network took 74.03 ms
[Step 5/11] Resizing network to match image sizes and given batch
[ INFO ] Network batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Network input 'image_info' precision FP32, dimensions (NC): 1 3
[ INFO ] Network input 'image_tensor' precision U8, dimensions (NCHW): 1 3 800 1365
[ INFO ] Network output 'masks' precision FP32, dimensions (NCHW): 100 90 33 33
[ INFO ] Network output 'reshape_do_2d' precision FP32, dimensions (NC): 100 7
[Step 7/11] Loading the model to the device
[ ERROR ] Function contains several inputs and outputs with one friendly name!
Traceback (most recent call last):
  File "C:\Program Files (x86)\Intel\openvino_2021.4.582\python\python3.6\openvino\tools\benchmark\main.py", line 255, in run
    exe_network = benchmark.load_network(ie_network)
  File "C:\Program Files (x86)\Intel\openvino_2021.4.582\python\python3.6\openvino\tools\benchmark\benchmark.py", line 63, in load_network
    num_requests=1 if self.api_type == 'sync' else self.nireq or 0)
  File "ie_api.pyx", line 367, in openvino.inference_engine.ie_api.IECore.load_network
  File "ie_api.pyx", line 379, in openvino.inference_engine.ie_api.IECore.load_network
RuntimeError: Function contains several inputs and outputs with one friendly name!

But the CPU and the GPU plugins work fine!

I attached also my mask rcnn model to this answer.

Regards,

Mouhammad

[Zulkifli_Intel]

Hello Mouhammad Abomoharam,

Thank you for sharing with us the model. On our side, we also see similar behavior when using the -d HETERO:GPU,CPU argument on mask_rcnn_resnet50_atrous_coco model.

We noticed you also open a new thread on GitHub #6645 for a similar issue. We will continue the investigation on GitHub. Therefore, we will close this thread and focus on GitHub. Are you okay with this arrangement?

Sincerely,

Zulkifli

[MAbom]

Hello Zulkifli,

Thanks for your reply.

Yes I think you can close this thread and focus on GitHub.

Thanks and regards,

Mouhammad

[Zulkifli_Intel]

Hello Mouhammad Abomoharam,

Thank you for your feedback. This thread will no longer be monitored since this issue will continue to be discussed on Github: https://github.com/openvinotoolkit/openvino/issues/6645

Sincerely,

Zulkifli