Hetero Plugin doesn't increase performance

[Di_Fiore__Cristian]

Hi I'm using OpenVINO 2019_R3, I created my application in Python using the Inference Engine and did some performance tests. In my application I process an image and therefore I decided to test with 600 images to have the average value. When I use the FP32 model on a CPU I have obtained almost the same results as the FP16 on CPU (10 FPS), the use of the FP32 on a GPU increases the performance (20 FPS) which increase with the FP16 model (34 FPS) . I checked the documentation and saw the Hetero plugin to use all the devices I have at my best. Why does my performance decrease if I use HETERO: GPU, CPU (22 FPS) or HETERO: CPU, GPU (11 FPS) with FP16 model?

If I use FP32, the results are: 10 FPS (HETERO:CPU,GPU) and  18 FPS (HETERO:GPU,CPU).

Is it possible that in my application and with my devices the best result is obtained only with the GPU?

Best Regards,

Cristian.

[SuryaPSC_Intel]

Hi Cristian,

Can you please confirm if the result (HETRO: GPU, CPU (22 FPS) or HETERO: CPU, GPU (11 FPS))  is with FP32 or FP16 model?

Best Regards,

Surya

[Di_Fiore__Cristian]

Hi Surya,

The results are with FP16 model. If I use FP32, the results are: 10 FPS (HETERO:CPU,GPU) and  18 FPS (HETERO:GPU,CPU).

Best Regards,
Cristian

[Di_Fiore__Cristian]

Any advice?

Thanks. 

[SuryaPSC_Intel · Accepted solution]

Hi Cristian,

HETRO mode uses the fallback policy to perform inference on multiple devices. However it does not ensure enhanced performance as transmitting of data from one part of network to another part in heterogeneous mode may take relatively much time.

So, if the network is supported in a single device it is recommended to use that device.

Best Regards,

Surya