Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

Hetero Plugin doesn't increase performance

Di_Fiore__Cristian
523 Views

Hi I'm using OpenVINO 2019_R3, I created my application in Python using the Inference Engine and did some performance tests. In my application I process an image and therefore I decided to test with 600 images to have the average value. When I use the FP32 model on a CPU I have obtained almost the same results as the FP16 on CPU (10 FPS), the use of the FP32 on a GPU increases the performance (20 FPS) which increase with the FP16 model (34 FPS) . I checked the documentation and saw the Hetero plugin to use all the devices I have at my best. Why does my performance decrease if I use HETERO: GPU, CPU (22 FPS) or HETERO: CPU, GPU (11 FPS) with FP16 model?

If I use FP32, the results are: 10 FPS (HETERO:CPU,GPU) and  18 FPS (HETERO:GPU,CPU).

Is it possible that in my application and with my devices the best result is obtained only with the GPU?

Best Regards,

Cristian.

0 Kudos
1 Solution
SuryaPSC_Intel
Employee
523 Views

Hi Cristian,

HETRO mode uses the fallback policy to perform inference on multiple devices. However it does not ensure enhanced performance as transmitting of data from one part of network to another part in heterogeneous mode may take relatively much time.

So, if the network is supported in a single device it is recommended to use that device.

Best Regards,

Surya

View solution in original post

0 Kudos
4 Replies
SuryaPSC_Intel
Employee
523 Views

Hi Cristian,

Can you please confirm if the result (HETRO: GPU, CPU (22 FPS) or HETERO: CPU, GPU (11 FPS))  is with FP32 or FP16 model?

Best Regards,

Surya

0 Kudos
Di_Fiore__Cristian
523 Views

Hi Surya,

The results are with FP16 model. If I use FP32, the results are: 10 FPS (HETERO:CPU,GPU) and  18 FPS (HETERO:GPU,CPU).

Best Regards,
Cristian

0 Kudos
Di_Fiore__Cristian
523 Views

Any advice?

Thanks. 

 

0 Kudos
SuryaPSC_Intel
Employee
524 Views

Hi Cristian,

HETRO mode uses the fallback policy to perform inference on multiple devices. However it does not ensure enhanced performance as transmitting of data from one part of network to another part in heterogeneous mode may take relatively much time.

So, if the network is supported in a single device it is recommended to use that device.

Best Regards,

Surya

0 Kudos
Reply