Intel® Distribution of OpenVINO™ Toolkit
Community support and discussions about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all things computer vision-related on Intel® platforms.

Slow fully connected layer performance

sbsky
Beginner
171 Views

I'm attempting to run a linear/dense/fully connected based network to openvino. The conversion works, however running the model with openvino is no faster than pytorch. This seems strange to me, as for CNNs I've found that openvino is 10-15x faster than pytorch.

I did some profiling, and it seems that openvino isn't using AVX. This is an except from the profiling log:

 

Add_2262 EXECUTED layerType: FullyConnected realTime: 8992 cpu: 8992 execType: jit_gemm_FP32
Mul_2283 EXECUTED layerType: Eltwise realTime: 128 cpu: 128 execType: ref_FP32
Add_2284 EXECUTED layerType: Eltwise realTime: 158 cpu: 158 execType: ref_FP32
Total time: 789103 microseconds
Total CPU time: 789103 microseconds

How do I enable the AVX operators for linear layers?

 

 

 

0 Kudos
1 Solution
Adli
Moderator
151 Views

Hi sbsky,

 

Thank you for reaching out to us. There are some functions to check what set of AVX instructions are supported by the CPU. Please refer to the following link:

https://docs.openvinotoolkit.org/latest/ie_plugin_api/group__ie__dev__api__system__conf.html

 

If CPU supports AVX instructions, they will be used (if it's possible for a certain layer) with no need to enable them.

 

Besides, please also refer to the CPU optimization topic at the following link: https://docs.openvinotoolkit.org/latest/openvino_docs_optimization_guide_dldt_optimization_guide.htm...

 

As additional information, you could also check the approximate performance values for certain topologies including ones converted from Pytorch to IR. Please refer to the following link: https://docs.openvinotoolkit.org/downloads/benchmark_files/OV-2021.2-Download-Excel.xlsx

 

Regards,

Adli


View solution in original post

3 Replies
Adli
Moderator
152 Views

Hi sbsky,

 

Thank you for reaching out to us. There are some functions to check what set of AVX instructions are supported by the CPU. Please refer to the following link:

https://docs.openvinotoolkit.org/latest/ie_plugin_api/group__ie__dev__api__system__conf.html

 

If CPU supports AVX instructions, they will be used (if it's possible for a certain layer) with no need to enable them.

 

Besides, please also refer to the CPU optimization topic at the following link: https://docs.openvinotoolkit.org/latest/openvino_docs_optimization_guide_dldt_optimization_guide.htm...

 

As additional information, you could also check the approximate performance values for certain topologies including ones converted from Pytorch to IR. Please refer to the following link: https://docs.openvinotoolkit.org/downloads/benchmark_files/OV-2021.2-Download-Excel.xlsx

 

Regards,

Adli


View solution in original post

sbsky
Beginner
147 Views

Hi Adli, the performance values were useful thank you. Would be great if this is displayed more prominently in the documentation

Adli
Moderator
138 Views

Hi sbsky,


This thread will no longer be monitored since this issue has been resolved. If you need any additional information from Intel, please submit a new question.


Regards,

Adli


Reply