topic why smaller openvino model get longer inference time in Intel® Distribution of OpenVINO™ Toolkit
https://community.intel.com/t5/Intel-Distribution-of-OpenVINO/why-smaller-openvino-model-get-longer-inference-time/m-p/1213406#M20860
<P>I have two model, the larger one has 4.09M params and 2.99GFlops, the smaller one has 2.17M params and 1.82GFlops. Those two models have different structure, the larger one use Mobilenetv1 as backbone and several convs as head, the smaller one use Mobilenetv2 as backbone, several convs and convTransposes as neck, and several convs as head. When i use C++ interface to inference those two models, the larger one's avg inference time is 70ms, but the smaller one's avg inference time is 80ms, anyone can help me to figure out why? </P>Tue, 29 Sep 2020 10:12:58 GMTmarshall19932020-09-29T10:12:58Zwhy smaller openvino model get longer inference time
https://community.intel.com/t5/Intel-Distribution-of-OpenVINO/why-smaller-openvino-model-get-longer-inference-time/m-p/1213406#M20860
<P>I have two model, the larger one has 4.09M params and 2.99GFlops, the smaller one has 2.17M params and 1.82GFlops. Those two models have different structure, the larger one use Mobilenetv1 as backbone and several convs as head, the smaller one use Mobilenetv2 as backbone, several convs and convTransposes as neck, and several convs as head. When i use C++ interface to inference those two models, the larger one's avg inference time is 70ms, but the smaller one's avg inference time is 80ms, anyone can help me to figure out why? </P>Tue, 29 Sep 2020 10:12:58 GMThttps://community.intel.com/t5/Intel-Distribution-of-OpenVINO/why-smaller-openvino-model-get-longer-inference-time/m-p/1213406#M20860marshall19932020-09-29T10:12:58ZRe: why smaller openvino model get longer inference time
https://community.intel.com/t5/Intel-Distribution-of-OpenVINO/why-smaller-openvino-model-get-longer-inference-time/m-p/1213822#M20886
<P><SPAN style="font-size: 14px;">Greetings,</SPAN></P>
<P> </P>
<P><SPAN style="font-size: 14px;">Generally, as you mentioned, both of these models are using different backbone with a couple more different architecture.</SPAN></P>
<P> </P>
<P><SPAN style="font-size: 14px;">Fyi, In mobilenetV1 the pointwise convolution either kept the number of channels the same or doubled them. In mobilenetV2 it does the opposite: it makes the number of channels smaller. This is why this layer is now known as the </SPAN><STRONG style="font-size: 14px;">projection layer</STRONG><SPAN style="font-size: 14px;"> — it projects data with a high number of dimensions (channels) into a tensor with a much lower number of dimensions.</SPAN></P>
<P> </P>
<P><SPAN style="font-size: 14px;"> MobileNet V2 uses depthwise separable convolutions which are not directly supported in GPU firmware (the cuDNN library). Therefore, MobileNet V2 tends to be slower.</SPAN></P>
<P> </P>
<P><SPAN style="font-size: 14px;">This is one of the factor that made smaller sized model to be slower. It's not solely relies on size.</SPAN></P>
<P> </P>
<P><SPAN style="font-size: 14px;">you may also refer here: <A href="https://github.com/tensorflow/tensorflow/issues/21196" target="_blank" rel="noopener">https://github.com/tensorflow/tensorflow/issues/21196</A></SPAN></P>
<P> </P>
<P><SPAN style="font-size: 14px;">Hope this helps!</SPAN></P>
<P><SPAN style="font-size: 14px;">Sincerely,</SPAN></P>
<P><SPAN style="font-size: 14px;">Iffa</SPAN></P>
<P> </P>Wed, 30 Sep 2020 17:28:27 GMThttps://community.intel.com/t5/Intel-Distribution-of-OpenVINO/why-smaller-openvino-model-get-longer-inference-time/m-p/1213822#M20886Iffa_Intel2020-09-30T17:28:27ZRe:why smaller openvino model get longer inference time
https://community.intel.com/t5/Intel-Distribution-of-OpenVINO/why-smaller-openvino-model-get-longer-inference-time/m-p/1216733#M21008
<P>Greetings,</P><P><BR /></P><P><SPAN style="font-family: Calibri, sans-serif; font-size: 11pt;">Intel will no longer monitor this thread since we have provided a solution. If you need any additional information from Intel, please submit a new question</SPAN></P><P><BR /></P><P><SPAN style="font-family: Calibri, sans-serif; font-size: 11pt;">Sincerely,</SPAN></P><P><SPAN style="font-family: Calibri, sans-serif; font-size: 11pt;">Iffa</SPAN></P><BR />Mon, 12 Oct 2020 06:44:58 GMThttps://community.intel.com/t5/Intel-Distribution-of-OpenVINO/why-smaller-openvino-model-get-longer-inference-time/m-p/1216733#M21008Iffa_Intel2020-10-12T06:44:58Z