Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

Batch size on CPU

mfoglio
New Contributor I
740 Views

Are there any benefits in using a batch size bigger than 1 on CPU as opposed to allocating the same model twice other than saving some RAM?

I have 2 threads. Each one needs to access the OpenVINO model. As of now each model initialize and access a dedicated instance of the model. I am not concerned about memory usage, but about processing speed. Would it be beneficial to allocate only a single instance of the model with a batch size equal to 2?

I know that increasing the batch size might be beneficial on GPUs but this model is running on CPU. I am wondering if there is an expected speedup on CPUs.

Thank you.

0 Kudos
2 Replies
Iffa_Intel
Moderator
722 Views

Greetings,


The batch size depends on how you are feeding the model with.

Let's say , if you are using 4 RGB images at once, the input shape would be [4,3,277,277].

This is equivalent to using batch size 4.


There's pack of 4 images together and the inference would be done on all of these 4 at once.

This would produce slower response time compared to using 1 image.

However, the efficiency/fps, would be higher.


Sincerely,

Iffa


0 Kudos
Iffa_Intel
Moderator
698 Views

Greetings,


Intel will no longer monitor this thread since we have provided a solution. If you need any additional information from Intel, please submit a new question.


Sincerely,

Iffa


0 Kudos
Reply