Intel® Distribution of OpenVINO™ Toolkit
Community support and discussions about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all things computer vision-related on Intel® platforms.

Batchsize vs Async Inference requests

Patel
Novice
381 Views

Hello,

 

On increasing the Batchsize of the model in OpenVINO the fps drops to half. 

On increasing the Async inference requests the fps increases to a threshold point.

 

Is this nature of batch size normal? What is the reason behind it. 

Can be replicated using Multi-Channel Object Detection Yolov3 C++ Demo

 

./multi_channel_object_detection_demo_yolov3 -bs 5 -nireq 1

observed fps  - 5

 

./multi_channel_object_detection_demo_yolov3 -bs 1 -nireq 5

observed fps  - 63

 

./multi_channel_object_detection_demo_yolov3 -bs 5 -nireq 5

observed fps  - 12

 

 

 

 

Labels (2)
0 Kudos
1 Solution
Vladimir_Dudnik
Employee
352 Views

@Patel  I'd refer you to Introduction to Performance Topics article of OpenVINO online documentation, which says:

 

One way to increase computational efficiency is batching, which combines many (potentially tens) of input images to achieve optimal throughput. However, high batch size also comes with a latency penalty

 

Basically, batching can provide some performance benefits on GPU device, for CPU playing streams and number of threads is more beneficial.

View solution in original post

2 Replies
Vladimir_Dudnik
Employee
353 Views

@Patel  I'd refer you to Introduction to Performance Topics article of OpenVINO online documentation, which says:

 

One way to increase computational efficiency is batching, which combines many (potentially tens) of input images to achieve optimal throughput. However, high batch size also comes with a latency penalty

 

Basically, batching can provide some performance benefits on GPU device, for CPU playing streams and number of threads is more beneficial.

Wan_Intel
Moderator
315 Views

Hi Prince Patel`,

Thank you for reaching out to us.


This thread will no longer be monitored since this issue has been resolved. 

If you need any additional information from Intel, please submit a new question.


Regards,

Wan


Reply