Batchsize vs Async Inference requests

[Patel]

Hello,

On increasing the Batchsize of the model in OpenVINO the fps drops to half. 

On increasing the Async inference requests the fps increases to a threshold point.

Is this nature of batch size normal? What is the reason behind it. 

Can be replicated using Multi-Channel Object Detection Yolov3 C++ Demo - 

./multi_channel_object_detection_demo_yolov3 -bs 5 -nireq 1

observed fps  - 5

./multi_channel_object_detection_demo_yolov3 -bs 1 -nireq 5

observed fps  - 63

./multi_channel_object_detection_demo_yolov3 -bs 5 -nireq 5

observed fps  - 12

[Vladimir_Dudnik · Accepted solution]

@Patel  I'd refer you to Introduction to Performance Topics article of OpenVINO online documentation, which says:

One way to increase computational efficiency is batching, which combines many (potentially tens) of input images to achieve optimal throughput. However, high batch size also comes with a latency penalty

Basically, batching can provide some performance benefits on GPU device, for CPU playing streams and number of threads is more beneficial.

[Wan_Intel]

Hi Prince Patel`,

Thank you for reaching out to us.

This thread will no longer be monitored since this issue has been resolved. 

If you need any additional information from Intel, please submit a new question.

Regards,

Wan