multi-batch latency comparative experiment

[shirleyliu]

I want to test the relationship between batch size and parallel computing on GPU. I did the following experiment 

device: GPU(Intel UHD Graphics 630)

model input shape: (1,3,8,8)

input image shape:(8,8)

my command is like this:

python ./benchmark_app.py -m xxx_8x8_fp16_b1.xml -i 0801_8x8.png -d GPU -api sync -niter 1000 -b batchsize

we need to focus on latency so I set the -api synchronize mode 

and only thing I need to do is to change the batch size, the test result is in the following chart

batchsize	8	16	32	64	128	256	512	1024	1200	1280	1300	1500	1536	2000	2048	3000	3500	3700	3850	3900	4000	4096
Inputshape	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8	8x8
GPU usage	50%	53.5%	59%	68%	75%	83%	89%	95%	94%	97%	94%	95%	98%	97%	98%	97.5	98%	99%	99%	98%	100%	100%
latency	0.35	0.39	0.51	0.66	0.86	1.38	2.57	6.54	5.7	9.94	5.96	6.80	13.19	9.56	19.19	14.23	17.41	17.76	18.68	18.61	41.57	42.41

and look at the bold numbers. It's strange that the result doesn't show the regularity of the latency increases with the increase of batch size. and specifically, if the batch size is in a particular size, the GPU usage is slightly reduced and the latency  reduced a lot 

[Peh_Intel · Accepted solution]

Hi Shirley,

We noticed that this is a duplicated issue from the following thread. If you need any additional information from Intel, please submit a new question since this thread is no longer being monitored.

Sincerely,

Peh