Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

multi-batch latency comparative experiment

shirleyliu
New Contributor I
475 Views

I want to test the relationship between batch size and parallel computing on GPU. I did the following experiment 

device: GPU(Intel UHD Graphics 630)

model input shape: (1,3,8,8)

input image shape:(8,8)

my command is like this:

 

python ./benchmark_app.py -m xxx_8x8_fp16_b1.xml -i 0801_8x8.png -d GPU -api sync -niter 1000 -b batchsize

 

we need to focus on latency so I set the -api synchronize mode 

and only thing I need to do is to change the batch size, the test result is in the following chart

batchsize

8

16

32

64

128

256

512

1024

1200

1280

1300

1500

1536

2000

2048

3000

3500

3700

3850

3900

4000

4096

Inputshape

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

8x8

GPU usage

50%

53.5%

59%

68%

75%

83%

89%

95%

94%

97%

94%

95%

98%

97%

98%

97.5

98%

99%

99%

98%

100%

100%

latency

0.35

0.39

0.51

0.66

0.86

1.38

2.57

6.54

5.7

9.94

5.96

6.80

13.19

9.56

19.19

14.23

17.41

17.76

18.68

18.61

41.57

42.41

 

and look at the bold numbers. It's strange that the result doesn't show the regularity of the latency increases with the increase of batch size. and specifically, if the batch size is in a particular size, the GPU usage is slightly reduced and the latency  reduced a lot 

0 Kudos
1 Solution
Peh_Intel
Moderator
468 Views

Hi Shirley,

 

We noticed that this is a duplicated issue from the following thread. If you need any additional information from Intel, please submit a new question since this thread is no longer being monitored.

 


Sincerely,

Peh


View solution in original post

0 Kudos
1 Reply
Peh_Intel
Moderator
469 Views

Hi Shirley,

 

We noticed that this is a duplicated issue from the following thread. If you need any additional information from Intel, please submit a new question since this thread is no longer being monitored.

 


Sincerely,

Peh


0 Kudos
Reply