Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Huynh__Loc
Beginner
183 Views

Weird latency behavior - Multi models for multiple batch sizes

Hi all, hope you could help me on this.

I have tried to run mobilenetv1 with dynamic batch size but got the "RuntimeError: MKLDNNGraph::CreateGraph: such topology cannot be compiled for dynamic batch!" error. Properly due to the squeeze layer inside mobilenetv1 that changes the shape of the tensor.

So in the end, I've decided to create multiple models for each batch size and did some benchmarks. But I got weird latency/throughput behavior. 

My program is pretty simple

images = np.random.uniform(-1,1, size=[64, 3, 224, 224]).astype(np.float32)
for batch_size in range(1,16):
    #create model
    net = IENetwork(model=model_xml, weights=model_bin)
    net.batch_size = batch_size
    exec_net = plugin.load(network=net)

    input_blob = next(iter(net.inputs))

    #run inference
    batch = images[np.arange(batch_size)]
    res = exec_net.infer(inputs={input_blob: batch})

However, after 4 iterations, openvino started using only 1 single CPU core instead of all my CPU cores (I'm using Intel(R) Xeon(R) Gold 6140). 

Batch_size: 1, Throughput: 643.86 imgs/s
Batch_size: 2, Throughput: 924.83 imgs/s
Batch_size: 3, Throughput: 1064.74 imgs/s
Batch_size: 4, Throughput: 1245.72 imgs/s
Batch_size: 5, Throughput: 168.25 imgs/s
Batch_size: 6, Throughput: 168.66 imgs/s

Do you have any suggestions to fix this problem?

Thank you

 

0 Kudos
5 Replies
Khang_N_Intel
Employee
183 Views

Hi Loc,

Based on the error message that you provided, it seems like your application has something to do with DNN and you are not using Intel(R) MKL.

This is the Intel(R) MKL forum.  Let me see if I can find the correct forum for you to post your question.

 

Best regards,

Khang

Huynh__Loc
Beginner
183 Views

Oops, sorry. I posted in a wrong forum. Should have posted into OpenVino forum.

Khang_N_Intel
Employee
183 Views

Hi Loc,

No problem.  Yes, all the questions relating to OpenVino should post over there.

Best regards,

Khang

Huynh__Loc
Beginner
183 Views

Nguyen, Khang T (Intel) wrote:

Hi Loc,

No problem.  Yes, all the questions relating to OpenVino should post over there.

Best regards,

Khang

Can you help me delete this post? I'll repost to OpenVino forum.

Thanks a lot :)

Gennady_F_Intel
Moderator
183 Views

the thread has been moved to the OpenVINO forum

Reply