Dynamic batching not working in 2020 R1

[Raghavan_S_]

Even simple networks like LeNet do not work with dynamic batch size.  I wonder if this support has been discontinued.  The message error that I get is:

RuntimeError: MKLDNNGraph::CreateGraph: such topology cannot be compiled for dynamic batch!

This network does not have any unsupported layers, only conv, ReLU, Pool and FullyConnected.  Also note that the model optimizer version used is 10.  For unoptimized networks, dynamic batching seems to work.  I have attached the python code with models that replicate this problem.  The caffe model and prototxt files were taken from the examples provided with openvino.  I've also included the output log of the model optimizer below:

% python3 /opt/intel/openvino_2020.1.023/deployment_tools/model_optimizer/mo.py --input_proto models/SampLeNet.prototxt --input_model models/SampLeNet.caffemodel --output_dir models

[setupvars.sh] OpenVINO environment initialized
Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:     [...]/models/SampLeNet.caffemodel
        - Path for generated IR:        [...]/models
        - IR output name:       SampLeNet
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         Not specified, inherited from the model
        - Output layers:        Not specified, inherited from the model
        - Input shapes:         Not specified, inherited from the model
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
Caffe specific parameters:
        - Path to Python Caffe* parser generated from caffe.proto:      /opt/intel/openvino_2020.1.023/deployment_tools/model_optimizer/mo/front/caffe/proto
        - Enable resnet optimization:   True
        - Path to the Input prototxt:   .../models/SampLeNet.prototxt
        - Path to CustomLayersMapping.xml:      Default
        - Path to a mean file:  Not specified
        - Offsets for a mean file:      Not specified
Model Optimizer version:        2020.1.0-61-gd349c3ba4a

[ SUCCESS ] Generated IR version 10 model.
[ SUCCESS ] XML file: [...]/models/SampLeNet.xml
[ SUCCESS ] BIN file: [...]/models/SampLeNet.bin
[ SUCCESS ] Total execution time: 1.99 seconds.
[ SUCCESS ] Memory consumed: 93 MB.
Model conversion Successful

[Max_L_Intel · Accepted solution]

Hello, Raghavan S.

As I see, ReLU is not one of the supported layers by dynamic batching feature. Please find the list of supported layers here - https://docs.openvinotoolkit.org/latest/_docs_IE_DG_DynamicBatching.html
I think this might be the reason of that error you see.

Thanks.
Best regards, Max.

[Prof_Joshua_Smith]

I couldn't find any given models compatible with dynamic batch sizing even (For example, face detection with person age, gender, or mood recognition is a typical usage scenario.) the mentioned example in the docs. Is there any we can port the existing models and architecture to support DBS?? It'd great if could help. @Raghavan_S_ @Max_L_Intel 

[Max_L_Intel]

Hi @Prof_Joshua_Smith 

We want to inform you that old threads are no longer monitored by OpenVINO team, so please kindly create new thread for your question.

Thank you.