cannot convert LSTM keras model to IR files

[Kannan_K_Intel]

Hi,

I am trying  to optimize a keras LSTM model using openvino. I converted the .h5 file in keras to .pb file using this repo: https://github.com/amir-abdi/keras_to_tensorflow . While converting the .pb file to IR, I am facing the following error

command used: 

python mo_tf.py --input_model ~\Desktop\machine\pb_file\model.pb --output_dir ~\Desktop\machine\pb_file

error:

WARNING: Logging before flag parsing goes to stderr.
E1111 16:07:03.382606  9500 infer.py:180] Cannot infer shapes or values for node "lstm_2/strided_slice_11".
E1111 16:07:03.387628  9500 infer.py:181] 'shrink_axis_mask'
E1111 16:07:03.387628  9500 infer.py:182]
E1111 16:07:03.388611  9500 infer.py:183] It can happen due to bug in custom shape infer function <function StridedSlice.infer at 0x000001D119ECA620>.
E1111 16:07:03.388611  9500 infer.py:184] Or because the node inputs have incorrect values/shapes.
E1111 16:07:03.388611  9500 infer.py:185] Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
E1111 16:07:03.388611  9500 infer.py:194] Run Model Optimizer with --log_level=DEBUG for more information.
E1111 16:07:03.389612  9500 main.py:307] Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "lstm_2/strided_slice_11" node.

can anyone help me to solve this issue. The log file is attached.

[Gouveia__César]

Hi kannan, 

You have to provide the input shape dimensions to the openvino model converter using the --input_shape flag. Run the above command with -h flag for more information.

Hope it helps,

César.

[Kannan_K_Intel]

Hi César,

Thank you for the reply.

My lstm model is a simple model with the following structure:

model = Sequential()
model.add(LSTM(
         input_shape=(1, n_features),
         units=100,
         return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(
          units=50,
          return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(units=1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

Where the n_features can be 21 or 33.

As you said, I tried with passing input shape as [1,21]. But getting the same error as I have explained in the question.

Is there any option to convert keras .h5 file to openvino, other than converting to .pb file and then trying?

[Gouveia__César]

Hi Kannan, 

You need to use the same number of features in both your model and input shape flag. Another thing, you converted your keras model to a tensorflow model right?, so a set of keras layers were converted to tensorflow layers. Did you checked that all tensorflow layers of your model are supported by the tensorflow -> openvino model optimizer (converter)? You can check that here.

Hope it helps,

César.

[Nunez-Yanez__Jose]

Hello,

Is there a solution for this simple keras model with LSTM to openvino, please ?

I have a similar simple model that looks like 

model.add(tf.keras.layers.LSTM(units=64,input_shape=(210, 1)))

model.add(tf.keras.layers.Dense(units=time_window_size, activation='linear'))

That fails to compile:

Cannot infer shapes or values for node "lstm/TensorArrayUnstack/TensorListFromTensor".
E0325 11:02:38.290691 139646501234496 infer.py:179] Tensorflow type 21 not convertible to numpy dtype.

I am happy to share what I have if somebody has any ideas ?

Any examples that show a Keras model with LSTM layers compiling to openvino, please ?

Thanks

[Kannan_K_Intel]

Hey Nunez-Yanez, Jose,

I converted the model just by changing the --input_shape. 

I my case the model input shape was (1,21), so I tried to pass [1,1,21] as input_shape while converting and it worked.

I hope if you pass the input shape accordingly you will be able to convert the model. And try to use the latest version of openVINO.

Hope this helps. 

[Nunez-Yanez__Jose]

Thanks for the reply,

Any chance that you could share your simple full keras model as an example and commands, pleaaase ?

I have tried everything I can think of in terms of input_shape and other options and I cannot get pass the: 

Cannot infer shapes or values for node "lstm/TensorArrayUnstack/TensorListFromTensor".
E0325 11:02:38.290691 139646501234496 infer.py:179] Tensorflow type 21 not convertible to numpy dtype.

I am thinking that maybe it is a tensorflow/keras version problem. I am using tf 1.14.0 and keras that is part of tensorflow and openvino openvino_2020.1.023. 

Really desperate with this.

[Nunez-Yanez__Jose]

This is my model and the pb that I cannot optimized with a command like:

 mo.py --input_model ./lstm.pb --output_dir . --input_shape [1,210,1] --data_type FP32 --framework tf

model = tf.keras.models.Sequential()

model.add(tf.keras.layers.LSTM(units=64,input_shape=(210, 1)))

model.add(tf.keras.layers.Dense(units=210, activation='linear'))

[Nunez-Yanez__Jose]

Hello Kannan,

If you could just take the pb I attached and try to compile with the optimizer to see if it works I could check if the problem is the pb or my openvino tools ?

please,

Thanks,

[Kannan_K_Intel]

Hey Jose,

I tried to optimize your model on both 2020 R1 and 2019 R3. Getting the same issue as you described. 

Can you try to convert your model with tensorflow 1.13 or less. My model was converted on tf 1.13. 

And can you try to use the code from this repo, https://github.com/amir-abdi/keras_to_tensorflow to convert keras weights to .pb file.

Hope these helps.

[Nunez-Yanez__Jose]

Thanks,

I was already using that h5 to pb script but it seems that my tf1.14 has some issues.

After installing a fresh tf1.13 using conda I have managed to run the model optimizer with the lstm network using openvino 2020 R1 tools.

So either there is a problem with tf1.14 or more probable I have managed to polute it previous work and was given those errors with the lstm network.

Thanks for your help, I kept trying because you said that it was possible. 

Best regards,

[Kannan_K_Intel]

Happy to know that you were able to convert the model. 

I think the issues is with openVINO, not tensorflow. I have also faced similar situation with some other models.

Happy to help.

[Nunez-Yanez__Jose]

Hello,

So I got the point of having a working system using the CPU plugin and had a go with MYRIAD. I compiled with FP16 precision without problems but then after launching the application:

Check (env.config.ignoreUnknownLayers) failed: Failed to compile layer lstm/while/LoopCond/TensorIteratorCondition_/TensorIterator : AssertionFailed: !ieDims.empty()

I have seen other comments about this !ieDims.empty() problem but I am not sure if a solution is possible or if this type of keras LSTM does not work in Myriad ? Myriad does mention LSTM as a supported layer..

Thanks for all the advice,

[Kannan_K_Intel]

Hey Jose,

This is again an issue I faced. And again this is an issue with openVINO.

And the issue with IR is in 0D tensors that are not supported by VPU plugins. 0D tensor is a scalar that represented as a tensor without dimensions at all. Equivalent representation that is supported by VPU plugin is 1D tensor with single dimensions equal to "1". 

There is a work around to this problem; 

You can manually open xml file and change some output/input ports descriptions from 0D to 1D. For reference I will attach FP16 model I created.

Compare both xmls and change only the parts where you see this kind of change

From:

<layer id="17" name="some_name" type="Const" version="opset1">
    <data element_type="f16" offset="24606" shape="" size="2"/>
    <output>
        <port id="1" precision="FP16"/>
    </output>
</layer>

To:

<layer id="17" name="some_name" type="Const" version="opset1">
    <data element_type="f16" offset="24606" shape="" size="2"/>
    <output>
        <port id="1" precision="FP16">
            <dim>1</dim>
        </port>
    </output>
</layer>

Note that not all layer ports without dimension needs to be changed, only the ones you find while comparing. And do check if you followed all the steps to install VPU plugins.

[Nunez-Yanez__Jose]

Fantastic ! .

I went through the xml replacing all: 

 <port id="1" precision="FP16"/>     

with no dimensions   with:

 <port id="1" precision="FP16">
        <dim>1</dim>
  </port>
 

and it works now on Myiriad !

Thank you so much