Re:Keras | Tensorflow backend | Lstm Runtime Error

pankajrawat · ‎08-19-2020

Below are the software version

Enviroment Details
=====================
openvino_fpga_2020.4.287  -- Intel cloud environment
Keras==2.4.3
tensorflow==2.2.0
onnx==1.7.0
onnxconverter-common==1.7.0
onnxruntime==1.3.0

Approach 1 - Keras to Onnx to Openvino [Backend tensorflow]

Keras model - running and giving results
Onnx model - running and giving results
Openvino - Model optimizer error while converting onnx to IR

[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.RNNSequenceNormalizeToIE.RNNSequenceNormalize'>): Something bad has happened with graph! Data node "lstm/HiddenStateResize" has 0 producers

Approach 2 - Keras to Openvino [Backend tensorflow]

Model optimizer is giving below error while converting keras model to IR.

[ ERROR ]  Cannot infer shapes or values for node "StatefulPartitionedCall/sequential/lstm/StatefulPartitionedCall/TensorArrayUnstack/TensorListFromTensor".
[ ERROR ]  Tensorflow type 21 not convertible to numpy dtype.
[ ERROR ]  
[ ERROR ]  It can happen due to bug in custom shape infer function <function tf_native_tf_node_infer at 0x7ff3db149ea0>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "StatefulPartitionedCall/sequential/lstm/StatefulPartitionedCall/TensorArrayUnstack/TensorListFromTensor" node. 
 For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #38.

Munesh_Intel · ‎08-21-2020

Hi Pankaj,

The errors that you are seeing seem to occur due to input shapes not being defined correctly or some unsupported TensorFlow operations.

General information regarding specifying input shapes is available here:

https://docs.openvinotoolkit.org/2020.4/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html#when_to_specify_input_shapes

TensorFlow Supported Operations are listed here:

https://docs.openvinotoolkit.org/2020.4/openvino_docs_MO_DG_prepare_model_Supported_Frameworks_Layers.html#tensorflow_supported_operations

Please share more information about your model, the layers used if it's a custom model, command given to Model Optimizer to convert the trained model to Intermediate Representation (IR), and environment details (versions of OS, Python, CMake, etc.).

If possible, please share the trained model files for us to reproduce your issue.

Regards,

Munesh

pankajrawat · ‎08-25-2020

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"

$ python  --version
Python 3.6.10

$ cmake --version
cmake version 3.10.2

Command Used to convert Onnx to IR

(cenv) u47404@s099-n003:~/intelmac$ python /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --input_model models/keras/timeseries_enode.onnx --input_shape [128,50,2]
Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      /home/u47404/intelmac/models/keras/timeseries_enode.onnx
        - Path for generated IR:        /home/u47404/intelmac/.
        - IR output name:       timeseries_enode
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         Not specified, inherited from the model
        - Output layers:        Not specified, inherited from the model
        - Input shapes:         [128,50,2]
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
ONNX specific parameters:
Model Optimizer version: 
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.RNNSequenceNormalizeToIE.RNNSequenceNormalize'>): Something bad has happened with graph! Data node "lstm/HiddenStateResize" has 0 producers

Command Used to convert from Keras to IR

python /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --saved_model_dir models/keras/timeseries_enode.keras_tf/  --input_shape [128,50,2]

Sample Model - Layers i am using

obj = Sequential()
obj.add(LSTM(units=128, return_sequences=False, input_shape=(50, 2)))
obj.add(RepeatVector(20))
obj.add(LSTM(units=128, return_sequences=True))
obj.add(Dense(units=128))
obj.add(TimeDistributed(Dense(units=2)))
optimizer = Adam(0.001)

Munesh_Intel · ‎09-01-2020

Hi Pankaj,

“TensorArrayUnstack” is not a supported TensorFlow operation in OpenVINO. The TensorFlow supported operations are available at the following page: https://docs.openvinotoolkit.org/2020.4/openvino_docs_MO_DG_prepare_model_Supported_Frameworks_Layers.html#tensorflow_supported_operations

Regards,

Munesh

pankajrawat · ‎09-03-2020

Based on the error. It seems like the above mentioned tensor operation is part of LSTM Layer itself, do you know a working example of the same.

I Replaced below but same error is coming.

Note: This syntax is of tensorflow 2.0 and not keras, as the error is same for both tensorflow and keras with tensorflow as backend

lstm = tf.keras.layers.LSTM(hidden_nodes, return_sequences=True)(rv)

with

lstm = tf.keras.layers.RNN(tf.keras.layers.LSTMCell(hidden_nodes), return_sequences=True)(rv)

[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "StatefulPartitionedCall/functional_1/lstm/PartitionedCall/TensorArrayUnstack/TensorListFromTensor" node.

Munesh_Intel · ‎09-04-2020

Hi Pankaj,

TensorFlow2 support has only recently been implemented in OpenVINO and support for LSTM is not validated yet. Having said that, I would suggest you have a look at the following two methods:

(1) Configure the LSTM layer to both return sequences and return states, as follows:

lstm = tf.keras.layers.LSTM(1, return_sequences=True, return_state=True)

Related information is available at the following page:

https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM

(2) The following community thread provides a workaround, but this is for TensorFlow 1.

https://community.intel.com/t5/Intel-Distribution-of-OpenVINO/cannot-convert-LSTM-keras-model-to-IR-files/td-p/1138027

Regards,

Munesh

pankajrawat · ‎09-11-2020

As per suggestion i modified the code to add return state. After adding sate=True, the lstm layer was returning 3 output.

[<tf.Tensor 'lstm_1/PartitionedCall:1' shape=(None, 20, 128) dtype=float32>, <tf.Tensor 'lstm_1/PartitionedCall:2' shape=(None, 128) dtype=float32>, <tf.Tensor 'lstm_1/PartitionedCall:3' shape=(None, 128) dtype=float32>]

For our usecase we need complete output from 2nd LSTM => (None, 20, 128).

The Expected one was the 0th Index One. So modified code looks like.

    input_x = tf.keras.Input(shape=(COL_X, FEATURES), name='input_layer')
    
    lstm = tf.keras.layers.LSTM(hidden_nodes)(input_x)
    rv = tf.keras.layers.RepeatVector(COL_Y)(lstm)

    #lstm = tf.keras.layers.LSTM(hidden_nodes, return_sequences=True)(rv)
    lstm = tf.keras.layers.LSTM(hidden_nodes, 
                       return_sequences=True, 
                       return_state=True)(rv)
    #lstm = tf.keras.layers.RNN(tf.keras.layers.LSTMCell(hidden_nodes), return_sequences=True)(rv)
    print(lstm)
    dense = tf.keras.layers.Dense(128)(lstm[0])
    dense = tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(FEATURES))(dense)
    
    model = tf.keras.Model(inputs=input_x, outputs=dense)
    optimizer = tf.keras.optimizers.Adam(lr)
    model.compile(optimizer=optimizer, loss='mse', metrics=['mse'])

The code compiles and gives output in tensorflow, but when converted to openvino same error is coming.

[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "StatefulPartitionedCall/functional_1/lstm/PartitionedCall/TensorArrayUnstack/TensorListFromTensor" node. 
 For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #38.

Munesh_Intel · ‎09-15-2020

Hi Pankaj,

OpenVINO™ support for TensorFlow 2 models, which has only been recently implemented, is in preview (aka Beta), which means limited and not of production quality yet. Also, OpenVINO™ does not support models with Keras RNN and Embedding layers.

These information is provided at the following page:

https://docs.openvinotoolkit.org/2020.4/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html#Convert_From_TF2X

Regards,

Munesh

Munesh_Intel · ‎09-17-2020

Hi Pankaj,

This thread will no longer be monitored since we have provided references. If you need any additional information from Intel, please submit a new question.

Regards,

Munesh