Failed to initialize Inference Engine backend: AssertionFailed

[Peeren__Christian]

Dear all,

I just started playing around with the NCS2 and could fix a lot of potential pitfalls thanks to this excellent forum. Unfortunately, I'm currently struggling with an issue I couldn't resolve so far and couldn't find in this form. I'm using OpenVino v2019.1.144 and successfully converted my trained Tensorflow graph to IR using:

python3 mo_tf.py --input_model /modelpb/encoder.pb --input_shape [1,224,224,3] --data_type FP16

My custom model consists of a Keras and Tensorflow part, whereas the last layer is a Reshape layer.

Inference is run on a RaspberryPi3B+, so I copy the .xml and .bin to the RPI and run the following Python code:

import numpy as np
import cv2 as cv
# Load the model.
net = cv.dnn.readNet('encoder.xml',
                     'encoder.bin')
# Specify target device.
net.setPreferableTarget(cv.dnn.DNN_TARGET_MYRIAD)
# Read an image.
frame = cv.imread('/home/pi/projects/02_ncs_test/orange_89.jpg')
if frame is None:
    raise Exception('Image not found!')

# Prepare input blob and perform an inference.
blob = cv.dnn.blobFromImage(frame, size=(224, 224), ddepth=cv.CV_8U)

print(blob.shape)
net.setInput(blob)
out = net.forward()

The model is read successfully but unfortunately the program crashes with this message:

Traceback (most recent call last):
  File "run_ncs.py", line 18, in <module>
    out = net.forward()
cv2.error: OpenCV(4.1.0-openvino) /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/dnn/src/op_inf_engine.cpp:747: error: (-215:Assertion failed) Failed to initialize Inference Engine backend: AssertionFailed: perm.size() == 4 in function 'initPlugin'

In older threads I read about a permute layer which could lead to this issues, but I can see a layer like that in my .xml file. What am I doing wrong?

Thanks,

Christian

P.S.: The examples all run fine, so it must be something model specific.

[Shubha_R_Intel]

Dear Peeren, Christian,

We are on 2019R2.01 now and actually will release R3 any day now. OpenVino v2019.1.144 is really far too old. Many bugs have been fixed since that release. Can you kindly upgrade and try again ?

Thanks,

Shubha

[Peeren__Christian]

Hi Shubha,

excellent point. So I updated to version R2 (2019.2.242) on my RPI and to 2019.2.275 on my host machine. Unfortunately, still no sucess.

cv2.error: OpenCV(4.1.1-openvino) /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/dnn/src/op_inf_engine.cpp:477: error: (-215:Assertion failed) Failed to initialize Inference Engine backend: AssertionFailed: perm.size() == 4 in function 'initPlugin'

My second thought is the way how I froze the model might be faulty. On the other hand I got it working for another (simpler) Keras model.

Any idea what is going wrong here? I added the xml and bin file as zip below.

Thanks again,

Christian

Additional note: I'd have updated to a higher version, but wasn't really sure how to use that provided "inference_engine" folder in the R3 section of the Intel Download Center for Raspbian. All other versions seem to have a tar.gz file, which goes along with the installation instruction, whereas it is not mentioned how to proceed with just the "inference_engine" folder. Maybe it would be good to update the documentation to avoid similar topics for other users? I'm sorry if it is mentioned somewhere else and I just didn't see it.

[Shubha_R_Intel]

Dear Peeren, Christian,

Sorry that i didn't notice this earlier but it seems like you're not using Inference Engine Core API. Unfortunately, this forum is meant for Inference Engine and Model Optimizer tech support. If you're using OpenCV  backend (which of course calls Inference Engine also), you should post your question to :

https://github.com/opencv/opencv/issues

And by the way, we just released 2019R3 today.

Seeing the *.bin and *.xml in your code above it does appear that you've used Model Optimizer however, which is certainly appropriate for this forum.

For freezing a Keras model into Tensorflow, please use the following code:

# !/usr/bin/env python
"""
 Copyright (c) 2018 Intel Corporation

 Licensed under the Apache License, Version 2.0 (the 'License');
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an 'AS IS' BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
"""


import os
import sys

import argparse

from pathlib import Path

import tensorflow as tf
from tensorflow.python.framework import graph_util
from tensorflow.python.framework import graph_io

from keras import backend as K
from keras.models import load_model


def setKerasOptions():
    K._LEARNING_PHASE = tf.constant(0)
    K.set_learning_phase(False)
    K.set_learning_phase(0)
    K.set_image_data_format('channels_last')


def getInputParameters():
    parser = argparse.ArgumentParser()
    parser.add_argument('--input_model', '-m', required=True, type=str, help='Path to Keras model.')
    parser.add_argument('--num_outputs', '-no', required=False, type=int, help='Number of outputs. 1 by default.', default=1)

    return parser


def export_keras_to_tf(input_model, output_model, num_output):
    print('Loading Keras model: ', input_model)

    keras_model = load_model(input_model)

    print(keras_model.summary())

    predictions = [None] * num_output
    predrediction_node_names = [None] * num_output

    for i in range(num_output):
        predrediction_node_names = 'output_node' + str(i)
        predictions = tf.identity(keras_model.outputs, name=predrediction_node_names)

    sess = K.get_session()

    constant_graph = graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), predrediction_node_names)
    infer_graph = graph_util.remove_training_nodes(constant_graph) 

    graph_io.write_graph(infer_graph, '.', output_model, as_text=False)


def main():
    argv = getInputParameters().parse_args()

    input_model = argv.input_model
    num_output = argv.num_outputs
    output_model = str(Path(input_model).name) + '.pb'
    
    predrediction_node_names = export_keras_to_tf(input_model, output_model, num_output)

    print('Ouput nodes are:', predrediction_node_names)
    print('Saved as TF frozen model to: ', output_model)


if __name__ == '__main__':
  main()

Hope it helps,

Thanks,

Shubha

[Shubha_R_Intel]

Dear Peeren, Christian, it seems like someone on the dldt github forum has a very similar error to you, already tested on R3. This person is also using FP16 (NCS2). Is this you or is it just a coincidence ?

https://github.com/opencv/dldt/issues/265

This could actually be a real bug on 2019R3. I will reproduce it and let you know shortly.

Thanks,

Shubha

[Peeren__Christian]

Dear Shubha,

thanks again for your detailed response and the code. The person is definitely not me.

Please drop me a message when you know more.

Thanks again,

Christian

[Peeren__Christian]

Dear Shubha,

I know this is probably off-topic, but could you also explain how to update to R3 on Raspbian? I'm a little bit lost here (and excuse my newbie question).

Obviously, I cannot follow the official Intel instructions (as described here https://docs.openvinotoolkit.org/latest/_docs_install_guides_installing_openvino_raspbian.html) since there is no tar file as for the previous versions. I already figured there's a "opencv_4.1.2-624_debian9arm.tar.xz" in the folder "inference_engine", but I have no clue how to proceed from here.

Some useful hints would be much appreciated.

Thanks,

Christian

[WDomo]

I have asked this question here  https://software.intel.com/en-us/forums/computer-vision/topic/825405

It should show up in https://download.01.org/opencv/2019/openvinotoolkit/  The R2 branch holds the .242 release but the R3 has nothing

[Shubha_R_Intel]

Dear  Domoney, William and Peeren, Christian,

Of course you have a right to know when 2019R3 Raspbian will be available. I have asked this on your behalf. Will update as soon as i find out.

Thanks,

Shubha

[es__we]

I also have this probelm, is there any solution?