Difference Output Results when run model on CPU and VPU

[LAndr23]

I have a model convert to IR (data type : FP16) and start inference.

But when compare the outputs when changing the device:

plugin = IEPlugin(device="CPU")

plugin = IEPlugin(device="GPU")

plugin = IEPlugin(device="MYRIAD")

The output I get from "CPU" or "GPU" are correct (or very little difference) but when run on "VPU", I get results are matrix with values "0".

What's happen with the VPU? Can you give me some reason?

Thanks.

[AlHill]

What is the model number of your compute stick?

Doc

[LAndr23]

I'm using the Intell NCS2

[David_C_Intel]

Hi LAndr23,

Thanks for reaching out. Could you please answer the following:

• On which OS are you using the Intel®  NCS2? 
• Which OpenVINO™ toolkit version are you using?
• Which sample / demo are you using?
• Could you share the IR files and the model optimizer command you used?

Best regards,

David C.

[LAndr23]

Hi David C,

I'm working on Windows 10 64bit, OpenVINO version lastest 2020.2.

This's outputs I got when running on VPU device :

[['import/primaryQN/LSTM_hidden_cell_output': array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32), 'import/primaryQN/LSTM_hidden_state_output': array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,

    0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)}

But when running on CPU, outputs are not 0.

You can follow the attached files to get IR, PB file and my code I using to run on VPU.

Regards,

Andrew

[David_C_Intel]

Hi LAndr23,

Thanks for the files provided.

Could you please send the command you used to convert the frozen model to IR format?

Regards,

David C.

[LAndr23]

Hi David C,

Thank for your reply.

This is my command convert the frozen model to IR.

python mo_tf.py -m D3RQN_phase3.pb --data_type FP16 --input "import/primaryQN/preContextsFlattened[1 1 248],import/primaryQN/LSTMCellZeroState/zeros[1 248],import/primaryQN/LSTMCellZeroState/zeros_1[1 248]" --output import/primaryQN/norm/Contexts,import/primaryQN/LSTM_hidden_cell_output,import/primaryQN/LSTM_hidden_state_output

[David_C_Intel]

Hi LAndr23,

Thank you for the information.

We are currently looking into your issue, we will come back to you as soon as possible.

Regards,

David C.

[David_C_Intel]

Hi LAndr23,

Thank you for your patience. It appears that it is not working as expected on the MYRIAD device because there are layers that are not supported by the VPU.

Before running inference on a model it is a good idea to always check for unsupported layers. Here is the code that will help you on this:

plugin1 = IECore() dev = "MYRIAD" #hetero_dev = "HETERO:MYRIAD,CPU" #dev = "CPU" #Read Model IR net = IENetwork(model=model_xml, weights=model_bin) # Get the supported layers of the network supported_layers = plugin1.query_network(network=net, device_name=dev) # Check for any unsupported layers, and let the user # know if anything is missing. Exit the program, if so. unsupported_layers = [l for l in net.layers.keys() if l not in supported_layers] if len(unsupported_layers) != 0: print("The following layers are not supported by the plugin for specified device {}:\n {}".format(dev, ', '.join(unsupported_layers))) print("Please try to specify extensions library path in sample's command line parameters using -l or device specific extension, i.e. --cpu-extension command line argument") sys.exit(1) input_blob = next(iter(net.inputs)) t = time.time() # Load model exec_net1 = plugin1.load_network(net, hetero_dev)

The output for your model is:

The following layers are not supported by the plugin for specified device MYRIAD:

 import/primaryQN/preContextsFlattened, import/primaryQN/LSTMCellZeroState/zeros_1, import/primaryQN/LSTMCellZeroState/zeros, import/primaryQN/Contexts/Cast_1588_const, Copy_import/primaryQN/mask/Output_0/Data__const, Copy_import/primaryQN/mask/Output_0/Data_267_const

If you need to use MYRIAD plugin then we recommend to use the CPU as a backup device that can handle the unsupported layers. That can be done with the Heterogeneous plugin.

Best regards,

David C.