Nan Output for custom tensorflow network

[idata]

Hello,

I have been trying to run a 3 layer tensorflow neural network on Movidius NCS but haven’t been able to get the desired output. I would like to report the issue and appreciate any support available regarding this issue.

The details are as follows :

The model was build and trained in Tensorflow. The tensorflow checkpoint file was generated,this was then used to create a graph file using mvNCCompile and a python script was written to load the graph and run the model with the test data on Movidius NCS. The output received was an array of Nans.

For debugging the issue, I parsed the graph file generated , It seems like the movidius tensorflow parser is not accurately converting the tap values used in the model which could be the potential issue why the output is an array Nans. The output of the parsed graph file shows that the dimensions of the Tap Matrix during the Sum Operation in particular layers is not as expected. For example in layer 1 the tap dimension are expected to be 1x1x50 but the graph file shows its 1x50x50 (refer the bold values in the output details below)

Here are the graph parser output details -

Name: b'TensorFlow Network'

Number of Stages: 8Offset: 2080Shaves 0 to 110 Name: b'layer_1/MatMul' Op: FullyConnected InputDim: 1 x 1 x 9 TapDim: 1 x 9 x 50 OutputDum: 1 x 1 x 50 PostOp: NoOp bias pointer: 0 bias index: 0 opparam pointer: 0 opparam index: 0 input pointer: 0 input index: 1 taps pointer: 0 taps index: 3 output pointer: 400 output index: 41 Name: b'layer_1/Relu' Op: Sum InputDim: 1 x 1 x 50 TapDim: 1 x 50 x 50 OutputDum: 1 x 1 x 50 PostOp: Relu bias pointer: 0 bias index: 0 opparam pointer: 0 opparam index: 0 input pointer: 400 input index: 4 taps pointer: 0 taps index: 1 output pointer: 1360 output index: 52 Name: b'layer_2/MatMul' Op: FullyConnected InputDim: 1 x 1 x 50 TapDim: 1 x 50 x 100 OutputDum: 1 x 1 x 100 PostOp: NoOp bias pointer: 0 bias index: 0 opparam pointer: 0 opparam index: 0 input pointer: 1360 input index: 5 taps pointer: 960 taps index: 3 output pointer: 2720 output index: 63 Name: b'layer_2/Relu' Op: Sum InputDim: 1 x 1 x 100TapDim: 1 x 100 x 100 OutputDum: 1 x 1 x 100 PostOp: Relu bias pointer: 0 bias index: 0 opparam pointer: 0 opparam index: 0 input pointer: 2720 input index: 6 taps pointer: 0 taps index: 1 output pointer: 4576 output index: 74 Name: b'layer_3/MatMul' Op: FullyConnected InputDim: 1 x 1 x 100 TapDim: 1 x 100 x 50 OutputDum: 1 x 1 x 50 PostOp: NoOp bias pointer: 0 bias index: 0 opparam pointer: 0 opparam index: 0 input pointer: 4576 input index: 7 taps pointer: 11008 taps index: 3 output pointer: 6032 output index: 85 Name: b'layer_3/Relu' Op: Sum InputDim: 1 x 1 x 50TapDim: 1 x 50 x 50 OutputDum: 1 x 1 x 50 PostOp: Relu bias pointer: 0 bias index: 0 opparam pointer: 0 opparam index: 0 input pointer: 6032 input index: 8 taps pointer: 0 taps index: 1 output pointer: 6992 output index: 96 Name: b'output/MatMul' Op: FullyConnected InputDim: 1 x 1 x 50 TapDim: 1 x 50 x 1 OutputDum: 1 x 1 x 1 PostOp: NoOp bias pointer: 0 bias index: 0 opparam pointer: 0 opparam index: 0 input pointer: 6992 input index: 9 taps pointer: 21056 taps index: 3 output pointer: 7560 output index: 107 Name: b'output/add' Op: Sum InputDim: 1 x 1 x 1 TapDim: 1 x 1 x 1 OutputDum: 1 x 1 x 1 PostOp: NoOp bias pointer: 0 bias index: 0 opparam pointer: 0 opparam index: 0 input pointer: 7560 input index: 10 taps pointer: 0 taps index: 1 output pointer: 0 output index: 2

In conclusion, It seems like the Tap Dimensions in the layer where sum operation is done gives wrong dimensions for some reasons. I can provide the python scripts and data used to run the model on NCS so it can be recreated at the support end if need be.

Thanks,

snh