Intel® Distribution of OpenVINO™ Toolkit
Community support and discussions about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all things computer vision-related on Intel® platforms.
5612 Discussions

IndexError: list index out of range trying to compile TF model

idata
Community Manager
4,203 Views

Traceback (most recent call last):

 

File "/usr/local/bin/mvNCCompile", line 118, in

 

create_graph(args.network, args.inputnode, args.outputnode, args.outfile, args.nshaves, args.inputsize, args.weights)

 

File "/usr/local/bin/mvNCCompile", line 104, in create_graph

 

net = parse_tensor(args, myriad_config, debug=True)

 

File "/usr/local/bin/ncsdk/Controllers/TensorFlowParser.py", line 1007, in parse_tensor

 

output_size = [input_shape[0], 1, 1, outputs[3]]

 

File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_shape.py", line 521, in getitem

 

return self._dims[key]

 

IndexError: list index out of range

 

This occurs if I pass either my TF meta or pb file like so:

 

mvNCCompile model.pb -w incept3_ft_weights.h5 -s 12 -in inception_v3_input -on dense_25/Softmax -o graph

 

What am I doing wrong?

0 Kudos
29 Replies
idata
Community Manager
2,496 Views

I also see the following error on some other models:

 

[Error 5] Toolkit Error: Stage Details Not Supported: FusedBatchNorm inputs mean and variance are not defined. The graph is not created for inference.

 

This was of course me converting a Keras model -> to a TF protobuf and then trying to compile it, etc. Not sure if that is a valid work flow but I thought it should theoretically just work.

idata
Community Manager
2,496 Views

@pisymbol Can you provide your model files for testing (pb, meta)?

idata
Community Manager
2,496 Views

Uploading now….I really need to get this to work!

idata
Community Manager
2,497 Views

Those three links are two PBs and a meta. All of them are translated Keras -> TF models.

 

Note to get the 'FusedBatchNorm' issue I had to patch the NCSDK like so:

 

-- TensorFlowParser.py 2018-01-29 11:37:22.912714875 +0000 +++ TensorFlowParser.py.patch 2018-01-29 11:36:51.172559957 +0000 @@ -287,7 +287,7 @@ for a in node.outputs: print(" OUT:", a.name) if not inputfound: - if have_first_input(strip_tensor_id(node.outputs[0].name)): + if node.outputs and have_first_input(strip_tensor_id(node.outputs[0].name)): inputfound = True if debug: print('Starting to process')
idata
Community Manager
2,497 Views

Note, here is some code I've used to translate Keras to TF (taken from a StackOverflow post I believe):

 

import os import tensorflow as tf from tensorflow.python.framework.graph_util import convert_variables_to_constants from keras import backend as K from keras.models import load_model def freeze_session(session, keep_var_names=None, output_names=None, clear_devices=True): graph = session.graph with graph.as_default(): freeze_var_names = list(set(v.op.name for v in tf.global_variables()).difference(keep_var_names or [])) output_names = output_names or [] output_names += [v.op.name for v in tf.global_variables()] input_graph_def = graph.as_graph_def() if clear_devices: for node in input_graph_def.node: node.device = "" frozen_graph = convert_variables_to_constants(session, input_graph_def, output_names, freeze_var_names) return frozen_graph model = load_model(os.getcwd() + os.path.sep + 'my_model.h5') frozen_graph = freeze_session(K.get_session(), output_names=[model.output.op.name]) tf.train.write_graph(frozen_graph, os.getcwd(), "my_model.pb", as_text=False)
idata
Community Manager
2,496 Views

@Tome_at_Intel Were you able to download these models? I'd like to understand what is going on here.

idata
Community Manager
2,496 Views

@pisymbol FusedBatchNorm requires the input to have the mean and variance defined. You can double check your input to the FusedBatchNorm layer. You can see the exception at line 1539 of TensorFlowParser.py in /opt/movidius/NCSDK/ncsdk-x86_64/tk/Controllers:

 

if len(mean) == 0 or len(var) == 0: throw_error(ErrorTable.StageDetailsNotSupported, "FusedbatchNorm inputs mean and variance are not defined. The graph is not created for inference.")
idata
Community Manager
2,496 Views

@Tome_at_Intel: I understand the exception what I don't understand is what exactly am I dong wrong in converting my Keras Model to Tensorflow. I don't set FusedBatchNorm explicitly. Also, you didn't address the patch I posted above to the TensorFlowParser which assumes node.outputs never has zero length.

idata
Community Manager
2,496 Views

@Tome_at_Intel: Actually, they can be zero and should be during training: See here: https://www.tensorflow.org/api_docs/cc/class/tensorflow/ops/fused-batch-norm

idata
Community Manager
2,496 Views

@pisymbol I understand. Per the Guidance for Compiling TensorFlow Doc @ https://movidius.github.io/ncsdk/tf_compile_guidance.html, you have to remove all training features and prepare a version of the network that is going to be used for inference only. Can you try removing any training features from the network and see if you can compile the network?

idata
Community Manager
2,495 Views

@Tome_at_Intel I had to upgrade to TF 1.5 to get rid of the training features.

 

But now I see:

 

/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py:858: DeprecationWarning: builtin type EagerTensor has no __module__ attribute EagerTensor = c_api.TFE_Py_InitEagerTensor(_EagerTensorBase) /usr/local/lib/python3.5/dist-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead if d.decorator_argspec is not None), _inspect.getargspec(target)) /usr/local/lib/python3.5/dist-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead if d.decorator_argspec is not None), _inspect.getargspec(target)) /usr/local/lib/python3.5/dist-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead if d.decorator_argspec is not None), _inspect.getargspec(target)) Traceback (most recent call last): File "/usr/local/bin/mvNCCompile", line 118, in <module> create_graph(args.network, args.inputnode, args.outputnode, args.outfile, args.nshaves, args.inputsize, args.weights) File "/usr/local/bin/mvNCCompile", line 104, in create_graph net = parse_tensor(args, myriad_config) File "/usr/local/bin/ncsdk/Controllers/TensorFlowParser.py", line 1007, in parse_tensor output_size = [input_shape[0], 1, 1, outputs[3]] File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_shape.py", line 521, in __getitem__ return self._dims[key] IndexError: list index out of range

 

Updated model file: https://drive.google.com/file/d/1A5hLi6Pa1rd6TtQ6PnikqkUmr0VwJvgj/view?usp=sharing

 

I'm using the TF Graph Transform Tool to fold batch norms etc.:

 

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/graph_transforms/README.md

 

I actually think in your guidelines for TF a better idea would be to work with this tool which is suppose to remove training features and strip the graph of non-essential nodes for inference, etc.

idata
Community Manager
2,495 Views

@Tome_at_Intel : Is there anything you can suggest to make this go?

idata
Community Manager
2,495 Views

@pisymbol You can try to flatten or reshape your output to a single 1D vector.

idata
Community Manager
2,495 Views

@Tome_at_Intel Is this a bug in your parser? (also you didn't comment about the patch above?)

idata
Community Manager
2,495 Views

@pisymbol Since it is outside of the normal workflow, it's hard to say what it could be. We don't have support for Keras at the moment, but we've made a note that some of our Intel NCS users are using Keras and are interested in using these networks on the NCS.

 

Regarding your issue, I haven't done this exact method to convert a network for use on the NCS myself (Keras model to TF via script then stripping out the training layers via TF graph transform tool then trying to compile for NCS) so I can't recommend a well informed next plan of action.

idata
Community Manager
2,495 Views

@Tome_at_Intel : :'(

idata
Community Manager
2,496 Views

@pisymbol did you get this to work? I am trying the exact same and see the same error as you.

idata
Community Manager
2,496 Views

model.fit({'input': X}, {'targets': Y}, n_epoch = 5,

 

validation_set =({'input': test_x}, {'targets': test_y}),

 

snapshot_step = 500, show_metric = True, run_id = MODEL_NAME)

 

model.save(MODEL_NAME)

 

Run id: LeafDetection-0.001-6conv-basic.model

 

Log directory: log/

 

INFO:tensorflow:Summary name Accuracy/ (raw) is illegal; using Accuracy/__raw_ instead.

 

Exception in thread Thread-9:

 

Traceback (most recent call last):

 

File "C:\Users\NIAZI\Anaconda3\lib\threading.py", line 916, in _bootstrap_inner

 

self.run()

 

File "C:\Users\NIAZI\Anaconda3\lib\threading.py", line 864, in run

 

self._target(*self._args, **self._kwargs)

 

File "C:\Users\NIAZI\Anaconda3\lib\site-packages\tflearn\data_flow.py", line 201, in fill_batch_ids_queue

 

ids = self.next_batch_ids()

 

File "C:\Users\NIAZI\Anaconda3\lib\site-packages\tflearn\data_flow.py", line 215, in next_batch_ids

 

batch_start, batch_end = self.batches[self.batch_index]

 

IndexError: list index out of range

 

I face this error kindly help me

idata
Community Manager
654 Views

@Nomanniazi Can you provide a link to your model for issue reproduction?

Reply