mvNCCompile Fails - Illegal instruction

[idata]

My team has built a relatively straightforward convolutional neural network in Tensorflow. We are trying to profile the network using the Neural Compute SDK on the Up Board with the AI Core (not Neural Compute Stick). We have exported the .meta file and have tried to pass it through the Neural Compute compiler. The result is:

Illegal instruction (core dumped)

This is pretty unhelpful. A brief dig into _mvNCCompile.py_ shows that it's failing on line 81 of _Controllers/Scheduler.py_:

from Controllers.TensorFlowParser import parse_tensor

I confirmed this by navigating into /usr/local/bin/ncsdk/, starting Python3 and trying to import Controllers.ParseTensorFlow. It seems unresponsive for a while and then produces the "Illegal instruction" error.

The next step is to use gdb to try to trace which (presumably) .so is ultimately causing this error, but I wanted to start the discussion. My co-worker tried using the command on a different network on his own desktop (Ubuntu 16.04) and received a long list of tensorflow errors. It is baffling to receive tensorflow errors from a network that ran fine on Tensorflow.

Any help would be appreciated. It is turning into a significant time sink to simply get the Up Board and Neural Computer SDK up and running at all.

• Matt

[idata]

Hi all,

We have resolved this on our end but for anyone running into the same issue, the solution was to downgrade to TensorFlow 1.5, as per this GitHub issue:

https://github.com/tensorflow/tensorflow/issues/17411

This obviously is not a proper long term solution but the fault here lies with TensorFlow and this is a temporary workaround to get you past this error. (For anyone dismayed by this obviously incomplete solution, check nacl's March 17th explanation for why this is happening and the direction to take to fix it properly).

I want to note something though: The version of TensorFlow that was failing, 1.6 CPU (which apparently tries to use an unsupported AVX instruction, hence the failure), was the version auto-installed via the Neural Compute SDK v2 installer. This installation was done on a brand new, clean Ubuntu 16.04 install on the Up Board, so I can only assume this is happening to others also putting together their new Up Board setup. While the "Illegal instruction" error is an issue with TensorFlow, someone at Intel/Movidius should probably have noticed that if you follow all of the instructions on the Up Board wiki and in the Neural Compute docs to set up an Up Board and install the Neural Compute SDK, it will auto-install a broken version of TensorFlow and their own mvNCxxxx commands will fail if handed a TensorFlow _.meta_ file.

What doesn't make sense to me is that none of the provided Inception examples failed with an "Illegal instruction" error.

[idata]

I can confirm both the problem and the solution posted by @MWright. In my case I'm using the docker out-of-the box installer. If you'd like a simple copy&paste solution:

pip3 list | grep tensorflow # Confirm you have a tf version above 1.5 pip3 uninstall tensorflow pip3 install tensorflow==1.5

[idata]

Why are we downgrading tensorflow? Is that really the best solution?

[idata]

Like MWright mentioned, TensorFlow 1.6 and above uses AVX instructions and not all CPUs have AVX instructions. The current workaround is to use TensorFlow 1.5 until a proper solution is implemented by TensorFlow.