Different results from the same program on NCS1 and NCS2

[Wong__Mike]

Hi all,

I have a NCS first gen and have also recently purchased a NCS2, and I have been trying to set up the software enviroment and use it on Windows (with intel i7 CPU). I've installed the software dependencies including Visual Studio 2017, OpenVINO and the driver for MYRIAD.

I'm testing a simple network of densely connected layers with 1D inputs/outputs, like [1,X] as input and [1,Y] as output, and the variables are all floats. I use network converted with FP16 for MYRIAD devices, and FP32 for CPU. NCS1 works fine and outputs similar results as CPU. However, when I replace NCS1 with NCS2, the network runs but outputs incorrect results (the correct outputs are floats between 0 and 1, which is what NCS1 and CPU returns, but NCS2 outputs numbers far from these values, such as negative numbers and two-digit numbers).

The question is, I'm actually using exactly the same program (and the input arguments of -m "FP16/my_model.xml" -d MYRIAD), but I cannot seem to understand why would NCS1 and NCS2 output different results. Somehow, the sample program of classification with squeezenet returns similar results on NCS1 and NCS2, and the only modifications I used in my program that are different from the sample program are the input and output parsing, so my guess is does NCS2 have any different approach with parsing data? (For my program, I've specified FP32 precision in the host program, which reads and also outputs in float, and a float* pointer is used to marshall data into the interpret buffer, while FP16 is used for the network itself. This seems to work with no problem for NCS1, but I wonder if NCS2 might have a different approach in accepting input/output data?)

Or, to make the question shorter, I wonder what are the key differences one need to address in software, when programming NCS1 versus NCS2? Are they compatible with exactly the same software, or do one need special format/arguments for the NCS2? Also, would this be a version problem with OpenVINO or the driver (although I installed it a week ago so it should be a pretty updated version).

Thanks a lot!

P.S. the network is a Keras model, which uses the freeze-variable method to output the network data as a Tensorflow .pb file. The .pb file is converted using mo.py while specifying FP16/FP32, as well as input shape of [1,X]. (Here in the output sample below the network has input of [1,4] and output of [1,6]). I wonder if the IR model needs to be compiled differently for NCS2, or would the same xml&bin files suffice?

Attached are the outputs of the inference program running on CPU, NCS1, and NCS2:

CPU:
[ INFO ] InferenceEngine:
API version ............ 1.4
Build .................. 19154
arguments:1
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ] car.png
[ INFO ] Loading plugin

API version ............ 1.5

Build .................. win_20181005

Description ....... MKLDNNPlugin

[ INFO ] Loading network files:
FP32/my_model.xml
FP32/my_model.bin
[ INFO ] Preparing input blobs
input dim0 (size):1
input dim1 (variables):4
[ INFO ] Batch size is 1
[ INFO ] Preparing output blobs
[ INFO ] Loading model to the plugin
[ INFO ] Starting inference (1 iterations)
[ INFO ] Processing output blobs

output data:

0:0.0766812
1:0.271392
2:0.0494225
3:0.265109
4:0.0571161
5:0.472295

[ INFO ] Inference complete.

NCS1:
[ INFO ] InferenceEngine:
API version ............ 1.4
Build .................. 19154
arguments:1
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ] car.png
[ INFO ] Loading plugin

API version ............ 1.5

Build .................. 19154

Description ....... myriadPlugin

[ INFO ] Loading network files:
FP16/my_model.xml
FP16/my_model.bin
[ INFO ] Preparing input blobs
input dim0 (size):1
input dim1 (variables):4
[ INFO ] Batch size is 1
[ INFO ] Preparing output blobs
[ INFO ] Loading model to the plugin
[ INFO ] Starting inference (1 iterations)
[ INFO ] Processing output blobs

output data:

0:0.0767212
1:0.27124
2:0.0493469
3:0.266113
4:0.0570068
5:0.471924

[ INFO ] Inference complete.

NCS2:
[ INFO ] InferenceEngine:
API version ............ 1.4
Build .................. 19154
arguments:1
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ] car.png
[ INFO ] Loading plugin

API version ............ 1.5

Build .................. 19154

Description ....... myriadPlugin

[ INFO ] Loading network files:
FP16/my_model.xml
FP16/my_model.bin
[ INFO ] Preparing input blobs
input dim0 (size):1
input dim1 (variables):4
[ INFO ] Batch size is 1
[ INFO ] Preparing output blobs
[ INFO ] Loading model to the plugin
[ INFO ] Starting inference (1 iterations)
[ INFO ] Processing output blobs

output data:

0:20.7813
1:14.4219
2:6.92188
3:-21.1563
4:-8.58594
5:7.01563

[ INFO ] Inference complete.

[Wong__Mike]

I've also tested with other samples, such as the classification sample that uses "squeezenet1.1.xml", and the text_detection_demo sample program which uses "text-detection-0001.xml" to find text fields in an image. 

Again, both the CPU and NCS1 output correct results, while the NCS2 outputs similar (but different in the actual numbers) results for the classification sample, and takes a significant longer time to infer (over a minute, while NCS1 only takes 2s) for the text detection sample.

This again makes one wonder if there is any difference in the inference engine API call, program argument, or network IR compilation when using NCS vs NCS2? (since apparently using the same program and arguments returns different result from the two devices.) For instance, do they have different support for network layers? Or, do they have different internal precision implementation?

Alternatively, would it just be a faulty device? Is there some way one can check and debug the NCS/NCS2 devices?

[Hu__Yiqian]

Hi Mike,

I'm currently using ncs2 and have met a similiar problem. I've got a FP32 model and converted it to FP16 for NCS2 to run. The output is different between CPU and MYRIAD device.

Have you soloved this problem? Many thanks.

Harry

[Shubha_R_Intel]

Dear Hu, Yiqian,

Mike's tests were between NCS1 and NCS2. Yours are between NCS2 and CPU.

Shubha

[Wong__Mike]

Dear Shubha and Harry,

Thanks for the comment! Indeed, as Shubha suggested, our previous problem with different results was between an older NCS1 and the new NCS2 stick, although I remember that running with CPU, the results still agrees with NCS1 and are of reasonable values, while NCS2 outputs some random results (i.e. NCS1 and CPU are both correct, but the model somehow doesn't work with NCS2 despite using same program and same MYRIAD flag).

Actually I was intending to ask the same thing this week too. In fact I previously returned the NCS2 and used NCS1 for our research project (data for a paper), but recently we want to work on a similar project again, and NCS1 is not in our lab, so I'm contemplating whether to buy another NCS2, or to buy an outdated NCS1 just because it outputs more reliable results.

May I ask if there had been any updates since the previous post in March, or whether you might have heard any other people mentioning similar problems (if so, was there any possible cause or potential solution to the problem)? Because I would prefer not to buy the more outdated and slower 1st-gen stick (but might have to buy a 1st-gen, depending on whether the 2nd-gen still has that problem...)

Thanks!

Yours Sincerely,

Mike

[Shubha_R_Intel]

Dear Wong, Mike,

NCS2 is an improvement over NCS1 and in fact, please read the Discontinuation of NCS1 though we are currently supporting it as of today.  The best suggestion I can make to you is to try the latest and greatest OpenVino release, 2019 R2.01 and test it again. Does it still happen ? Hu, Yiqian reproduced the CPU/NCS2 discrepancy on OpenVino 2019 2.01.

Hope it helps,

thanks,

Shubha

[Ge__Zheng]

Dear Shubha.

Sorry to say this. The issue still exists even in latest version of OpenVino(2020.R1). My model is a tensorflow based FCOS. I got the correct results on CPU and NCS1, but got weird results on NCS2. Many similar problems on NCS2 have been reported by other users but I don't think those are fixed. Besides, my friend uses MXNet on NCS2 and get the expected results. This is really frustrating. I hope this issue can raise your attention and be fixed soon.

Thanks

Caleb Ge