Demo Classification result different for Neural Compute stick 2 ./demo_squeezenet_download_convert_run.sh

[GKund1]

I installed on a fresh ubuntu 16.04 and run the Image Classification verification script:

./demo_squeezenet_download_convert_run.sh

I do not get the listed example result of 0.836 sports car but the classification is much lower - only 0.671 (see attachment). There were no errors or warnings, does that point to a problem in the installation or is:

https://docs.openvinotoolkit.org/latest/_docs_install_guides_installing_openvino_linux.html

not up to date ?

Thx

[David_C_Intel]

Hi GKund1,

Thanks for reaching out. Could you tell us more about which generation of CPU and model are you using?

If you are using an unsupported generation of CPU it could be the reason why results differ compared to the documentation results. You can take a look at the supported hardware page for the supported CPU families and other hardware here if it helps. We have seen such behavior (.68 result) on an older CPU family (4th gen).

If you have more questions, feel free to ask.

Regards,

David C.

Intel Customer Support Technician

A Contingent Worker at Intel

[GKund1]

Thank you David,

I was running actually a VMWARE FUSION virtual machine with Ubuntu 64bit 16.04 on a MacBook Pro with 3.1 GHz Intel Core i7.

I followed your advice and run it on a INTEL NUC (system info below) with a ubuntu 16.04, the result is to a surprising level (see below)

the same I posted yesterday from the virtual machine.

Could you run it on a machine you think should work and verify that the demo still works as expected ?

I am curious what the mechanism would be that makes the results depended on the architecture (I run with FP32,

the result is .5 percent better), maybe you could be so kind to elaborate.

Thanks you very much, Gerd

Outputs:

Image /home/test/intel/openvino/deployment_tools/demo/car.png

classid probability label

------- ----------- -----

817   0.6708984  sports car, sport car

479   0.1922607  car wheel

511   0.0936890  convertible

436   0.0216064  beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon

751   0.0075760  racer, race car, racing car

656   0.0049667  minivan

717   0.0027428  pickup, pickup truck

581   0.0019779  grille, radiator grille

468   0.0014219  cab, hack, taxi, taxicab

661   0.0008636  Model T

[ INFO ] Execution successful

[ INFO ] This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool

run on:

Architecture:    x86_64

CPU op-mode(s):   32-bit, 64-bit

Byte Order:     Little Endian

CPU(s):       8

On-line CPU(s) list: 0-7

Thread(s) per core: 2

Core(s) per socket: 4

Socket(s):      1

NUMA node(s):    1

Vendor ID:      GenuineIntel

CPU family:     6

Model:        94

Model name:     Intel(R) Core(TM) i7-6770HQ CPU @ 2.60GHz

Stepping:      3

CPU MHz:       900.069

CPU max MHz:     3500.0000

CPU min MHz:     800.0000

BogoMIPS:      5184.00

Virtualization:   VT-x

L1d cache:      32K

L1i cache:      32K

L2 cache:      256K

L3 cache:      6144K

NUMA node0 CPU(s):  0-7

Flags:        fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp flush_l1d

[David_C_Intel]

Hi GKund1,

Thanks for the information provided.

We were able to reproduce the results you are seeing and have opened a bug with the development team to take a look on this issue. This seems to be specific to the OpenVINO™ toolkit 2019 R3.1 release as the previous version (2019 R2) shows the results as expected.

Feel free to contact us again if you have more questions.

Have a nice day!

Regards,

David C.

Intel Customer Support Technician

A Contingent Worker at Intel

[GKund1]

Hallo David,

I switched to 2.1 and it appears that some directory structures changed:

I found the caffe model in

~/openvino_models/models/FP16/public/squeezenet1.1/

rather than

~/openvino_models/models/FP16/classification/squeezenet/1.1/caffe

where the script is looking.

The model appears to run fine in 2.1

Image /opt/intel/openvino/deployment_tools/demo/car.png

classid probability label

------- ----------- -----

817   0.8364178  sports car, sport car

511   0.0945683  convertible

479   0.0419195  car wheel

751   0.0091233  racer, race car, racing car

436   0.0068038  beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon

656   0.0037315  minivan

586   0.0025940  half track

717   0.0016044  pickup, pickup truck

864   0.0012045  tow truck, tow car, wrecker

581   0.0005833  grille, radiator grille

[David_C_Intel]

Hi GKund1,

Thank you for your reply.

We are still investigating the change. Also, some file structures were made.

We apologize for the inconvenience.

Best regards,

David C.

Intel Customer Support Technician

A Contingent Worker at Intel

[GKund1]

Do I understand that it only effects R3 and that I can safely work in R2.1 - where would I look up the changed between the releases ?

Thanks Gerd

[David_C_Intel]

Hi GKund1,

You can find the release notes for the OpenVINO™ Toolkit releases at the following link:

Release_notes

We recommend using the latest release. However, you should be okay using the previous release if the lower accuracy is a limiting factor to your project.  

Best regards,

David C.

Intel Customer Support Technician

A Contingent Worker at Intel

[GKund1]

Hi,

is there any update on this topic ? I see that there is no new release ?

Respectfully Gerd