running speed: MYRIAD slower than CPU; FP16 is comparable with FP32?

[fariver]

Hi

I fall into two confusion: 1) my network run slower on MYRIAD than CPU; 2).  the FP16 network runs comparable speed with its FP32 version on MYRIAD device without any speedup.

	FP16	FP32
MYRIAD	1.0966s	1.0958s
CPU	0.7722s	0.7883s

All the time consuming result is averaged by 10 iterations. Attachment is my new network definition. Is this the expected performance? Hoping for your help. 

Bellow is some detail imformation.

My environment info:

Neural Compute Stick2

MAC OS 10.13

openvino_2021.4.582

My test program:

from openvino.inference_engine import IECore

ie = IECore()

xml_path = './baseline.xml'

net = ie.read_network(model=xml_path)

exec_net = ie.load_network(network=net, device_name=MYRIAD) 

[Peh_Intel]

Hi fariver,

Regarding the performance between CPU and Intel® Neural Compute Stick 2 (NCS2), you can have a look at this Benchmark Results. By looking into all the chosen models, the performance of CPU is always better than NCS2. Referring to these Supported Model Formats, the VPU plugins only support FP16 models.

Regards,

Peh

[fariver]

---

By looking into all the chosen models, the performance of CPU is always better than NCS2.

---

Hi Peh， 

As shown in Benchmark Results,  the NCS2 has much better performance than i5 CPU. However, the conclusion disappearance in my customized network. Can you help me profile what's happen? or it can also be helpful if you can tell me which operations run faster on CPU than MYRIAD. Hoping for replying.

[Peh_Intel]

Hi fariver,

Please note that the values in these two graphs are different. You can point to the bar to observe the exact value. Besides, also note that there are two comparison graphs in term of throughput (FPS) and value (FPS/$) as I noticed that you’re observing the value comparison graph.

Intel® Neural Compute Stick 2 (NCS 2) is a USB stick that offers you access to neural network functionality, without the need for large, expensive hardware. It is a plug-and-play device, so you are ready to start prototyping right away. However, the performance of CPU is always better than NCS2.

Regards,

Peh

[Peh_Intel]

Hi fariver,

This thread will no longer be monitored since we have provided answers. If you need any additional information from Intel, please submit a new question. 

Regards,

Peh