Results are different between CPU and NCS2 with the same IR model

[fariver]

Hi, 

I'm find the outputs of official model from cpu and myriad have different results, which block my development. Here is some details of my environment and codes.

my development evironment:

Neural Compute Stick2

MAC OS 10.13

openvino_2021.4.582

my inference alter from:

openvino_2021/deployment_tools/inference_engine/samples/python/object_detection_sample_ssd/object_detection_sample_ssd.py

and extract 'conv2/WithoutBiases' by 

# add extra output
net.add_outputs('conv2/WithoutBiases')
# extract the output of this layer
conv2_fm = res['conv2/WithoutBiases']
conv_img = sigmoid(conv2_fm.mean(aixs=(0,1)))
conv_img = (conv2_img * 255.).astype(np.uint8)
return conv_img

My original model download from intel official model 

the outputs from cpu and myriad post as below,  the bottom is difference image between them(red dot is different position).

The above result show that the result from MYRIAD is indeed not aligned with CPU even in official model and inference code. 

My customized task is deploy a segmentation-like model on NCS2, the difference between MYRIAD and CPU will bring some uncertain risks in real application. I'd be appreciate if anyone can help me out of this confusion.

[Wan_Intel]

Hi Fariver,

Thank you for reaching out to us.

I tried to replicate your issue by adding your snippet of code into object_detection_sample_ssd.py.

Unfortunately, I faced an unexpected error when executing the object_detection_sample_ssd.py.

Could you please share your modified object_detection_sample_ssd.py with us so that we can replicate your issue?

Regards,

Wan

[fariver]

Thanks for your attention. My modified object_detection_sample_ssd.py is compressed in code.zip.

[Wan_Intel]

Hi Fariver,

Thanks for your information.

I have validated your scripts with a custom input image using Intel® Core™ i7-8665U Processor (CPU Plugin) and Intel® Neural Compute Stick 2 (Myriad Plugin), on Windows 10.

The similarity for both inference results was 99.95%. I compared the inference results for both plugins using main.py from olivierbenard/differences-between-two-images.

May I know which script are you using to compare the inference results?

Can you compare your inference results using the above script from your side?

Regards,

Wan

[fariver]

Thanks for your replying. I compared the results of two devices by saving the output as image. Then I used a software named Beyond Compare by which you can easily know the different pixel between the output image. 

Or you can post the output images from myriad and cpu by my code.zip, then I can return you the difference between these two devices.

[Wan_Intel]

Hi Fariver,

For your information, I have compared CPU and MYRIAD inference results using Beyond Compare. I obtained the same result as you did when I set the Tolerance value to 0.

However, when I set the Tolerance value greater than 0, and activate Ignore Unimportant Differences, the similarity for both results was indistinguishable.

You may refer to How to Compare in the Picture Compare View for more information.

On another note, did you compare your inference results using main.py from olivierbenard/differences-between-two-images?

Based on the development team's response, we claim that the relative accuracy difference between any of the target platforms and the reference metrics should be within 1%.

Best regards,

Wan

[Wan_Intel]

Hi Fariver,

This thread will no longer be monitored since we have provided explanations.

If you need any additional information from Intel, please submit a new question.

Best regards,

Wan