Model Performace diffference NCS2 vs Tensorflow

[Qayyum__khushboo]

Hello,

I am using the NCS2 for inference of the popular FACENET CNN for face recognition. I have observed that when I use the model on the NCS2, I get a decreased accuracy as compared to Tensorflow output. Is this normal? To have a fair comparison, I gave the same input images to the model on both platforms. 

I thought that the performance of a model should remain the same on NCS as it was on the other platform (Tensorflow in my case). Is there some optimization that needs to be done in order to get the identical performance?

Thanks in Advance.

Khushboo 

[Cary_P_Intel1]

Hi, Qayyum, khushboo,

The accuracy should be the same as Tensorflow, can you share the model and code you are using? so I can reduplicate the issue, thank you.

[Qayyum__khushboo]

Hello Cary P,

You can fine the codes and models here: https://drive.google.com/open?id=1V7FzhudkpcZF1DvtcG4lKot1ywbSmrJl

I was trying to upload files here but somehow I wasn't able to.

Regards,

Khushboo

[Cary_P_Intel1]

Hi, Khushboo,

The model is pretty big, have you tried the facenet with OpenVINO on CPU? is the result also different from Tensorflow?

[Cary_P_Intel1]

Hi, Khushboo,

I've tested your code and model, the performance of CPU and VPU are better than tensorflow run on CPU. Also I can't see the accuracy drop issue, can you let me know how do you compare the result accuracy between tf and OpenVINO?

[Qayyum__khushboo]

Hello Cary,

I am giving the same image as input to both the scripts (tensorflow and VCS2) and compare the output of the model. I am attaching here the out put of the same model via tensorflow and openVino NCS2.

tensorflow
front_face.jpg (u'khushboo', 82.0228824217682)
face_2.jpg (u'khushboo', 70.19255365615679)
face_side.jpg(u'khushboo', 86.12229337944406)
face_3.jpg ('Unknown', 54.716366131550984)

intel stick

Front_face.jpg [('khushboo', 79.44055289492539)]
face_2.jpg [('khushboo', 70.27896715998985)]
face_side.jpg [('khushboo', 75.9920282922568)]
face_3.jpg [('Unknown', 54.853669788701346)]

[Cary_P_Intel1]

Hi, Khushboo,

Can I get the images you are testing? I guess the discrepancy might come from the pre-processing.

[Qayyum__khushboo]

Hello Cary,

Sure, I have attached the test images.

[Cary_P_Intel1]

Hi, Khushboo,

Is there any format difference between the images of face_2.jpg and face_side.jpg? I am able to execute the python file with "face_2.jpg" or "face_3.jpg" but when I try to test with "face_side.jpg" and "front_face.jpg", openCV throw out the error as attached. Have you encountered this problem before?

[Qayyum__khushboo]

Hello Cary,

No they should have the same format. I have never encountered this problem before. I also tried but so far I haven't got this error. Are you trying with tensorflow or the intel stick code?

[Cary_P_Intel1]

Hi, Khushboo,

I tested with both TF and intel stick code, but I can run all of them after re-name the image file name, and I didn't see the discrepancy between results from TF and Intel Stick code as attachment.

[Qayyum__khushboo]

Hello Cary,

Please try these experiments with the database file attached. The issue ma y not be  visible because in the results all the data is labelled unknown.