Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

Model Performace diffference NCS2 vs Tensorflow

Qayyum__khushboo
Beginner
1,175 Views

Hello,

I am using the NCS2 for inference of the popular FACENET CNN for face recognition. I have observed that when I use the model on the NCS2, I get a decreased accuracy as compared to Tensorflow output. Is this normal? To have a fair comparison, I gave the same input images to the model on both platforms. 

I thought that the performance of a model should remain the same on NCS as it was on the other platform (Tensorflow in my case). Is there some optimization that needs to be done in order to get the identical performance?

 

Thanks in Advance.

Khushboo 

0 Kudos
11 Replies
Cary_P_Intel1
Employee
1,175 Views

Hi, Qayyum, khushboo,

The accuracy should be the same as Tensorflow, can you share the model and code you are using? so I can reduplicate the issue, thank you.

 

0 Kudos
Qayyum__khushboo
Beginner
1,175 Views

Hello Cary P,

You can fine the codes and models here: https://drive.google.com/open?id=1V7FzhudkpcZF1DvtcG4lKot1ywbSmrJl

I was trying to upload files here but somehow I wasn't able to.

 

Regards,

Khushboo

0 Kudos
Cary_P_Intel1
Employee
1,175 Views

Hi, Khushboo,

The model is pretty big, have you tried the facenet with OpenVINO on CPU? is the result also different from Tensorflow?

0 Kudos
Cary_P_Intel1
Employee
1,175 Views

Hi, Khushboo,

I've tested your code and model, the performance of CPU and VPU are better than tensorflow run on CPU. Also I can't see the accuracy drop issue, can you let me know how do you compare the result accuracy between tf and OpenVINO?

0 Kudos
Qayyum__khushboo
Beginner
1,175 Views

Hello Cary,

I am giving the same image as input to both the scripts (tensorflow and VCS2) and compare the output of the model. I am attaching here the out put of the same model via tensorflow and openVino NCS2.

 

tensorflow
front_face.jpg (u'khushboo', 82.0228824217682)
face_2.jpg (u'khushboo', 70.19255365615679)
face_side.jpg(u'khushboo', 86.12229337944406)
face_3.jpg ('Unknown', 54.716366131550984)


intel stick

Front_face.jpg [('khushboo', 79.44055289492539)]
face_2.jpg [('khushboo', 70.27896715998985)]
face_side.jpg [('khushboo', 75.9920282922568)]
face_3.jpg [('Unknown', 54.853669788701346)]

 

0 Kudos
Cary_P_Intel1
Employee
1,175 Views

Hi, Khushboo,

Can I get the images you are testing? I guess the discrepancy might come from the pre-processing.

0 Kudos
Qayyum__khushboo
Beginner
1,175 Views

Hello Cary,

Sure, I have attached the test images.

 

0 Kudos
Cary_P_Intel1
Employee
1,175 Views

Hi, Khushboo,

Is there any format difference between the images of face_2.jpg and face_side.jpg? I am able to execute the python file with "face_2.jpg" or "face_3.jpg" but when I try to test with "face_side.jpg" and "front_face.jpg", openCV throw out the error as attached. Have you encountered this problem before?

 

0 Kudos
Qayyum__khushboo
Beginner
1,175 Views

Hello Cary,

No they should have the same format. I have never encountered this problem before. I also tried but so far I haven't got this error. Are you trying with tensorflow or the intel stick code?

 

0 Kudos
Cary_P_Intel1
Employee
1,175 Views

Hi, Khushboo,

I tested with both TF and intel stick code, but I can run all of them after re-name the image file name, and I didn't see the discrepancy between results from TF and Intel Stick code as attachment.

0 Kudos
Qayyum__khushboo
Beginner
1,175 Views

Hello Cary,

Please try these experiments with the database file attached. The issue ma y not be  visible because in the results all the data is labelled unknown. 

0 Kudos
Reply