OpenVino converts the faceNet model to FP32 is correct, but convert to FP16 runs incorrectly on Movidius

[chen__shuo]

I use OpenVino converted the faceNet model(20180402-114759) to FP32, it runs correctly in cpu, but if I converted the model to FP16 , it runs incorrectly on Movidius

The FaceNet model Link:https://github.com/davidsandberg/facenet

I converted the model to FP32 use:

python3 ./mo_tf.py --input_model model/20180408-102900.pb --freeze_placeholder_with_value "phase_train->False" --reverse_input_channels

I use --data_type=fp16 to convert model to movidius:

python3 ./mo_tf.py --input_model model/20180408-102900.pb --freeze_placeholder_with_value "phase_train->False" --reverse_input_channels --data_type FP16

the result of using model FP32 to Extracting Face Features, the result is correct:

-0.0303093 -0.0069193 0.0673719 0.0191522 0.0839787 0.0183519 0.0102432 -0.016102 -0.0557591 -0.051992 -0.0583747 0.00479728 -0.0458586 0.0157291 0.0380102 0.00401546 0.0291528 -0.0109071 -0.000333144 0.0313997 0.0251056 -0.0896044 0.0290247 -0.0023481 0.0147871 -0.0205698 0.0121236 0.081593 0.034786 0.00569561 0.0616632 0.0607331 -0.0139996 0.0848528 -0.000542952 0.0250864 0.0456123 0.0227162 0.0226587 0.0452475 0.0591343 0.00393377 -0.0399091 -0.0498996 -0.0580236 -0.00902976 -0.0186846 -0.00152146 -0.0372101 -0.0839904 0.036758 0.00357341 -0.0649666 0.00296314 0.0962272 -0.0233325 0.0125292 -0.0407852 -0.0450948 -0.0784623 0.0125671 0.0112286 -0.0514212 -0.00544102 0.0270228 0.00232056 0.000279993 -0.0661163 0.0626728 -0.10613 0.077301 -0.016827 -0.0992868 -0.0455369 -0.00453928 0.00792284 -0.0373775 -0.0640373 0.0195536 -0.0597935 -0.0113326 0.00508127 0.0181655 0.0636499 0.062529 0.0301514 0.048107 -0.0534238 -0.0257291 -0.0561252 0.0531831 0.0143683 -0.051425 -0.0164832 0.00920025 0.0321136 0.040807 0.0280781 -0.0825003 0.0414518 0.0479673 -0.0608024 0.000331064 -0.0401369 0.0643712 -0.0011322 -0.000881701 -0.0502676 -0.0731829 -0.0289641 -0.0143488 -0.048839 0.0548656 0.00623099 0.0884519 0.0180087 0.0158212 -0.0406758 -0.00589386 0.012359 0.0333257 0.0313446 -0.0185623 -0.00757585 -0.0269076 0.0576905 -0.0665087 -0.0814487 -0.0814507 0.015162 0.0448395 0.0958095 0.0249909 -0.0140888 -0.0383492 -0.0339562 -0.0352145 0.0104023 -0.0567233 0.0236233 0.0766157 -0.0119436 -0.0391956 0.0293403 0.047081 -0.0376553 0.0582097 0.0723744 -0.00966117 0.0380036 -0.0126938 -0.0100318 -0.0386953 -0.0247702 0.0182403 -0.0371625 0.0295725 0.0225616 0.027327 -0.00114575 -0.0277443 -0.0284961 -0.0794621 0.0106357 0.00939375 0.0126513 -0.0326162 0.0149674 -0.020906 0.0244328 -0.0137134 0.0433781 0.0773455 -0.00312938 0.0597768 0.0405744 -0.0656238 0.0265835 0.0701471 -0.0561795 -0.0721568 -0.0296266 0.0442357 -0.0472046 0.0329284 0.0624285 0.0375223 -0.00373118 -0.028729 -0.0451934 -0.0175522 -0.0525724 0.03412 0.0443992 -0.0146092 -0.0129767 0.0286949 -0.0785776 0.0389373 0.0334287 0.00540069 0.0116089 0.0581521 0.000463655 -0.0484372 -0.0657498 -0.0258909 -0.052913 -0.00353711 -0.077772 0.0546322 -0.00486849 -0.0908816 -0.0351685 -0.0475095 0.0124364 0.0353708 -0.0225474 -0.0629055 -0.0877562 -0.0100317 -0.0242093 -0.000984329 0.0438599 0.02429 0.0275248 0.0591982 -0.00989004 -0.0318442 0.0322847 0.0584652 -0.0359303 0.0249034 -0.0299864 0.0327249 0.00363593 0.0211814 0.0431675 -0.0475301 -0.074582 0.0414153 0.0426292 -0.088903 0.0184509 -0.0773675 0.0624114 0.0184458 0.0212691 0.0385956 -0.0132264 0.000994564 0.0274678 0.0114034 0.0249954 0.0347586 -0.0578911

the result of using model FP32 to Extracting Face Features, the result is incorrect:

-1 -1 1 1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1 1 -1 -1 1 1 -1 1 -1 1 -1 1 1 1 1 1 1 -1 1 -1 1 1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 -1 -1 1 1 -1 1 1 -1 1 -1 -1 -1 1 1 -1 -1 1 1 1 -1 1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 -1 1 1 1 1 1 1 -1 -1 -1 1 1 -1 -1 1 1 1 1 -1 1 1 -1 1 -1 1 -1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 -1 -1 1 1 1 -1 -1 -1 1 -1 -1 -1 1 1 1 1 -1 -1 -1 -1 1 -1 1 1 -1 -1 1 1 -1 1 1 -1 1 -1 -1 -1 -1 1 -1 1 1 1 -1 -1 -1 -1 1 1 1 -1 1 -1 1 -1 1 1 -1 1 1 -1 1 1 -1 -1 -1 1 -1 1 1 1 -1 -1 -1 -1 -1 1 1 -1 -1 1 -1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 1 -1 -1 -1 -1 -1 -1 1 1 1 1 -1 -1 1 1 -1 1 -1 1 1 1 1 -1 -1 1 1 -1 1 -1 1 1 1 1 -1 1 1 1 1 1 -1 -1 -1 1 -1 1 -1 1 1 1 -1 -1 1 1 1 -1 1 1 1 1 -1 -1 1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 1 -1 1 -1 -1 -1 1 1 -1 1 1 1 1 -1 1 -1 1 -1 1 1 1 1 -1 -1 1 1 -1 1 -1 1 1 -1 1 -1 -1 -1 1 -1 1 -1 -1 1 1 1 1 -1 1 1 -1 1 1 -1 -1 1 -1 -1 -1 -1 1 1 -1 1 -1 1 1 1 1 1 1 1 1 -1 1 1 -1 -1 -1 1 -1 -1 1 1 1 1 1 1 1 1 1 -1 1 -1 -1 1 1 -1 -1 -1 -1 -1 -1 1 -1 -1 -1 1 1 1 1 1 1 -1 1 1 1 -1 -1 -1 1 -1 1 1 1 1 -1 -1 -1 1 -1 -1 1 -1 -1 1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 -1 -1 1 -1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1 1 -1 -1 1 -1 1 -1 -1 1 -1 1 1 -1 -1 -1 1 -1 1 -1 -1 -1 1 -1 -1 1 -1 -1 -1 -1 1 1 1 -1 -1 -1 -1 1 -1 -1 -1 1 1 -1 -1 1 1 -1 1 1 1 1 -1 -1 -1 -1 1 -1 1 

What should I do to transform the FP16 model correctly?
Thanks for help~

[JIANHUI_D_Intel]

I have the same problem. Anyone can help?

Thank you.

[JIANHUI_D_Intel]

The same facenet FP16 model runs successfully on GPU plugin, but failed on NCS movidius plugin.

So the model itself is good, and perhaps not supported by NCS movidius plugin?

[qiqiqi__qiqiqi]

When testing interactive_face_detection_demo, R4 and R5 package both got wrong result on NCS device with facial-landmarks-35-adas-0001.xml model.

Can any one help? Thank you.

[shakya__sanjay]

I think that FP16 model of facenet is giving incorrect results due to normalization layer in the model definition file(facenet.xml) at line no.6670 <layer id="314" name="normalize" precision="FP16" type="Normalize"><data across_spatial="0" channel_shared="0" eps="1.000000013351432e-10"/> and line 7054 <edge from-layer="313" from-port="3" to-layer="314" to-port="0"/> . There can be two solutions
1. comment the normalization layer

2.change the eps value

On commenting the normalization layer .I am getting correct facenet array .But Still the result doesnot match orginal tensorflow resultant array.

Orginal Tensorflow result

[[-0.02016883  0.07294131 -0.06137365  0.13047621 -0.10882749  0.10714169
  -0.08949636  0.05453004  0.05977161  0.05744238 -0.07827419 -0.04914774
   0.0793832  -0.13196917 -0.07499827 -0.00375911  0.05725604  0.04510152
   0.00851381  0.03481326  0.02916214  0.23850687  0.01904107  0.04875557
  -0.01149567  0.03791498 -0.03363566 -0.02189376  0.01407445  0.05537404
  -0.09964523  0.02504354 -0.03001506 -0.10081876  0.24061945 -0.09872237
  -0.10051446  0.01396536  0.13352244  0.02885238 -0.05039185  0.04528062
  -0.09655323 -0.00528629  0.04987795  0.03139931  0.08220865 -0.07368948
   0.02490031  0.16983372  0.00643123  0.05590355  0.09619643 -0.11701696
   0.05046166 -0.01447951  0.1609236  -0.19752064 -0.19571348 -0.09357896
   0.02159576 -0.17709012  0.0476344  -0.14661884  0.0190397  -0.05705429
   0.08652931  0.03676815 -0.06852241 -0.15693901  0.00259933  0.05497146
  -0.08401819 -0.08796083  0.17946047 -0.07930267 -0.10966888  0.06514623
  -0.00162298  0.02012202  0.10749369 -0.07394527  0.03188538  0.06221956
   0.11273629  0.12770137 -0.04906961  0.02203626 -0.16503599  0.00410052
   0.05542625  0.11863745 -0.0453809   0.11116403  0.04378612  0.04046849
   0.03046808  0.09850886 -0.13603202 -0.18397573  0.07327073  0.02570553
  -0.03675752 -0.04966513 -0.09762211 -0.09500827 -0.13967887  0.07866212
  -0.02596954 -0.03171893  0.00724166  0.01481481  0.02925692  0.09408037
  -0.08739417 -0.07733101 -0.01750433 -0.08060329  0.07513442  0.07714925
   0.00189818 -0.143401   -0.0672382  -0.08290038  0.01673577 -0.05139204
  -0.1433048   0.00261288]]

Openvino implementation

[[-0.19702148 -0.45507812 -0.5395508  -0.43554688  0.21484376 -0.484375
   0.04431152 -0.17773438 -0.04174805 -0.15100098  0.05651856 -0.36376953
  -0.05297852 -0.41918945 -0.1989746   0.10803223  0.20349123 -0.37622073
   0.1373291  -0.6767578  -0.43188477 -0.05578614 -0.29223633  0.04187012
   0.46679688 -0.04846192  0.58203125  0.1838379  -0.2619629  -0.20043945
  -0.46264648  0.0402832  -0.10510254 -0.15246582  0.2927246  -0.19384766
   0.26049805  0.08874512  0.31030273 -0.03479004 -0.03210449  0.17834473
  -0.3569336  -0.08972168 -0.4873047  -0.5332031  -0.20629881 -0.3154297
   0.15808105  0.1303711  -0.5551758   0.31176758  0.26611328 -0.5083008
  -0.06896972  0.24511717  0.43847656 -0.09741211 -0.0657959  -0.77441406
  -0.30493164 -0.1496582  -0.35498047  0.31713867 -0.10693359  0.62841797
  -0.02868652  0.10253906  0.07299804 -0.49877927 -0.11938477 -0.37646484
  -0.15771484 -0.7578125  -0.14904785 -0.09399414 -0.21606445  0.09924316
  -0.25268555 -0.6713867  -0.5566406   0.03564453 -0.13452148  0.15979004
   0.47583008  0.09643555  0.1652832  -0.16027832 -0.5727539   0.87597656
   0.22631836 -0.02478027 -0.22497559 -0.05932617  0.01574707  0.1809082
  -0.0493164  -0.6401367  -0.6479492   0.14453125 -0.21252441  0.47802737
   0.33813477 -0.79052734 -0.11120605  0.08398438 -0.37524417  0.10559082
  -0.33422852 -0.5449219   0.19665527  0.05517578  0.27685547  0.15783691
  -0.00549316  0.17114258 -0.24731445  0.34765625  0.41845703  0.15148926
   0.09802246 -0.08325196 -0.11657715 -0.23461914  0.22949217 -0.60595703
   0.26831055 -0.42114255]]

Have anybody solved this issue?

[shakya__sanjay]

As it turns out facenet has also implemented normalization on prelogits in facenet/train_tripletloss line 136 embeddings = tf.nn.l2_normalize(prelogits, 1, 1e-10, name='embeddings').

So it cannot be commented in facenet model definition xml file

Still looking for a solution.

[Zhang__Long]

Anyone solve this problem?

[Tolbert__Donnie]

Hello,

im having this same problem.  I don’t think it’s related to normalization since the signs of the elements of the embedding vector differ between FP16 on Myriad and FP32 on CPU.  

I also noticed that when loading in my FP16 model the inference engine reports that inputs and outputs have precision FP32– I have to call setPrecision to set them to FP16.  I think that could be part of the problem— perhaps the FP16 weights or logits are being interpreted as FP32 during the load or inference?

intel folks: any pointers would be appreciated!

thanks,

-Donnie

[Tolbert__Donnie]

I believe this problem has to do with the normalization algorithms on the NCS with OpenVINO.  I created a notebook to highlight the issue.  Here are the results of a simple normalization layer running on both the NCS via the Myriad plugin, and the CPU using OpenVINO:

CPU: https://gist.github.com/donniet/0d0e8f604eb30aeb0215968dc4e20071

Myriad: https://gist.github.com/donniet/fffdc95526ffa5fa625068db4cdb0507
 

To me it appears that the CPU plugin behaves as expected, but the myriad plugin returns essentially the sign function on each element of the channel vector-- basically x/|x|.  I posted a new topic on the intel forum (hasn't been released by moderators yet) and a post on the movidius forum:

https://ncsforum.movidius.com/discussion/1666/global-response-normalization-difference-between-cpu-and-myraid

[Mas__Juan]

Hello,

I have the same problem with FaceNet. Investigating, I realized FaceNet pre-trained models are all Inception Resnet v1 which does not appear in the compatibility list of models in the Myriad Plugin doc https://docs.openvinotoolkit.org/latest/_docs_IE_DG_supported_plugins_MYRIAD.html , only Inception Resnet v2 is supported. Could this be the source of the problem? David Sandberg published an Inception Resnet v2 version of the model but no pre-trained models. Had anyone tried this model?

Any help from Intel would be appreciated

[shakya__sanjay]

@juan is correct. I have also researched and tested but forgot to reply. I came to the following conclusion.

-David sandberg  trained the Facenet model using Inception Resnet v1.
-Facenet model using Inception Resnet v2 is not available any where.
-Movidius Neural compute stick 2 only support Intel Openvino SDK but Openvino and myriad 2 does not support Inception Resnet v1 architecture.
-NCSDK1 AND NCSDK2 libraries support Inception Resnet v1 Architecture but It doesnot support Movidius Neural compute stick 2 . So If by any chance you have old myriad stick 1 then you can run via NCSDK1 libraries

link- https://ncsforum.movidius.com/discussion/1308/ncsdk-support-ncs-2?Sort=date
link - https://software.intel.com/en-us/articles/OpenVINO-Using-TensorFlow#converting-a-tensorflow-facenet-model

The only solution is to train the model on Inception resnet v2 or to use compute stick myriad 1 until intel allow support for Incpetoin resnet v1 on new myriad devices.

[Shubha_R_Intel]

Dearest shakya, sanjay,

Thank you for the EXCELLENT synopsis. Keep in mind that Intel is no longer supporting NCS1. However I have filed a feature request on your behalf to support inception resnet v1 on NCS2.

 Thanks for your patience,

Shubha

[佟__志林]

@shakya,do you solve the problem that tensorflow resultant array do not match the result from openvino! Thank you!

[Chen__Jiandan]

Since the new release improved batch support, the L2-norm in normalization layer from the model definitions file (facenet.xml) normalized feature vector in wrong dimension. We need change data across_spatial="0" to data across_spatial="1" as the following:

<layer id="314" name="normalize" precision="FP16" type="Normalize">
            <data across_spatial="1" channel_shared="0" eps="1e-10"/> 

[Chen__Jiandan1]

Since the new OpenVino release improved batch support,  the L2-norm in the converted model normalized feature vector in wrong dimension.  The normalize layer in model definition file(facenet.xml) needs change data across_spatial="0" to across_spatial="1" as the following:

 <layer id="314" name="normalize" precision="FP16" type="Normalize">

        <data across_spatial="1" channel_shared="0" eps="1e-10"/>