Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.
6392 Discussions

FP16 model Inference on GPU gives all Nan values in output array

iJema
Novice
1,575 Views

Hi there,

 

I'm working on a project that involves Token2Token-Vision Transformer for classification task. Information on model here. During conversion from Pytorch weights to IR through onnx, some layers weren't supported with opset version 9, but I managed to export with opset version 12. INT8 & FP16 model works without any problem, but FP16 GPU inference outputs all Nan values. Is it because of version incompatibility? 

 

I'm using the latest version of Openvino 2022.1.  Please see the attached screenshot.  Any suggestion would be appreciated.

 

Thanks and regards

iJema

Labels (2)
0 Kudos
5 Replies
IntelSupport
Community Manager
1,543 Views

Hi iJema,

 

Thanks for reaching out.

 

Does the Nan values output occurs only when running on GPU? From the list of T2T-Vit Models, which specified model you are using?

 

Please share all the necessary files and inputs to reproduce this issue from our end. You can put all the files in Google Drive and share the link here or privately to my email:

noor.aznie.syaarriehaahx.binti.baharuddin@intel.com

 


Regards,

Aznie


0 Kudos
iJema
Novice
1,525 Views

Hi Aznie,

 

Sorry for late replying. Yes, the Nan output occurs only when running on GPU. Also, I specifically tested T2T-VIT-14 model. Please find the code here . The test_vmmr_img.py has been set up for GPU inference. Just run the file, in console you should be able to see the output from model. Please let me know if you face any problem running the code.

 

Thanks and regards,

iJema

0 Kudos
IntelSupport
Community Manager
1,508 Views

Hi iJema,

 

I observed the same output from my end and also encountered an IndexError when running the FP16 precision model on GPU.

fp16_gpu_LI (2).jpg

 

The performance of T2T-VIT-14 with FP16 precision on GPU was expected and the error encountered was due to missing transposes to MatMul. Our development team is working to rectify this and potentially, the fixes will be available in the next releases.

 

For the 2022.1 release, I would suggest you use INT8 with GPU instead.

 

 

Regards,

Aznie

 

iJema
Novice
1,458 Views

Hi @IntelSupport 

 

Sorry for late replying and thanks for the suggestion! I'll try the model again with next release of OpenVino.

 

Regards,

iJema

0 Kudos
IntelSupport
Community Manager
1,493 Views

Hi iJema,


This thread will no longer be monitored since we have provided a solution. If you need any additional information from Intel, please submit a new question.



Regards,

Aznie


0 Kudos
Reply