Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.
6392 Discussions

How to optimize models for Myriad VPU by using openvino pot or nncf

hussein34345
Beginner
619 Views

I have 4 models that am running on the intel UP Squared card but am having a problem with getting my optimized models to work on it, the fps when the program starts is about 1-2 fps ,so i optimized the models using the OpenVino pot toolkit ,but recently after discovering that Fake Quantize layers are not supported on Myriad VPU device so am looking for alternative ways to speed up my models with minimum accuracy drop, my questions are:

  1. i have read that nncf have multiple compression algorithms so can i use one of them to optimize my models and get it working on my vpu device? if so which one can i use?
  2. can i load a pretrained model and apply nncf optimization on it with out the need of re-training it?
0 Kudos
2 Replies
Zulkifli_Intel
Moderator
578 Views

Hello Hussein,

 

Thank you for reaching you to us.

 

Low precision 8-bit inference is not supported for the Myriad plugin, as mentioned here.

 

Quantization-Aware Training using OpenVINO compatible training frameworks supports models written on TensorFlow QAT or PyTorch NNCF, with optimization extensions.

 

The Quantize Aware-Training is based on the Fake Quantization operation which, in turn, can be represented by a pair of Quantize/Dequantize operations. The important feature of the proposed software solution is the automatic insertion of the Fake Quantization operations, and Fake Quantization layers are automatically inserted in the model graph. Therefore, you cannot use quantized models trained using NNCF with Myriad plugin.

 

For your second question, you must retrain the model to use the NNCF. The following paper, Introducing a Training Add-on for OpenVINO toolkit: Neural Network Compression Framework, explains the steps needed to use NNCF to implement optimization methods using supported training samples as well as through integration into the custom training code.

 

The training samples are available at:

https://github.com/openvinotoolkit/nncf/tree/develop/examples

 

Sincerely,

Zulkifli 


0 Kudos
Zulkifli_Intel
Moderator
526 Views

Hello Hussein,


This thread will no longer be monitored since we have provided a solution. If you need any additional information from Intel, please submit a new question.


Sincerely,

Zulkifli


0 Kudos
Reply