How to convert a pruned Keras model to IR without losing the model sparsity?

[publio]

Hello! I have a Keras DNN model that I am trying to compare with Google's Coral Edge TPU. What I am doing is after training i am using tensorflow pruning tool to add sparsity my model and compress it and then converting it to OpenVino format and TFLite. 

Although, for the Edge TPU case, Tensorflow has a pruning-quantization-aware training method that lets us quantize the model before converting the model to the accelerator required file format. 

My model has a test accuracy of 91% before converting to NCS2, after pruning I get 91.33%. However, when reducing the model precision to float16 for the NCS2 device, I get 91%  test accuracy.

Whereas on Edge TPU, I get 93.33% using the PQAT technique.   From what I deduced, is that the OpenVino tool will not keep the sparsity from a pruned tensorflow model when reducing the model precision. I believe my trained model may benefit from pruning + precision reduction in terms of test accuracy. 

I'd like to know if there is any way to preserve the model sparsity after I use pruning and convert it to IR format?

[Wan_Intel]

Hi Publio,

Thanks for reaching out to us.

Based on your description, OpenVINO™ Model Optimizer does not preserve the model sparsity that was pruned with preserving quantization-aware training (PQAT). We regret to inform you that PQAT is not validated with OpenVINO™.

For your information, Neural Network Compression Framework (NNCF) provides a suite of advanced algorithms for neural network optimization in OpenVINO™ with minimal accuracy drop. It supports various compression algorithms that support TensorFlow as shown as follows:

• Quantization
• Sparsity
• Filter pruning

Hope it helps.

Regards,

Wan

[Wan_Intel]

Hi Publio,

Thanks for your question.

This thread will no longer be monitored since we have provided a suggestion. 

If you need any additional information from Intel, please submit a new question.

Regards,

Wan