Low performance on quantized models and dynamic batch models

[Shravanthi]

Hi,

I converted keras InceptionV3 to IR format (multiple batches upto 256) and collected inferences for all the batches. There was no improvement in performance after batching, the results were similar for all batches, seems like batching has no impact here. Also after applying quantization for InceptionV3 when run inference on GPU the numbers got decreased instead of increasing, for CPU it was increasing. I have attached model conversion, inference and quantization scripts along with results sheet. Could you please help why the performance is so low ?

And the dynamic batching should improve performance right ? I tried applying dynamic batching in 2 ways

1. Using inference_engine it throws error '[GPU] Attempt to set user property DYN_BATCH_ENABLED (YES) which was not regiseterd' 

2. The Source Tensorflow model input accepts dynamic batch_size, same was converted to IR format then read the model to runtime environment using core.read_model() followed by reshaping to specific batch size. The results were way off (it was less than 1) 

Regards,

Shravanthi J

[Aznie_Intel]

Hi Shravanthi J,

Thanks for reaching out.

We are checking on this and will update you with the information soon. Can you elaborate your workaround for us to validate this from our end.

Meanwhile, there are two alternatives in case of substantial accuracy degradation after applying quantization method:

1. Hyperparameters tuning.
2. AccuracyAwareQuantization algorithm

Refer to Improving accuracy after the Default Quantization.

Regards,

Aznie

[Aznie_Intel]

Hi Shravanthi J,

Thank you for your question. If you need any additional information from Intel, please submit a new question as this thread is no longer being monitored. 

Regards,

Aznie