AccuracyChecker in DefaultQuantization mode - doubt

[Buvana_R]

Hello,

I understand that the POTS tool supports DefaultQuantization with engines, AccuracyCheck and Simplified:

https://docs.openvinotoolkit.org/latest/pot_docs_BestPractices.html 

"The DefaultQuantization algorithm implies two different usage scenarios (engines

• With AccuracyChecker under the hood. Here the POT relies on the AccuracyChecker execution flow so that all the required settings (the same as for FP32 model) must be provided in order to get correct results for the model.
• As a part of Simplified mode with minimum configuration effort. Currently, this mode can be used when the model or dataset are not supported by the AccuracyChecker tool but it has several limitations:"
  
  

Aside from DefaultQuantization, I see that there is an AccuracyAware Quantization and I fully understand its purpose and operations.

However, what I do not understand is the need for the 'AccuracyChecker' while operating the DefaultQuantization algorithm. When does it kick in? It does not appear that the algorithm 'needs' to evaluate the accuracy while performing the quantization (unlike the AccuracyAware Quantization algorithm)

In my experience with the SSD512 model (from mxnet), DefaultQuantization with 'AccuracyChecker' and 'Simplfied' engines produces quantized models that have accuracy within 0.5% of each other.

So, I am confused about the purpose that the AccuracyCheck engine serves to the DefaultQuantization. Does it kick in only after the quantization is completed?

Kindly explain or point me to more documentation. Your explanation from the web documentation above is unclear.

thanks,

Buvana

[Iffa_Intel]

Hi,

First and foremost, POT and accuracy checker are both tools that have different purposes.

Accuracy Checker tool is used to check/validate a specific model.

The POT quantize a model for better performance (latency, etc)

In POT, you will need 2 configs file:

1. a json file : this is where the DefaultQuantization Algorithm is used
2. a yml file : This is the accuracy_checker config file. You can search according to your model in <openvino_location>\deployment_tool\tools\post_training_optimization_toolkit\open_model_zoo\tools\accuracy_checker\configs
3. Here is how you can use the POT together with the configs

You might want to refer here for more details on accuracy checker tool.

Sincerely,

Iffa

[Iffa_Intel]

Greetings,

Intel will no longer monitor this thread since we have provided a solution. If you need any additional information from Intel, please submit a new question. 

Sincerely,

Iffa