Openvino Issue Calibration Tool

[V_Marco]

Hi,

I'm using openvino to accelerate my depth estimation network, i followed all the steps and generate IR files and done the inference, and it works all fine.

Now i want to use  Low-Precision 8-bit Integer Inference to calibrate my network but i don't understand if it's possible, i only found instruction for classification and obj detection. now i'd like to know if it's possible to use accuracy checker and calibration tool using a KITTI depth estimation dataset and custom adapters and configuration files. I also tried to use calibration tool in simplified mode but the output is totally uncorrect(only a yellow screen).

So i can use openvino to calibrate my custom model(depth estimation using disparity maps) on a custom dataset or it's not possible at the moment?

and if it's possible there are any instructions to set the correct parameters?

[V_Marco]

any advice ?

[V_Marco]

any advice?

[HemanthKum_G_Intel]

Hi Marco,

Calibration Tool has flexible and extensible mechanism of enabling new data set and metrics. Each network has its own dedicated network metric and dataset where network was trained. Dataset description and network metrics can be reused for different network.

To plug new dataset you need to develop YML file. To develop new metric you need to develop Python* module implementing metric and describe in YML. Please, refer to Accuracy Checker Tool for details.

[V_Marco]

Hi,

please could you give me some indication on which file i have to modify to generate a custom accuracy checking for post training quantization?  it's not a topology supported by accuracy checker, i have custom dataset  and for groundtruth i have image disparity(.npy) for depth estimation (that are contained in a folder). I'm stucked on which files i have to extend (representation,annotation converter, metrics, adapter ...) so a workflow will be awesome

[Max_L_Intel]

Hello Marco.

If you already have your FP32 or FP16 model trained, then you could proceed with a calibration tool.
Please see this step-by-step guide starting with prerequisites section - https://docs.openvinotoolkit.org/2019_R3.1/_inference_engine_tools_calibration_tool_README.html
And also the first post from Shubha with the summary of what needs to be done in this topic https://github.com/opencv/dldt/issues/171 

You could also check the following page with the example of generating a dataset in DL Workbench - https://docs.openvinotoolkit.org/latest/_docs_Workbench_DG_Generate_Datasets.html

Hope this helps.

Best regards, Max.

[V_Marco]

I've already read that post, but, analyzing the classes, in order to try to fit my network on the tool  i need to modify all classes (representation, adapters, annotation, metrics) and as i can see accuracy checker needs to be modified correctly to work and , so it's feasible to try to modify internal classes? and if so there is a template (i've only saw a Sampleannotation.py) but for the others there isn't a workflow

[Max_L_Intel]

Hello Marco.

You should be able to edit these functions you pointed as well. Unfortunately, we don't have a template for this one, but you could find some additional information on them in the following links:

Adapters - https://docs.openvinotoolkit.org/2019_R3.1/_tools_accuracy_checker_accuracy_checker_adapters_README.html
Annotation converters - https://docs.openvinotoolkit.org/2019_R3.1/_tools_accuracy_checker_accuracy_checker_annotation_converters_README.html
Metrics and representation - https://docs.openvinotoolkit.org/2019_R3.1/_tools_accuracy_checker_accuracy_checker_metrics_README.html

Please also have a chance to learn and test a completely new calibration tool within Post-Training Optimization toolkit as a part of OpenVINO toolkit latest 2020.1 build - http://docs.openvinotoolkit.org/latest/_README.html
You can download it here https://software.intel.com/en-us/openvino-toolkit/choose-download