model optimization using openvino workbench

[hussein34345]

i have 4 model that i should optimize the first two are face recognition and face detection after optimizing this two models using int8 default optimization, i got double the fps on my device with 1% to 3% accuracy drop but doing the same process on the other two face spoof models (one is trained on depth images and other on RGB images ) the accuracy dropped about 30% ,so is this normal or dose the problem come from my implementation of the whole process.
more notes to better explain the problem:
1-for each model i used similar dataset that the model was originally trained on.
2-for the detection and spoof models i used a dataset without annotation.
3-for spoof models i checked the accuracy of the models using a pycharm script.
4-i tried different datasets size but the minimum accuracy drop was 30%.
5-am using the openvino DL Workbench

[Iffa_Intel]

Greetings,

Besides focusing on the models themselves, you need to properly design and evaluate your inferencing process. The way you handle and interpret models and inputs in the algorithm is really important.

You could train proper models, each with specific functions (which you already did), and read the input being fed, frame by frame. This way, an independent analysis of each frame can be done.

You may refer to this Face Recognition Python Demos as an example.

Generally, we would prepare frame 1 and inference 1, then move to prepare frame 2 and inference 2. This would cost a longer processing time and waiting for the pipeline to finish in synchronous mode stage by stage can kill the performance. This is how things are done in the synchronous mode.

Instead, switch to the asynchronous mode where we send the inference request and directly prepare the next frame. When the inference requests are not blocking the execution, preparation of the frame and inference operation can be done in parallel. This would give in high throughput improvement.

You may review these:

1. OpenVINO synchronous Inference Request
2. OpenVINO Asynchronous Inference Engine

The OpenVINO Inference Engine API supports asynchronous operation on CPU, GPU, VPU and, FPGA. For the application example, you may refer to the OpenVINO Image Classification Async Python Sample.

Sincerely,

Iffa

[VladimirG]

Hi hussein3434,
thank you for reaching out.

From the fact that you used a dataset without annotation during optimization process, I can assume that you've quantized your model using the Default algorithm. This algorithm aims to quantize as many layers as it possibly can with no regard for the resulting network's accuracy. This is most likely the cause of the observed drop in accuracy, and to fix that you'll have to quantize using the AccuracyAware algorithm. 

Keep in mind that in order to utilize this option, you'll have to reformat your dataset to include some sort of annotation readable by the Accuracy Checker tool. Since face spoofing boils down to a case of classification with 2 classes, ImageNet seems like the simplest and most fitting choice. You can refer to DL Workbench's documentation on supported dataset types to learn how to properly format your data, then upload your new set to DL Workbench and configure a new project.

In that project, provide a valid configuration, then create an accuracy report to verify that the accuracy matches that which you obtain using your own script. Refer to the Accuracy Checker tool documentation in case you need to manually adjust the config using the Advanced mode in DL Workbench.

After your report is complete, you may then quantize your model with the aforementioned algorithm and check if the issue still persists.

Regards,
Vladimir

[hussein34345]

thank you for you answers that was really helpful ,but i think that i need a little more help with applying the Accuracy Aware algorithm. am using my dataset for the spoof models (binary classification) and am wondering how should i use the annotation converter to write the annotation file and can i just write it like the image-nets val.txt file? and if so how should the configuration file look like? and what should i write in the dataset: section.

[Iffa_Intel]

Example for annotation conversion usage (describe annotation conversion in conf file):

annotation_conversion:
# Converter name which will be called for conversion.
converter: sample
# Converter specific parameters, can be different depend on converter realization.
data_dir: sample/sample_dataset
# (Optional) subsample generation. Can be also used with prepared annotation file.
subsample_size: 1000
# (Optional) paths to store annotation files for following usage. In the next evaluation these files will be directly used instead running conversion.
annotation: sample_dataset.pickle
dataset_meta: sample_dataset.json

Convert process via cmd:

convert_annotation <converter_name> <converter_specific parameters>

You may refer here for more info.

Sincerely,

Iffa

[hussein34345]

thank you for sharing this page, but my question is:
which converter should i use for my data(images of real faces and spoof faces) i tried using imageNet as the converter.

and wrote the annotation file like this val.txt:
0001.jpg 0
0002.jpg 0
0003.jpg 0
0004.jpg 0
0005.jpg 1
0006.jpg 1
0007.jpg 1
0008.jpg 1

and the config file is like this:

name: mask_attack_model_org

launchers:

  - adapter:

      type: classification

    batch: 1

    device: CPU

    framework: dlsdk

    model: $MODEL_PATH/saved_model.xml

    weights: $MODEL_PATH/saved_model.bin

datasets:

  - annotation_conversion:

      annotation_file: $DATASET_PATH/val.txt

      converter: imagenet

      has_background: false

      images_dir: $DATASET_PATH

    data_source: $DATASET_PATH

    metrics:

      - presenter: print_vector

        type: accuracy

    name: dataset

    preprocessing:

      - type: auto_resize

    subsample_size: 100%

After the accuracy report is created am getting no errors but the predicted labels all are 0 but for the same images on my machine it is giving a different result (mixed 0 and 1) ,so the model is not working properly with this configurations so what should i change to solve this?

[VladimirG]

The configuration seems solid enough for most classification cases, so the fact that it doesn't produce the expected results may suggest some discrepancies with the script you utilized earlier. 

Things you could try to look into to resolve this are:

• Use pre-/postprocessing options in Accuracy Checker to match those used in the script, if any (refer to the respective sections in AC documentation provided before)
• Provide additional arguments in the 'adapter' and 'metrics' sections of the config, suck as top_k for your metric and 'argmax_output' for your adapter (if you use an argmax layer as output)

Other than that, I'd require a bit more information about the model and dataset to determine the cause and try to replicate it locally.

P.S. to add to Iffa's suggestion, while the antispoofing converter may be more appropriate for your task, it's quite unlikely that you'll be able to upload such a dataset to DL Workbench due to its current limitation on the range of supported tasks and formats, which is why I originally suggested formatting to Imagenet to circumvent these issues.

Regards,
Vladimir

[Iffa_Intel]

It seems that you are using the dataset for spoof model (binary classification),you can use the antispoofing as converter which you could refer from the link here: Annotation Converters — OpenVINO™ documentation

You need to define the below parameters in your respective config file:

• data_dir : path to root folder of the dataset
• annotation_file : path to json file containing annotations to the dataset ({index: {path:”…”, labels:[…], bbox:[…] (optional), …})
• label_id : number of label in the annotation file representing spoof/real labels
• dataset_meta_file : path to json file with dataset meta (e.g. label_map)

You may refer to this link on how to write the config file: open_model_zoo/accuracy-check.yml at master · openvinotoolkit/open_model_zoo (github.com)

Sincerely,

Iffa

[Iffa_Intel]

Greetings,

Intel will no longer monitor this thread since we have provided a solution. If you need any additional information from Intel, please submit a new question. 

Sincerely,

Iffa