Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.
6404 Discussions

model optimization using openvino workbench

hussein34345
Beginner
844 Views

i have 4 model that i should optimize the first two are face recognition and face detection after optimizing this two models using int8 default optimization, i got double the fps on my device with 1% to 3% accuracy drop but doing the same process on the other two face spoof models (one is trained on depth images and other on RGB images ) the accuracy dropped about 30% ,so is this normal or dose the problem come from my implementation of the whole process.
more notes to better explain the problem:
1-for each model i used similar dataset that the model was originally trained on.
2-for the detection and spoof models i used a dataset without annotation.
3-for spoof models i checked the accuracy of the models using a pycharm script.
4-i tried different datasets size but the minimum accuracy drop was 30%.
5-am using the openvino DL Workbench

0 Kudos
8 Replies
Iffa_Intel
Moderator
806 Views

Greetings,

 

Besides focusing on the models themselves, you need to properly design and evaluate your inferencing process. The way you handle and interpret models and inputs in the algorithm is really important.

 

You could train proper models, each with specific functions (which you already did), and read the input being fed, frame by frame. This way, an independent analysis of each frame can be done.

You may refer to this Face Recognition Python Demos as an example.

 

Generally, we would prepare frame 1 and inference 1, then move to prepare frame 2 and inference 2. This would cost a longer processing time and waiting for the pipeline to finish in synchronous mode stage by stage can kill the performance. This is how things are done in the synchronous mode.

 

Instead, switch to the asynchronous mode where we send the inference request and directly prepare the next frame. When the inference requests are not blocking the execution, preparation of the frame and inference operation can be done in parallel. This would give in high throughput improvement.

 

You may review these:

  1. OpenVINO synchronous Inference Request
  2. OpenVINO Asynchronous Inference Engine

 

The OpenVINO Inference Engine API supports asynchronous operation on CPU, GPU, VPU and, FPGA. For the application example, you may refer to the OpenVINO Image Classification Async Python Sample.

 

Sincerely,

Iffa

 

0 Kudos
VladimirG
Employee
792 Views

Hi hussein3434,
thank you for reaching out.

 

From the fact that you used a dataset without annotation during optimization process, I can assume that you've quantized your model using the Default algorithm. This algorithm aims to quantize as many layers as it possibly can with no regard for the resulting network's accuracy. This is most likely the cause of the observed drop in accuracy, and to fix that you'll have to quantize using the AccuracyAware algorithm. 


Keep in mind that in order to utilize this option, you'll have to reformat your dataset to include some sort of annotation readable by the Accuracy Checker tool. Since face spoofing boils down to a case of classification with 2 classes, ImageNet seems like the simplest and most fitting choice. You can refer to DL Workbench's documentation on supported dataset types to learn how to properly format your data, then upload your new set to DL Workbench and configure a new project.

 

In that project, provide a valid configuration, then create an accuracy report to verify that the accuracy matches that which you obtain using your own script. Refer to the Accuracy Checker tool documentation in case you need to manually adjust the config using the Advanced mode in DL Workbench.

 

After your report is complete, you may then quantize your model with the aforementioned algorithm and check if the issue still persists.

Regards,
Vladimir

0 Kudos
hussein34345
Beginner
765 Views

thank you for you answers that was really helpful ,but i think that i need a little more help with applying the Accuracy Aware algorithm. am using my dataset for the spoof models (binary classification) and am wondering how should i use the annotation converter to write the annotation file and can i just write it like the image-nets val.txt file? and if so how should the configuration file look like? and what should i write in the dataset: section.

0 Kudos
Iffa_Intel
Moderator
757 Views

Example for annotation conversion usage (describe annotation conversion in conf file):

 

annotation_conversion:
# Converter name which will be called for conversion.
converter: sample
# Converter specific parameters, can be different depend on converter realization.
data_dir: sample/sample_dataset
# (Optional) subsample generation. Can be also used with prepared annotation file.
subsample_size: 1000
# (Optional) paths to store annotation files for following usage. In the next evaluation these files will be directly used instead running conversion.
annotation: sample_dataset.pickle
dataset_meta: sample_dataset.json

 

 

Convert process via cmd:

convert_annotation <converter_name> <converter_specific parameters>

 

You may refer here for more info.

 

Sincerely,

Iffa

0 Kudos
hussein34345
Beginner
744 Views

thank you for sharing this page, but my question is:
which converter should i use for my data(images of real faces and spoof faces) i tried using imageNet as the converter.

and wrote the annotation file like this val.txt:
0001.jpg 0
0002.jpg 0
0003.jpg 0
0004.jpg 0
0005.jpg 1
0006.jpg 1
0007.jpg 1
0008.jpg 1

and the config file is like this:

namemask_attack_model_org
launchers:
  - adapter:
      typeclassification
    batch1
    deviceCPU
    frameworkdlsdk
    model$MODEL_PATH/saved_model.xml
    weights$MODEL_PATH/saved_model.bin
datasets:
  - annotation_conversion:
      annotation_file$DATASET_PATH/val.txt
      converterimagenet
      has_backgroundfalse
      images_dir$DATASET_PATH
    data_source$DATASET_PATH
    metrics:
      - presenterprint_vector
        typeaccuracy
    namedataset
    preprocessing:
      - typeauto_resize
    subsample_size100%
 
After the accuracy report is created am getting no errors but the predicted labels all are 0 but for the same images on my machine it is giving a different result (mixed 0 and 1) ,so the model is not working properly with this configurations so what should i change to solve this?
0 Kudos
VladimirG
Employee
712 Views

The configuration seems solid enough for most classification cases, so the fact that it doesn't produce the expected results may suggest some discrepancies with the script you utilized earlier. 

Things you could try to look into to resolve this are:

  • Use pre-/postprocessing options in Accuracy Checker to match those used in the script, if any (refer to the respective sections in AC documentation provided before)
  • Provide additional arguments in the 'adapter' and 'metrics' sections of the config, suck as top_k for your metric and 'argmax_output' for your adapter (if you use an argmax layer as output)

Other than that, I'd require a bit more information about the model and dataset to determine the cause and try to replicate it locally.

 

P.S. to add to Iffa's suggestion, while the antispoofing converter may be more appropriate for your task, it's quite unlikely that you'll be able to upload such a dataset to DL Workbench due to its current limitation on the range of supported tasks and formats, which is why I originally suggested formatting to Imagenet to circumvent these issues.

 

Regards,
Vladimir

0 Kudos
Iffa_Intel
Moderator
723 Views

It seems that you are using the dataset for spoof model (binary classification),you can use the antispoofing as converter which you could refer from the link here: Annotation Converters — OpenVINO™ documentation

 

You need to define the below parameters in your respective config file:

  • data_dir : path to root folder of the dataset
  • annotation_file : path to json file containing annotations to the dataset ({index: {path:”…”, labels:[…], bbox:[…] (optional), …})
  • label_id : number of label in the annotation file representing spoof/real labels
  • dataset_meta_file : path to json file with dataset meta (e.g. label_map)

 

You may refer to this link on how to write the config file: open_model_zoo/accuracy-check.yml at master · openvinotoolkit/open_model_zoo (github.com)



Sincerely,

Iffa


0 Kudos
Iffa_Intel
Moderator
680 Views

Greetings,


Intel will no longer monitor this thread since we have provided a solution. If you need any additional information from Intel, please submit a new question. 



Sincerely,

Iffa


0 Kudos
Reply