- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
i have 4 model that i should optimize the first two are face recognition and face detection after optimizing this two models using int8 default optimization, i got double the fps on my device with 1% to 3% accuracy drop but doing the same process on the other two face spoof models (one is trained on depth images and other on RGB images ) the accuracy dropped about 30% ,so is this normal or dose the problem come from my implementation of the whole process.
more notes to better explain the problem:
1-for each model i used similar dataset that the model was originally trained on.
2-for the detection and spoof models i used a dataset without annotation.
3-for spoof models i checked the accuracy of the models using a pycharm script.
4-i tried different datasets size but the minimum accuracy drop was 30%.
5-am using the openvino DL Workbench
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Greetings,
Besides focusing on the models themselves, you need to properly design and evaluate your inferencing process. The way you handle and interpret models and inputs in the algorithm is really important.
You could train proper models, each with specific functions (which you already did), and read the input being fed, frame by frame. This way, an independent analysis of each frame can be done.
You may refer to this Face Recognition Python Demos as an example.
Generally, we would prepare frame 1 and inference 1, then move to prepare frame 2 and inference 2. This would cost a longer processing time and waiting for the pipeline to finish in synchronous mode stage by stage can kill the performance. This is how things are done in the synchronous mode.
Instead, switch to the asynchronous mode where we send the inference request and directly prepare the next frame. When the inference requests are not blocking the execution, preparation of the frame and inference operation can be done in parallel. This would give in high throughput improvement.
You may review these:
The OpenVINO Inference Engine API supports asynchronous operation on CPU, GPU, VPU and, FPGA. For the application example, you may refer to the OpenVINO Image Classification Async Python Sample.
Sincerely,
Iffa
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi hussein3434,
thank you for reaching out.
From the fact that you used a dataset without annotation during optimization process, I can assume that you've quantized your model using the Default algorithm. This algorithm aims to quantize as many layers as it possibly can with no regard for the resulting network's accuracy. This is most likely the cause of the observed drop in accuracy, and to fix that you'll have to quantize using the AccuracyAware algorithm.
Keep in mind that in order to utilize this option, you'll have to reformat your dataset to include some sort of annotation readable by the Accuracy Checker tool. Since face spoofing boils down to a case of classification with 2 classes, ImageNet seems like the simplest and most fitting choice. You can refer to DL Workbench's documentation on supported dataset types to learn how to properly format your data, then upload your new set to DL Workbench and configure a new project.
In that project, provide a valid configuration, then create an accuracy report to verify that the accuracy matches that which you obtain using your own script. Refer to the Accuracy Checker tool documentation in case you need to manually adjust the config using the Advanced mode in DL Workbench.
After your report is complete, you may then quantize your model with the aforementioned algorithm and check if the issue still persists.
Regards,
Vladimir
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
thank you for you answers that was really helpful ,but i think that i need a little more help with applying the Accuracy Aware algorithm. am using my dataset for the spoof models (binary classification) and am wondering how should i use the annotation converter to write the annotation file and can i just write it like the image-nets val.txt file? and if so how should the configuration file look like? and what should i write in the dataset: section.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Example for annotation conversion usage (describe annotation conversion in conf file):
annotation_conversion:
# Converter name which will be called for conversion.
converter: sample
# Converter specific parameters, can be different depend on converter realization.
data_dir: sample/sample_dataset
# (Optional) subsample generation. Can be also used with prepared annotation file.
subsample_size: 1000
# (Optional) paths to store annotation files for following usage. In the next evaluation these files will be directly used instead running conversion.
annotation: sample_dataset.pickle
dataset_meta: sample_dataset.json
Convert process via cmd:
convert_annotation <converter_name> <converter_specific parameters>
You may refer here for more info.
Sincerely,
Iffa
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
thank you for sharing this page, but my question is:
which converter should i use for my data(images of real faces and spoof faces) i tried using imageNet as the converter.
and wrote the annotation file like this val.txt:
0001.jpg 0
0002.jpg 0
0003.jpg 0
0004.jpg 0
0005.jpg 1
0006.jpg 1
0007.jpg 1
0008.jpg 1
and the config file is like this:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The configuration seems solid enough for most classification cases, so the fact that it doesn't produce the expected results may suggest some discrepancies with the script you utilized earlier.
Things you could try to look into to resolve this are:
- Use pre-/postprocessing options in Accuracy Checker to match those used in the script, if any (refer to the respective sections in AC documentation provided before)
- Provide additional arguments in the 'adapter' and 'metrics' sections of the config, suck as top_k for your metric and 'argmax_output' for your adapter (if you use an argmax layer as output)
Other than that, I'd require a bit more information about the model and dataset to determine the cause and try to replicate it locally.
P.S. to add to Iffa's suggestion, while the antispoofing converter may be more appropriate for your task, it's quite unlikely that you'll be able to upload such a dataset to DL Workbench due to its current limitation on the range of supported tasks and formats, which is why I originally suggested formatting to Imagenet to circumvent these issues.
Regards,
Vladimir
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It seems that you are using the dataset for spoof model (binary classification),you can use the antispoofing as converter which you could refer from the link here: Annotation Converters — OpenVINO™ documentation
You need to define the below parameters in your respective config file:
- data_dir : path to root folder of the dataset
- annotation_file : path to json file containing annotations to the dataset ({index: {path:”…”, labels:[…], bbox:[…] (optional), …})
- label_id : number of label in the annotation file representing spoof/real labels
- dataset_meta_file : path to json file with dataset meta (e.g. label_map)
You may refer to this link on how to write the config file: open_model_zoo/accuracy-check.yml at master · openvinotoolkit/open_model_zoo (github.com)
Sincerely,
Iffa
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Greetings,
Intel will no longer monitor this thread since we have provided a solution. If you need any additional information from Intel, please submit a new question.
Sincerely,
Iffa
![](/skins/images/7B0AB6865064EAE32B30EC9A4E94B48A/responsive_peak/images/icon_anonymous_message.png)
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page