Intel® Distribution of OpenVINO™ Toolkit
Community support and discussions about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all things computer vision-related on Intel® platforms.
5924 Discussions

Is there any similar application as the Validation Application in OpenVINO 2019_R1.01 Version?

IDPBY
Beginner
143 Views

Hi, I am using OpenVINO 2021.4 version. I am looking for a similar application as the Validation Application in OpenVINO 2019_R1.01 (https://docs.openvino.ai/2019_R1.01/_inference_engine_samples_validation_app_README.html). I would like to measure the AP of each class of my custom object detection model (Figure below). Is there any recommendation for it?

IDPBY_0-1655815143739.png

The Figure above is taken from the example in attached link.

Thank you.

0 Kudos
4 Replies
Megat_Intel
Moderator
119 Views

Hi IDPBY,

Thank you for reaching out to us

 

For OpenVINO™ 2021.4 release, you can use Accuracy Checker to measure the Average Precision (AP) of each class and Mean Average Precision (map). Details on usage of Accuracy Checker can be found here.

 

Please refer to Accuracy Checker Metrics for details on the metrics you want to use. Note that the metrics require specific representation format. (eg. map expects detection annotation and detection prediction for evaluation).

 

Every metric has parameters available for configuration. The metric and its parameters are set through the configuration file.

 

 

Regards,

Megat


IDPBY
Beginner
95 Views

Hi @Megat_Intel ,

 

Thanks for your solution. I have tried the Accuracy Checker to measure the AP and mAP of my custom model using different devices (eg: CPU and HDDL). I found out that there is a slight difference in the AP and mAP for different devices. Is this normal, and what is the reason of this difference?

 

CPU:

AP (CPU device)AP (CPU device)

 

HDDL:

AP (HDDL device)AP (HDDL device)

By the way, in the list of supported adapters in AccuracyChecker, I could not find any adapter for YOLOv4 model. Does the adapter for YOLOv3 and YOLOv4 share the same thing, which is yolo_v3?

 

Thank you.

 

Megat_Intel
Moderator
65 Views

Hi IDPBY,

Based on your results, the mAP@mean difference between the CPU and HDDL is 0.04%. This result is within acceptable range, as the accuracy difference between any of the target platforms and the reference metrics should be within 1%. 

 

Different target platform devices such as CPU and HDDL may work differently to get results which contribute to slightly different results.

 

For the adapter, you are correct, there is no adapter specifically for the YOLOv4 model. You can use the adapter for YOLOv3, however, you may need to modify some parameters. Refer to Yolo_V4 Accuracy-check.yml for our public pre-trained yolo-v4-tf model configuration file template which uses YOLOv3 adapter with modified parameters.

 

 

Regards,

Megat


Hairul_Intel
Moderator
33 Views

Hi IDPBY,

This thread will no longer be monitored since we have provided explanation and suggestion. If you need any additional information from Intel, please submit a new question.

 

 

Regards,

Hairul


Reply