Is there any similar application as the Validation Application in OpenVINO 2019_R1.01 Version?

[IDPBY]

Hi, I am using OpenVINO 2021.4 version. I am looking for a similar application as the Validation Application in OpenVINO 2019_R1.01 (https://docs.openvino.ai/2019_R1.01/_inference_engine_samples_validation_app_README.html). I would like to measure the AP of each class of my custom object detection model (Figure below). Is there any recommendation for it?

The Figure above is taken from the example in attached link.

Thank you.

[Megat_Intel]

Hi IDPBY,

Thank you for reaching out to us

For OpenVINO™ 2021.4 release, you can use Accuracy Checker to measure the Average Precision (AP) of each class and Mean Average Precision (map). Details on usage of Accuracy Checker can be found here.

Please refer to Accuracy Checker Metrics for details on the metrics you want to use. Note that the metrics require specific representation format. (eg. map expects detection annotation and detection prediction for evaluation).

Every metric has parameters available for configuration. The metric and its parameters are set through the configuration file.

Regards,

Megat

[IDPBY]

Hi @Megat_Intel ,

Thanks for your solution. I have tried the Accuracy Checker to measure the AP and mAP of my custom model using different devices (eg: CPU and HDDL). I found out that there is a slight difference in the AP and mAP for different devices. Is this normal, and what is the reason of this difference?

CPU:

AP (CPU device)

HDDL:

AP (HDDL device) 

 

By the way, in the list of supported adapters in AccuracyChecker, I could not find any adapter for YOLOv4 model. Does the adapter for YOLOv3 and YOLOv4 share the same thing, which is yolo_v3?

Thank you.

[Megat_Intel]

Hi IDPBY,

Based on your results, the mAP@mean difference between the CPU and HDDL is 0.04%. This result is within acceptable range, as the accuracy difference between any of the target platforms and the reference metrics should be within 1%. 

Different target platform devices such as CPU and HDDL may work differently to get results which contribute to slightly different results.

For the adapter, you are correct, there is no adapter specifically for the YOLOv4 model. You can use the adapter for YOLOv3, however, you may need to modify some parameters. Refer to Yolo_V4 Accuracy-check.yml for our public pre-trained yolo-v4-tf model configuration file template which uses YOLOv3 adapter with modified parameters.

Regards,

Megat

[Hairul_Intel]

Hi IDPBY,

This thread will no longer be monitored since we have provided explanation and suggestion. If you need any additional information from Intel, please submit a new question.

Regards,

Hairul