Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.
6480 Discussions

Why is the coco_precision of YOLOv3-tf INT8 higher than that of YOLOv3-tf FP32 with accuracy checker

WT_Jacko
Beginner
4,598 Views

Dear sirs,

I use the accuracy checker to run Yolov3-tf INT8 after POT, and its coco_precision is even higher than Yolov3-tf FP32. It is not in line with my perception.

My test environment information as below:

  1. CPU : Intel Core i5-1145GRE
  2. OS : linux Ubuntu 20.04 LTS, 64bits
  3. Openvino versoin : 2021.3.394
  4. Python version : 3.8.5
  5. Yolov3-tf : https://docs.openvinotoolkit.org/latest/omz_models_model_yolo_v3_tf.html

          IR Model download: https://drive.google.com/file/d/1MiB5W-                                   

          khMNRCSw6sB5tM5MvvUQWyLiJe/view

  1. Coco datasets json file: https://drive.google.com/file/d/1CpEwYdtC0rdYabpiUjWT4rYvmASA5v9e/view?usp=sharing

          Or you can download (instances_val2017.json) here:

          http://images.cocodataset.org/annotations/annotations_trainval2017.zip

  1. Coco datasets JPG file : http://images.cocodataset.org/zips/val2017.zip
  2. Yml file : https://drive.google.com/file/d/1jpDutCxPP6htjIcaH2RoGJhIsAhOQhy4/view?usp=sharing

 Run yoloV3-tf IR FP32 accuracy checker:

jackowt@jackowt:/opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/accuracy_checker$ accuracy_check –c accuracy-check-omg.yml –m /home/openvino/openvino_models/public/yolo-v3-tf/FP32 –s /home/openvino/coco_dataset    

1.png      

 

 

 

 

 

 

 

The result is same as yolo-v3-tf Accuracy as below:

2.png

Question 1:

Accuracy check for INT8.

coco_precision: 67.84% higher than FP32: 67.7% , It doesn’t make sense.

3.png

Question 2:

When accuracy check with INT8, why precision of input info shows FP32?

4.png

Question 3:

Why GPU not work?

5.png

 

Please help me to check.

Thanks for your suppoort.

BR

Jacko Chiu

0 Kudos
13 Replies
Syamimi_Intel
Moderator
4,558 Views

Hi Jacko Chiu,

Thank you for reaching out and sharing the files with us. We are currently investigating this issues and will get back to you. 

 

 

Regards,

Syamimi

 

0 Kudos
Syamimi_Intel
Moderator
4,507 Views

Hi Jacko Chiu,

Thank you for waiting.

This is the answer to your first question:

The accuracy value of the yolo-v3-tf model is validated using the mAP (mean average precision) metric, not coco_precision.

 

Thus, the below outcome of your replication has confirmed our documentation that 8-bit computations (referred to as int8) offer better performance compared to the results of inference in higher precision (for example, fp32) because they allow loading more data into a single processor instruction. Usually, the cost for a significant boost is reduced accuracy.

 

FP32 mAP: 62.95%

 

INT8 mAP: 62.8%

 

Therefore, such reduction is expected and you should not be comparing using coco_precision.

 

Second answer:

I will check this issue with the developer team and get back to you.

 

Third answer:

You need to make sure that your environment supports the GPU plugin. After that, you can have a try to run the Accuracy Checker by using the following command:

accuracy_check -c accuracy-chech.omg.yaml -m /home/openvino/openvino_models/public/yolo-v3-tf/FP32 -td GPU

 

 

 

Regards,

Syamimi


0 Kudos
WT_Jacko
Beginner
4,462 Views

Hi Syamimi,

Thanks for your response.

I  have a little question about your answer to Question 1.

Here is your response:

"The accuracy value of the yolo-v3-tf model is validated using the mAP (mean average precision) metric, not coco_precision."

and Please refer to this link :  https://docs.openvinotoolkit.org/latest/openvino_docs_performance_int8_vs_fp32.html

In this form, the metric name used by yolo_v3-tf is CoCo mAP.

WT_Jacko_0-1624526607770.png

In this form, I have seen that one of the metric names is mAP.

WT_Jacko_1-1624526804711.png

Question 1: 

In your comment, yolo-v3-tf model is validated using the mAPThis is different from the yolo_v3-tf:coco mAP mentioned in the link form.

 

Question 2:

Are coco_precision and coco mAP the same? or Are mAP and coco mAP the same?

 

Question 3:

You say "yolo-v3-tf model is validated using the mAP (mean average precision) metric", Is there any document mentioned? Why can only be MAP, not coco_precision.

 

Thanks your help.

BR

Jacko

 

 

0 Kudos
Syamimi_Intel
Moderator
4,490 Views

Hi Jacko Chiu,

Thank you for waiting. For the question number 2, this is not a bug. It is default behaviour of InferenceEngine Python API to use FP32 as input precision for data.

Such configuration/info was made for user convenience because not all data type are platform agnostics (e.g. FP16), casting from FP32 to real used in model precision happens inside runtime.

 

Without this feature, user may need to manually pre-process data to specific required precision which is might not be trivial in some cases.

 

 

Regards,

Syamimi


0 Kudos
Syamimi_Intel
Moderator
4,437 Views

Hi Jacko Chiu,

Question 1: 

In your comment, yolo-v3-tf model is validated using the mAPThis is different from the yolo_v3-tf:coco mAP mentioned in the link form.

 

The link that you referred to [INT8 vs FP32 Comparison on Select Networks and Platforms - OpenVINO™ Toolkit (openvinotoolkit.org)] is used to show the absolute accuracy drop that is calculated as the difference in accuracy between the FP32 representation of a model and its INT8 representation.

This is merely for comparison between model accuracy value within several platform configuration.

In this table, COCO mAP is chosen as selected metric to compare the model accuracy across these identified configuration.

 

For OpenVINO model accuracy validation, all models are validated and tested using their respective metrics that can be found here: Trained Models - OpenVINO™ Toolkit (openvinotoolkit.org)

Accuracy metrics obtained on COCO* validation dataset for converted model of yolo-v3-tf as shown here: yolo-v3-tf - OpenVINO™ Toolkit (openvinotoolkit.org)

 

Question 2:

Are coco_precision and coco mAP the same? or Are mAP and coco mAP the same?

 

Under COCO context, there is no difference between AP and mAP as mentioned here: COCO - Common Objects in Context (cocodataset.org)

From OpenVINO context, mAP and coco_precision carries different perception/metrics as in the link: https://docs.openvinotoolkit.org/latest/omz_tools_accuracy_checker_metrics.html

  • coco_precision - MS COCO Average Precision metric for keypoints recognition and object detection tasks. Supported representations: PoseEstimationAnnotation, PoseEstimationPrediction, DetectionAnnotation, DetectionPrediction.
    • max_detections - max number of predicted results per image. If you have more predictions,the results with minimal confidence will be ignored.
    • threshold - intersection over union threshold. You can specify one value or comma separated range of values. This parameter supports precomputed values for standard COCO thresholds (.5, .75, .5:.05:.95).
  • map - mean average precision. Supported representations: DetectionAnnotation, DetectionPrediction.
    • overlap_threshold - minimal value for intersection over union that allows to make decision that prediction bounding box is true positive.
    • overlap_method - method for calculation bbox overlap. You can choose between intersection over union (iou), defined as area of intersection divided by union of annotation and prediction boxes areas, and intersection over area (ioa), defined as area of intersection divided by ara of prediction box.
    • include_boundaries - allows include boundaries in overlap calculation process. If it is True then width and height of box is calculated by max - min + 1.
    • ignore_difficult - allows to ignore difficult annotation boxes in metric calculation. In this case, difficult boxes are filtered annotations from postprocessing stage.
    • distinct_conf - select only values for distinct confidences.
    • allow_multiple_matches_per_ignored - allows multiple matches per ignored.
    • label_map - the field in annotation metadata, which contains dataset label map (Optional, should be provided if different from default).
    • integral - integral type for average precision calculation. Pascal VOC 11point and max approaches are available.

 

Question 3:

You say "yolo-v3-tf model is validated using the mAP (mean average precision) metric", Is there any document mentioned? Why can only be MAP, not coco_precision.

 

As mentioned in this link [yolo-v3-tf - OpenVINO™ Toolkit (openvinotoolkit.org)]. yolo-v3-tf accuracy is validated using mAP & COCO mAP.

Example of other Object Detection Models that validated on coco_precision specifically are ssd_mobilenet_v1_coco & faster_rcnn_inception_v2_coco

 

Additional references:

https://towardsdatascience.com/map-mean-average-precision-might-confuse-you-5956f1bfa9e2

https://blog.roboflow.com/mean-average-precision/

 

Hope these clarifies.

 

 

 

Regards,

Syamimi

0 Kudos
WT_Jacko
Beginner
4,408 Views

Hi Syamimi,

Thanks for your response.

I  have a little question about coco_precision and coco_mAP.

First,  you are right. Under COCO context, there is no difference between coco_ap and coco_mAP.

As mentioned in this link :[https://docs.openvinotoolkit.org/latest/omz_tools_accuracy_checker_metrics.html] .

The Openvino Object Detection only support these metrics as below:

WT_Jacko_0-1625472848381.png

but in this link :[https://docs.openvinotoolkit.org/latest/omz_models_model_yolo_v3_tf.html ] .

WT_Jacko_1-1625473064138.png

 you say "yolo-v3-tf accuracy is validated using mAP & COCO mAP."

 

Question 1:

COCO mAP is not in metric support list.  so yolo-v3-tf  accuracy is only validated using mAP?

 

Question 2:

Are COCO mAP and COCO precision the same in openvino?

 

Thanks for your help!!

BR

Jacko

 

 

 

 

0 Kudos
Syamimi_Intel
Moderator
4,376 Views

Hi Jacko Chiu,

Question 1:

COCO mAP is not in metric support list. so yolo-v3-tf accuracy is only validated using mAP?

 

All the supported metrics available in the link: https://docs.openvinotoolkit.org/latest/omz_tools_accuracy_checker_metrics.html are used and defined when you want to compare the predicted data with annotated data and perform quality measurement using OpenVINO accuracy_checker tool. You may use any available metrics based on your dataset representation and optionally reference field for metrics.

 

As for the model itself, yolo-v3-tf is tested and validated using both mAP and COCO mAP metrics for the converted model prior to adding it into our OMZ repository.

Therefore if you are referring to the model validation - yolo-v3-tf is validated using these metrics as per our documentation: yolo-v3-tf - OpenVINO™ Toolkit (openvinotoolkit.org)

 

Question 2:

Are COCO mAP and COCO precision the same in openvino?

 

coco_precision in accuracy_checker metric is using the similar threshold value as defined in Common Objects in Context (COCO) validation dataset.

  • coco_precision - MS COCO Average Precision metric for key points recognition and object detection tasks. Metric is calculated as a percentage. Direction of metric's growth is higher-better. Supported representations: PoseEstimationAnnotation, PoseEstimationPrediction, DetectionAnnotation, DetectionPrediction.
  • max_detections - max number of predicted results per image. If you have more predictions,the results with minimal confidence will be ignored.
  • threshold - intersection over union threshold. You can specify one value or comma separated range of values. This parameter supports precomputed values for standard COCO thresholds (.5, .75, .5:.05:.95).

 

 

Can you assist to confirm that you are looking to utilize coco_precision value as your main reference rather than mAP?

If yes, we may re-engage with our development team to further elaborate your initial Question 1 - coco_precision: 67.84% higher than FP32: 67.7% , It doesn’t make sense.

 

 

Regards,

Syamimi


0 Kudos
WT_Jacko
Beginner
4,369 Views

Hi Syamimi,

Thanks for your response.

Let me summarize.  Please help me confirm my concept in red.

First, COCO mAP is not in metrics support list in openvino. so yolo-v3-tf accuracy is only validated using mAP in openvino? Is That right?

WT_Jacko_0-1625649013504.png

WT_Jacko_1-1625649037486.png

 

Second, As you mentioned before,   for the model yolo-v3-tf  itself, it is tested and validated using both mAP and COCO mAP metrics for the converted model prior to adding it into our OMZ repository. => This is the validated metrics of the model itself before the conversion, which has nothing to do with openvino.

 

Third, the metrics that openvino supports are all in this link: [https://docs.openvinotoolkit.org/latest/omz_tools_accuracy_checker_metrics.html] ,   In other words, the metrics supported by openvino do not include COCO mAP, so for yolo-v3-tf, I can only focus on map.

WT_Jacko_2-1625649576687.png

Is the concept correct for my answer in the red letter above?

Thanks for your help!!

BR

Jacko Chiu

 

0 Kudos
Syamimi_Intel
Moderator
4,346 Views

Hi Jacko Chui,

First and foremost, sorry for the confusions that we might induced in providing our replies to your questions.

 

As for your subsequent follow-up questions, the answers as below:

 

Question 1: First, COCO mAP is not in metrics support list in openvino. so yolo-v3-tf accuracy is only validated using mAP in openvinoIs That right?

 

Accuracy_checker tools allows to reproduce validation pipeline and collect aggregated quality indicators for popular datasets both for networks in source frameworks and in the OpenVINO™ supported formats.

 

As yolo-v3 is object detection model, you can use any of these metrics to validate the accuracy when using OpenVINO accuracy_checker tool – not only mAP, depending on your preferences.

 

Object Detection:

  • map
  • recall
  • miss_rate
  • coco_precision
  • coco_recall

 

Question 2: This is the validated metrics of the model itself before the conversion, which has nothing to do with openvino.

 

You are correct. Our development team utilized these two metrics (mAP and COCO mAP) to calculate the accuracy prior to add them into the OMZ repository

 

Question 3: In other words, the metrics supported by openvino do not include COCO mAP, so for yolo-v3-tf, I can only focus on map.

  

This was the confusion that we previously replied to you.

 

As mentioned in Question 1, you can use any of available metrics in Object Detection to calculate the accuracy and we think that you can use coco_precision with specifying the threshold under this metric – to follow the same COCO mAP metric defined here: COCO - Common Objects in Context (cocodataset.org)

 

However, we can say mAP was the most common metric used to calculate the model accuracy.

  • coco_precision - MS COCO Average Precision metric for key points recognition and object detection tasks. Metric is calculated as a percentage. Direction of metric's growth is higher-better. Supported representations: PoseEstimationAnnotation, PoseEstimationPrediction, DetectionAnnotation, DetectionPrediction.
  • max_detections - max number of predicted results per image. If you have more predictions, the results with minimal confidence will be ignored.
  • threshold - intersection over union threshold. You can specify one value or comma separated range of values. This parameter supports precomputed values for standard COCO thresholds (.5, .75, .5:.05:.95).

 

Looking at your original question, the accuracy result that you get for INT8 using coco_precision was 67.84% higher than FP32: 67.7% and I believe you want to understand this further.

Please confirm this so that I can re-engage our engineering team to elaborate further on the reasoning behind the result obtained.

 

 

Regards,

Syamimi


0 Kudos
WT_Jacko
Beginner
4,338 Views

Hi Syamimi,

Thanks for your response.

Looking at your original question, the accuracy result that you get for INT8 using coco_precision was 67.84% higher than FP32: 67.7% and I believe you want to understand this further.

Please confirm this so that I can re-engage our engineering team to elaborate further on the reasoning behind the result obtained.

YES, Please.

Thanks

BR

Jacko

 

0 Kudos
Syamimi_Intel
Moderator
4,308 Views

Hi Jacko Chiu,

I have escalated this case to our development team and still waiting for them to share their findings with us. We will get back to you soon.

 

 

Regards,

Syamimi


0 Kudos
Syamimi_Intel
Moderator
4,275 Views

Hi Jacko Chiu,

Thank you for waiting. This is the answer from our development team:

 

We do see the same behaviour on our internal test result for the respective model and this is something expected. The similar results were captured for some other quantized models too due to reduced variance in weights coefficients of quantized model. It does not says that overall accuracy of quantized model become better or worse on real life data, this is just a metric on validation dataset (which is always limited compared with infinite real life data).

 

Sometimes, INT8 accuracy is better than FP32 which may indicated that the model is not well tuned or the validation dataset is not optimal. In this particular case/model, the difference is not big thus our development team don't see a problem and we don't have concrete concerns towards coco_precision.

 

 

Regards,

Syamimi


0 Kudos
Syamimi_Intel
Moderator
4,235 Views

Hi Jacko Chiu,

This thread will no longer be monitored since we have provided a solution. If you need any additional information from Intel, please submit a new question.



Regards,

Syamimi


0 Kudos
Reply