Intel® Distribution of OpenVINO™ Toolkit
Community support and discussions about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all things computer vision-related on Intel® platforms.
5541 Discussions

OpenVINO DL Workbench incorrectly detects COCO dataset as inpainting/style transfer

kkhode
Employee
484 Views

I'm working with the attached dataset and when I try converting a custom trained Yolov4 FP32 model to INT8, OpenVINO DL Workbench complains that such a conversion is not possible, since I'm providing it a dataset that isn't compatible with an Object Detection task.

 

My dataset, model can be found here: https://drive.google.com/drive/folders/12XmkqjB9Li0ptJhV47i9uKlUeICQHOsC?usp=sharing

 

Following images show the optimization screen where DL workbench shows calibration dataset as Task: Inpainting/Style Transfer and Optimization option isn't available.

 

Error message showing Type of calibration dataset and option to Optimize being unableError message showing Type of calibration dataset and option to Optimize being unable

projectWithDatasetMondel.PNG

Labels (1)
0 Kudos
8 Replies
Wan_Intel
Moderator
426 Views

Hi Kkhode,

Thank you for reaching out to us and thank you for using OpenVINO™ Deep Learning Workbench!

 

I have clicked on the link you provided. It is an empty folder and I could not find any dataset or model.
Empty_folder.JPG

Could you please re-attach your dataset and model for us to replicate your issue from our side?

 

On another note, I have imported yolo-v4-tf (from Public Pre-Trained Models), and COCO Dataset (2017 Val images and Train/Val annotations) in OpenVINO™ Deep Learning Workbench. OpenVINO™ Deep Learning Workbench correctly detects COCO datasets as Object Detection, Instance Segmentation, Inpainting, Style Transfer. You may refer to the attachment below:
int_8_coco_dataset_yolov4_1.jpg

 

Also, I’m able to perform INT8 calibration and Accuracy Checker as shown as follows:

int_8_coco_dataset_yolov4_3.jpg

You may download and import the COCO Dataset (2017 Val images and Train/Val annotations) in OpenVINO™ Deep Learning Workbench to check whether the problem still exists.

 

The structure of the COCO dataset and the JSON file with annotations is available at the following page:

https://docs.openvinotoolkit.org/2021.4/workbench_docs_Workbench_DG_Dataset_Types.html#coco_structur...

 

If you want to save your time when loading original datasets into OpenVINO™ Deep Learning Workbench, you may refer to Cut Datasets for more information.

 

 

Regards,

Wan

 

kkhode
Employee
394 Views

Thanks for the fast response. FYI, I'm using openvino/workbench:latest docker image.

 

I forgot to upload the dataset into the folder that I shared, but it should be there now. Could you check them out and see if you're able to reproduce the same issue I'm seeing?

 

I converted the dataset into COCO format from darknet and used (2017 Val images and Train/Val annotations) alongside the description provided in https://docs.openvinotoolkit.org/2021.4/workbench_docs_Workbench_DG_Dataset_Types.html#coco_structur... as a reference.

 

I even performed the Default optimization i.e. the non AccuracyAware optimization step, however, I'm unable to download the INT8 model. I don't see the download button mentioned here: https://docs.openvinotoolkit.org/2019_R3/_docs_Workbench_DG_Int_8_Quantization.html

 

Screenshot of my project window:

kkhode_0-1634576940631.png

 

 

Wan_Intel
Moderator
302 Views

Hi Kkhode,

Thanks for your information.

 

I’m able to download the dataset and model that you shared in the previous post.

 

For your information, I was able to reproduce the same issue as you did when I upload your dataset to OpenVINO™ Deep Learning Workbench. OpenVINO™ Deep Learning Workbench detects your dataset as inpainting/style transfer.
customer_dataset.JPG

 

Based on the development team’s response, the source of the problem is indeed the subfolder hierarchy in which you store your images.

 

During the dataset validation process, OpenVINO™ Deep Learning Workbench utilizes Accuracy Checker's annotation converters to check the presence of all images relative to the specified image directory.

 

We suggest you perform dataset reformatting into a flat hierarchy should resolve this issue.

 

On another note, I was able to perform INT8 calibration with your model and your dataset using the OpenVINO™ Deep Learning Workbench 2021.4.1.1 (Latest Tag
success.JPG

 

Steps to perform INT8 calibration using OpenVINO™ Deep Learning Workbench 2021.4.1.1 are:

 

1.   From the Project page, “Click” the Perform tab, then “Click” the Optimize Performance Tab. “Select” the Optimization Method, and “Click” the Optimize button.

1.JPG

2.   “Select” your Calibration Dataset, Subset Size, Optimization Methods, and Calibration Schemes. Then “Click” the Optimize button.

2.JPG

Steps to install OpenVINO™ Deep Learning Workbench 2021.4.1.1 is available at the following page:

https://docs.openvino.ai/2021.4/workbench_docs_Workbench_DG_Run_Locally.html

 

 

Regards,

Wan

 

kkhode
Employee
237 Views

1. I changed my directory structure from:

kkhode_0-1634922824216.png

To:

kkhode_1-1634922898511.png

Is this what you meant by flat heirarchy? Even after this I'm getting the same error.

kkhode_2-1634923008533.png

 

I've given you my dataset, could you modify it to a working version? Maybe share a screenshot or upload the working version to my drive?

 

2. Unfortunately the instructions to convert model INT8 did not help, since I had already done these steps. My question pertinent to "Accuracy Unaware" optimization was that after the conversion, I'm unable to download the model. The action's drop down menu in the screenshot doesn't provide a download button. I can't find instructions to get the converted INT8 model from some place. I need to get the XML and .bin file for INT8.

kkhode_3-1634923274615.png

 

 

VladimirG
Employee
163 Views

Hi kkhode,

 

After taking a look at the dataset provided, it seems that the issue is with the annotation file you provide. Namely, some of the fields inside such as category_id and bbox are typed as strings. Accuracy Checker expects these to be int, so when the Workbench attempts to validate the dataset through AC, it inevitably fails due to this faulty annotation. Fixing the typing of mentioned fields should resolve the issue

 

Regards,

Vladimir

VladimirG
Employee
158 Views

Attaching a corrected version of the annotation file. The bbox and category_id fields have been converted to float and int respectively. This should allow your dataset to be recognized as OD and perform AccuracyAware optimization.

Regards,
Vladimir

Wan_Intel
Moderator
142 Views

Hi VladimirG,

Thank you for sharing in the OpenVINO™ community!

 

 

Hi Kkhode,

Thanks for your patience.

 

I have checked the structure of your custom COCO dataset that you provided in your Google Drive. The structure of your custom COCO dataset and the structure of the JSON file with annotations were correct.

 

However, VladimirG had provided an answer in the previous post. We noticed that there were wrong data types in your sub-section (annotations) from your JSON file. For your information, there are several data types for the sub-section in the JSON file.

 

I have generated a custom COCO dataset by taking one image from your dataset, and I have edited your instances_val.json file. DL Workbench successfully detected the dataset as Object Detection, Style Transfer, Inpainting, etc.
Correct_COCO_Format.JPG

You may refer to the custom dataset shared by us. You may regenerate your annotation file with the correct data type for each sub-section.

 

For more information on the COCO dataset format, you can refer to COCO Dataset Format – Complete Walkthrough.

 

To answer your second question, you can download your optimized model with the following steps:

 

1.   “Open” the Project page.

 

2.   “Click” the Perform tab.

 

3.   “Click” Export Project tab.

 

4.   “Select” Yes for Include Model.

 

5.   “Click” Export.

export_int_8_model.JPG

 

 

Regards,

Wan

 

Wan_Intel
Moderator
87 Views

Hi Kkhode,

Thank you for your question.


This thread will no longer be monitored since we have provided a solution.

If you need additional information from Intel, please submit a new question.



Regards,

Wan


Reply