I'm working with the attached dataset and when I try converting a custom trained Yolov4 FP32 model to INT8, OpenVINO DL Workbench complains that such a conversion is not possible, since I'm providing it a dataset that isn't compatible with an Object Detection task.
My dataset, model can be found here: https://drive.google.com/drive/folders/12XmkqjB9Li0ptJhV47i9uKlUeICQHOsC?usp=sharing
Following images show the optimization screen where DL workbench shows calibration dataset as Task: Inpainting/Style Transfer and Optimization option isn't available.
Thank you for reaching out to us and thank you for using OpenVINO™ Deep Learning Workbench!
I have clicked on the link you provided. It is an empty folder and I could not find any dataset or model.
Could you please re-attach your dataset and model for us to replicate your issue from our side?
On another note, I have imported yolo-v4-tf (from Public Pre-Trained Models), and COCO Dataset (2017 Val images and Train/Val annotations) in OpenVINO™ Deep Learning Workbench. OpenVINO™ Deep Learning Workbench correctly detects COCO datasets as Object Detection, Instance Segmentation, Inpainting, Style Transfer. You may refer to the attachment below:
The structure of the COCO dataset and the JSON file with annotations is available at the following page:
If you want to save your time when loading original datasets into OpenVINO™ Deep Learning Workbench, you may refer to Cut Datasets for more information.
Thanks for the fast response. FYI, I'm using openvino/workbench:latest docker image.
I forgot to upload the dataset into the folder that I shared, but it should be there now. Could you check them out and see if you're able to reproduce the same issue I'm seeing?
I converted the dataset into COCO format from darknet and used (2017 Val images and Train/Val annotations) alongside the description provided in https://docs.openvinotoolkit.org/2021.4/workbench_docs_Workbench_DG_Dataset_Types.html#coco_structur... as a reference.
I even performed the Default optimization i.e. the non AccuracyAware optimization step, however, I'm unable to download the INT8 model. I don't see the download button mentioned here: https://docs.openvinotoolkit.org/2019_R3/_docs_Workbench_DG_Int_8_Quantization.html
Screenshot of my project window:
Thanks for your information.
I’m able to download the dataset and model that you shared in the previous post.
For your information, I was able to reproduce the same issue as you did when I upload your dataset to OpenVINO™ Deep Learning Workbench. OpenVINO™ Deep Learning Workbench detects your dataset as inpainting/style transfer.
Based on the development team’s response, the source of the problem is indeed the subfolder hierarchy in which you store your images.
During the dataset validation process, OpenVINO™ Deep Learning Workbench utilizes Accuracy Checker's annotation converters to check the presence of all images relative to the specified image directory.
We suggest you perform dataset reformatting into a flat hierarchy should resolve this issue.
On another note, I was able to perform INT8 calibration with your model and your dataset using the OpenVINO™ Deep Learning Workbench 2021.4.1.1 (Latest Tag
Steps to perform INT8 calibration using OpenVINO™ Deep Learning Workbench 2021.4.1.1 are:
1. From the Project page, “Click” the Perform tab, then “Click” the Optimize Performance Tab. “Select” the Optimization Method, and “Click” the Optimize button.
2. “Select” your Calibration Dataset, Subset Size, Optimization Methods, and Calibration Schemes. Then “Click” the Optimize button.
Steps to install OpenVINO™ Deep Learning Workbench 2021.4.1.1 is available at the following page:
1. I changed my directory structure from:
Is this what you meant by flat heirarchy? Even after this I'm getting the same error.
I've given you my dataset, could you modify it to a working version? Maybe share a screenshot or upload the working version to my drive?
2. Unfortunately the instructions to convert model INT8 did not help, since I had already done these steps. My question pertinent to "Accuracy Unaware" optimization was that after the conversion, I'm unable to download the model. The action's drop down menu in the screenshot doesn't provide a download button. I can't find instructions to get the converted INT8 model from some place. I need to get the XML and .bin file for INT8.
After taking a look at the dataset provided, it seems that the issue is with the annotation file you provide. Namely, some of the fields inside such as category_id and bbox are typed as strings. Accuracy Checker expects these to be int, so when the Workbench attempts to validate the dataset through AC, it inevitably fails due to this faulty annotation. Fixing the typing of mentioned fields should resolve the issue
Thank you for sharing in the OpenVINO™ community!
Thanks for your patience.
I have checked the structure of your custom COCO dataset that you provided in your Google Drive. The structure of your custom COCO dataset and the structure of the JSON file with annotations were correct.
However, VladimirG had provided an answer in the previous post. We noticed that there were wrong data types in your sub-section (annotations) from your JSON file. For your information, there are several data types for the sub-section in the JSON file.
I have generated a custom COCO dataset by taking one image from your dataset, and I have edited your instances_val.json file. DL Workbench successfully detected the dataset as Object Detection, Style Transfer, Inpainting, etc.
You may refer to the custom dataset shared by us. You may regenerate your annotation file with the correct data type for each sub-section.
For more information on the COCO dataset format, you can refer to COCO Dataset Format – Complete Walkthrough.
To answer your second question, you can download your optimized model with the following steps:
1. “Open” the Project page.
2. “Click” the Perform tab.
3. “Click” Export Project tab.
4. “Select” Yes for Include Model.
5. “Click” Export.
Thank you for your question.
This thread will no longer be monitored since we have provided a solution.
If you need additional information from Intel, please submit a new question.