I need help with OpenVino PTQ.

[kucha]

OpenVino2023
OpenCV4.9

1. Test_Original_fp32.xml and Test_Original_fp32.bin work well.

2. I compressed the model using the code below.

import openvino.runtime as ov
from nncf import compress_weights, CompressWeightsMode

# Instantiate your uncompressed model
model = ov.Core().read_model("Test_Original_fp32.xml")

# Compress weights
model = compress_weights(model, mode=CompressWeightsMode.INT8_ASYM)

# save the model
ov.save_model(model, "Test_Compress.xml", compress_to_fp16=False)

There is no problem with operation.

3. A PTQ was conducted using the code below.

import nncf
import openvino.runtime as ov
import torch
from torchvision import datasets, transforms
import numpy as np
import glob

# Instantiate your uncompressed model
model = ov.Core().read_model("Test_Original_fp32.xml")

# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = datasets.ImageFolder("nncf_anpr/dataset", transform=transforms.Compose([transforms.ToTensor()]))
dataset_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1)

# Step 1: Initialize transformation function
def transform_fn(data_item):
images, _ = data_item
return images

# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(dataset_loader, transform_fn)

# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(model, calibration_dataset)

# save the model
ov.save_model(quantized_model, "Test_Quantized.xml", compress_to_fp16=False)

However, detection is not possible at all.

I need help with what to check.

As always, thank you for your help.

[kucha]

Please note that I am testing using subsource.

"https://github.com/Y-T-G/YOLO-NAS-OpenVino-cpp"

As always, thank you for your help.

[Iffa_Intel]

Hi,

Could you share specifically your nncf_anpr/dataset files (perhaps simple version of it) for us to test out that outcome.

So far as you mentioned, all the other models work:

Original model:

Compressed model:

Cordially,

Iffa

[kucha]

Hi Iffa

First of all, thank you very much for your reply.

In the case of Compress model, it also works normally.

nncf Quantize does not work properly.

Does nncf Quantize work properly?

Or do you need a Dataset to test nncf Quantize?

[Iffa_Intel]

Yes, I do need dataset to test out the model that has issue, specifically your dataset so that we are on the same page of your use case as I mentioned previously.

Please help to share it if you can. If not, I'll try with any other dataset which might be different than yours.

Plus, could you describe how you determine the detection is not working properly? (eg: you trained for oranges but the model couldn't detect it)

Cordially,

Iffa

[kucha]

Thank you very much for your reply.

We will simply send you the files used for learning.

The number of datasets is small

"CoreKor_Object.xml" --> fp32 --> ok
"CoreKor_Object_compress.xml" --> ok
"CoreKor_Object_Quantized.xml" --> detection fail

I'm not sure how to solve this.

Cordially,

Kucha

[Iffa_Intel]

@kucha I've checked your dataset with the model.

First and foremost, I had to change the model's shape from 1,3,640,640 into 1,3,480,640 to match your dataset

Your dataset need to have class classification folders in it:

As for the result, the model is not exactly failed to detect but it actually requires a more refined dataset for it to learn.

For example, I checked your car images. Most of them are far with open surroundings. The model might not be able to learn which one is a "car that you want it to learn". 

I suggest you try to use Roboflow to get your custom dataset with proper annotations.

Upload your images, annotate then generate the dataset there.

Cordially,

Iffa

[kucha]

Iffa Thank you very much for your reply.

The image size was resized to the upload capacity and then attached.

The actual size is 1920x1080.

1. Is the problem of not being able to detect after PTQ caused by not classifying the Class folder during PTQ?

2. There are multiple classes in one image. How do I classify them into folders?

3. Can you attach the class folder and code where PTQ was conducted?

Cordially,

Kucha

[Iffa_Intel]

Your model needs to match with the tensor shape.

As mentioned previously, You need to properly prepare your dataset, the reason I recommend Roboflow is because it will handle the class, annotations and folders for you. The one that I tried was just a dummy multiclass classification folders for a trial and error based on what you shared.

Dataset plays a major role in obtaining a model that has high perfomance.

This is actually out from OpenVINO scope and you are required to explore further in handling and preparing your custom dataset.

Cordially,

Iffa

[kucha]

Iffa Thank you very much for your reply.

I must proceed with a custom dataset.

First, let's conduct a PTQ test by dividing the Dataset folder by class.

Thank you so much.

[Iffa_Intel]

Generally, there are 3 crucial stages in your use case:

1. Model creation (this step is covered in the GitHub link you shared, where an ONNX model is exported) - This part is done since this indicate you are using Transfer Learning
2. Model training (you need to train your custom dataset on the model for it to learn) - You need to work on this, as I suggested previously, you may refer to How to Train YOLO-Nas on a custom dataset or OpenVINO Training Extensions
3. Model inferencing (create codes for the trained model to detect trained objects) - This is where you mainly use OpenVINO, especially to run them on Intel hardwares (CPU,GPU,NPU).  This part also covered in step 3 - you already generated an exe file for inferencing. You may also explore this official OpenVINO Object Detection Demo.

Good luck & all the best to you.

Cordially,

Iffa

[Iffa_Intel]

Hi,

Intel will no longer monitor this thread since we have provided a solution. If you need any additional information from Intel, please submit a new question.

Cordially,

Iffa