Re: Does OpenVINO2022.x support QuantizeLinear/DequantizeLinear? - Page 2

North-man · ‎04-11-2023

Hi,
My post is a similar issue to the post below.

https://community.intel.com/t5/Intel-Distribution-of-OpenVINO/openVino-inference-using-onnx-model/m-p/1460146

I believe OpenVINO supports QuantizeLinear/DeauantizeLinear layers and ONNX format models.

However, the model using QuantizeLinear/DeauantizeLinear layers (=Model1) gives different results depending on the OpenVINO 2021.4 and 2022.3.

For comparison, the model that does not use QuantizeLinear/DeauantizeLinear (=Models2) gives the same result for OpenVINO 2021.4 and 2022.3.

For further comparison, I also compared results when using different inference frameworks (e.g. TensorRT).

The conditions are as follows.

・model format is onnx.

・Windows10 and C++.

・output size = 1000 (using Generic imageNet)

The results are summarized below.

・The numbers in the table below are the first 10 of 1000

	OpenVINO 2021.4	OpenVINO 2022.3	Other framework (TensorRT)
use QuantizeLinear/DeauantizeLinear (=Model1)	1.82228	2.83418	1.84796
	-5.29731	-3.76114	-5.31353
	-2.57535	-3.80294	-2.56074
	0.815923	0.491049	0.816834
	-5.83255	-4.85166	-5.85728
	-3.84568	-1.81016	-3.84114
	-4.65202	-5.18312	-4.6788
	3.07942	1.29188	3.09532
	2.51693	1.51579	2.54098
	10.021	6.2427	10.0298
does not use QuantizeLinear/DeauantizeLinear (=Model2)	1.92688	1.92688	1.92688
	-4.5917	-4.5917	-4.5917
	-3.09134	-3.09134	-3.09134
	0.355102	0.355103	0.355103
	-6.01955	-6.01955	-6.01955
	-4.48821	-4.48821	-4.48821
	-5.00235	-5.00235	-5.00235
	2.16591	2.16591	2.16591
	2.74276	2.74276	2.74276
	11.9108	11.9108	11.9108

Only the combination of the model using QuantizeLinear/DeauantizeLinear and OpenVINO2022.3 has a large numerical difference.

Why is this?

Please let me know if you get similar results in your environment and if so what the problem is.

I wanted to attach the model and the input image, but it was not possible due to size restrictions. Please let me know if there is a better way.

Thank you for your support.

Hari_B_Intel · ‎07-27-2023

Hi North-man

Apology for the late response, as we are investigating the issue on the latest OpenVINO 2023.0, which causes the results not to be similar to the oldest version of OpenVINO.

For your information, this issue only seems for the oldest CPU, and we cannot reproduce it on Intel Gen 9 processors and above.

For your case, we suggest using the older version of OpenVINO, which is the OpenVINO 2021 LTS version based on your processor.

I hope this information helps, and we would like to thank you for bringing up this bug so that we can further improve.

Thank you

North-man · ‎07-27-2023

Hi Hari_B_Intel,

Thank you for your report.

Here's the CPU I'm using:

CPU : Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz 3.70 GHz

It's a Xeon, so I don't know the generation number, but I think it's older than the Gen 9. (Gen 6?)

Ok, I'd like to check again with a Gen 9 processors and above.

Regards,

North-man.

Wan_Intel · ‎08-05-2023

Hi North-man,

I hope everything is going well for you. Just wanted to know if the issue has been resolved when using Gen 9 processors and above.

Regards,

Wan

North-man · ‎08-07-2023

Hi Wan,
Thank you for your response.

We are currently checking the behavior with the 10th generation Core (Intel (R) Core (TM) i7-10610U),
but the results look similar to the previous post (result of old CPU).
Also, we are currently ordering 12th generation CPUs and will confirm once they are available.
After that, I would like to check the results again.
This may take about 2 weeks.

Regards,
North-man.

Wan_Intel · ‎08-07-2023

Hi North-man,

Thanks for the update.

We'll wait for the inference result using Gen 12 from your end. Meanwhile, could you please share the inference script with us so that we can replicate the issue from our end?

Regards,

Wan

North-man · ‎08-08-2023

Hi Wan,

Thank you for your response.

OK,
I will email you the code, model, data, and results that I used for my evaluation.

Reagrds,
North-man.

Wan_Intel · ‎08-08-2023

Hello North-man,

I've received your email. We'll investigate the issue and update you at the earliest.

Regards,

Wan

Wan_Intel · ‎08-09-2023

Hello North-man,

Thanks for your patience.

We've replicated the issue from our end using Gen 10 processors. We encountered the same issue as yours when using Gen 10 processors, and our developers are investigating the issue.

On another note, the inference results are correct when using Gen 11 processor and above. Please get back to us after you have checked with Gen 12 processors from your end.

Regards,

Wan

Wan_Intel · ‎08-16-2023

Hello North-man,

Thanks for your patience. We've received feedback from relevant team.

Here are the root causes of the quantized model not showing the right results on older CPUs:

We can have a saturation issue if the CPU doesn't have VVNI. In official documents, "the Intel® Deep Learning Boost (Intel® DL Boost)" is used instead of VVNI.
The issue will be resolved if you use a not quantized model, which means using FP32 because INT8 will show a saturation issue on a CPU that doesn't have VVNI.
Another workaround is to set up a DNNL_MAX_CPU_ISA=AVX2 environment during quantization (+ accuracy check algorithm) if the target platform is AVX2.

Here is the documentation on the saturation issue: https://docs.openvino.ai/2023.0/pot_saturation_issue.html. The issue is not OpenVINO runtime or CPU defect.

Regards,

Wan

North-man · ‎08-22-2023

Hi Wan,

Thank you for the information on the cause of this issue.

I understand the root cause and occurrence conditions.

Preparations for the 12th generation CPU, which is my work, are delayed,

but I would like to close the issue if I can confirm it.

Regards,
North-man.

Wan_Intel · ‎08-24-2023

Hello North-man,

Noted. We'll wait for your replication results. Please get back to us if you have the latest update.

Regards,

Wan

North-man · ‎08-25-2023

Hi Wan,

Sorry for the late reply.

I checked the results on a 10th generation CPU and 12th generation CPU.

・Program : classification_sample_async.exe (C++ Sample)
・Model : Model1_softmax.onnx (QuantizeLinear/DequantizeLinear model)
・Input : Zebra-1_227x227.jpg or bmp

■Core i7-10610U 1.8GHz (10th Generation)

OpenVINO Version	2021.4.752	2022.3.0	2023.0.0
Top10 result(classId, value)	549	0.2040894	904	0.1225632	904	0.1246897
922	0.0968276	418	0.0508875	418	0.0513055
446	0.0806766	318	0.0427833	318	0.0437959
815	0.0592603	563	0.0225132	563	0.0227482
794	0.0499254	772	0.0199804	772	0.019438
971	0.0252059	697	0.0187601	697	0.019129
591	0.0228805	311	0.0181506	311	0.0184334
721	0.0208825	77	0.0178132	77	0.017797
916	0.0178964	316	0.0160175	316	0.016454
611	0.0163776	700	0.0149515	700	0.0149051

■Core i7-12700 2.1GHz (12th Generation)

OpenVINO Version	2021.4.752	2022.3.0	2023.0.0
Top10 result(classId, value)	549	0.2040894	549	0.2034392	549	0.2034393
922	0.0968276	922	0.0914842	922	0.0914842
446	0.0806766	446	0.080296	446	0.0802961
815	0.0592603	815	0.059889	815	0.059889
794	0.0499254	794	0.0473791	794	0.0473791
971	0.0252059	971	0.0242755	971	0.0242755
591	0.0228805	591	0.0239744	591	0.0239744
721	0.0208825	721	0.0197706	721	0.0197706
916	0.0178964	916	0.0175369	916	0.0175369
611	0.0163776	611	0.017226	611	0.017226

12th generation CPUs have better results!
OpenVINO2022 and 2023 are almost the same result as 2021.

you are right!
>>For your case, we suggest using the older version of OpenVINO, which is the OpenVINO 2021 LTS version based on your processor.

Thank you for your help in resolving the problem!

Regards,
Notrh-man.

North-man · ‎08-27-2023

The table was misaligned, so I reshaped it.

・Program : classification_sample_async.exe (C++ Sample)
・Model : Model1_softmax.onnx (QuantizeLinear/DequantizeLinear model)
・Input : Zebra-1_227x227.jpg or bmp

■Core i7-10610U 1.8GHz (10th Generation)

OpenVINO Version	2021.4.752	2022.3.0	2023.0.0
Top10 result (classId, value)	549, 0.2040894	904, 0.1225632	904, 0.1246897
	922, 0.0968276	418, 0.0508875	418, 0.0513055
	446, 0.0806766	318, 0.0427833	318, 0.0437959
	815, 0.0592603	563, 0.0225132	563, 0.0227482
	794, 0.0499254	772, 0.0199804	772, 0.019438
	971, 0.0252059	697, 0.0187601	697, 0.019129
	591, 0.0228805	311, 0.0181506	311, 0.0184334
	721, 0.0208825	77, 0.0178132	77, 0.017797
	916, 0.0178964	316, 0.0160175	316, 0.016454
	611, 0.0163776	700, 0.0149515	700, 0.0149051

■Core i7-12700 2.1GHz (12th Generation)

OpenVINO Version	2021.4.752	2022.3.0	2023.0.0
Top10 result (classId, value)	549, 0.2040894	549, 0.2034392	549, 0.2034393
	922, 0.0968276	922, 0.0914842	922, 0.0914842
	446, 0.0806766	446, 0.080296	446, 0.0802961
	815, 0.0592603	815, 0.059889	815, 0.059889
	794, 0.0499254	794, 0.0473791	794, 0.0473791
	971, 0.0252059	971, 0.0242755	971, 0.0242755
	591, 0.0228805	591, 0.0239744	591, 0.0239744
	721, 0.0208825	721, 0.0197706	721, 0.0197706
	916, 0.0178964	916, 0.0175369	916, 0.0175369
	611, 0.0163776	611, 0.017226	611, 0.017226

On 12th Generation CPU, OpenVINO2022 and 2023 are almost the same result as 2021.

For 10th generation CPU, I will use OpenVINO2021.
For 11th generation CPU and above, I can also use OpenVINO2022 or 2023.

Wan_Intel · ‎08-30-2023

Hello North-man,

Thanks for sharing your replication result with us.

Glad to know that your issue has been resolved. This case will no longer be monitored as the issue has been resolved. If you need additional information from Intel, please submit a new question.

Best,

Wan