Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.
6392 Discussions

RuntimeError: Inconsistent number of per-sample metric values

dopeuser
Novice
1,169 Views

Hi, I am trying to convert my model to int8 using accuracy aware optimization. However, when I try to optimize it I get this error

RuntimeError: Inconsistent number of per-sample metric values

I am not able to find what this means.

I have attached my configuration file below. I have renamed it to txt as I am not allowed to upload .json. I have also attached annotation.txt file of my dataset. 

The model converts successfully when I use Default Optimization.

0 Kudos
1 Solution
Iffa_Intel
Moderator
1,163 Views
0 Kudos
7 Replies
Iffa_Intel
Moderator
1,164 Views

Greetings,


Are you trying to implement INT8 Quantization?

If yes, you need to take the dataset types into consideration.

Only these 5 are supported:

You can refer to these links for further info:

  1. https://docs.openvinotoolkit.org/latest/_docs_Workbench_DG_Dataset_Types.html
  2. https://docs.openvinotoolkit.org/latest/_docs_Workbench_DG_Int_8_Quantization.html



Sincerely,

Iffa




0 Kudos
bell__sefi
Beginner
1,114 Views

I have the same problem when trying to convert to 8bit ("Inconsistent number of per-sample...")

I am trying to use Resnet50 on Imagenet with a slightly different data structure (specified below).

Any suggestions how to solve this problem?

 

My Imagenet structure is:

- Imagenet

   - train (NOT USED)

      - ....

   -val

      - annotation.txt

      - n03187595

         - ILSVRC2012_val_00034672.JPEG

         - ILSVRC2012_val_00037631.JPEG

         - .....

      - n04356056

         - ILSVRC2012_val_00039734.JPEG

         - .....

      - ....

 

0 Kudos
dopeuser
Novice
1,104 Views

@bell__sefi 

Dataset wasn't an issue in my case. What worked for me was a combination of .json and .yaml files. Have a look at the samples in the C:\Program Files (x86)\IntelSWTools\openvino_2020.4.287\deployment_tools\tools\post_training_optimization_toolkit\configs folder.

I have attached a few files. Maybe try splitting your .json into two such files. Let me know if this works. 

0 Kudos
bell__sefi
Beginner
1,072 Views

@dopeuser  - Thanks a lot!

Your yaml and json solved the issue!

 

Where did you find the documentation explaining the use of them?

0 Kudos
dopeuser
Novice
1,060 Views

@bell__sefi 

You are welcome. I didn't find it in any documentation. I spent a lot of time going through the directories and the source code. Their documentation is pretty inconsistent. I suggest going through all the examples to see if you find something similar to your use case. 

0 Kudos
bell__sefi
Beginner
1,051 Views

@dopeuser If you have the experience, maybe you can help?

I am trying to quantize to 8bit several Imagenet models (e.g. resnet50, mobilenetv2, v3, efficientnets etc.).

Do you have any insights regarding how much accuracy drop to expect? how many images\steps are needed for that?

Any information could assist.

 

Thanks a lot

0 Kudos
dopeuser
Novice
1,038 Views

@bell__sefi 

I didn't explore this further. It always removed quantization from all the layers when doing accuracy aware quantization. I really can't help you with this

0 Kudos
Reply