Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Novice
176 Views

RuntimeError: Inconsistent number of per-sample metric values

Jump to solution

Hi, I am trying to convert my model to int8 using accuracy aware optimization. However, when I try to optimize it I get this error

RuntimeError: Inconsistent number of per-sample metric values

I am not able to find what this means.

I have attached my configuration file below. I have renamed it to txt as I am not allowed to upload .json. I have also attached annotation.txt file of my dataset. 

The model converts successfully when I use Default Optimization.

0 Kudos

Accepted Solutions
Highlighted
Moderator
170 Views
0 Kudos
7 Replies
Highlighted
Moderator
171 Views
0 Kudos
Highlighted
Beginner
121 Views

I have the same problem when trying to convert to 8bit ("Inconsistent number of per-sample...")

I am trying to use Resnet50 on Imagenet with a slightly different data structure (specified below).

Any suggestions how to solve this problem?

 

My Imagenet structure is:

- Imagenet

   - train (NOT USED)

      - ....

   -val

      - annotation.txt

      - n03187595

         - ILSVRC2012_val_00034672.JPEG

         - ILSVRC2012_val_00037631.JPEG

         - .....

      - n04356056

         - ILSVRC2012_val_00039734.JPEG

         - .....

      - ....

 

0 Kudos
Highlighted
Novice
111 Views

@bell__sefi 

Dataset wasn't an issue in my case. What worked for me was a combination of .json and .yaml files. Have a look at the samples in the C:\Program Files (x86)\IntelSWTools\openvino_2020.4.287\deployment_tools\tools\post_training_optimization_toolkit\configs folder.

I have attached a few files. Maybe try splitting your .json into two such files. Let me know if this works. 

0 Kudos
Highlighted
Beginner
79 Views

@dopeuser  - Thanks a lot!

Your yaml and json solved the issue!

 

Where did you find the documentation explaining the use of them?

0 Kudos
Highlighted
Novice
67 Views

@bell__sefi 

You are welcome. I didn't find it in any documentation. I spent a lot of time going through the directories and the source code. Their documentation is pretty inconsistent. I suggest going through all the examples to see if you find something similar to your use case. 

0 Kudos
Highlighted
Beginner
58 Views

@dopeuser If you have the experience, maybe you can help?

I am trying to quantize to 8bit several Imagenet models (e.g. resnet50, mobilenetv2, v3, efficientnets etc.).

Do you have any insights regarding how much accuracy drop to expect? how many images\steps are needed for that?

Any information could assist.

 

Thanks a lot

0 Kudos
Highlighted
Novice
45 Views

@bell__sefi 

I didn't explore this further. It always removed quantization from all the layers when doing accuracy aware quantization. I really can't help you with this

0 Kudos