workbench

[taka]

I want to convert the learned SSD model to INT8 using workbench.
The SSD model is two label names that do not exist in the Pascal VOC.

However, if a label name that does not exist in the Pascal VOC is written to <name> </ name> of the "/annotations/***.xml" file, an error will occur.

   example:

  .xml description     <name>truck</name>

   ⇒ Error occurred during Int 8 optimization.

       KeyError: 'truck'

  By the way, If "car" is written, no error will occur.

I think it is necessary to give the correspondence between the identification number of the SSD model and the label name. Where should the label information be written?
 

The folder structure is as follows:
VOCTEST ---- Annotations
                  |        0001.xml
                  |                :
                  | --- ImageSets
                  |           | ---- Main
                  |                      train.txt

                  | --- JPEGImages
                               0001.jpg

                                   :

[Mikhail_F_Intel]

Hi,

I believe VOC dataset can be extended with your classes though it might require some effort. Or you can create your own "VOC like"dataset but you need to follow the VOC format to get it accepted by DL Workbench.

I suspect you have the understanding of dependencies between components in VOC dataset, but just in case I'll state them.

1) JPEGImages contains images (jpg files) making your dataset. E.g. for VOC2007 the first file 000001.jpg contains person and his dog. 

2) Annotations folder contains annotation file corresponding to each jpg file from JPEGImages directory (000001.xml annotates 000001.jpg etc). In our case (in addition to aux information) it contains bounding box and classification of the object (bbox for person, class: person, bbox for dog, class: dog)

3) ImageSets folder contains list of classes. Each class file (e.g. dog_test.txt) contains the full list of pictures with -1 or 1 against each of them marking if the corresponding image contains target class or not. So, our dog_text.txt first line is "000001 1" marking that we have dog on picture 000001.jpg, second line "000002 -1" marking that dog class is not present on picture 000002.jpg etc. If you check person_test.txt file your will see the same picture ("000001 1", "000002 -1")

So, if you add your class, you will need to do updates in all 3 places in order to have it (add pictures with your class, add annotation to those pictures, add <your class>_test.txt file and mark pictures with your class there.) 

If your dataset is really big, it might be challenging to do. The good point is that for Int8 calibration you might need only limited (though ideally representative from class coverage dataset). So, you might try to make a subset of several hundred pictures with inclusion of your own classes and run Int8 calibration. 

If you have big dataset and you would like to add many pictures, suspect you'd need to do some coding to automate the preparation. 

Hope I interpreted your problem correctly and managed to clarify it. If further questions - do not hesitate.   

Regards,

Mikhail

[taka]

hi

Own "VOC like"dataset is "truck" and "bus" only.

The three points you point out have been updated.

When "<name> truck </ name>"  is described in xml, the following error occurs in workbench

    "Error occurred during Int 8 optimization."

     "KeyError: 'truck'"

The index of the output layer of own SSD model is "truck" is "1" and "bus" is "2".

Where should I define the correspondence between the SSD output index and labelname?

The original VOC labels are as follows:
aeroplane
bicycle
bird
boat
bottle
bus
car
cat
chair
cow
diningtable
dog
horse
motorbike
person
pottedplant
sheep
sofa
train
tvmonitor

The following is not correct.....

I use workbench by changing <name> of * .xml to “truck” ⇒ “aeroplane”, “bus” ⇒ “bicycle”.

Accuracy evaluation and conversion to INT8 can be performed.

I want to know how to give a label name to workbench

[Mikhail_F_Intel]

Hi again,

Ok, thanks for explanation. I did check in the code and can state that by default DL Workbench works with original classes of VOC only when we speak about Int8 calibration and accuracy checking. So it really expects for class 1 the label to be "aeroplane" and so on. 

I found the place in the code of accuracy checker used in DL Workbench for Int8 calibration and accuracy measureemtn there the labels are tagged to numbers. You can "hack" parameters to suit to your use case. But it requires changes in one of the files. I did not have your model and tweaked dataset, so I did not have the chance to check that recommended solution 100% guarantee as working, still there is a good probability that it should. 

What you need to do is to update file pascal_voc.py in docker container. The easiest way for me was to copy it to local host:

docker cp workbench:/opt/intel/openvino/python/python3.5/openvino/tools/accuracy_checker/accuracy_checker/annotation_converters/pascal_voc.py /tmp/

then modify _VOC_CLASSES_DETECTION with your labels (plus strip _SEGMENTATION_COLORS to correspond to number of classes) and push it back  

docker cp /tmp/pascal_voc.py workbench:/opt/intel/openvino/python/python3.5/openvino/tools/accuracy_checker/accuracy_checker/annotation_converters/

Though it's going to be working while you have the container running. If you would like to store it for further use you can 

docker commit container-ID <your_new_workbench_image_name>

I apologize, but currently I can't see other options to enable your labels (with exception of the workaround that you already stated). The issue stems from the fact that vanilla VOC dataset doesn't have explicit mapping between labels and corresponding values present explicitly in dataset.

Regards,
Mikhail

[taka]

In OpenVINO2019R2, I was able to do well with the method  presented previously.

But,the following file does not exist in OpenVINO2019R3.1

/opt/intel/openvino/python/python3.5/openvino/tools/accuracy_checker/accuracy_checker/annotation_converters/pascal_voc.py

Which file should be modified in OpenVINO2019R3.1?
 

[taka]

Please answer the above questions.