Calibrate Object Detection Model PROBLEM

[Stefano_M_]

I'm trying to convert to Int8 Precision using calibration tools but the tools gives me this error:

Progress: [....................] 100.00% done
  FP32 Accuracy: 99.95%
[ INFO ] Verification of network accuracy if all possible layers converted to INT8
Validate int8 accuracy, threshold for activation statistics = 100.00
[ ERROR ] Inference problem:
Number of priors must match number of confidence predictions.
C:\Intel\computer_vision_sdk_2018.5.445\deployment_tools\inference_engine\src\extension\ext_detectionoutput.cpp:62
..\src\mkldnn_plugin\nodes\mkldnn_generic_node.cpp:49
c:\intel\computer_vision_sdk_2018.5.445\inference_engine\include\details/ie_exception_conversion.hpp:71

I'm running the following command:

-d CPU -t OD -ODa "D:/I8/Annotations" -i "D:/I8/" -m "C:/vehicle-license-plate-detection-barrier-0106.xml" -ODc "D:/I8/test_classes.txt" -ODsubdir JPEGImages -output "D:/I8/vehicle-license-plate-detection-barrier-0106_I8.xml"

on my dataset of 40 images. [attached an example of annotation]

How can I solve?

Thanks

[nikos1]

Hi Stefano,

Hard to repro the exact same issue as yours without your data. Best way to resolve this is to build a debug version of the calibration tool ( on Windows it is under C:\Intel\computer_vision_sdk_2018.5.445\deployment_tools\inference_engine\samples\calibration_tool ), debug and see what the issue issue is with your own data and configuration parameters. Initially I thought I had the same issue as yours but after debugging my issue was different.

Nikos

Just some ideas below.

You may have to modify the calibration_tool for your application. It supports SSD

( -ODkind <kind>          Type of an Object Detection model. Options: SSD ) 

You could use the existing source code in omputer_vision_sdk_2018.5.445/deployment_tools/inference_engine/samples/calibration_tool/main.cpp as a template and implement the inference part for your network so that you do not get the error above.

I believe the solution to your problem is to implement a function in the calibration_tool that can process the output of vector of vehicle-license-plate-detection-barrier-0106.xml - maybe you can see how this is done in the /security_barrier_camera_demo sample ( ~/intel/computer_vision_sdk_2018.5.445/deployment_tools/inference_engine/samples/security_barrier_camera_demo/main.cpp ).

[Stefano_M_]

nikos wrote:

Hi Stefano,

Hard to repro the exact same issue as yours without your data. Best way to resolve this is to build a debug version of the calibration tool ( on Windows it is under C:\Intel\computer_vision_sdk_2018.5.445\deployment_tools\inference_engine\samples\calibration_tool ), debug and see what the issue issue is with your own data and configuration parameters. Initially I thought I had the same issue as yours but after debugging my issue was different.

Nikos

Just some ideas below.

You may have to modify the calibration_tool for your application. It supports SSD

( -ODkind <kind>          Type of an Object Detection model. Options: SSD ) 

You could use the existing source code in omputer_vision_sdk_2018.5.445/deployment_tools/inference_engine/samples/calibration_tool/main.cpp as a template and implement the inference part for your network so that you do not get the error above.

I believe the solution to your problem is to implement a function in the calibration_tool that can process the output of vector of vehicle-license-plate-detection-barrier-0106.xml - maybe you can see how this is done in the /security_barrier_camera_demo sample ( ~/intel/computer_vision_sdk_2018.5.445/deployment_tools/inference_engine/samples/security_barrier_camera_demo/main.cpp ).

Thanks, I will try.

I succeded with the conversion of vehicle-detection-adas-0002, person-vehicle-bike-detection-crossroad-0078 and pedestrian-and-vehicle-detector-adas-0001 but I obtain with some strange results:

running this models on Xeon Silver 4116 with 1000 images the time performances get worse:

• person-vehicle-bike-detection-crossroad-0078 -> 36.50 ms per image with FP32 and 38.15 ms per image with I8
• pedestrian-and-vehicle-detector-adas-0001 -> 15.96 ms per image with FP32 and 23.94 ms per image with I8

only with vehicle-detection-adas-0002 I obtain a better result:

• vehicle-detection-adas-0002 -> 13.68 ms per image with FP32 and 9.89 ms per image with I8

Can this results be influenced by the reduced and incomplete dataset I used for the calibration tool?

[nikos1]

Hi Stefano,

> I succeeded with the conversion of vehicle-detection-adas-0002, person-vehicle-bike-detection-crossroad-0078 and pedestrian-and-vehicle-detector-adas-0001 but I obtain with some strange results:

Good progress! Just wanted to ask you how you obtained the validation data subset to perform calibration? Also what was the size of the calibration dataset you used?

> person-vehicle-bike-detection-crossroad-0078 -> 36.50 ms per image with FP32 and 38.15 ms per image with I8

There are a number of reasons this could happen. Best way is if you could attach your performance counters for all layers in the case of FP32 and I8. Then we could see exactly what got converted to I8 and what got slower. You could use the -pc sample parameter. Look for  execType: *avx2*I8

> Can this results be influenced by the reduced and incomplete dataset I used for the calibration tool?

Yes, in theory this could be the case too but there are also networks that have layers not supported for FP32 -> I8 so this could be the case too. The pedestrian-and-vehicle-detector-adas-0001 is a MobileNet SSD so that explain the improvement. Maybe not too many layers got converted for person-vehicle-bike-detection-crossroad-0078 - let's see the perf counters.

Cheers,

Nikos

[Ermak__Aleksei]

Hello Nikos,

can you tell me please, how can I build a debug version of the calibration tool under Ubuntu 16.04?

Thanks