validation app produces 0% mAP (mean average precision)

[Onyalo__Sem]

Hi,

I ran the validation app using a model that I trained and I am getting an mAP of 0. I've used this same model in OpenCV's TensorFlow Object Detection API with success so I know I should be getting a higher mAP than zero. I've pasted my run conditions below. Can you let me know what I am doing wrong? Thanks!

Run Conditions

OS: Windows 10

Trained Model: MobileNet-SSD v1

Model, Images and Annotations: intel-openvino-boxing.zip

Command:

validation_app.exe -d CPU -t OD -ODa "C:\temp\intel-openvino-boxing\voc-export\annotations" -i "C:\temp\intel-openvino-boxing" -m "C:\temp\intel-openvino-boxing\transformed_frozen_inference_graph.xml" -ODc "C:\temp\intel-openvino-boxing\VOC_SSD_Classes.txt" -ODsubdir images

Output

[ INFO ] InferenceEngine:
        API version ............ 1.4
        Build .................. 19154
[ INFO ] Parsing input parameters
[ INFO ] Loading plugin

        API version ............ 1.5
        Build .................. win_20181005
        Description ....... MKLDNNPlugin
[ INFO ] Loading network files
[ INFO ] Preparing input blobs
[ INFO ] Batch size is 1
name: image_tensor [ out: FP32/l=NCHW ]
name: Preprocessor/mul/FusedPower_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_0/Conv2D [ in: FP32/l=NCHW; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_1_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_2_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_2_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_3_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_4_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_5_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_6_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_6_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_7_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_7_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_8_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_8_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_9_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_9_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_10_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_10_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_11_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_11_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_12_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_12_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_13_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_13_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_1_Conv2d_2_1x1_256/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_2_3x3_s2_512/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_1_Conv2d_3_1x1_128/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_1_Conv2d_4_1x1_128/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_1_Conv2d_5_1x1_64/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_5_3x3_s2_128/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_5/ClassPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_5/ClassPredictor/Conv2D_nChw8c_nchw_BoxPredictor_5/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_5/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_5/ClassPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_5/ClassPredictor/BiasAdd/Reshape___concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_5/BoxEncodingPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_5/BoxEncodingPredictor/Conv2D_nChw8c_nchw_PriorBoxClustered_5 [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: PriorBoxClustered_5 [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_5/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_5/BoxEncodingPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_5/BoxEncodingPredictor/BiasAdd/Reshape___concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_4/ClassPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_4/ClassPredictor/Conv2D_nChw8c_nchw_BoxPredictor_4/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_4/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_4/ClassPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_4/ClassPredictor/BiasAdd/Reshape___concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_4/BoxEncodingPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_4/BoxEncodingPredictor/Conv2D_nChw8c_nchw_PriorBoxClustered_4 [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: PriorBoxClustered_4 [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_4/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_4/BoxEncodingPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_4/BoxEncodingPredictor/BiasAdd/Reshape___concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_3/ClassPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_3/ClassPredictor/Conv2D_nChw8c_nchw_BoxPredictor_3/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_3/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_3/ClassPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_3/ClassPredictor/BiasAdd/Reshape___concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_3/BoxEncodingPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_3/BoxEncodingPredictor/Conv2D_nChw8c_nchw_PriorBoxClustered_3 [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: PriorBoxClustered_3 [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_3/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_3/BoxEncodingPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_3/BoxEncodingPredictor/BiasAdd/Reshape___concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_2/ClassPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_2/ClassPredictor/Conv2D_nChw8c_nchw_BoxPredictor_2/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_2/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_2/ClassPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_2/ClassPredictor/BiasAdd/Reshape___concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_2/BoxEncodingPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_2/BoxEncodingPredictor/Conv2D_nChw8c_nchw_PriorBoxClustered_2 [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: PriorBoxClustered_2 [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_2/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_2/BoxEncodingPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_2/BoxEncodingPredictor/BiasAdd/Reshape___concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_1/ClassPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_1/ClassPredictor/Conv2D_nChw8c_nchw_BoxPredictor_1/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_1/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_1/ClassPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_1/ClassPredictor/BiasAdd/Reshape___concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_1/BoxEncodingPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_1/BoxEncodingPredictor/Conv2D_nChw8c_nchw_PriorBoxClustered_1 [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: PriorBoxClustered_1 [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_1/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_1/BoxEncodingPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_1/BoxEncodingPredictor/BiasAdd/Reshape___concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_0/ClassPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_0/ClassPredictor/Conv2D_nChw8c_nchw_BoxPredictor_0/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_0/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_0/ClassPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_0/ClassPredictor/BiasAdd/Reshape___concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: do_ExpandDims_conf [ in: FP32/l=CHW; out: FP32/l=NCHW ]
name: do_ExpandDims_conf/sigmoid [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: do_reshape_conf [ in: FP32/l=NCHW; out: FP32/l=NC ]
name: BoxPredictor_0/BoxEncodingPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_0/BoxEncodingPredictor/Conv2D_nChw8c_nchw_PriorBoxClustered_0 [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: PriorBoxClustered_0 [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: ConcatPriorBoxesClustered [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_0/BoxEncodingPredictor/Conv2D_nChw8c_nchw_BoxPredictor_0/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_0/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_0/BoxEncodingPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_0/BoxEncodingPredictor/BiasAdd/Reshape___concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: do_reshape_loc [ in: FP32/l=CHW; out: FP32/l=NC ]
name: DetectionOutput [ in: FP32/l=NC; out: FP32/l=NCHW ]
name: out_DetectionOutput [ in: FP32/l=NCHW;  ]
[ INFO ] Device: CPU
[ INFO ] Collecting VOC annotations from C:\temp\intel-openvino-boxing\voc-export\annotations
[ INFO ] 498 annotations collected
[ INFO ] Starting inference
Progress: [....................] 100.00% done
[ INFO ] Processing output blobs
[ INFO ] Inference report:
        Network load time: 164.55ms
        Model: C:\temp\intel-openvino-boxing\transformed_frozen_inference_graph.xml
        Model Precision: FP32
        Batch size: 1
        Validation dataset: C:\temp\intel-openvino-boxing
        Validation approach: Object detection network
[ INFO ] Average infer time (ms): 66.29 (15.09 images per second with batch size = 1)
Average precision per class table:

Class   AP
1       0.000

Mean Average Precision (mAP): 0.0000

[Shubha_R_Intel]

Dear Sem:

Sometimes the reason for validation app issues is incorrect pre-processing parameters fed into model conversion using the Model Optimizer.

For example, we need to the check reverse_channel parameter of the model optimizer. It must be specified if the user trained his/her model in RGB layout because OpenVino samples usually use OpenCV to read an input image and OpenCV returns BGR.

Second, there are other pre-processing parameters to be checked, e.g. mean/scale values. It’s important to specify proper values here during the MO conversion. These details are described in the MO Documentation  for converting of tensorflow models.

And finally, you can use other object_detection samples/demos to infer this model. Do OpenVino samples infer on your model correctly ?

Sem, please also download the latest 2019 R1 Release and see if the problem persists despite that you are feeding the correct pre-processing parameters into the Model Optimizer tool.

Thanks,

Shubha

[Onyalo__Sem]

Hi Shubha,

I used the object_detection_sample_ssd sample with my model and it worked as expected.

Do you know why that sample would work but the validation app would not?

Thanks for your help.

[Shubha_R_Intel]

Dear Onyalo, Sem,

If the object_detection_sample_ssd worked as expected then I suspect a bug in the validation app. I will reproduce the issue with your attached zip file and definitely file a bug on your behalf.

Thanks for using OpenVino !

Shubha