- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I ran the validation app using a model that I trained and I am getting an mAP of 0. I've used this same model in OpenCV's TensorFlow Object Detection API with success so I know I should be getting a higher mAP than zero. I've pasted my run conditions below. Can you let me know what I am doing wrong? Thanks!
Run Conditions
OS: Windows 10
Trained Model: MobileNet-SSD v1
Model, Images and Annotations: intel-openvino-boxing.zip
Command:
validation_app.exe -d CPU -t OD -ODa "C:\temp\intel-openvino-boxing\voc-export\annotations" -i "C:\temp\intel-openvino-boxing" -m "C:\temp\intel-openvino-boxing\transformed_frozen_inference_graph.xml" -ODc "C:\temp\intel-openvino-boxing\VOC_SSD_Classes.txt" -ODsubdir images
Output
[ INFO ] InferenceEngine:
API version ............ 1.4
Build .................. 19154
[ INFO ] Parsing input parameters
[ INFO ] Loading plugin
API version ............ 1.5
Build .................. win_20181005
Description ....... MKLDNNPlugin
[ INFO ] Loading network files
[ INFO ] Preparing input blobs
[ INFO ] Batch size is 1
name: image_tensor [ out: FP32/l=NCHW ]
name: Preprocessor/mul/FusedPower_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_0/Conv2D [ in: FP32/l=NCHW; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_1_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_2_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_2_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_3_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_4_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_5_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_6_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_6_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_7_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_7_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_8_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_8_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_9_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_9_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_10_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_10_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_11_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_11_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_12_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_12_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_13_depthwise/depthwise [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_13_pointwise/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_1_Conv2d_2_1x1_256/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_2_3x3_s2_512/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_1_Conv2d_3_1x1_128/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_1_Conv2d_4_1x1_128/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_1_Conv2d_5_1x1_64/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_5_3x3_s2_128/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_5/ClassPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_5/ClassPredictor/Conv2D_nChw8c_nchw_BoxPredictor_5/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_5/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_5/ClassPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_5/ClassPredictor/BiasAdd/Reshape___concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_5/BoxEncodingPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_5/BoxEncodingPredictor/Conv2D_nChw8c_nchw_PriorBoxClustered_5 [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: PriorBoxClustered_5 [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_5/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_5/BoxEncodingPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_5/BoxEncodingPredictor/BiasAdd/Reshape___concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_4/ClassPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_4/ClassPredictor/Conv2D_nChw8c_nchw_BoxPredictor_4/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_4/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_4/ClassPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_4/ClassPredictor/BiasAdd/Reshape___concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_4/BoxEncodingPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_4/BoxEncodingPredictor/Conv2D_nChw8c_nchw_PriorBoxClustered_4 [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: PriorBoxClustered_4 [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_4/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_4/BoxEncodingPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_4/BoxEncodingPredictor/BiasAdd/Reshape___concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_3/ClassPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_3/ClassPredictor/Conv2D_nChw8c_nchw_BoxPredictor_3/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_3/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_3/ClassPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_3/ClassPredictor/BiasAdd/Reshape___concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_3/BoxEncodingPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_3/BoxEncodingPredictor/Conv2D_nChw8c_nchw_PriorBoxClustered_3 [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: PriorBoxClustered_3 [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_3/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_3/BoxEncodingPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_3/BoxEncodingPredictor/BiasAdd/Reshape___concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_2/ClassPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_2/ClassPredictor/Conv2D_nChw8c_nchw_BoxPredictor_2/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_2/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_2/ClassPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_2/ClassPredictor/BiasAdd/Reshape___concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_2/BoxEncodingPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_2/BoxEncodingPredictor/Conv2D_nChw8c_nchw_PriorBoxClustered_2 [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: PriorBoxClustered_2 [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_2/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_2/BoxEncodingPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_2/BoxEncodingPredictor/BiasAdd/Reshape___concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_1/ClassPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_1/ClassPredictor/Conv2D_nChw8c_nchw_BoxPredictor_1/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_1/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_1/ClassPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_1/ClassPredictor/BiasAdd/Reshape___concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_1/BoxEncodingPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_1/BoxEncodingPredictor/Conv2D_nChw8c_nchw_PriorBoxClustered_1 [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: PriorBoxClustered_1 [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_1/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_1/BoxEncodingPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_1/BoxEncodingPredictor/BiasAdd/Reshape___concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_0/ClassPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_0/ClassPredictor/Conv2D_nChw8c_nchw_BoxPredictor_0/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_0/ClassPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_0/ClassPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_0/ClassPredictor/BiasAdd/Reshape___concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: concat_1 [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: do_ExpandDims_conf [ in: FP32/l=CHW; out: FP32/l=NCHW ]
name: do_ExpandDims_conf/sigmoid [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: do_reshape_conf [ in: FP32/l=NCHW; out: FP32/l=NC ]
name: BoxPredictor_0/BoxEncodingPredictor/Conv2D [ in: FP32/l=BLOCKED; out: FP32/l=BLOCKED ]
name: BoxPredictor_0/BoxEncodingPredictor/Conv2D_nChw8c_nchw_PriorBoxClustered_0 [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: PriorBoxClustered_0 [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: ConcatPriorBoxesClustered [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: BoxPredictor_0/BoxEncodingPredictor/Conv2D_nChw8c_nchw_BoxPredictor_0/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=BLOCKED; out: FP32/l=NCHW ]
name: BoxPredictor_0/BoxEncodingPredictor/Conv2D/Permute_ [ in: FP32/l=NCHW; out: FP32/l=NCHW ]
name: BoxPredictor_0/BoxEncodingPredictor/BiasAdd/Reshape [ in: FP32/l=NCHW; out: FP32/l=CHW ]
name: BoxPredictor_0/BoxEncodingPredictor/BiasAdd/Reshape___concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: concat [ in: FP32/l=CHW; out: FP32/l=CHW ]
name: do_reshape_loc [ in: FP32/l=CHW; out: FP32/l=NC ]
name: DetectionOutput [ in: FP32/l=NC; out: FP32/l=NCHW ]
name: out_DetectionOutput [ in: FP32/l=NCHW; ]
[ INFO ] Device: CPU
[ INFO ] Collecting VOC annotations from C:\temp\intel-openvino-boxing\voc-export\annotations
[ INFO ] 498 annotations collected
[ INFO ] Starting inference
Progress: [....................] 100.00% done
[ INFO ] Processing output blobs
[ INFO ] Inference report:
Network load time: 164.55ms
Model: C:\temp\intel-openvino-boxing\transformed_frozen_inference_graph.xml
Model Precision: FP32
Batch size: 1
Validation dataset: C:\temp\intel-openvino-boxing
Validation approach: Object detection network
[ INFO ] Average infer time (ms): 66.29 (15.09 images per second with batch size = 1)
Average precision per class table:
Class AP
1 0.000
Mean Average Precision (mAP): 0.0000
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear Sem:
Sometimes the reason for validation app issues is incorrect pre-processing parameters fed into model conversion using the Model Optimizer.
For example, we need to the check reverse_channel parameter of the model optimizer. It must be specified if the user trained his/her model in RGB layout because OpenVino samples usually use OpenCV to read an input image and OpenCV returns BGR.
Second, there are other pre-processing parameters to be checked, e.g. mean/scale values. It’s important to specify proper values here during the MO conversion. These details are described in the MO Documentation for converting of tensorflow models.
And finally, you can use other object_detection samples/demos to infer this model. Do OpenVino samples infer on your model correctly ?
Sem, please also download the latest 2019 R1 Release and see if the problem persists despite that you are feeding the correct pre-processing parameters into the Model Optimizer tool.
Thanks,
Shubha
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Shubha,
I used the object_detection_sample_ssd sample with my model and it worked as expected.
Do you know why that sample would work but the validation app would not?
Thanks for your help.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear Onyalo, Sem,
If the object_detection_sample_ssd worked as expected then I suspect a bug in the validation app. I will reproduce the issue with your attached zip file and definitely file a bug on your behalf.
Thanks for using OpenVino !
Shubha

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page