Yolo v3 tiny 0 map when converting to openvino

[Ruz_Gómez__Rafael]

I'm having trouble converting yolov3 tiny from darknet to openvino IR format. When I run the accuracy_check script, I get 0 map (mean average precision). Here are the commands I exeecuted to convert the model to IR and evaluate:

• First I convert from darknet weights to tensorflow frozen graph:

python3 /scratch/tfg-adas/models/yolo_v3/original/tensorflow-yolo-v3/convert_weights_pb.py \
	--class_names /scratch/tfg-adas/datasets/kitti/kitti_custom.names  \
	--data_format NHWC \
	--weights_file /scratch/tfg-adas/models/yolo_v3/kitti_finetuned/yolo_v3_tiny_original_training_50000.weights \
	--output_graph /scratch/tfg-adas/models/yolo_v3/kitti_finetuned/frozen_darknet_yolo_v3_tiny_FP32_model.pb \
	--data_type FP32 \
	--tiny

File /scratch/tfg-adas/datasets/kitti/kitti_custom.names contains:

person
bycicle
car

/scratch/tfg-adas/models/yolo_v3/kitti_finetuned/yolo_v3_tiny_original_training_50000.weights are the trained weights on custom dataset.

• Second, I use the model optimizer to conver from the frozen graph to IR:

python3 ~/intel/openvino2020/deployment_tools/model_optimizer/mo_tf.py \
	--input_model /scratch/tfg-adas/models/yolo_v3/kitti_finetuned/frozen_darknet_yolo_v3_tiny_FP32_model.pb  \
	--transformations_config ~/intel/openvino2020/deployment_tools/model_optimizer/extensions/front/tf/yolo_v3_tiny.json \
	--output_dir /scratch/tfg-adas/models/yolo_v3/kitti_finetuned/IR/FP32 \
	--model_name yolo_v3_tiny \
	--data_type FP32 \
	--batch 1 

File ~/intel/openvino2020/deployment_tools/model_optimizer/extensions/front/tf/yolo_v3_tiny.json contains:

[
  {
    "id": "TFYOLOV3",
    "match_kind": "general",
    "custom_attributes": {
      "classes": 3,
      "anchors": [10, 14, 23, 27, 37, 58, 81, 82, 135, 169, 344, 319],
      "coords": 4,
      "num": 6,
      "masks": [[3, 4, 5], [0, 1, 2]],
      "entry_points": ["detector/yolo-v3-tiny/Reshape", "detector/yolo-v3-tiny/Reshape_4"]
    }
  }
]

• Finally, I run the accuracy_check script:

accuracy_check \
	-c /scratch/tfg-adas/models/yolo_v3/configs/yolo-v3-tf.yml  \
	-m /scratch/tfg-adas/models/yolo_v3/kitti_finetuned/IR/FP32 \
	-s /scratch/tfg-adas/datasets/kitti/voc_converted/ \
	-a /scratch/tfg-adas/datasets/kitti/openvino_annotations/ \
	-tf dlsdk \
	-tt FP32 \
	-td CPU

File /scratch/tfg-adas/models/yolo_v3/configs/yolo-v3-tf.yml contains:

models:
	- name: yolo_v3_tiny
		launchers:
		- framework: dlsdk
			tags:
			- FP32
			model:   yolo_v3_tiny.xml
			weights: yolo_v3_tiny.bin
			adapter:
			type: yolo_v3
			anchors: "10,14,  23,27,  37,58,  81,82,  135,169,  344,319"
			num: 6
			coords: 4
			classes: 3
			anchor_masks: [[3, 4, 5], [0, 1, 2]]
		
		datasets:
		- name: VOC2012
			preprocessing:
			- type: resize
			  size: 416
			  aspect_ratio_scale: fit_to_window
			- type: padding
			  size: 416
			  
			data_source: VOCdevkit/VOC2012/JPEGImages
			annotation: voc12_test.pickle
			dataset_meta: voc12_test.json
			
			postprocessing:
			  - type: resize_prediction_boxes
			  - type: filter
				apply_to: prediction
				min_confidence: 0.001
				remove_filtered: True
			  - type: nms
				overlap: 0.5
			  - type: clip_boxes
				apply_to: prediction
				
			metrics:
			  - type: map
				integral: 11point
				ignore_difficult: true
				presenter: print_vector
			  - type: coco_precision
				max_detections: 100
				threshold: 0.5

• Results:

1500 objects processed in 77.542 seconds
map@map@person: 0.01%
map@map@bicycle: 0.00%
map@mean: 0.00%
coco_precision: 0.00%

I've tried changing the preprocessing in the .yml file to just resizing to fixed shape 416x416 ignoring the aspect ratio, but I was getting same results. Since YOLOv3 keeps aspect ratio, I tried adding the fit_to_window, but it says the dimensions cannot be reshaed to (1,3,416,416). If I add padding like the version above, it works now but I get same results. Any help?

[Munesh_Intel]

Hi Rafael,

It seems that you are using two different datasets, namely Kitti dataset for training and VOC2012 dataset for validation.

I would suggest you try accuracy check using the training dataset first to confirm that Model Optimizer to Intermediate Representation (IR) conversion is fine.

If you still get the same result as you posted on this question, then the problem might be in the Model Optimizer conversion process. Thereafter, I would suggest you try these two approaches:

(i) Adding the --reverse_input_channels parameter in your Model Optimizer conversion command.

More information can be found at the following page:

https://docs.openvinotoolkit.org/2020.3/_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html

(ii) Update your TensorFlow version

OpenVINO supports TensorFlow r1.2 or higher, upto r1.15 only.

Regards,

Munesh

[Ruz_Gómez__Rafael]

Hello Munseh,

first of all, thank you for your answer. Just to clarify, it's the same dataset, named KITTI dataset. It has it's own way to represent the bounding boxes annotations, so VOC2012 is just the same dataset but converted to PASCAL VOC format. I use a training/validation partition of this dataset, but I've also tried different tensorflow models like SSD and SSDLite and they are working fine (they don't give 0 map when evaluating) on the same dataset.

(i) About your suggestions, I've tried the parameter --reverse_input_channels parameter in the Model Optimizer but unfortunately I'm getting the same results.

(ii) I'm using tensorflow 1.14 as the object detection installing tutorial suggested, but I'm not actually using the framework. For the conversion from darknet weights to tensorflow frozen graph I'm using the github repository of mystic123 as the openvino documentation for converting YOLO models says.

Regards,

Rafael

[Munesh_Intel]

Hi Rafael,

I would suggest you validate the YoloV3 model with Kitti dataset (in the original format) to make sure the conversion to IR is correct.

Regards,

Munesh