the performance goes down after converting my customized yolov3 model into IR model

[mo__liheng]

Hi

I follow the instruction  here https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_tf_specific_Convert_YOLO_From_Tensorflow.html to convert the yolov3 model that trained with my own dataset into IR model and work on the NCS. But, I found that  the IR model present a worse performance than the original tensorflow model. with the original tensorflow model the mAP@0.5 is about 95% on my test set while the  the IR model is just 63%. the prediction of the coordinates of the boxes is not good. is there any idea that  I can try to improve the performance of the IR model and make the predicted coordinates more accurently? 

thank you.

[Luis_at_Intel]

Hi liheng,

Thanks for reaching out. Not sure what could be happening but I have seen other users run into issues because of how they are converting their model (missing flags, wrong parameters, etc.). May I ask how are you converting the model (i.e. full command)? If its possible please share your model so we can attempt to convert it, if you'd like sharing it via Private Message let me know. Are you using the latest OpenVINO release (R3.1) and which OS?

Regards,

Luis

[mo__liheng]

Hi Luis.

Thank you for your reply. 

The way I converted the model is using the following command on Linux with the openvino_2019.1.144:

python3 mo_tf.py --input_model /home/bo/yolo-v3/darknet_yolov3_meter.pb --tensorflow_use_custom_operations_config /home/bo/yolo-v3/yolo_v3_changed.json --input_shape [1,416,416,3] --data_type FP16 --output_dir /home/bo/yolo-v3/

The pb model and json file is attached, I am so glad if you can help and attempt to convert it for me. 

Regards,

Liheng

[Luis_at_Intel]

Thanks for the .pb and .json files, could you share the .cfg file as well? Also I can see that you are using an older version of OpenVINO, have you tested the latest release (R3.1) and observed the same issue? 

Regards,

Luis

[mo__liheng]

Hi

Luis.

Here is the cfg file and weights file I used.

I am also trying on the latest R3 to see whether the same issue will appear.

Regards

Liheng.

[Luis_at_Intel]

Thanks for the cfg file, I am able to convert the model to IR format. I have tried running the model using the object_detection_demo_yolov3_async.py sample script but I can't see any detections, may I ask if you have any sample images that I can test with or what object(s) your model is supposed to detect?

Also let me know if you are using the demo/sample applications provided in the OpenVINO toolkit, as you may need to use the --reverse_input_channels flag when converting the model to IR.

Regards,

Luis

[mo__liheng]

thank you for your help.

here is some test images and the classes name of my model, you can try.

Yes, I am also using the object_detection_demo_yolov3_async.py sample script to do the evaluation. And I also tried the R3 latest version but still have the same issues.

Will the reverse_input_channels flag have influence on the predictions of the bounding boxes?

Regards,

Liheng.

[mo__liheng]

the classes name of my model

[Luis_at_Intel · Accepted solution]

mo, liheng wrote:

Will the reverse_input_channels flag have influence on the predictions of the bounding boxes?

It shouldn't influence the prediction of the bounding boxes. The Inference Engine samples load input images in BGR channels order, most TF models are trained on images loaded in RGB channels order, so using this flag takes care of that. You can find more information about such flag here.

I have tested converting the model with such flag and I can see the accuracy you mentioned with the sample images provided. Please give a try at converting the model to IR using the flag mentioned, let me know if it works for you.

This is the command I used to convert to IR:

 python3 mo_tf.py --input_model ~/darknet_yolov3_meter.pb --tensorflow_use_custom_operations_config ~/yolo_v3_changed.json --input_shape [1,416,416,3] --reverse_input_channels --output_dir ~/ir_format/

Regards,

Luis

[mo__liheng]

thank you for your help.

Can you also share me the IR model that you have converted and I can try to do the evaluation by myself.

Regards,

Liheng

[Luis_at_Intel]

Sure, find the .zip file with the IR model files attached. I'm also including a screenshot of the results for both conversions; to the left  detection without the flag and to the right using the --reverse_input_channels flag.

Check if you see the same results as I did. Hope this helps, let me know if you need anything else.

Regards,

Luis

[mo__liheng]

Hi 

Thank you for your help and I finally fixed this problem.

But I have another problem that when I use the same  object_detection_demo_yolov3_async.py script on the NCS device with raspberry pi with the latest openvinoR3 version, the output of the detection result is strange. when using the CPU the issue doesn't happened. the photo is the comparison of while using -d CPU and -d MYRIAD.

the command I use for CPU:

python3 object_detection_demo_yolov3_async.py --model lrmodels/yolov3_meter_r3/FP32/r_channel/darknet_yolov3_meter.xml -d CPU --cpu_extension /opt/intel/openvino_2019.3.376/deployment_tools/inference_engine/lib/intel64/libcpu_extension_sse4.so

the command I use for MYRIAD on NCS raspberry pi:

python3 pi_object_detection_demo_yolov3_async.py --model lrmodels/yolov3_meter_r3/FP16/r_channel/darknet_yolov3_meter.xml -d MYRIAD

Could you also help me with this problem?

Regards

Liheng.