YOLO v3 detection accuracy on NCS2 is much lower than on NCS1

[Ezzat__Ahmed]

Test Configuration:

• YOLOv3 model with input size (320, 320).
• Windows 10 Machine.
• OpenVino Release 2019 R1 (133).
• Detection test runs on a video file.

Test Description:

Running the YOLOv3 detector on a video sequence ("test.mp4") using the NCS usb stick. The output video (frames overlayed with detection boxes) is saved.

I attached the whole test package with the output videos inside detection_out.

There are two video recordings for the output of the YOLOv3 detector on each stick NCS1 and NCS2.

The NCS v2 misses a lot of detection boxes, while the NCS v1 is working fine. 

Why does this happen? 

Is this issue expected to be fixed in the next release? 

Regards,

Ahmed

[Shubha_R_Intel]

Dear Ezzat, Ahmed,

First of all I see that you are using an old OpenVino Release 2019 R1 (133).. We are on OpenVino 2019 R1.1 (144) and are soon to release R2.

You should not see different accuracy between NCS1 and NCS2 on OpenVino

You state that you are using YOLOv3 model with input size (320, 320). Did you pass the correct input shape into Model Optimizer ?

Please try the latest OpenVino release and report your results here.

Thanks,

Shubha

[Ezzat__Ahmed]

Shubha R. (Intel) wrote:

Dear Ezzat, Ahmed,

First of all I see that you are using an old OpenVino Release 2019 R1 (133).. We are on OpenVino 2019 R1.1 (144) and are soon to release R2.

You should not see different accuracy between NCS1 and NCS2 on OpenVino

You state that you are using YOLOv3 model with input size (320, 320). Did you pass the correct input shape into Model Optimizer ?

Please try the latest OpenVino release and report your results here.

Thanks,

Shubha

Dear Shubha,

Thanks for fast reply.

1. For the OpenVino Latest release (1.148) I tried it and obtained the same results.
2. Of course I passed the correct size to the model optimizer (320, 320). Also, I tried the YOLOv3 with the (416, 416) input size and got the same results (OF course i run the model optimizer for this case with 416, 416).
3. Could you please run the test quickly at your side and verify my findings? You will notice the difference from the first few frames of the video if you do not have time to continue the whole video test.

Regards,

Ahmed

[Shubha_R_Intel]

Dear Ezzat, Ahmed,

Yes I will run your code and see for myself. Thanks for your patience,

Shubha

[Shubha_R_Intel]

Dear Ezzat, Ahmed,

We just released OpenVino 2019R2. Can you kindly try again ? Please report back here with your results.

Thanks !

Shubha

[Ezzat__Ahmed]

Dear Shubha,

I tried with the new release R2. The results are the same as I posted before.

I tried NCS1, NCS2 and GPU. The inference output from the NCS1 and the GPU are matched with accurate detection output.

The NCS2 output is significantly different. Most of the detection boxes are not detected. 

Still I cannot figure out what is wrong in case of NCS version 2?

Please help.

Regards,

Ahmed 

[Shubha_R_Intel]

Dear Ezzat, Ahmed,

Thanks for trying OpenVino R2. Yeah, this is definitely strange, I agree. Are you changing yolo_v3.json at all ? And which one are you using yolo_v3.json or yolo_v3_voc.json ? In your MO command are you using the *.json from 2019R2 ?  I promise to take a look this week.

Thanks for your patience,

Shubha

[Ezzat__Ahmed]

Dear Shubha,

I follow the yolo_v3.json since I am using the MS COCO YOLO trained model.

Regards,

Ahmed

[Shubha_R_Intel]

Dear Ezzat, Ahmed,

Please give me your frozen.pb also. I didn't see it in that archive which I downloaded. Is it just the default pre-trained frozen.pb ? Also, one more question - did you regenerate IR using OpenVino 2019R2 or did you use the old IR from previous releases ? If you didn't regenerate IR, please use OpenVino 2019R2 and re-generate IR, then try again.

I am looking forward to your response.

Thanks,

Shubha

[Shubha_R_Intel]

Dear Dear Ezzat, Ahmed,

This looks like a legitimate bug. Thanks for your patience. I regenerated the IR using OpenVino 2019R2 and in fact reproduced your results. NCS1 is drastically more accurate than NCS2.

When NCS2 detects, it does a good job - but it misses a lot of stuff which is detected by NCS1.

Thanks again for your patience,

Shubha

[Ezzat__Ahmed]

Thanks Shubha for your attention.

[Shubha_R_Intel]

Dear Ezzat, Ahmed,

Of course. Stay tuned. I will report back on this forum.

Shubha

[Shubha_R_Intel]

Dear Ezzat, Ahmed,

While I did reproduce the problem using your model, there is a problem with your model. Your model looks different from what we expect. Perhaps you forgot to use --tensorflow_use_custom_operations_config in your model optimizer command ?

If you follow These instructions to the letter you will get a perfect IR. From there you can run object_detection_demo_yolov3_async.py and the results between NCS1 and NCS2 are great (and similar).

The IR which you provided is missing RegionYolo layers. 

Hope it helps,

Thanks,

Shubha

[Ezzat__Ahmed]

Dear Shbha,

Yes I used a different approach of generating the YOLO model from the mystic123 repo. I cut the model at the layers mentioned in the yolov3.json (Reshape, Reshape_4, Reshape_8) and then writes the other parts (which are not Neural Network Layers) in numpy. I tested my code in the following platforms:

• TensorFlow               --> Working Fine
• OpenVino + GPU     --> Working Fine
• OpenVino + NCS1    --> Working Fine
• OpenVINO + NCS2   --> Not Working Well (as I reported and you reproduced)

So, for me it is obvious there is a problem in the NCS2.

Regards,

Ahmed

[Shubha_R_Intel]

Dearest Ahmed

NCS2 requires the original Yolo model to be unmodified as discussed.

Shubha

[Ezzat__Ahmed]

Dear Shubha,

Thanks for your help. I converted the model as in the OpenVino Documentation and it worked. My postprocessing is changed slightly to adapt for the converted model (removed the external sigmoid, which I figured out is carried inside the NCS and adapt the processing to the output shapes). I tried with both the code of the Example in the Open Model Zoo and my code. I found my code postprocessing is much faster.

Anyway, the problem is now solved. Thanks again.

Regards,

Ahmed

[Shubha_R_Intel]

Dear Ezzat, Ahmed,

Thanks for sharing your findings with the OpenVino community !

Shubha