YOLOv5 Accuracy difference between CPU and MYRIAD X

[anujd9]

Hi. I have trained a YOLOv5s model taken from this repository [https://github.com/ultralytics/yolov5] on a custom dataset with 2 classes. 

I have converted the model from PyTorch to ONNX and to OpenVINO IR [OpenVINO version 2021.4.1 LTS]. 

There is a discrepancy between the model accuracy while inferencing using IR files on the CPU vs NCS2. When using CPU as the target, the model works well and both classes are detected with a high confidence of around 90 %. On the other hand, when I run the same IR files with NCS2, the accuracy for one class remains around 80%, but for the other class it drops down to 0.1% or so.

This means that when I am using CPU as the target, I can use a confidence threshold of around 0.2 to 0.5, whereas for NCS, this value drops down to 0.001 with some False Positives (due to a very low threshold).

I am using the latest version of OpenVINO that says it has the support and fixes specifically for YOLOv5, but still I see these issues. Can anyone please help me with this?

Thanks

[Zulkifli_Intel]

Hello Anuj Dutt,

Greetings to you.

Yolov5 is not officially supported by OpenVINO and has not been validated by us. So it may or may not work. Could you try converting the model using --scale 255?

Sincerely,

Zulkifli

[anujd9]

Hi @Zulkifli_Intel , thanks for your reply. Yes, I am using --scale255 while doing the conversion. The model works fine on the CPU as target but as I set MYRIADX as target, the model still works, but the accuracy for one of the two classes drops drastically.

I have also tried using my model with a Luxonis OAK-D device [using depthai API,  https://shop.luxonis.com/products/1098obcenclosure] that also uses a MYRIADX and the same model works perfectly fine there, but as soon as I run it on a NCS2 stick, the accuracy drops.

I don't know what's the difference there as both devices have the MYRIADX, same OpenVINO versions etc.

Also, OpenVINO 2021.4.1 LTS release notes say that YOLOv5 is supported [https://software.intel.com/content/www/us/en/develop/articles/openvino-2021-4-lts-relnotes.html], see below:

```

MYRIAD plugin:

• Performance fixes

• Fixed most of the sporadic performance drops down to 10% by prioritizing runtime threads

• Accuracy fixes

• Accuracy and working capability on Yolo-v3 model through ONNX importer were fixed
• Fixed Yolo-v5 accuracy

```

Thanks,

Anuj

[Zulkifli_Intel]

Hello Anuj Dutt,

Please share with us your model, MO command used to convert to IR, steps to reproduce the issue, and any other relevant info to help identify the possible issues.

Sincerely,

Zulkifli

[anujd9]

Hi @Zulkifli_Intel , attached is the zip file containing the ONNX model, IR model xml/bin files as well as the model blob file. This also contains sample images to test with and a demo script. I have tried to add as much info as possible in the ReadMe.md file.

Also, I have added the sample outputs from inferencing with the same model with different targets so that you can see the issue clearly.

Please let me know if you need anything else.

Thanks,

Anuj

[Zulkifli_Intel]

Hello Anuj Dutt,

Thank you for sharing the model and information with us. We will investigate this and revert the information to you soon.

Sincerely,

Zulkifli

[anujd9]

Hi @Zulkifli_Intel , any update on the issue?

Thanks.

[Zulkifli_Intel]

Hello Anuj Dutt,

In the Release Notes, the fixes for Yolo-V5 accuracy are based on the developer running the Accuracy_Check using the dataset for validation. The developer found out that disabling HW optimization for convolution solves the issue with accuracy and the root cause was the layer (the very first convolution in the net) that is the culprit for accuracy drop.

The Fixed Yolo-v5 accuracy for the MYRIAD plugin mentioned in the Release Note was based on fixing the above problem.

We suggest you utilize our Accuracy Checker tool to validate your trained model for both CPU and MYRIAD.

Sincerely,

Zulkifli 

[Zulkifli_Intel]

Hello Anuj Dutt,

This thread will no longer be monitored since we have provided the information on this issue. If you need any additional information from Intel, please submit a new question.

Sincerely,

Zulkifli