Benchmark tool failed using Movidius IntelMyriad X VPU

[JunalTinga]

Hello,

Please help. Tried to run DL Workbench inference performance using Intel Movidius Myriad X VPU as target device but failed due to the following remark "Benchmark tool failed".

Other target devices were successful - CPU and GPU.

OS: Ubuntu 18.04 LTS

Please see attached Error Log

ValueError: Did not find a value for number of MYRIAD streams

Thank you.

[Wan_Intel]

Hi JunalTinga,

Thanks for reaching out to us.

For your information, we encountered the same error when running DL Workbench with Docker command using Intel Movidius Myriad X VPU as the target device.


We are investigating this issue and we will update you at the earliest.

Regards,

Wan

[artyom_tugaryov]

Hello @JunalTinga ,

Thank you for using  OpenVINO DL Workbench and thank you for reporting the problem.

Now we are investigating to the problem that you reported and we will try to find a solution as soon as possible.

It would be great if you can help us answering for the question. Could you say which exactly way do you use to start DL Workbench? Do you run DL Workbench using python package openvino-workbench? Or using docker command directly? Can you share the command line that you use to run DL Workbench?

Thank you,

Artyom Tugarev

[JunalTinga]

Hello @artyom_tugaryov ,

Thank you for your reply.

To answer your follow up questions;

1. Could you say which exactly way do you use to start DL Workbench? Do you run DL Workbench using python package openvino-workbench? Or using docker command directly? --> using docker

2. Can you share the command line that you use to run DL Workbench? --> docker run -p 0.0.0.0:5665:5665 --name workbench --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -1) --device-cgroup-rule 'c 189:* rmw' -v /dev/bus/usb:/dev/bus/usb -it openvino/workbench:2021.4.2

Please note:

1. Docker command taken from https://docs.openvino.ai/latest/workbench_docs_Workbench_DG_Run_Locally.html

2. I have 3 NCS2 sticks, i tried 3 of them and produced the same error when used as target device. (DL Workbench on Localhost)

3. I have tried to use the DevCloud: https://notebooks.edge.devcloud.intel.com/services/workbench/dashboard/1/projects/41

but still encountered error when using NCS2 as target device. Error --> Got unrecognized code error from the server. JOB SCRIPT ERROR: Remote job failed, code: 1.

Target machine:

Machine: idc004nc2

Thank you.

Junal Tin-ga

Device: Intel Movidius Myriad X VPU

[artyom_tugaryov]

Thank you for your observation!

We are working on the fix and will provide it in one of the next release. 

Thank you,

Artyom Tugarev

[JunalTinga]

hello @artyom_tugaryov ,

Thank you for the information. However, may i ask when will be the next release with the fix? It is because i need to perform the test using the Myriad for my school's final capstone, and I am on a tight timeframe. Thank you for your support.

Regards,

Junal

[JunalTinga]

Hi @artyom_tugaryov , is there any work-around for this problem? or any other alternative i can use until the permanent fix will be released?

Thank you.

Regards,

Junal

[artyom_tugaryov]

Hello @JunalTinga ,

You can measure performance inside the DL Workbench docker container directly using benchmark application

To connect inside the container you have to use the command:

docker exec -it workbench bash

After running the command you will attached to the docker container and you can run the benchmark application:

source /opt/intel/openvino_2021/bin/setupvars.sh

python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /home/workbench/.workbench/models/2/original/person-detection-retail-0013.xml -i /home/workbench/.workbench/datasets/5/val/hard_hat_workers249.png -d MYRIAD -b 1 -nireq 3 -t 20 --report_type no_counters --report_folder

When the benchmark application will done you can find performance information in the output.

More information about benchmark app you can find by the following link: https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html#:~:text=You%20can%20quick%20start%20with,on%20various%20Intel%C2%AE%20platforms

If you have any questions about DL Workbench or benchmark application, please free free to ask in these thread

Artyom

[JunalTinga]

Hello @artyom_tugaryov , thank you. though it the benchmark tool doesn't have a gui, hopefully you will have the next release of the DL Workbench soon.

Thank you.

Regards,

Junal

[artyom_tugaryov]

Hello @JunalTinga 

We've  released the 2022.1 version of the DL Workbench. You can get it using docker pull command:

docker pull openvino/workbench:2022.1

And then run the new version of DL Workbench:

docker run -p 0.0.0.0:5665:5665 --name workbench_2022.1 --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -1) --device-cgroup-rule 'c 189:* rmw' -v /dev/bus/usb:/dev/bus/usb -it openvino/workbench:2022.1

 The bug with MYRIAD should be fixed in this version.

Thanks,

Artyom

[JunalTinga]

hello @artyom_tugaryov,

I tested the new release and its working! thank you very much!

Regards,

Junal

[Wan_Intel]

Hi Artyom_Tugaryov,

Thanks for sharing your answer in the OpenVINO™ community!

Hi JunalTinga,

Thanks for your question.

Glad to know that your issue has been resolved.

Therefore, this thread will no longer be monitored since this issue has been resolved. 

If you need any additional information from Intel, please submit a new question.

Regards,

Wan