OpenVINO demo hangs on iris_xe_max nodes

[Elvis_F_Intel]

I have an OpenVINO application that hangs on the iris_xe_max nodes. Looking into it, I also realized that the demos do hang as well. Please find below directions to reproduce:

Build the openvino demo:

source /opt/intel/openvino/bin/setupvars.sh

cd /opt/intel/openvino/deployment_tools/inference_engine/demos

./build_demos.sh

(This should build the demos to a subdirectory in your home directory: ~/omz_demos_build/intel64/Release)

Get pre-trained models:

cd ~/omz_demos_build/intel64/Release

 wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/1/face-detection-adas-0001/FP32/face-detection-adas-0001.xml

wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/1/face-detection-adas-0001/FP32/face-detection-adas-0001.bin

wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/1/head-pose-estimation-adas-0001/FP32/head-pose-estimation-adas-0001.bin

wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/1/head-pose-estimation-adas-0001/FP32/head-pose-estimation-adas-0001.xml

Get a sample video:

git clone https://github.com/intel-iot-devkit/sample-videos.git

Run demo on login node: (THIS RUNS FINE AND RETURNS IN ABOUT A MINUTE OR TWO)

./interactive_face_detection_demo -i sample-videos/head-pose-face-detection-female-and-male.mp4 -m face-detection-adas-0001.xml -m_hp head-pose-estimation-adas-0001.xml -no_show

Run demo on iris_xe_max_node: (THIS HANGS AND DOESN’T RETURN)

qsub -I -l nodes=1:iris_xe_max:ppn=2,walltime=08:0:0

cd ~/omz_demos_build/intel64/Release

source /opt/intel/openvino/bin/setupvars.sh

./interactive_face_detection_demo -i sample-videos/head-pose-face-detection-female-and-male.mp4 -m face-detection-adas-0001.xml -m_hp head-pose-estimation-adas-0001.xml -no_show

./interactive_face_detection_demo -i sample-videos/head-pose-face-detection-female-and-male.mp4 -m face-detection-adas-0001.xml -m_hp head-pose-estimation-adas-0001.xml -no_show -d GPU

Any assistance is appreciated,

Thanks,

Elvis.

[Iffa_Intel]

Greetings,

May I know which Linux distribution are you using? (eg Ubuntu 18.04).

Are you able to run the verification script ./demo_squeezenet_download_convert_run.sh. This is important to ensure that  you had set up the toolkit correctly as in here: https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_linux.html

Besides, It is recommended for you to use the latest OpenVINO toolkit version which is 2021.2, and ensures that you had the updated plugin for your hardware.

I had some issues when inferencing in Raspbian with NCS2 previously, but this latest version fixes it.

You may also get the latest model here:

https://download.01.org/opencv/2021/openvinotoolkit/2021.2/

Plus, for running the interactive_face_detection_demo try this:

./interactive_face_detection_demo -i 0 -m intel/face-detection-adas-0001/FP16/face-detection-adas-0001.xml -m_hp intel/head-pose-estimation-adas-0001/FP16/head-pose-estimation-adas-0001.xml -d GPU -d_hp GPU

Sincerely,

Iffa

[Elvis_F_Intel]

Iffa,

Thanks for your response.

I am running on DevCloud. I checked the Linux distribution and it is Ubuntu 20.04.

I also downloaded OpenVINO 2021.2 and used that instead of the default OpenVINO installation on DevCloud. There was a slight improvement as the models were now loaded onto the CPU. However, it still hangs trying to load the models to the GPU. Find below the output:

~/omz_demos_build/intel64/Release$ ./interactive_face_detection_demo -i sample-videos/head-pose-face-detection-female-and-male.mp4 -m face-detection-adas-0001.xml -m_hp head-pose-estimation-adas-0001.xml -no_show -d GPU -d_hp GPU
InferenceEngine: API version ......... 2.1
Build ........... 2021.2.0-1877-176bdf51370-releases/2021/2
[ INFO ] Parsing input parameters
[ INFO ] Loading device GPU

(It hangs at this point)

By the way, I have run this application successfully on my local server, and also on other DevCloud nodes. This issue only occurs on nodes with iris_xe_max GPUs.

Hope this further information is helpful. Any further assistance is appreciated.

Thanks,

Elvis.

[Iffa_Intel]

Fyi, this is the Update for DG1 (Iris® Xe MAX Graphics) GPU is supported starting with OV 2021.2 release - https://software.intel.com/content/www/us/en/develop/articles/openvino-relnotes.html#inpage-nav-1-2-3

You may try to check prerequisites (GPU driver) as per https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_linux.html#additional-GPU-steps

Sincerely,

Iffa

[Elvis_F_Intel]

Iffa,

Thanks for the response.

I dug into it a little more and it looks like the issue is coming from the Inference Engine trying to query the devices on the iris_xe_max systems. 

This is evident when building and running "hello_query_device" located at  <OpenVINO directory>/deployment_tools/inference_engine/samples/cpp.

It hangs when running GetAvailableDevices() to get the devices on the iris_xe_max system.

Can you confirm that as of now, one can successfully query a device through OpenVINO on an iris_xe_max DevCloud system? (Or by extension run a demo using an iris_xe_max GPU?)

Is it possible to get someone on the DevCloud side of things to ensure that the drivers etc needed to run OpenVINO are in place on the iris_xe_max nodes?

Thanks once more for your assistance.

Elvis. 

[Iffa_Intel]

Greetings,

Since the Iris® Xe MAX Graphics GPU is supported starting with OV 2021.2 release , the inferencing should work but you need to use this specific version of OpenVINO(2021.2): - https://software.intel.com/content/www/us/en/develop/articles/openvino-relnotes.html#inpage-nav-1-2-3

You may do the following on OV 2021.2: install GPU pre-requisites as per https://docs.openvinotoolkit.org/2021.2/openvino_docs_install_guides_installing_openvino_linux.html#additional-GPU-steps and ensure this is installed without any errors.

Then to run the verification script (demo squeezenet with -d GPU parameter).

If you still see any error for the verification script, then please help to create a new ticket in DevCloud for Edge forum - https://community.intel.com/t5/Intel-DevCloud-for-Edge/bd-p/devcloud-edge

Sincerely,

Iffa

[Elvis_F_Intel]

Iffa,

Thanks for your response.

As you can see from the output below, I am using OpenVINO 2021.2:

~/omz_demos_build/intel64/Release$ ./interactive_face_detection_demo -i sample-videos/head-pose-face-detection-female-and-male.mp4 -m face-detection-adas-0001.xml -m_hp head-pose-estimation-adas-0001.xml -no_show -d GPU -d_hp GPU
InferenceEngine: API version ......... 2.1
Build ........... 2021.2.0-1877-176bdf51370-releases/2021/2
[ INFO ] Parsing input parameters
[ INFO ] Loading device GPU

I don't know if creating a ticket on DevCloud for Edge will be helpful because as far as I know, there are no Iris_xe_max GPUs on DevCloud for Edge. The application runs fine on DevCloud for Edge using the integrated GPUs. The hangs only happen on DevCloud for oneAPI with the dedicated Iris_xe_max nodes. Please let me know where to file a bug report on DevCloud for oneAPI.

On another note, one cannot run install_NEO_OCL_driver.sh in the link you sent as a regular user, root privileges are required.

https://docs.openvinotoolkit.org/2021.2/openvino_docs_install_guides_installing_openvino_linux.html#additional-GPU-steps 

That's the reason I asked that you get someone on the DevCloud side of things to make sure all drivers are working properly on the Iris_xe_max nodes.

Like I mentioned previously, the Inference Engine fails when trying to query devices using GetAvailableDevices(). If you can get someone to verify and fix this, that will be great.

Thanks,

Elvis.

[RaeesaM_Intel]

Hi,

We have informed the DevCloud Admin team regarding the driver issue. We will get back to you on the updates.

Raeesa

[RaeesaM_Intel]

Hi ,

The issue with drivers have been resolved by the team. Could you please check whether the demo is working fine now .

Raeesa

[Elvis_F_Intel]

Hi Raeesa,

Yes, updating the drivers fixed the issues. The application runs fine now.

Thanks,

Elvis.

[RaeesaM_Intel]

Hi,

Thank you for the response. Glad that your issue got resolved . We are discontinuing monitoring this thread. Please raise a new one incase of any issues.

Regards,

Raeesa

[RaeesaM_Intel]

Hi,

Thank you for the response. Glad that your issue got resolved . We are discontinuing monitoring this thread. Please raise a new one incase of any issues.

Regards,

Raeesa