Application Main Thread is hung when loading GPU as a device

[Aankesh]

When loading GPU as a device  to run the inference, the app main thread is unresponsive for sometime. But this is not the case with CPU or other devices. The same behviour happens when loading complex models like semantic segmentation and object detection. 

-Aankesh

[Rizal_Intel]

Hi Aankesh,

This is a one-time overhead (order of few seconds) of compiling the OpenCL kernels.

The compilation happens upon loading the network to the GPU plugin and does not affect the inference time.

You can read more about GPU optimization here.

If you simply don’t want your main thread from freezing you could run the model asynchronously by running it on another thread.

Regards,

Rizal.

[Aankesh]

hi,

the actual issue is, while the compilation or Running in Async mode, the main threads are getting chocked, even when we are spinning other thread for loading the model. This is seen especially when loading complex models and also durin the GPU device loading.

-Aankesh

[Rizal_Intel]

Hi Aankesh,

The all thread freezing behavior is not expected.

However, you can use cl_cache method as a workaround to shorten the time for GPU loading.

Regards,

Rizal

[Aankesh]

Rizal, 

thanks . We wanted to understand why the thread is getting chocked. Wanted to understand if any interrupts are being masked which we are not able to get it through our debugging. 

-Aankesh 

[Rizal_Intel]

Hi Aankesh,

I did some testing, at it seems at least the keyboards interrupt is registered but deferred (executed later after the GPU plugin finished loading).

How was your debugging result, did the interrupts missing completely (not registered at all) or is it also deferred?

Regards,

Rizal

[Aankesh]

hi Rizal, 

from our end, all the interrupts are getting blocked. We built a Python GUI app that connects to the openvino. The moment GPU load begins, the entire GUI is hung and all the signals are getting blocked until the gpu load is successful even though we spin a separate thread for the openvino process. 

-Aankesh

[Aankesh]

hi Rizal,

do you have any updates for me on the issue?

-Aankesh 

[Ray_Lo_Intel]

Is there a way to 'cache' the compiled kernel? It is one of the deal breaker for demo as it hang way too long before anything works.

[Rizal_Intel]

Hi Raymond,

You can use cl_cache method as a workaround to shorten the time for subsequent GPU loading.

Regards,
Rizal

[Rizal_Intel]

Hi Aankesh,

We are still investigating this issue, we will inform you on any updates or development on the matter.

Regards,

Rizal

[Rizal_Intel]

Hi Aankesh,

The investigation shows that this script hits the well-known Python GIL problem, which is described in links below along with some clues on how to workaround it:

https://realpython.com/python-gil/

https://hackernoon.com/has-the-python-gil-been-slain-9440d28fa93d?gi=a9cc9f35638c 

So the problem reported could not be considered as an OpenVINO bug.

Regards,

Rizal

[Rizal_Intel]

Hi Aankesh,

Intel will no longer monitor this thread since we have provided a solution. If you need any additional information from Intel, please submit a new question. 

Regards,

Rizal

[Aankesh]

hi Rizal,

This issue is happening in c++ as well. Please validate from your end also.

-Aankesh 

[Rizal_Intel]

Hi Aankesh,

Would you kindly provide more details for the issue in C++?

Please provide some replication code if possible for me to verify this issue on C++.

Regards,

Rizal

[Rizal_Intel]

Hi Aankesh,

Thank you for your question.

Intel will no longer monitor this thread . If you need any additional information from Intel, please submit a new question. 

Regards,

Rizal