Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

opencv-vino 4.2.0 DNN exception on exit

Behrens__Henrik
Beginner
1,136 Views

Hi, I have installed openvino_2020.1.023 on macos catalina successfully.
Tests are running, samples are running, python examples with openvino ie are running.
All tests made on CPU and on MYRIAD (NCS2) - all ok.
OpenCV DNN coming with openvino distribution is also running, so where is the problem?
Here is a minimal python test which causes the error:

import cv2
net = cv2.dnn.readNetFromCaffe("face-detection-retail-0044.prototxt", "face-detection-retail-0044.caffemodel")
net.setPreferableTarget(cv2.dnn.DNN_TARGET_MYRIAD)
frame = cv2.imread("test.jpg")
(h, w) = frame.shape[:2]
blob = cv2.dnn.blobFromImage(cv2.resize(frame, (300, 300)), 1.0, (300, 300), (104.0, 177.0, 123.0))
net.setInput(blob)
net.forward()
exit(0)

net.forward runs perfectly (not shown in the example above, because not relevant), but in exit() the following exception is thrown:

libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument

There's no exception thrown when running the same example on CPU (net.setPreferableTarget(cv2.dnn.DNN_TARGET_CPU))

==> I can run OpenCV DNN on live video streams on MYRIAD or CPU for hours without any problem
==> The exception is thrown only at exit()
==> It seems that there is a problem in stopping/closing/releasing the cv2.dnn when connected to the OpenVino DNN_TARGET = MYRIAD device.

I can provide the system error report, when necessary.
Henrik
 

0 Kudos
5 Replies
Maksim_S_Intel
Employee
1,136 Views

This is known problem with Myriad plugin on OSX, as a workaround try to create another instance of InferenceEngine::Core and enumerate devices:

# NOTE: syntax can be slightly wrong as I'm not familiar with IE Python API
core = IECore()
print(core.available_devices)
0 Kudos
Behrens__Henrik
Beginner
1,136 Views

Hi Maksim,
I'm not sure about your intention ... here is the output of you code fragment:
['CPU', 'MYRIAD']
This is clear.
I do not have a problem running my models on MYRIAD, I have a problem in exit() - when exiting the application!
When exiting the program, there's definitely a problem cleaning up cv2.dnn after MYRIAD was used.
!! This only affects opencv.dnn - when using IE directly, exit works fine - so, the problem can be isolated to opencv.dnn coming with your OpenVino distribution. And this problem is limited to the MYRIAD plugin.
Below I have added the upper interesting part of the system error report, along with some comments.
Do you need the complete report?
By the way: The small NCS2 is pretty cool! Did some cv detections with several models on live video streams - 10-15 fps, compared to 15-30 fps on my 10-core i9 ...
Best Regards,
Henrik

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Application Specific Information:
terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
abort() called

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib            0x00007fff6e0f87fa __pthread_kill + 10
1   libsystem_pthread.dylib           0x00007fff6e1b5bc1 pthread_kill + 432
2   libsystem_c.dylib                 0x00007fff6e07fa1c abort + 120
3   libc++abi.dylib                   0x00007fff6b11dbe8 abort_message + 231
4   libc++abi.dylib                   0x00007fff6b11dd84 demangling_terminate_handler() + 238
5   libobjc.A.dylib                   0x00007fff6cc45792 _objc_terminate() + 104
6   libc++abi.dylib                   0x00007fff6b12adc7 std::__terminate(void (*)()) + 8
7   libc++abi.dylib                   0x00007fff6b12ad79 std::terminate() + 41
## COMMENT First call: libmyriadPlugin.dylib destructor of InferenceEngine::PluginBase<vpu::MyriadPlugin::Engine>::~PluginBase()
8   libmyriadPlugin.dylib             0x0000000132a5696a InferenceEngine::PluginBase<vpu::MyriadPlugin::Engine>::~PluginBase() + 170
## COMMENT Second call: libmyriadPlugin.dylib destructor of InferenceEngine::PluginBase<vpu::MyriadPlugin::Engine>::~PluginBase()
9   libmyriadPlugin.dylib             0x0000000132a5424c InferenceEngine::PluginBase<vpu::MyriadPlugin::Engine>::~PluginBase() + 28

## COMMENT Seems, that destructing InferenceEngine::InferencePlugin causes the problem ...
10  libinference_engine.dylib  
       0x0000000128d10f90 std::__1::__tree<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, InferenceEngine::InferencePlugin>, std::__1::__map_value_compare<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, InferenceEngine::InferencePlugin>, InferenceEngine::details::CaselessLess<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, InferenceEngine::InferencePlugin> > >::destroy(std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, InferenceEngine::InferencePlugin>, void*>*) + 96
11  libinference_engine.dylib         0x0000000128d10f6c std::__1::__tree<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, InferenceEngine::InferencePlugin>, std::__1::__map_value_compare<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, InferenceEngine::InferencePlugin>, InferenceEngine::details::CaselessLess<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, InferenceEngine::InferencePlugin> > >::destroy(std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, InferenceEngine::InferencePlugin>, void*>*) + 60

12  libinference_engine.dylib         0x0000000128d02dd2 InferenceEngine::Core::Impl::~Impl() + 258
## COMMENT Here you can see libopencv_dnn.4.2.0.dylib calling the destructor of InferenceEngine::Core::Impl
13 
libopencv_dnn.4.2.0.dylib         0x000000010fe61bf1 InferenceEngine::Core::~Core() + 49
## COMMENT Here you can see libopencv_dnn.4.2.0.dylib calling the destructor of InferenceEngine::Core

14  libsystem_c.dylib                 0x00007fff6e05a446 __cxa_finalize_ranges + 319
15  libsystem_c.dylib                 0x00007fff6e05a71c exit + 55
16  org.python.python                 0x000000010dff8042 Py_Exit + 30
17  org.python.python                 0x000000010dffe11c handle_system_exit + 314
18  org.python.python                 0x000000010dffdd7c PyErr_PrintEx + 54
19  org.python.python                 0x000000010dffd563 PyRun_SimpleFileExFlags + 963
20  org.python.python                 0x000000010e015abe pymain_main + 5445
21  org.python.python                 0x000000010e01612c _Py_UnixMain + 56
22  libdyld.dylib                     0x00007fff6dfb17fd start + 1

 

0 Kudos
Maksim_S_Intel
Employee
1,136 Views

No, additional information is not needed. The intention is to keep one instance of InferenceEngine::Core object to keep Myriad plugin from unloading. It helps in my C++ reproducer application at least. So your application will look like this:

import cv2
from ie_api import IECore

core = IECore() # we keep this instance in our application to avoid problem with unloading
print(core.available_devices) # this will load libMyriadPlugin.dylib

net = cv2.dnn.readNetFromCaffe("face-detection-retail-0044.prototxt", "face-detection-retail-0044.caffemodel")
net.setPreferableTarget(cv2.dnn.DNN_TARGET_MYRIAD)
frame = cv2.imread("test.jpg")
(h, w) = frame.shape[:2]
blob = cv2.dnn.blobFromImage(cv2.resize(frame, (300, 300)), 1.0, (300, 300), (104.0, 177.0, 123.0))
net.setInput(blob)
net.forward()
exit(0)

 

0 Kudos
Behrens__Henrik
Beginner
1,136 Views

Hi, I tested with

from openvino.inference_engine.ie_api import IECore

because ie_api can't be accessed without path on my system.
Output is ['CPU', 'MYRIAD'] as expected, but in exit(0), I get the same uncaught exception as before.

libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument

Any other idea about that?
Best Regards,
Henrik

0 Kudos
Behrens__Henrik
Beginner
1,136 Views

I installed 2020.2 - unfortunately, same error on exit:

libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
Abort trap: 6

This exception is thrown in exit(0) - the app is running perfectly.

Info about used Python and cv2:

Python 3.7.6 (default, Dec 30 2019, 19:38:26)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.3.0-openvino'

Is there anyone who encounters the same problem?
Best Regards,
Henrik
 

0 Kudos
Reply