Segfault / terminate when loading models in different threads

[Ant__Michael]

Segfault / sigabrt sometimes occurs when networks are loaded in different threads. The problem can be reproduced with code in attached sample_code_multithread.zip

The problem is reproduced on the latest OpenVino 2020.2 release with both new and old models' versions (10 and 7), for example, face-detection-retail-0005. Backtrace for this case is attached as bt_2020R2.txt

The problem is also reproduced on the OpenVino 2020.1 release, but only for new version of the models - v10, for older models it seems to be not reproduced. Backtrace for this OpenVino version and the same model (face-detection-retail-0005) is attached as bt_2020R1.txt

Checked at different cpus: 
Core(TM) i7-8700 CPU @ 3.20GHz
Xeon(R) CPU E5-2697A v4 @ 2.60GHz

Xeon(R) Gold 6150 CPU @ 2.70GHz

Can you please confirm that segfault is reproduced or may I probably add some additional data to help with reproducing?

[dangu]

Hi Michael,

Saw your reply on the other thread (https://software.intel.com/en-us/forums/intel-distribution-of-openvino-toolkit/topic/852212), and thought I'd share my experience.

Experimentally I discovered Core::ReadNetwork was causing the issue. After protecting those calls with a mutex, I was able to call Infer on separate threads with different models and things seemed to work fine. This isn't based on a solid understanding of the code or anything, so your mileage may vary.

Surya's note about over-subscription was spot on though- the CPU plugin already appears to be making maximal use of the CPU, so inferring multiple models in parallel probably won't yield any performance improvement unless you're using other devices (GPU, VPU, etc.)

Hope this is helpful!

[SuryaPSC_Intel]

Hi Michael,

Hope dangu's findings help you to resolve the issue. You may refer to Optimization guide for more information on Multi threading in OpenVINO.

Best Regards,

Surya

[Ant__Michael]

Surya,

Yes, dangus' solution can help to avoid segfaults, but networks will be loaded sequentially at this case, that will significantly increase the total time of loading networks. So, is there any way of loading networks in different threads at the same time? 

Best Regards,
Michael

[Max_L_Intel]

Hi Michael.

For the inference on CPU device the Multi-threading is already implemented on CPU plugin level, so you don't need to develop a specific application for that purpose. You can have some control over it via "KEY_CPU_THREADS_NUM" and "KEY_CPU_BIND_THREAD" parameters - https://docs.openvinotoolkit.org/latest/_docs_IE_DG_supported_plugins_CPU.html#supported_configuration_parameters ;

For NUMA systems please also see tips in this section - https://docs.openvinotoolkit.org/latest/_docs_optimization_guide_dldt_optimization_guide.html#note-on-numa

Hope this helps.
Best regards, Max.