Pardiso always crashing on Linux

[Essex_E_]

I consistently have Pardiso crash when running on Linux. I've trimmed it down to a relatively simple test case of building a Poisson matrix and solving it with Pardiso.  The same program runs fine on Windows 8 and 10. It segfaults on both Centos 6.7 and Ubuntu 14.04. I assume I'm doing something wrong, but I can't tell what.

I'm building and testing on Centos 6.7 (upgraded from 6.5, not originally installed as 6.7).  I'm using g++ 4.8.2.  I put a fresh install of MKL from "parallel_studio_xe_2016_composer_edition_for_cpp_update2.tgz" into my home directory. I'm intentionally using the TBB version of MKL, and not using OpenMP at all. When I started working on this, I was using OpenMP and Pardiso was still crashing in OpenMP instead of TBB.

GDB shows the segfault is during Pardiso phase 11, inside some tbbmalloc stuff.

Full Source file: https://drive.google.com/open?id=0B1-EScYNsy0uTDZNX2Etck9NM1U

Makefile: https://drive.google.com/file/d/0B1-EScYNsy0uNEZLQmx5dl80ZUk/view?usp=sharing

Lots of console output, versions, make, gdb, ldd: https://drive.google.com/file/d/0B1-EScYNsy0uTjFqU2lyQTVjdUU/view?usp=sharing

Any suggestions on what I should look at?

-Essex

[Gennady_F_Intel]

Essex, could you please check if the problem exists with the latest version of MKL we released a couple of days ago? This is version 11.3 update 3. The similar issue has been fixed into this update. Thanks. 

[Essex_E_]

Thank you very much for your response.

I have just installed update 3 (parallel_studio_xe_2016_composer_edition_for_cpp_update3.tgz).  I had to run install.sh twice. On the first install, mkl_tbb_thread.lib was missing, so I re-ran the installer to modify the installation and add TBB support to MKL.

My test program now runs without crashing, but it's only running a single thread. Running fine on a single thread is not new. It used to do that already. The segfault is in some parallel section. Unfortunately, I can't figure out how to get it to run with more threads. Changing the TBB_NUM_THREADS environment variable and calling mkl_set_num_threads(16) don't change anything. It used to just do it automatically.

If I switch from linking with mkl_tbb_thread.a to linking with mkl_intel_thread.a and libiomp5.a, then it runs with 4 threads and doesn't crash. If need be, I guess I can work with this, but the rest of my application is threaded with TBB. Are there correctness problems from using mkl_intel_thread in a TBB-parallelized application?

[Roman_A_Intel]

Hi Essex,

Actually, there is no way to set number of threads for TBB. In case of TBB threading Pardiso automatically detects the number of threads. Also if you set msglvl = 1 then you will see that number of threads always equal to 1 in case of TBB, but in fact Pardiso runs on multiple threads. The correct output in case of msglvl=1 will be fixed in one of the following releases.

Regards, Roman

[Essex_E_]

Okay, I see what you're saying.  The verbose output was saying things like "Parallel Direct Factorization is running on 1 OpenMP" and I interpreted this to mean that it's running on 1 thread, since there is no other output about threading. But if I watch the process in "top", that I see it's running 400% CPU (on my 4-core virtual machine).

To recap, 

On 11.3 update 2, I always segfault, due to some sort of bug in MKL.

On 11.3 update 3, using mkl_tbb_thread.a, the verbose Pardiso output doesn't say how many threads it used and reports it's only using 1 (irrelevant) OpenMP thread. I was confused by this, but "top" reveals it's working fine.

I think this is all resolved. Thanks.