problem running inspector using command line

[Marc4]

Hello,

I try to run inspector from the terminal, with this command:

OMP_NUM_THREADS=4 inspxe-cl -collect ti3 -result-dir=./inspector/test1 -- ./install/Intel/Debug/test1_omp

Collection started. To stop the collection, either press CTRL-C or enter from another console window: inspxe-cl -r /export/home/tajchman/Workspace2/test/inspector/test1 -command stop.
Warning: Encountered an issue while loading result [/export/home/tajchman/Workspace2/test/inspector/test1]. Run terminated abnormally. Some data may be lost.

0 new problem(s) found

The OpenMP test code contains an error that was not flagged by Inspector (or that was not saved on disk) : 

__________________

double a, b;
#pragma omp parallel private(a)
{

  a = omp_get_thread_num() + 1.0; 
  // OpenMP error : shared variable "b" modified by several threads

  b = mysqrt(a);
}

__________________

(complete source code in attachment)

Can you tell me what's wrong with my command ?

Thanks in advance,

Marc

[Marc4]

The log file (inspector/test1/log/realtime_mode.log) shows that the code doesn't even start :

<?xml version="1.0" encoding="UTF-8"?>
<feedback>
<message severity="verbose">Analysis started...</message>
<nop/>
<message severity="info">Collection started. To stop the collection, either press CTRL-C or enter from another console window: inspxe-cl -r /export/home/tajchman/Workspace2/test/inspector/test1 -command stop.</message>
<nop/>
<message severity="verbose">Result file: /export/home/tajchman/Workspace2/test/inspector/test1/test1.inspxe </message>
<nop/>
<message severity="verbose">Found target process /export/home/tajchman/Workspace2/test/install/Intel/Debug/test1_omp (PID = 1859879). Analysis started... </message>
<nop/>
<message severity="verbose">Loaded module: /export/home/tajchman/Workspace2/test/install/Intel/Debug/test1_omp.</message>
<nop/>
<message severity="verbose">Loaded module: /lib64/ld-linux-x86-64.so.2. </message>
<nop/>
<message severity="verbose">Loaded module: [vdso]. </message>
<nop/>
<message severity="verbose">Loaded module: /lib64/libstdc++.so.6. </message>
<nop/>
<message severity="verbose">Loaded module: /lib64/libm.so.6. </message>
<nop/>
<message severity="verbose">Loaded module: /local/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin/libiomp5.so. </message>
<nop/>
<message severity="verbose">Loaded module: /lib64/libc.so.6. </message>
<nop/>
<message severity="verbose">Loaded module: /lib64/libgcc_s.so.1. </message>
<nop/>
<message severity="verbose">Loaded module: /lib64/librt.so.1. </message>
<nop/>
<message severity="verbose">Loaded module: /lib64/libpthread.so.0. </message>
<nop/>
<message severity="verbose">Loaded module: /lib64/libdl.so.2. </message>
<nop/>
<message severity="verbose">Loaded module: /local/opt/intel/oneapi/inspector/2021.4.0/lib64/runtime/libittnotify.so. </message>
<nop/>
<message severity="verbose">Loaded module: /local/opt/intel/oneapi/inspector/2021.4.0/lib64/libpinjitprofiling.so. </message>
<nop/>
<message severity="verbose">Application exit code: 139 </message>
<nop/>
<message severity="verbose">Result file: /export/home/tajchman/Workspace2/test/inspector/test1/test1.inspxe </message>
<nop/>
<message severity="verbose">Analysis completed</message>
<nop/>
<message severity="warning">Encountered an issue while loading result [/export/home/tajchman/Workspace2/test/inspector/test1]. Run terminated abnormally. Some data may be lost. </message>
<nop/>
<message severity="info"> </message>
<nop/>
<message severity="info">0 new problem(s) found </message>
<nop/>
</feedback>

[Michael_T]

Hello Marc,

The command line looks good. Return code 139 means that application crashed during analysis.

Could you please double-check if it runs fine in the same environment/console without Inspector. If it works fine without Inspector, but crashes during analysis, please compress and attach entire result directory here to investigate the root cause.

[Marc4]

Hello Michael,

Thanks for your suggestion.

It seems that the problem comes from the installation of intel compiler/tools on my personal computer (Windows 10).

I did some more tests:

• the application runs fine outside of inspector
• if I run the command on my company's cluster (CentOS 7), Intel Inspector finds the openmp error in my code. That's what I wanted
• Intel tools versions on the cluster are : 
  • icc -v
    icc version 19.1.0.166 (gcc version 4.8.5 compatibility)
  • inspxe-cl -version
    Intel(R) Inspector 2020 (build 603904) Command Line tool
• Intel tools versions on my personal computer :
  • icl.exe
    Intel(R) C++ Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.5.0 Build 20211109_000000
  • inspxe-cl.exe -version
    Intel(R) Inspector 2022.0 (build 605868) Command Line tool

You will find a copy of the results directory in my computer.

Thanks a lot

[Marc4]

I forget to say that I have the same problem on my desktop machine (Ubuntu 21.10):

icc -v
icc version 2021.4.0 (gcc version 8.5.0 compatibility)

inspxe-cl -version
Intel(R) Inspector 2021.4 (build 605683) Command Line tool

My first messages contains output from this machine

[Michael_T]

It looks like a bug in the tool. Windows version crashes with STATUS_STACK_BUFFER_OVERRUN. Could you please attach your application binary with all dependencies (e.g. include your version of omp5 library) so I can run it on my side? Either Linux on Windows whichever you prefer.

So far, I tried to build your example on my side. It works fine and expected data race is reported. Let's see if your binary behaves differently.

Also, don't you configure environment in some special way (e.g. limit stack size)?

[Marc4]

Hi Michael,

My desktop OS is Fedora 35 (I said that it was Ubuntu 21.10 : my mistake)

> uname -a
Linux is243723.intra.cea.fr 5.14.10-300.fc35.x86_64 #1 SMP Thu Oct 7 20:48:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

> ldd test_omp1 

linux-vdso.so.1 (0x00007ffee45f6000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f86f6a3c000)
libm.so.6 => /lib64/libm.so.6 (0x00007f86f6960000)
libiomp5.so => /opt/intel/oneapi/lib/intel64/libiomp5.so (0x00007f86f6540000)
libc.so.6 => /lib64/libc.so.6 (0x00007f86f6336000)
/lib64/ld-linux-x86-64.so.2 (0x00007f86f6c86000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f86f631c000)
librt.so.1 => /lib64/librt.so.1 (0x00007f86f6317000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f86f6310000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f86f630b000)

I send you "test_omp1" binary and its dependencies (couldn't find .

Also, I compile my code with Intel compiler icpc + gcc 8.5 because the native gcc on the computer (gcc 11.2) has incompatibilities with icpc (compilation errors).

But it seems that there are no dependencies on gcc 8.5 in the compiled code.

Best

Marc

[Marc4]

Oh, didn't finish my sentence : "couldn't find linux-vdso.so.1"

Sorry

[Marc4]

and I didn't configure my environment in a special way as far as I remember

> ulimit -a
real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63111
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 63111
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[Michael_T]

Hi Marc,

Thank you. I’ve reproduced the problem locally using your binaries. Investigating root cause of this failure right now.

[Michael_T]

Hi Marc,

We found one root cause of this crash on Linux. It is related to libc.so version included in Fedora 35 distribution. The fix will be included in next Inspector release. Intermediate workaround would be switching to older libc version for now.

You also observed crash on Windows, but it might be different issue. Don't you mind to share your Windows binaries, so we check that scenario as well?

[Marc4]

Sorry, I was away from my computers.

I will send you the windows binaries of my toy code this evening

[Marc4]

My test code on 2 machines:

- Windows 10 Entreprise (20H2 version 19042.1526) : Intel Inspector didn't catch the error

- Windows 11 Professional :  Intel Inspector catch the error

The sources and binaries in joint pieces

Thank you for your time

[Michael_T]

Hi Marc,

Thank you for reproducer and sorry for the delay. I had to contact OpenMP team to clarify details. It seems that we have discrepancy in code generated by compiler and OpenMP library that causes access violation issue when this library is loaded under analysis tool. The library has extra code to notify Inspector on worker threads and synchronization events which doesn’t work well if we use different versions of compiler to build the application and OpenMP library loaded in runtime. This problem is fixed in oneAPI 2022 package, but some older library versions could still lead to this crash.

Could you please try the following?

• Build application using latest oneAPI compiler
• Check folders in PATH environment variables and make sure “<oneAPI root>\compiler\latest\windows\compiler\lib\intel64” folder is before other folders that might have its own versions of OpenMP.

In your result folder I noticed that libiomp5md.dll is taken from “oneapi\intelpython\latest\library\bin” instead. It might be older incompatible version. If everything is correct, but you still experience the crash, try to set KMP_VERSION=1 and run analysis again to see exact version of OpenMP library used during this run.

[Gopika_Intel]

Hi,

We have not heard from you. Did you get to try Michael_T’s suggestions? Is your issue resolved, can we discontinue monitoring this thread?

Regards

Gopika

[Gopika_Intel]

Hi,

We assume that your issue is resolved. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Regards

Gopika