Missing traceback with OpenMP threads activated

[Olivier_C_]

To get a traceback, we set the -traceback option to the compiler.
However, after meeting an OpenMP instruction (threads created), there is no more traceback.
Here is a sample program:
$ cat toto.f90
program toto
implicit none
integer :: iexit
integer(KIND=4) :: ier
!$ integer(kIND=4),external :: omp_get_num_threads
!$OMP PARALLEL
!$OMP MASTER
!$ print*,'Threads=',omp_get_num_threads()
!$OMP END MASTER
!$OMP END PARALLEL
do iexit=-2, 2
print*,'iexit=',1/iexit
enddo
end program toto
$ ifort -V -traceback -g -C -openmp -c toto.f90 -o toto.o && ifort -g -openmp -o a.toto toto.o && ./a.toto ; ldd ./a.toto
Intel Fortran Intel 64 Compiler Professional for applications running on Intel 64, Version 11.0 Build 20080930 Package ID: l_cprof_p_11.0.069
Copyright (C) 1985-2008 Intel Corporation. All rights reserved.
FOR NON-COMMERCIAL USE ONLY

Intel Fortran 11.0-1549
toto.f90(6): (col. 9) remark: OpenMP DEFINED REGION WAS PARALLELIZED.
Threads= 4
iexit= 0
iexit= -1
Aborted
linux-vdso.so.1 => (0x00007fff647fd000)
libm.so.6 => /lib64/libm.so.6 (0x00002ac2465f7000)
libiomp5.so => /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002ac246879000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002ac246a09000)
libc.so.6 => /lib64/libc.so.6 (0x00002ac246c24000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002ac246f79000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002ac247186000)
/lib64/ld-linux-x86-64.so.2 (0x00002ac2463dc000)
Does anyone have an Intel compiler release/platform that provides a traceback ?
Without OpenMP, we get
$ ifort -V -traceback -g -C -c toto.f90 -o toto.o && ifort -g -openmp -o a.toto toto.o && ./a.toto
Intel Fortran Intel 64 Compiler Professional for applications running on Intel 64, Version 11.0 Build 20080930 Package ID: l_cprof_p_11.0.069
Copyright (C) 1985-2008 Intel Corporation. All rights reserved.
FOR NON-COMMERCIAL USE ONLY

Intel Fortran 11.0-1549
iexit= 0
iexit= -1
forrtl: severe (71): integer divide by zero
Image PC Routine Line Source
a.toto 000000000040302E MAIN__ 12 toto.f90
a.toto 0000000000402F7C Unknown Unknown Unknown
libc.so.6 00002B08A385D074 Unknown Unknown Unknown
a.toto 0000000000402EA9 Unknown Unknown Unknown
which is what is expected from the -traceback option.

[Kevin_D_Intel]

Thanks for the convenient reproducer. Also occurs with our 11.1 release on non-Mandriva. I noted you also submitted to Intel Premier so I will try to coordinateinvestigation with the assigned Support Engineer and update as we learn more.

[Olivier_C_]

Quoting - Kevin Davis (Intel)

Thanks for the convenient reproducer. Also occurs with our 11.1 release on non-Mandriva. I noted you also submitted to Intel Premier so I will try to coordinateinvestigation with the assigned Support Engineer and update as we learn more.

Today I made the test on an old Xeon with a CentOS, a three years old one.
I got the traceback with 11.0.69 !
As well, I tried the test on an old Pentium III. It failed with 10.1.008 but passed with 9.1.039 !
Since the problem seems to be very operating system related, I think a survey in that forum might be more fruitful there than by the traditional support.
Thank you for your feedback, I need more feedback from other users.
Sincerely,
Olivier Cessenat.

[Olivier_C_]

Quoting - cessenat@free.fr

Today I made the test on an old Xeon with a CentOS, a three years old one.
I got the traceback with 11.0.69 !
As well, I tried the test on an old Pentium III. It failed with 10.1.008 but passed with 9.1.039 !
Since the problem seems to be very operating system related, I think a survey in that forum might be more fruitful there than by the traditional support.
Thank you for your feedback, I need more feedback from other users.
Sincerely,
Olivier Cessenat.

Apologies, on Pentium is OK with 9.1.039 and sometimes with 10.1.008 (not always) but fails on 11.0.069. In fact it depends on the libguide runtime library since the same executable is OK or not depending on libguide as follows:
<<
KO:
$ a.toto_chaillot_ifort10
Threads= 1
iexit= 0
iexit= -1
forrtl: severe (71): integer divide by zero
Image PC Routine Line Source

Stack trace terminated abnormally.
$ ldd a.toto_chaillot_ifort10
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0xb7f5a000)
libguide.so => /opt/intel/Compiler/11.0/069/lib/ia32/libguide.so (0xb7ee4000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7ed2000)
libdl.so.2 => /lib/libdl.so.2 (0xb7ece000)
libc.so.6 => /lib/tls/libc.so.6 (0xb7d9b000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7d90000)
/lib/ld-linux.so.2 (0xb7f99000)
<<
OK:
a.toto_chaillot_ifort10
Threads= 1
iexit= 0
iexit= -1
forrtl: severe (71): integer divide by zero
Image PC Routine Line Source
a.toto_chaillot_i 0804A3BF MAIN__ 12 toto.f90
a.toto_chaillot_i 0804A2DD Unknown Unknown Unknown
libc.so.6 B7E41E40 Unknown Unknown Unknown
a.toto_chaillot_i 0804A201 Unknown Unknown Unknown
$ ldd a.toto_chaillot_ifort10
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0xb7f38000)
libguide.so => /opt/intel/fc/9.1.039/lib/libguide.so (0xb7f03000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7ef1000)
libdl.so.2 => /lib/libdl.so.2 (0xb7eed000)
libc.so.6 => /lib/tls/libc.so.6 (0xb7dba000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7daf000)
/lib/ld-linux.so.2 (0xb7f77000)
>>
I also provide a C reproducer :
$ cat toto.c
#include
#include
int main () {
int i;
#pragma omp parallel
{
#pragma omp master
{
#if _OPENMP
fprintf(stdout,"#Threads=%dn",omp_get_num_threads()) ;
#endif
}
}
system("sleep 10") ;
for (i=-2; i<1;i++) {
fprintf(stdout,"iexit=%dn",1/i) ;
}
exit(0) ;
}
$ icc -openmp toto.c && ./a.out
toto.c(5): (col. 1) remark: OpenMP DEFINED REGION WAS PARALLELIZED.
#Threads=4
iexit=0
iexit=-1
Floating point exception

[Kevin_D_Intel · Accepted solution]

From Development: the OpenMP RTLs handle interrupts by default. This will be changed in a future release.

(Internal tracking id: DPD200140048)

For now, disable OpenMP RTL handling by setting the environment variable KMP_HANDLE_SIGNALS to 0.

The variable setting applies equally to 11.x, 10.x, 9.x compiler releases. (11.1.056 shown below).

$ ifort -V -g -traceback -openmp -o a.toto toto.f90
Intel Fortran Intel 64 Compiler Professional for applications running on Intel 64, Version 11.1 Build 20090827 Package ID: l_cprof_p_11.1.056
Copyright (C) 1985-2009 Intel Corporation. All rights reserved.

Intel Fortran 11.1-2582
GNU ld version 2.17.50.0.6-5.el5 20061020

$ ./a.toto
Threads= 8
iexit= 0
iexit= -1
Aborted

$ export KMP_HANDLE_SIGNALS=0
$ ./a.toto
Threads= 8
iexit= 0
iexit= -1
forrtl: severe (71): integer divide by zero
Image PC Routine Line Source
a.toto 000000000040358B MAIN__ 12 toto.f90
a.toto000000000040335CUnknownUnknown Unknown
libc.so.6 0000003D0FC1D8A4 Unknown Unknown Unknown
a.toto 0000000000403269 Unknown Unknown Unknown

[jimdempseyatthecove]

Modify your test program such that the exit(0); is .NOT. issued from OpenMP team member number 0 if(omp_get_thread_num() != 0) exit(0)

When team member number ==0 this is the "master thread" of the team. In an un-Nested parallel region this is your main thread.

Should the exit(0) occure in the main thread then the unwinding of the stack will show path up through main.
IOW you will have a false indication that everything works as expected.

Only when traceback works properly for thread .NOT. being main thread can you then declare the code is working as expected.

Jim Dempsey

[Olivier_C_]

Quoting - Kevin Davis (Intel)

From Development: the OpenMP RTLs handle interrupts by default. This will be changed in a future release.

For now, disable OpenMP RTL handling by setting the environment variable KMP_HANDLE_SIGNALS to 0.

The variable setting applies equally to 11.x, 10.x, 9.x compiler releases. (11.1.056 shown below).

$ ifort -V -g -traceback -openmp -o a.toto toto.f90
Intel Fortran Intel 64 Compiler Professional for applications running on Intel 64, Version 11.1 Build 20090827 Package ID: l_cprof_p_11.1.056
Copyright (C) 1985-2009 Intel Corporation. All rights reserved.

Intel Fortran 11.1-2582
GNU ld version 2.17.50.0.6-5.el5 20061020

$ ./a.toto
Threads= 8
iexit= 0
iexit= -1
Aborted

$ export KMP_HANDLE_SIGNALS=0
$ ./a.toto
Threads= 8
iexit= 0
iexit= -1
forrtl: severe (71): integer divide by zero
Image PC Routine Line Source
a.toto 000000000040358B MAIN__ 12 toto.f90
a.toto000000000040335CUnknownUnknown Unknown
libc.so.6 0000003D0FC1D8A4 Unknown Unknown Unknown
a.toto 0000000000403269 Unknown Unknown Unknown

Thanks a lot, I had the same answer from a colleague today.
I tested at home and it works.
Could you indicate some documentation about what signals are handled by the OpenMP RTL ?
Thank you.
Olivier Cessenat.

[Olivier_C_]

Quoting - jimdempseyatthecove

Modify your test program such that the exit(0); is .NOT. issued from OpenMP team member number 0 if(omp_get_thread_num() != 0) exit(0)

When team member number ==0 this is the "master thread" of the team. In an un-Nested parallel region this is your main thread.

Should the exit(0) occure in the main thread then the unwinding of the stack will show path up through main.
IOW you will have a false indication that everything works as expected.

Only when traceback works properly for thread .NOT. being main thread can you then declare the code is working as expected.

Jim Dempsey

I didn't get your point. The exit function is outside the parallel region and never reached.
You are right for the traceback inside an OpenMP region, which may be my next subject.
Thanks for you answer and attention.
Sincerely,
Olivier Cessenat.

[Kevin_D_Intel]

Quoting - cessenat@free.fr

Could you indicate some documentation about what signals are handled by the OpenMP RTL ?

Please excuse the delay Olivier. In regards to this question, Development says:

List of handled signals on Windows: int, ill, abrt, fpe, segv, term.
List of handled signals on Linux/Mac OS: hup, int, quit, ill, abrt, fpe, bus, segv, sys (if any), term, pipe.

This information nor the previous mentioned environment variable are currently documented so I requested both be documented in a future release.