Intel® MPI Library
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.
2153 Discussions

Issues with the I_MPI_HYDRA_BOOTSTRAP* envs, Intel MPI 4.0.1, SLES11 SP1

Christian_Guggenberg
528 Views
I see some reproducible glibc backtraces with the hydra process manager, when I_MPI_HYDRA_BOOTSTRAP* is defined and not overridden at command line:

#:~/pi> export I_MPI_HYDRA_BOOTSTRAP_EXEC=ssh
#:~/pi> mpiexec.hydra -n 8 ./pi
pi is 3.14159265358989 (error is 9.681144774731365E-014 )
*** glibc detected *** mpiexec.hydra: free(): invalid pointer: 0x00007fffff4f7c6c ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75018)[0x2acea8130018]
/lib64/libc.so.6(cfree+0x6c)[0x2acea8134f6c]
mpiexec.hydra[0x42b270]
mpiexec.hydra[0x4037bf]
/lib64/libc.so.6(__libc_start_main+0xe6)[0x2acea80d9bc6]
mpiexec.hydra[0x402bca]
======= Memory map: ========
00400000-00445000 r-xp 00000000 00:15 97290328 mpiexec.hydra
00545000-00547000 rw-p 00045000 00:15 97290328 mpiexec.hydra
00547000-0056b000 rw-p 00000000 00:00 0 [heap]
2acea7610000-2acea762e000 r-xp 00000000 fd:01 24 /lib64/ld-2.11.1.so
2acea762e000-2acea762f000 rw-p 00000000 00:00 0
2acea782e000-2acea782f000 r--p 0001e000 fd:01 24 /lib64/ld-2.11.1.so
2acea782f000-2acea7830000 rw-p 0001f000 fd:01 24 /lib64/ld-2.11.1.so
2acea7830000-2acea7831000 rw-p 00000000 00:00 0
2acea7831000-2acea7833000 r-xp 00000000 fd:01 54 /lib64/libdl-2.11.1.so
2acea7833000-2acea7a33000 ---p 00002000 fd:01 54 /lib64/libdl-2.11.1.so
2acea7a33000-2acea7a34000 r--p 00002000 fd:01 54 /lib64/libdl-2.11.1.so
2acea7a34000-2acea7a35000 rw-p 00003000 fd:01 54 /lib64/libdl-2.11.1.so
2acea7a35000-2acea7a4a000 r-xp 00000000 fd:01 29 /lib64/libnsl-2.11.1.so
2acea7a4a000-2acea7c49000 ---p 00015000 fd:01 29 /lib64/libnsl-2.11.1.so
2acea7c49000-2acea7c4a000 r--p 00014000 fd:01 29 /lib64/libnsl-2.11.1.so
2acea7c4a000-2acea7c4b000 rw-p 00015000 fd:01 29 /lib64/libnsl-2.11.1.so
2acea7c4b000-2acea7c4e000 rw-p 00000000 00:00 0
2acea7c4e000-2acea7ca3000 r-xp 00000000 fd:01 81 /lib64/libm-2.11.1.so
2acea7ca3000-2acea7ea2000 ---p 00055000 fd:01 81 /lib64/libm-2.11.1.so
2acea7ea2000-2acea7ea3000 r--p 00054000 fd:01 81 /lib64/libm-2.11.1.so
2acea7ea3000-2acea7ea4000 rw-p 00055000 fd:01 81 /lib64/libm-2.11.1.so
2acea7ea4000-2acea7eba000 r-xp 00000000 fd:01 123 /lib64/libgcc_s.so.1
2acea7eba000-2acea80b9000 ---p 00016000 fd:01 123 /lib64/libgcc_s.so.1
2acea80b9000-2acea80ba000 r--p 00015000 fd:01 123 /lib64/libgcc_s.so.1
2acea80ba000-2acea80bb000 rw-p 00016000 fd:01 123 /lib64/libgcc_s.so.1
2acea80bb000-2acea820f000 r-xp 00000000 fd:01 35 /lib64/libc-2.11.1.so
2acea820f000-2acea840f000 ---p 00154000 fd:01 35 /lib64/libc-2.11.1.so
2acea840f000-2acea8413000 r--p 00154000 fd:01 35 /lib64/libc-2.11.1.so
2acea8413000-2acea8414000 rw-p 00158000 fd:01 35 /lib64/libc-2.11.1.so
2acea8414000-2acea841b000 rw-p 00000000 00:00 0
2aceac000000-2aceac021000 rw-p 00000000 00:00 0
2aceac021000-2aceb0000000 ---p 00000000 00:00 0
7fffff4e4000-7fffff4f8000 rw-p 00000000 00:00 0 [stack]
7fffff5f7000-7fffff5f8000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted

However:
#:~/pi> mpiexec.hydra -bootstrap-exec ssh -n 8 ./pi
pi is 3.14159265358989 (error is 9.681144774731365E-014 )
#:~/pi> mpiexec.hydra -bootstrap-exec $I_MPI_HYDRA_BOOTSTRAP_EXEC -n 8 ./pi
pi is 3.14159265358989 (error is 9.681144774731365E-014 )

Anyone seen this issue as well ?

0 Kudos
2 Replies
Dmitry_K_Intel2
Employee
528 Views
Hi Christian,

Thank you for the message! This is a bug in the code handling some environment variables. The only workaround for this issue is using of command line options.
If you cannot use command line options and need environment variables support we can provide a patch, but you need to submit a request at Premier Support against Intel MPI Library.

Best regards!
Dmitry
0 Kudos
Christian_Guggenberg
528 Views
Thanks a lot, Dimitry. I still have to think of pros and cons wether I should write a wrapper around mpiexec.hdyra using the -bootstrap and -bootsrap-exec arguments or I should use the environment approach (my main goal is to get mpiexec.hydra tightly integrated into SGE, which is quite easy with both approaches)...
0 Kudos
Reply