Issues with the I_MPI_HYDRA_BOOTSTRAP* envs, Intel MPI 4.0.1, SLES11 SP1

[Christian_Guggenberg]

I see some reproducible glibc backtraces with the hydra process manager, when I_MPI_HYDRA_BOOTSTRAP* is defined and not overridden at command line:

#:~/pi> export I_MPI_HYDRA_BOOTSTRAP_EXEC=ssh
#:~/pi> mpiexec.hydra -n 8 ./pi
pi is 3.14159265358989 (error is 9.681144774731365E-014 )
*** glibc detected *** mpiexec.hydra: free(): invalid pointer: 0x00007fffff4f7c6c ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75018)[0x2acea8130018]
/lib64/libc.so.6(cfree+0x6c)[0x2acea8134f6c]
mpiexec.hydra[0x42b270]
mpiexec.hydra[0x4037bf]
/lib64/libc.so.6(__libc_start_main+0xe6)[0x2acea80d9bc6]
mpiexec.hydra[0x402bca]
======= Memory map: ========
00400000-00445000 r-xp 00000000 00:15 97290328 mpiexec.hydra
00545000-00547000 rw-p 00045000 00:15 97290328 mpiexec.hydra
00547000-0056b000 rw-p 00000000 00:00 0 [heap]
2acea7610000-2acea762e000 r-xp 00000000 fd:01 24 /lib64/ld-2.11.1.so
2acea762e000-2acea762f000 rw-p 00000000 00:00 0
2acea782e000-2acea782f000 r--p 0001e000 fd:01 24 /lib64/ld-2.11.1.so
2acea782f000-2acea7830000 rw-p 0001f000 fd:01 24 /lib64/ld-2.11.1.so
2acea7830000-2acea7831000 rw-p 00000000 00:00 0
2acea7831000-2acea7833000 r-xp 00000000 fd:01 54 /lib64/libdl-2.11.1.so
2acea7833000-2acea7a33000 ---p 00002000 fd:01 54 /lib64/libdl-2.11.1.so
2acea7a33000-2acea7a34000 r--p 00002000 fd:01 54 /lib64/libdl-2.11.1.so
2acea7a34000-2acea7a35000 rw-p 00003000 fd:01 54 /lib64/libdl-2.11.1.so
2acea7a35000-2acea7a4a000 r-xp 00000000 fd:01 29 /lib64/libnsl-2.11.1.so
2acea7a4a000-2acea7c49000 ---p 00015000 fd:01 29 /lib64/libnsl-2.11.1.so
2acea7c49000-2acea7c4a000 r--p 00014000 fd:01 29 /lib64/libnsl-2.11.1.so
2acea7c4a000-2acea7c4b000 rw-p 00015000 fd:01 29 /lib64/libnsl-2.11.1.so
2acea7c4b000-2acea7c4e000 rw-p 00000000 00:00 0
2acea7c4e000-2acea7ca3000 r-xp 00000000 fd:01 81 /lib64/libm-2.11.1.so
2acea7ca3000-2acea7ea2000 ---p 00055000 fd:01 81 /lib64/libm-2.11.1.so
2acea7ea2000-2acea7ea3000 r--p 00054000 fd:01 81 /lib64/libm-2.11.1.so
2acea7ea3000-2acea7ea4000 rw-p 00055000 fd:01 81 /lib64/libm-2.11.1.so
2acea7ea4000-2acea7eba000 r-xp 00000000 fd:01 123 /lib64/libgcc_s.so.1
2acea7eba000-2acea80b9000 ---p 00016000 fd:01 123 /lib64/libgcc_s.so.1
2acea80b9000-2acea80ba000 r--p 00015000 fd:01 123 /lib64/libgcc_s.so.1
2acea80ba000-2acea80bb000 rw-p 00016000 fd:01 123 /lib64/libgcc_s.so.1
2acea80bb000-2acea820f000 r-xp 00000000 fd:01 35 /lib64/libc-2.11.1.so
2acea820f000-2acea840f000 ---p 00154000 fd:01 35 /lib64/libc-2.11.1.so
2acea840f000-2acea8413000 r--p 00154000 fd:01 35 /lib64/libc-2.11.1.so
2acea8413000-2acea8414000 rw-p 00158000 fd:01 35 /lib64/libc-2.11.1.so
2acea8414000-2acea841b000 rw-p 00000000 00:00 0
2aceac000000-2aceac021000 rw-p 00000000 00:00 0
2aceac021000-2aceb0000000 ---p 00000000 00:00 0
7fffff4e4000-7fffff4f8000 rw-p 00000000 00:00 0 [stack]
7fffff5f7000-7fffff5f8000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted

However:
#:~/pi> mpiexec.hydra -bootstrap-exec ssh -n 8 ./pi
pi is 3.14159265358989 (error is 9.681144774731365E-014 )
#:~/pi> mpiexec.hydra -bootstrap-exec $I_MPI_HYDRA_BOOTSTRAP_EXEC -n 8 ./pi
pi is 3.14159265358989 (error is 9.681144774731365E-014 )

Anyone seen this issue as well ?

[Dmitry_K_Intel2]

Hi Christian,

Thank you for the message! This is a bug in the code handling some environment variables. The only workaround for this issue is using of command line options.
If you cannot use command line options and need environment variables support we can provide a patch, but you need to submit a request at Premier Support against Intel MPI Library.

Best regards!
Dmitry

[Christian_Guggenberg]

Thanks a lot, Dimitry. I still have to think of pros and cons wether I should write a wrapper around mpiexec.hdyra using the -bootstrap and -bootsrap-exec arguments or I should use the environment approach (my main goal is to get mpiexec.hydra tightly integrated into SGE, which is quite easy with both approaches)...