Here is a more verbose log of the error: [puneets@host01 bin]$ mpirun -genv I_MPI_HYDRA_DEBUG=1 -hostfile hosts.txt -genv I_MPI_DEBUG=5 -np 4 -ppn 2 ./bt.E.4.mpi_io_full host: host02 host: host03 ================================================================================================== mpiexec options: ---------------- Base path: /home/soft/intel2015//impi/5.0.3.048/intel64/bin/ Launcher: ssh Debug level: 1 Enable X: -1 Global environment: ------------------- I_MPI_PERHOST=allcores LD_LIBRARY_PATH=/home/soft/intel2015//impi/5.0.3.048/intel64/lib:/home/soft/intel2015//itac/9.0.3.051/mic/slib:/home/soft/intel2015//itac/9.0.3.051/intel64/slib:/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/mpirt/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/../compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/tools/intel64/perfsys:/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/mkl/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/tbb/lib/intel64/gcc4.4:/home/soft/intel2015/composer_xe_2015.3.187/debugger/libipt/intel64/lib:/home/soft/intel2015//impi/5.0.3.048/intel64/lib:/home/soft/intel2015//impi/5.0.3.048/mic/lib MKLROOT=/home/soft/intel2015/composer_xe_2015.3.187/mkl MANPATH=/home/soft/intel2015//impi/5.0.3.048/man:/home/soft/intel2015//itac/9.0.3.051/man:/home/soft/intel2015/composer_xe_2015.3.187/man/en_US:/home/soft/intel2015/composer_xe_2015.3.187/man/en_US:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64/share/man/:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/share/man/:/usr/share/man GDB_HOST=/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/bin/gdb-ia-mic INSPECTOR_2015_DIR=/home/soft/intel2015/inspector_xe_2015.1.2.379161 HOSTNAME=host01 VT_MPI=impi4 INTEL_LICENSE_FILE=28518@licmngr2.org.in:/home/soft/intel2015/licenses:/opt/intel/licenses:/root/intel/licenses:/home/soft/intel2015/composer_xe_2015.3.187/licenses:/opt/intel/licenses:/root/intel/licenses IPPROOT=/home/soft/intel2015/composer_xe_2015.3.187/ipp SHELL=/bin/bash TERM=xterm HISTSIZE=1000 SSH_CLIENT=10.193.43.236 48078 22 GDBSERVER_MIC=/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/target/mic/bin/gdbserver LIBRARY_PATH=/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/../compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/mkl/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/tbb/lib/intel64/gcc4.4 QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/2 MIC_LD_LIBRARY_PATH=/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/mpirt/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/mkl/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/tbb/lib/mic LOCATE_PATH=/var/lib/mlocate/mlocate.db:/var/lib/mlocate/mlocate.app.db:/var/lib/mlocate/mlocate.soft.db USER=puneets LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36: I_MPI_MPIRUN=mpirun MIC_LIBRARY_PATH=/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/mpirt/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/tbb/lib/mic VTUNE_AMPLIFIER_XE_2015_DIR=/home/soft/intel2015/vtune_amplifier_xe_2015.3.0.403110 CPATH=/home/soft/intel2015/composer_xe_2015.3.187/ipp/include:/home/soft/intel2015/composer_xe_2015.3.187/mkl/include:/home/soft/intel2015/composer_xe_2015.3.187/tbb/include NLSPATH=/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64/locale/%l_%t/%N:/home/soft/intel2015/composer_xe_2015.3.187/ipp/lib/intel64/locale/%l_%t/%N:/home/soft/intel2015/composer_xe_2015.3.187/mkl/lib/intel64/locale/%l_%t/%N:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/share/locale/%l_%t/%N:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64/share/locale/%l_%t/%N PATH=/home/soft/intel2015/bin:/home/soft/intel2015/advisor_xe_2015.1.10.380555/bin64:/home/soft/intel2015/vtune_amplifier_xe_2015.3.0.403110/bin64:/home/soft/intel2015/inspector_xe_2015.1.2.379161/bin64:/home/soft/intel2015//impi/5.0.3.048/intel64/bin:/home/soft/intel2015//itac/9.0.3.051/intel64/bin:/home/soft/intel2015/composer_xe_2015.3.187/bin/intel64:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/bin:/usr/lib64/qt-3.3/bin:/opt/pbs/default/bin:/opt/pbs/default/sbin:/usr/kerberos/sbin:/usr/kerberos/bin:/opt/pbs/default/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/pbs/default/bin/:/home/puneets/bin MAIL=/var/spool/mail/puneets VT_ADD_LIBS=-ldwarf -lelf -lvtunwind -lnsl -lm -ldl -lpthread TBBROOT=/home/soft/intel2015/composer_xe_2015.3.187/tbb PWD=/home/puneets/AppBench/NPB/NPB3.3.1/NPB3.3-MPI/bin MLM_LICENSE_FILE=27000@licmngr1.org.in,27000@licmngr2org.in,27000@licmanager.org.in _LMFILES_=/home/soft/modules/suite/intel/parallelStudio LANG=en_IN KDE_IS_PRELINKED=1 GDB_CROSS=/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/bin/gdb-mic MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles:/opt/mellanox/bupc/2.18.0/modules:/home/soft/modules PBS_TZID=Asia/Kolkata ADVISOR_XE_2015_DIR=/home/soft/intel2015/advisor_xe_2015.1.10.380555 KDEDIRS=/usr LOADEDMODULES=suite/intel/parallelStudio VT_LIB_DIR=/home/soft/intel2015//itac/9.0.3.051/intel64/lib VT_ROOT=/home/soft/intel2015//itac/9.0.3.051 HISTCONTROL=ignoredups SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass MPM_LAUNCHER=/home/soft/intel2015/composer_xe_2015.3.187/debugger/mpm/bin/start_mpm.sh INTEL_PYTHONHOME=/home/soft/intel2015/composer_xe_2015.3.187/debugger/python/intel64/ HOME=/home/puneets SHLVL=2 VT_ARCH=intel64 VT_SLIB_DIR=/home/soft/intel2015//itac/9.0.3.051/intel64/slib LOGNAME=puneets MPI_ROOT=/home/soft/intel2015/impi/5.0.3.048 CVS_RSH=ssh QTLIB=/usr/lib64/qt-3.3/lib SSH_CONNECTION=10.193.43.236 48078 10.113.30.6 22 CLASSPATH=/home/soft/intel2015//itac/9.0.3.051/intel64/lib MODULESHOME=/usr/share/Modules LESSOPEN=||/usr/bin/lesspipe.sh %s INFOPATH=/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64/share/info/:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/share/info/ G_BROKEN_FILENAMES=1 SCRATCH=/home/puneets I_MPI_ROOT=/home/soft/intel2015//impi/5.0.3.048 BASH_FUNC_module()=() { eval `/usr/bin/modulecmd bash $*` } _=/home/soft/intel2015//impi/5.0.3.048/intel64/bin/mpiexec.hydra Hydra internal environment: --------------------------- MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1 GFORTRAN_UNBUFFERED_PRECONNECTED=y User set environment: --------------------- I_MPI_HYDRA_DEBUG=1 I_MPI_DEBUG=5 Proxy information: ********************* [1] proxy: host02 (2 cores) Exec list: ./bt.E.4.mpi_io_full (2 processes); [2] proxy: host03 (2 cores) Exec list: ./bt.E.4.mpi_io_full (2 processes); ================================================================================================== [mpiexec@host01] Timeout set to -1 (-1 means infinite) [mpiexec@host01] Got a control port string of host01:54130 Proxy launch args: /home/soft/intel2015//impi/5.0.3.048/intel64/bin/pmi_proxy --control-port host01:54130 --debug --pmi-connect lazy-cache --pmi-aggregate -s 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 1474035545 --usize -2 --proxy-id Arguments being passed to proxy 0: --version 3.1.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname host02 --global-core-map 0,2,4 --pmi-id-map 0,0 --global-process-count 4 --auto-cleanup 1 --pmi-kvsname kvs_11264_0 --pmi-process-mapping (vector,(0,2,2)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 67 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/home/soft/intel2015//impi/5.0.3.048/intel64/lib:/home/soft/intel2015//itac/9.0.3.051/mic/slib:/home/soft/intel2015//itac/9.0.3.051/intel64/slib:/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/mpirt/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/../compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/tools/intel64/perfsys:/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/mkl/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/tbb/lib/intel64/gcc4.4:/home/soft/intel2015/composer_xe_2015.3.187/debugger/libipt/intel64/lib:/home/soft/intel2015//impi/5.0.3.048/intel64/lib:/home/soft/intel2015//impi/5.0.3.048/mic/lib' 'MKLROOT=/home/soft/intel2015/composer_xe_2015.3.187/mkl' 'MANPATH=/home/soft/intel2015//impi/5.0.3.048/man:/home/soft/intel2015//itac/9.0.3.051/man:/home/soft/intel2015/composer_xe_2015.3.187/man/en_US:/home/soft/intel2015/composer_xe_2015.3.187/man/en_US:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64/share/man/:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/share/man/:/usr/share/man' 'GDB_HOST=/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/bin/gdb-ia-mic' 'INSPECTOR_2015_DIR=/home/soft/intel2015/inspector_xe_2015.1.2.379161' 'HOSTNAME=host01' 'VT_MPI=impi4' 'INTEL_LICENSE_FILE=28518@licmngr2.org.in:/home/soft/intel2015/licenses:/opt/intel/licenses:/root/intel/licenses:/home/soft/intel2015/composer_xe_2015.3.187/licenses:/opt/intel/licenses:/root/intel/licenses' 'IPPROOT=/home/soft/intel2015/composer_xe_2015.3.187/ipp' 'SHELL=/bin/bash' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=10.193.43.236 48078 22' 'GDBSERVER_MIC=/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/target/mic/bin/gdbserver' 'LIBRARY_PATH=/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/../compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/mkl/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/tbb/lib/intel64/gcc4.4' 'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/2' 'MIC_LD_LIBRARY_PATH=/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/mpirt/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/mkl/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/tbb/lib/mic' 'LOCATE_PATH=/var/lib/mlocate/mlocate.db:/var/lib/mlocate/mlocate.app.db:/var/lib/mlocate/mlocate.soft.db' 'USER=puneets' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:' 'I_MPI_MPIRUN=mpirun' 'MIC_LIBRARY_PATH=/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/mpirt/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/tbb/lib/mic' 'VTUNE_AMPLIFIER_XE_2015_DIR=/home/soft/intel2015/vtune_amplifier_xe_2015.3.0.403110' 'CPATH=/home/soft/intel2015/composer_xe_2015.3.187/ipp/include:/home/soft/intel2015/composer_xe_2015.3.187/mkl/include:/home/soft/intel2015/composer_xe_2015.3.187/tbb/include' 'NLSPATH=/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64/locale/%l_%t/%N:/home/soft/intel2015/composer_xe_2015.3.187/ipp/lib/intel64/locale/%l_%t/%N:/home/soft/intel2015/composer_xe_2015.3.187/mkl/lib/intel64/locale/%l_%t/%N:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/share/locale/%l_%t/%N:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64/share/locale/%l_%t/%N' 'PATH=/home/soft/intel2015/bin:/home/soft/intel2015/advisor_xe_2015.1.10.380555/bin64:/home/soft/intel2015/vtune_amplifier_xe_2015.3.0.403110/bin64:/home/soft/intel2015/inspector_xe_2015.1.2.379161/bin64:/home/soft/intel2015//impi/5.0.3.048/intel64/bin:/home/soft/intel2015//itac/9.0.3.051/intel64/bin:/home/soft/intel2015/composer_xe_2015.3.187/bin/intel64:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/bin:/usr/lib64/qt-3.3/bin:/opt/pbs/default/bin:/opt/pbs/default/sbin:/usr/kerberos/sbin:/usr/kerberos/bin:/opt/pbs/default/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/pbs/default/bin/:/home/puneets/bin' 'MAIL=/var/spool/mail/puneets' 'VT_ADD_LIBS=-ldwarf -lelf -lvtunwind -lnsl -lm -ldl -lpthread' 'TBBROOT=/home/soft/intel2015/composer_xe_2015.3.187/tbb' 'PWD=/home/puneets/AppBench/NPB/NPB3.3.1/NPB3.3-MPI/bin' 'MLM_LICENSE_FILE=27000@licmngr1.org.in,27000@licmngr2.org.in,27000@licmanager.org.in' '_LMFILES_=/home/soft/modules/suite/intel/parallelStudio' 'LANG=en_IN' 'KDE_IS_PRELINKED=1' 'GDB_CROSS=/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/bin/gdb-mic' 'MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles:/opt/mellanox/bupc/2.18.0/modules:/home/soft/modules' 'PBS_TZID=Asia/Kolkata' 'ADVISOR_XE_2015_DIR=/home/soft/intel2015/advisor_xe_2015.1.10.380555' 'KDEDIRS=/usr' 'LOADEDMODULES=suite/intel/parallelStudio' 'VT_LIB_DIR=/home/soft/intel2015//itac/9.0.3.051/intel64/lib' 'VT_ROOT=/home/soft/intel2015//itac/9.0.3.051' 'HISTCONTROL=ignoredups' 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass' 'MPM_LAUNCHER=/home/soft/intel2015/composer_xe_2015.3.187/debugger/mpm/bin/start_mpm.sh' 'INTEL_PYTHONHOME=/home/soft/intel2015/composer_xe_2015.3.187/debugger/python/intel64/' 'HOME=/home/puneets' 'SHLVL=2' 'VT_ARCH=intel64' 'VT_SLIB_DIR=/home/soft/intel2015//itac/9.0.3.051/intel64/slib' 'LOGNAME=puneets' 'MPI_ROOT=/home/soft/intel2015/impi/5.0.3.048' 'CVS_RSH=ssh' 'QTLIB=/usr/lib64/qt-3.3/lib' 'SSH_CONNECTION=10.193.43.236 48078 10.113.30.6 22' 'CLASSPATH=/home/soft/intel2015//itac/9.0.3.051/intel64/lib' 'MODULESHOME=/usr/share/Modules' 'LESSOPEN=||/usr/bin/lesspipe.sh %s' 'INFOPATH=/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64/share/info/:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/share/info/' 'G_BROKEN_FILENAMES=1' 'SCRATCH=/home/puneets' 'I_MPI_ROOT=/home/soft/intel2015//impi/5.0.3.048' 'BASH_FUNC_module()=() { eval `/usr/bin/modulecmd bash $*` }' '_=/home/soft/intel2015//impi/5.0.3.048/intel64/bin/mpiexec.hydra' --global-user-env 2 'I_MPI_HYDRA_DEBUG=1' 'I_MPI_DEBUG=5' --global-system-env 2 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 2 --mpi-cmd-env mpirun -genv I_MPI_HYDRA_DEBUG=1 -hostfile hosts.txt -genv I_MPI_DEBUG=5 -np 4 -ppn 2 ./bt.E.4.mpi_io_full --exec --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir /home/puneets/AppBench/NPB/NPB3.3.1/NPB3.3-MPI/bin --exec-args 1 ./bt.E.4.mpi_io_full Arguments being passed to proxy 1: --version 3.1.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname host03 --global-core-map 0,2,4 --pmi-id-map 0,2 --global-process-count 4 --auto-cleanup 1 --pmi-kvsname kvs_11264_0 --pmi-process-mapping (vector,(0,2,2)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 67 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/home/soft/intel2015//impi/5.0.3.048/intel64/lib:/home/soft/intel2015//itac/9.0.3.051/mic/slib:/home/soft/intel2015//itac/9.0.3.051/intel64/slib:/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/mpirt/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/../compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/tools/intel64/perfsys:/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/mkl/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/tbb/lib/intel64/gcc4.4:/home/soft/intel2015/composer_xe_2015.3.187/debugger/libipt/intel64/lib:/home/soft/intel2015//impi/5.0.3.048/intel64/lib:/home/soft/intel2015//impi/5.0.3.048/mic/lib' 'MKLROOT=/home/soft/intel2015/composer_xe_2015.3.187/mkl' 'MANPATH=/home/soft/intel2015//impi/5.0.3.048/man:/home/soft/intel2015//itac/9.0.3.051/man:/home/soft/intel2015/composer_xe_2015.3.187/man/en_US:/home/soft/intel2015/composer_xe_2015.3.187/man/en_US:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64/share/man/:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/share/man/:/usr/share/man' 'GDB_HOST=/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/bin/gdb-ia-mic' 'INSPECTOR_2015_DIR=/home/soft/intel2015/inspector_xe_2015.1.2.379161' 'HOSTNAME=host01' 'VT_MPI=impi4' 'INTEL_LICENSE_FILE=28518@licmngr2.org.in:/home/soft/intel2015/licenses:/opt/intel/licenses:/root/intel/licenses:/home/soft/intel2015/composer_xe_2015.3.187/licenses:/opt/intel/licenses:/root/intel/licenses' 'IPPROOT=/home/soft/intel2015/composer_xe_2015.3.187/ipp' 'SHELL=/bin/bash' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=10.193.43.236 48078 22' 'GDBSERVER_MIC=/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/target/mic/bin/gdbserver' 'LIBRARY_PATH=/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/../compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/ipp/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/mkl/lib/intel64:/home/soft/intel2015/composer_xe_2015.3.187/tbb/lib/intel64/gcc4.4' 'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/2' 'MIC_LD_LIBRARY_PATH=/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/mpirt/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/mkl/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/tbb/lib/mic' 'LOCATE_PATH=/var/lib/mlocate/mlocate.db:/var/lib/mlocate/mlocate.app.db:/var/lib/mlocate/mlocate.soft.db' 'USER=puneets' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:' 'I_MPI_MPIRUN=mpirun' 'MIC_LIBRARY_PATH=/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/mpirt/lib/mic:/home/soft/intel2015/composer_xe_2015.3.187/tbb/lib/mic' 'VTUNE_AMPLIFIER_XE_2015_DIR=/home/soft/intel2015/vtune_amplifier_xe_2015.3.0.403110' 'CPATH=/home/soft/intel2015/composer_xe_2015.3.187/ipp/include:/home/soft/intel2015/composer_xe_2015.3.187/mkl/include:/home/soft/intel2015/composer_xe_2015.3.187/tbb/include' 'NLSPATH=/home/soft/intel2015/composer_xe_2015.3.187/compiler/lib/intel64/locale/%l_%t/%N:/home/soft/intel2015/composer_xe_2015.3.187/ipp/lib/intel64/locale/%l_%t/%N:/home/soft/intel2015/composer_xe_2015.3.187/mkl/lib/intel64/locale/%l_%t/%N:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/share/locale/%l_%t/%N:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64/share/locale/%l_%t/%N' 'PATH=/home/soft/intel2015/bin:/home/soft/intel2015/advisor_xe_2015.1.10.380555/bin64:/home/soft/intel2015/vtune_amplifier_xe_2015.3.0.403110/bin64:/home/soft/intel2015/inspector_xe_2015.1.2.379161/bin64:/home/soft/intel2015//impi/5.0.3.048/intel64/bin:/home/soft/intel2015//itac/9.0.3.051/intel64/bin:/home/soft/intel2015/composer_xe_2015.3.187/bin/intel64:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/bin:/usr/lib64/qt-3.3/bin:/opt/pbs/default/bin:/opt/pbs/default/sbin:/usr/kerberos/sbin:/usr/kerberos/bin:/opt/pbs/default/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/pbs/default/bin/:/home/puneets/bin' 'MAIL=/var/spool/mail/puneets' 'VT_ADD_LIBS=-ldwarf -lelf -lvtunwind -lnsl -lm -ldl -lpthread' 'TBBROOT=/home/soft/intel2015/composer_xe_2015.3.187/tbb' 'PWD=/home/puneets/AppBench/NPB/NPB3.3.1/NPB3.3-MPI/bin' 'MLM_LICENSE_FILE=27000@licmngr1.org.in,27000@licmngr2.org.in,27000@licmanager.org.in' '_LMFILES_=/home/soft/modules/suite/intel/parallelStudio' 'LANG=en_IN' 'KDE_IS_PRELINKED=1' 'GDB_CROSS=/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/bin/gdb-mic' 'MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles:/opt/mellanox/bupc/2.18.0/modules:/home/soft/modules' 'PBS_TZID=Asia/Kolkata' 'ADVISOR_XE_2015_DIR=/home/soft/intel2015/advisor_xe_2015.1.10.380555' 'KDEDIRS=/usr' 'LOADEDMODULES=suite/intel/parallelStudio' 'VT_LIB_DIR=/home/soft/intel2015//itac/9.0.3.051/intel64/lib' 'VT_ROOT=/home/soft/intel2015//itac/9.0.3.051' 'HISTCONTROL=ignoredups' 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass' 'MPM_LAUNCHER=/home/soft/intel2015/composer_xe_2015.3.187/debugger/mpm/bin/start_mpm.sh' 'INTEL_PYTHONHOME=/home/soft/intel2015/composer_xe_2015.3.187/debugger/python/intel64/' 'HOME=/home/puneets' 'SHLVL=2' 'VT_ARCH=intel64' 'VT_SLIB_DIR=/home/soft/intel2015//itac/9.0.3.051/intel64/slib' 'LOGNAME=puneets' 'MPI_ROOT=/home/soft/intel2015/impi/5.0.3.048' 'CVS_RSH=ssh' 'QTLIB=/usr/lib64/qt-3.3/lib' 'SSH_CONNECTION=10.193.43.236 48078 10.113.30.6 22' 'CLASSPATH=/home/soft/intel2015//itac/9.0.3.051/intel64/lib' 'MODULESHOME=/usr/share/Modules' 'LESSOPEN=||/usr/bin/lesspipe.sh %s' 'INFOPATH=/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64/share/info/:/home/soft/intel2015/composer_xe_2015.3.187/debugger/gdb/intel64_mic/share/info/' 'G_BROKEN_FILENAMES=1' 'SCRATCH=/home/puneets' 'I_MPI_ROOT=/home/soft/intel2015//impi/5.0.3.048' 'BASH_FUNC_module()=() { eval `/usr/bin/modulecmd bash $*` }' '_=/home/soft/intel2015//impi/5.0.3.048/intel64/bin/mpiexec.hydra' --global-user-env 2 'I_MPI_HYDRA_DEBUG=1' 'I_MPI_DEBUG=5' --global-system-env 2 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 2 --mpi-cmd-env mpirun -genv I_MPI_HYDRA_DEBUG=1 -hostfile hosts.txt -genv I_MPI_DEBUG=5 -np 4 -ppn 2 ./bt.E.4.mpi_io_full --exec --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir /home/puneets/AppBench/NPB/NPB3.3.1/NPB3.3-MPI/bin --exec-args 1 ./bt.E.4.mpi_io_full [mpiexec@host01] Launch arguments: /usr/bin/ssh -x -q host02 /home/soft/intel2015//impi/5.0.3.048/intel64/bin/pmi_proxy --control-port host01:54130 --debug --pmi-connect lazy-cache --pmi-aggregate -s 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 1474035545 --usize -2 --proxy-id 0 [mpiexec@host01] Launch arguments: /usr/bin/ssh -x -q host03 /home/soft/intel2015//impi/5.0.3.048/intel64/bin/pmi_proxy --control-port host01:54130 --debug --pmi-connect lazy-cache --pmi-aggregate -s 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 1474035545 --usize -2 --proxy-id 1 [mpiexec@host01] STDIN will be redirected to 1 fd(s): 9 [proxy:0:0@host02] Start PMI_proxy 0 [proxy:0:0@host02] STDIN will be redirected to 1 fd(s): 9 [proxy:0:0@host02] got pmi command (from 6): init pmi_version=1 pmi_subversion=1 [proxy:0:0@host02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@host02] got pmi command (from 6): get_maxes [proxy:0:0@host02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@host02] got pmi command (from 6): barrier_in [proxy:0:0@host02] got pmi command (from 7): init pmi_version=1 pmi_subversion=1 [proxy:0:0@host02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@host02] got pmi command (from 7): get_maxes [proxy:0:0@host02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@host02] got pmi command (from 7): barrier_in [mpiexec@host01] [pgid: 0] got PMI command: cmd=barrier_in [proxy:0:0@host02] forwarding command (cmd=barrier_in) upstream [proxy:0:1@host03] Start PMI_proxy 1 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 30136 RUNNING AT host03 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)