Mpirun problem on running test.f90 on a cluster

giulio_t_ · ‎02-11-2014

Dear all,

I have installed on a cluster intel mpi. I run ./sshconnectivity.exp machines.LINUX

on a cluster with on master and 3 nodes. using the following machines.LINUX

gotpeumet-node01

gotpeumet-node02

gotpeumet-node03

gotpeumet-node04

Than I installed mpi on all the nodes using the install.sh.

Everything went great...apparently

It's three days that I am trying to figure out what is going wrong, because if I run a simple code like testf90 (taken the test.f90 source) on one of the nodes it works perfectly, while if i try to run

mpirun -d -n 2 -machinefile ~/machine_file -genvall ./test/testf90

I get the following

forrtl: severe (174): SIGSEGV, segmentation fault occurred

Image PC Routine Line Source

testf90 000000000046FC79 Unknown Unknown Unknown

testf90 000000000046E5F0 Unknown Unknown Unknown

testf90 0000000000440282 Unknown Unknown Unknown

testf90 0000000000424D73 Unknown Unknown Unknown

testf90 0000000000403A4B Unknown Unknown Unknown

libpthread.so.0 0000003E1DA0F710 Unknown Unknown Unknown

libmpi.so.4 00007F310A7C612E Unknown Unknown Unknown

libmpi.so.4 00007F310A860B24 Unknown Unknown Unknown

libmpi.so.4 00007F310A85F035 Unknown Unknown Unknown

libmpi.so.4 00007F310A860847 Unknown Unknown Unknown

libmpi.so.4 00007F310A70FC90 Unknown Unknown Unknown

libmpi.so.4 00007F310A84F69D Unknown Unknown Unknown

libmpi.so.4 00007F310A825E7F Unknown Unknown Unknown

libmpi.so.4 00007F310A819D18 Unknown Unknown Unknown

libmpigf.so.4 00007F310ACE331F Unknown Unknown Unknown

testf90 000000000040307B Unknown Unknown Unknown

testf90 0000000000403036 Unknown Unknown Unknown

libc.so.6 0000003E1D61ED1D Unknown Unknown Unknown

testf90 0000000000402F29 Unknown Unknown Unknown

forrtl: severe (174): SIGSEGV, segmentation fault occurred

Image PC Routine Line Source

testf90 000000000046FC79 Unknown Unknown Unknown

testf90 000000000046E5F0 Unknown Unknown Unknown

testf90 0000000000440282 Unknown Unknown Unknown

testf90 0000000000424D73 Unknown Unknown Unknown

testf90 0000000000403A4B Unknown Unknown Unknown

libpthread.so.0 000000301100F710 Unknown Unknown Unknown

libmpi.so.4 00007F0DC63D712E Unknown Unknown Unknown

libmpi.so.4 00007F0DC6471B24 Unknown Unknown Unknown

libmpi.so.4 00007F0DC6470035 Unknown Unknown Unknown

libmpi.so.4 00007F0DC6471847 Unknown Unknown Unknown

libmpi.so.4 00007F0DC6320C90 Unknown Unknown Unknown

libmpi.so.4 00007F0DC646069D Unknown Unknown Unknown

libmpi.so.4 00007F0DC6436E7F Unknown Unknown Unknown

libmpi.so.4 00007F0DC642AD18 Unknown Unknown Unknown

libmpigf.so.4 00007F0DC68F431F Unknown Unknown Unknown

testf90 000000000040307B Unknown Unknown Unknown

testf90 0000000000403036 Unknown Unknown Unknown

libc.so.6 0000003010C1ED1D Unknown Unknown Unknown

testf90 0000000000402F29 Unknown Unknown Unknown

while the -d option give me the following information:

host: gotpeumet-node01
host: gotpeumet-node02
host: gotpeumet-node03
host: gotpeumet-node04

==================================================================================================
mpiexec options:
----------------
Base path: /opt/intel/impi/4.1.3.045/intel64/bin/
Launcher: ssh
Debug level: 1
Enable X: -1

Global environment:
-------------------
I_MPI_PERHOST=allcores
LD_LIBRARY_PATH=/opt/intel/impi/4.1.3.045/intel64/lib:/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/mpirt/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64:/opt/intel/impi/4.1.3.045/intel64/lib:/opt/intel/impi/4.1.3.045/mic/lib
MKLROOT=/opt/intel/composer_xe_2013_sp1.1.106/mkl
MANPATH=/opt/intel/impi/4.1.3.045/man:/opt/intel/composer_xe_2013_sp1.1.106/man/en_US:/opt/intel/composer_xe_2013_sp1.1.106/man/en_US:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man:
HOSTNAME=gotpeumet-node01
SELINUX_ROLE_REQUESTED=
INTEL_LICENSE_FILE=/opt/intel/composer_xe_2013_sp1.1.106/licenses:/opt/intel/licenses:/home/giulio/intel/licenses
SHELL=/bin/bash
TERM=ansi
HISTSIZE=1000
SSH_CLIENT=169.143.244.115 24708 22
GDBSERVER_MIC=/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/target/mic/bin/gdbserver
LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64
SELINUX_USE_CURRENT_RANGE=
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
SSH_TTY=/dev/pts/2
MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.1.106/mpirt/lib/mic:/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/mic
USER=giulio
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
CPATH=/opt/intel/composer_xe_2013_sp1.1.106/mkl/include
NLSPATH=/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64_mic/py26/share/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64/py26/share/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.1.106/debugger/intel64/locale/%l_%t/%N
PATH=/opt/intel/impi/4.1.3.045/intel64/bin:/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64:/opt/intel/composer_xe_2013_sp1.1.106/mpirt/bin/intel64:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64_mic/py26/bin:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64/py26/bin:/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64:/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64_mic:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gui/intel64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/dell/srvadmin/bin:/home/giulio/bin
MAIL=/var/spool/mail/giulio
PWD=/home/giulio
IDB_HOME=/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64
LANG=en_US.UTF-8
GDB_CROSS=/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64_mic/py26/bin/gdb-mic
SELINUX_LEVEL_REQUESTED=
HISTCONTROL=ignoredups
HOME=/home/giulio
SHLVL=2
LOGNAME=giulio
CVS_RSH=ssh
QTLIB=/usr/lib64/qt-3.3/lib
LC_CTYPE=UTF-8
SSH_CONNECTION=169.143.244.115 24708 10.67.135.11 22
LESSOPEN=|/usr/bin/lesspipe.sh %s
INCLUDE=/opt/intel/composer_xe_2013_sp1.1.106/mkl/include
G_BROKEN_FILENAMES=1
I_MPI_ROOT=/opt/intel/impi/4.1.3.045
_=/opt/intel/impi/4.1.3.045/intel64/bin/mpiexec.hydra

Hydra internal environment:
---------------------------
MPICH_ENABLE_CKPOINT=1
GFORTRAN_UNBUFFERED_PRECONNECTED=y

Proxy information:
*********************
[1] proxy: gotpeumet-node01 (1 cores)
Exec list: ./test/testf90 (1 processes);

[2] proxy: gotpeumet-node02 (1 cores)
Exec list: ./test/testf90 (1 processes);

==================================================================================================

[mpiexec@gotpeumet-node01] Timeout set to -1 (-1 means infinite)
[mpiexec@gotpeumet-node01] Got a control port string of gotpeumet-node01:40839

Proxy launch args: /opt/intel/impi/4.1.3.045/intel64/bin/pmi_proxy --control-port gotpeumet-node01:40839 --debug --pmi-connect lazy-cache --pmi-aggregate -s 0 --rmk slurm --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 2011340091 --proxy-id

[mpiexec@gotpeumet-node01] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
Arguments being passed to proxy 0:
--version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname gotpeumet-node01 --global-core-map 0,1,1 --filler-process-map 0,1,1 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname kvs_27468_0 --pmi-process-mapping (vector,(0,4,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 42 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/opt/intel/impi/4.1.3.045/intel64/lib:/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/mpirt/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64:/opt/intel/impi/4.1.3.045/intel64/lib:/opt/intel/impi/4.1.3.045/mic/lib' 'MKLROOT=/opt/intel/composer_xe_2013_sp1.1.106/mkl' 'MANPATH=/opt/intel/impi/4.1.3.045/man:/opt/intel/composer_xe_2013_sp1.1.106/man/en_US:/opt/intel/composer_xe_2013_sp1.1.106/man/en_US:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man:' 'HOSTNAME=gotpeumet-node01' 'SELINUX_ROLE_REQUESTED=' 'INTEL_LICENSE_FILE=/opt/intel/composer_xe_2013_sp1.1.106/licenses:/opt/intel/licenses:/home/giulio/intel/licenses' 'SHELL=/bin/bash' 'TERM=ansi' 'HISTSIZE=1000' 'SSH_CLIENT=169.143.244.115 24708 22' 'GDBSERVER_MIC=/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/target/mic/bin/gdbserver' 'LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64' 'SELINUX_USE_CURRENT_RANGE=' 'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/2' 'MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.1.106/mpirt/lib/mic:/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/mic' 'USER=giulio' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:' 'CPATH=/opt/intel/composer_xe_2013_sp1.1.106/mkl/include' 'NLSPATH=/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64_mic/py26/share/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64/py26/share/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.1.106/debugger/intel64/locale/%l_%t/%N' 'PATH=/opt/intel/impi/4.1.3.045/intel64/bin:/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64:/opt/intel/composer_xe_2013_sp1.1.106/mpirt/bin/intel64:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64_mic/py26/bin:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64/py26/bin:/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64:/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64_mic:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gui/intel64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/dell/srvadmin/bin:/home/giulio/bin' 'MAIL=/var/spool/mail/giulio' 'PWD=/home/giulio' 'IDB_HOME=/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64' 'LANG=en_US.UTF-8' 'GDB_CROSS=/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64_mic/py26/bin/gdb-mic' 'SELINUX_LEVEL_REQUESTED=' 'HISTCONTROL=ignoredups' 'HOME=/home/giulio' 'SHLVL=2' 'LOGNAME=giulio' 'CVS_RSH=ssh' 'QTLIB=/usr/lib64/qt-3.3/lib' 'LC_CTYPE=UTF-8' 'SSH_CONNECTION=169.143.244.115 24708 10.67.135.11 22' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'INCLUDE=/opt/intel/composer_xe_2013_sp1.1.106/mkl/include' 'G_BROKEN_FILENAMES=1' 'I_MPI_ROOT=/opt/intel/impi/4.1.3.045' '_=/opt/intel/impi/4.1.3.045/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 2 'MPICH_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --genv-prop all --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/giulio --exec-args 1 ./test/testf90

[mpiexec@gotpeumet-node01] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
Arguments being passed to proxy 1:
--version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname gotpeumet-node02 --global-core-map 1,1,0 --filler-process-map 1,1,0 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname kvs_27468_0 --pmi-process-mapping (vector,(0,4,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 42 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/opt/intel/impi/4.1.3.045/intel64/lib:/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/mpirt/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64:/opt/intel/impi/4.1.3.045/intel64/lib:/opt/intel/impi/4.1.3.045/mic/lib' 'MKLROOT=/opt/intel/composer_xe_2013_sp1.1.106/mkl' 'MANPATH=/opt/intel/impi/4.1.3.045/man:/opt/intel/composer_xe_2013_sp1.1.106/man/en_US:/opt/intel/composer_xe_2013_sp1.1.106/man/en_US:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man:' 'HOSTNAME=gotpeumet-node01' 'SELINUX_ROLE_REQUESTED=' 'INTEL_LICENSE_FILE=/opt/intel/composer_xe_2013_sp1.1.106/licenses:/opt/intel/licenses:/home/giulio/intel/licenses' 'SHELL=/bin/bash' 'TERM=ansi' 'HISTSIZE=1000' 'SSH_CLIENT=169.143.244.115 24708 22' 'GDBSERVER_MIC=/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/target/mic/bin/gdbserver' 'LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64' 'SELINUX_USE_CURRENT_RANGE=' 'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/2' 'MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.1.106/mpirt/lib/mic:/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/mic' 'USER=giulio' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:' 'CPATH=/opt/intel/composer_xe_2013_sp1.1.106/mkl/include' 'NLSPATH=/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64_mic/py26/share/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64/py26/share/locale/%l_%t/%N:/opt/intel/composer_xe_2013_sp1.1.106/debugger/intel64/locale/%l_%t/%N' 'PATH=/opt/intel/impi/4.1.3.045/intel64/bin:/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64:/opt/intel/composer_xe_2013_sp1.1.106/mpirt/bin/intel64:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64_mic/py26/bin:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64/py26/bin:/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64:/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64_mic:/opt/intel/composer_xe_2013_sp1.1.106/debugger/gui/intel64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/dell/srvadmin/bin:/home/giulio/bin' 'MAIL=/var/spool/mail/giulio' 'PWD=/home/giulio' 'IDB_HOME=/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64' 'LANG=en_US.UTF-8' 'GDB_CROSS=/opt/intel/composer_xe_2013_sp1.1.106/debugger/gdb/intel64_mic/py26/bin/gdb-mic' 'SELINUX_LEVEL_REQUESTED=' 'HISTCONTROL=ignoredups' 'HOME=/home/giulio' 'SHLVL=2' 'LOGNAME=giulio' 'CVS_RSH=ssh' 'QTLIB=/usr/lib64/qt-3.3/lib' 'LC_CTYPE=UTF-8' 'SSH_CONNECTION=169.143.244.115 24708 10.67.135.11 22' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'INCLUDE=/opt/intel/composer_xe_2013_sp1.1.106/mkl/include' 'G_BROKEN_FILENAMES=1' 'I_MPI_ROOT=/opt/intel/impi/4.1.3.045' '_=/opt/intel/impi/4.1.3.045/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 2 'MPICH_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --genv-prop all --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/giulio --exec-args 1 ./test/testf90

[mpiexec@gotpeumet-node01] Launch arguments: /opt/intel/impi/4.1.3.045/intel64/bin/pmi_proxy --control-port gotpeumet-node01:40839 --debug --pmi-connect lazy-cache --pmi-aggregate -s 0 --rmk slurm --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 2011340091 --proxy-id 0
[mpiexec@gotpeumet-node01] Launch arguments: /usr/bin/ssh -x -q gotpeumet-node02 /opt/intel/impi/4.1.3.045/intel64/bin/pmi_proxy --control-port gotpeumet-node01:40839 --debug --pmi-connect lazy-cache --pmi-aggregate -s 0 --rmk slurm --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 2011340091 --proxy-id 1
[mpiexec@gotpeumet-node01] STDIN will be redirected to 1 fd(s): 9
[proxy:0:0@gotpeumet-node01] Start PMI_proxy 0
[proxy:0:0@gotpeumet-node01] STDIN will be redirected to 1 fd(s): 15
[proxy:0:0@gotpeumet-node01] got pmi command (from 10): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@gotpeumet-node01] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@gotpeumet-node01] got pmi command (from 10): get_maxes

[proxy:0:0@gotpeumet-node01] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0@gotpeumet-node01] got pmi command (from 10): barrier_in

[mpiexec@gotpeumet-node01] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:0@gotpeumet-node01] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@gotpeumet-node02] Start PMI_proxy 1
[proxy:0:1@gotpeumet-node02] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@gotpeumet-node02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@gotpeumet-node02] got pmi command (from 6): get_maxes

[proxy:0:1@gotpeumet-node02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[mpiexec@gotpeumet-node01] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@gotpeumet-node01] PMI response to fd 6 pid 6: cmd=barrier_out
[mpiexec@gotpeumet-node01] PMI response to fd 7 pid 6: cmd=barrier_out
[proxy:0:1@gotpeumet-node02] got pmi command (from 6): barrier_in

[proxy:0:1@gotpeumet-node02] forwarding command (cmd=barrier_in) upstream
[proxy:0:0@gotpeumet-node01] PMI response: cmd=barrier_out
[proxy:0:0@gotpeumet-node01] got pmi command (from 10): get_ranks2hosts

[proxy:0:0@gotpeumet-node01] PMI response: put_ranks2hosts 48 2
16 gotpeumet-node01 0, 16 gotpeumet-node02 1,
[proxy:0:1@gotpeumet-node02] PMI response: cmd=barrier_out
[proxy:0:0@gotpeumet-node01] got pmi command (from 10): get_appnum

[proxy:0:0@gotpeumet-node01] PMI response: cmd=appnum appnum=0
[proxy:0:1@gotpeumet-node02] got pmi command (from 6): get_ranks2hosts

[proxy:0:0@gotpeumet-node01] got pmi command (from 10): get_my_kvsname

[proxy:0:0@gotpeumet-node01] PMI response: cmd=my_kvsname kvsname=kvs_27468_0
[proxy:0:1@gotpeumet-node02] PMI response: put_ranks2hosts 48 2
16 gotpeumet-node01 0, 16 gotpeumet-node02 1,
[proxy:0:1@gotpeumet-node02] got pmi command (from 6): get_appnum

[proxy:0:1@gotpeumet-node02] PMI response: cmd=appnum appnum=0
[proxy:0:0@gotpeumet-node01] got pmi command (from 10): get_my_kvsname

[proxy:0:1@gotpeumet-node02] got pmi command (from 6): get_my_kvsname

[proxy:0:1@gotpeumet-node02] PMI response: cmd=my_kvsname kvsname=kvs_27468_0
[proxy:0:0@gotpeumet-node01] PMI response: cmd=my_kvsname kvsname=kvs_27468_0
[proxy:0:1@gotpeumet-node02] got pmi command (from 6): get_my_kvsname

[proxy:0:1@gotpeumet-node02] PMI response: cmd=my_kvsname kvsname=kvs_27468_0
[proxy:0:0@gotpeumet-node01] got pmi command (from 10): barrier_in

[proxy:0:0@gotpeumet-node01] forwarding command (cmd=barrier_in) upstream
[mpiexec@gotpeumet-node01] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@gotpeumet-node01] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@gotpeumet-node01] PMI response to fd 6 pid 6: cmd=barrier_out
[mpiexec@gotpeumet-node01] PMI response to fd 7 pid 6: cmd=barrier_out
[proxy:0:0@gotpeumet-node01] PMI response: cmd=barrier_out
[proxy:0:1@gotpeumet-node02] got pmi command (from 6): barrier_in

[proxy:0:1@gotpeumet-node02] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@gotpeumet-node02] PMI response: cmd=barrier_out

Can you please help me in understanding why i am getting always segmentation fault to run a very simple code that just tests the system?

thanks

James_T_Intel · ‎03-05-2014

What is the output from the following commands?

[plain]which mpirun

ldd ./test/testf90

env | grep I_MPI[/plain]