problem executing in symmteric mode

[Wasim_M_]

Hi,

    I am trying to execute mic code from host. I am facing the following problem.

 Launch arguments: /usr/bin/ssh -x -q cm1 sh -c 'export I_MPI_ROOT="/opt/intel//impi/5.0.1.035" ; export PATH="/opt/intel/impi/5.0.1.035/intel64/bin/:${I_MPI_ROOT}/intel64/bin:${PATH}" ; exec "$0" "$@"' pmi_proxy --control-port 10.1.1.249:35336 --debug --pmi-connect lazy-cache --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/impi/5.0.1.035/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 1970575286 --usize -2 --proxy-id 0
[mpiexec@compute-0-4.local] STDIN will be redirected to 1 fd(s): 9
pmi_proxy: /opt/intel/impi/5.0.1.035/intel64/bin/pmi_proxy: cannot execute binary file
pmi_proxy: /opt/intel/impi/5.0.1.035/intel64/bin/pmi_proxy: Success

(standalone code on mic is working fine).

Plz help.

Thanks. 

[Artem_R_Intel]

Hi,
Could you please specify the whole MPI command line and IMPI specific environment variables?

[Wasim_M_]

Hi,

   Thanks for the reply.

    mpi command: /opt/intel/impi/5.0.1.035/intel64/bin/mpiexec.hydra  -genv LD_LIBRARY_PATH $MIC_LD_LIBRARY_PATH   -genv I_MPI_HYDRA_DEBUG=1  -f machines <my_executable_mic>

   machines file contains mic0

   IMPI env vars:

   I_MPI_MIC=1
   I_MPI_ROOT=/opt/intel//impi/5.0.1.035

   Also, my current directory as well /opt is nfs mounted on mic0.

[Artem_R_Intel1]

Could you please try to specify '-n <# of processes>' option for mpiexec.hydra.
As far as I know without this option the results may be unexpected.

[Wasim_M_]

with '-n  #' option also the same problem.

Launch arguments: /usr/bin/ssh -x -q cm1 sh -c 'export I_MPI_ROOT="/opt/intel//impi/5.0.1.035" ; export PATH="/opt/intel/impi/5.0.1.035/intel64/bin/:${I_MPI_ROOT}/intel64/bin:${PATH}" ; exec "$0" "$@"' pmi_proxy --control-port 10.1.1.249:35336 --debug --pmi-connect lazy-cache --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/impi/5.0.1.035/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 1970575286 --usize -2 --proxy-id 0
[mpiexec@compute-0-4.local] STDIN will be redirected to 1 fd(s): 9
pmi_proxy: /opt/intel/impi/5.0.1.035/intel64/bin/pmi_proxy: cannot execute binary file
pmi_proxy: /opt/intel/impi/5.0.1.035/intel64/bin/pmi_proxy: Success

Is the PATH variable correctly set in the above Launch arguments?

or should it be "/opt/intel/impi/5.0.1.035/mic/bin/" ?

Thanks.

[Artem_R_Intel1]

The symptoms look like I_MPI_MIC isn't set.
Could you please double check that I_MPI_MIC is set?
It can be set via 'export I_MPI_MIC=1' or within MPI run 'env I_MPI_MIC=1 mpiexec.hydra ...'.
If it doesn't help could you please provide full hydra console output?

One more proposal is to check the whole workflow - example of the correct usage:
 - Source IMPI environment:
$ . /opt/intel/impi/5.0.1.035/intel64/bin/mpivars.sh
 - Set necessary environment variables:
$ export I_MPI_MIC=1 
 - Run the MPI application:
$ mpiexec.hydra -n 2 ... -f hosts.txt <exe>

One more trick:
If you need MIC specific LD_LIBRARY_PATH you can set on the host:
$ export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${MIC_LD_LIBRARY_PATH}"
The linker will choose the libraries with correct architecture automatically.

 

[Wasim_M_]

Tried all 3 things but the problem stays.

(I guess the problem is with proxy setting for mic)

Here is the console output:

mpiexec.hydra -genv I_MPI_MIC=1 -genv I_MPI_HYDRA_DEBUG 1  -host mic0 -n 20 ./exec_mic
host: mic0

==================================================================================================
mpiexec options:
----------------
  Base path: /opt/intel//impi/5.0.1.035/intel64/bin/
  Launcher: ssh
  Debug level: 1
  Enable X: -1

  Global environment:
  -------------------
    I_MPI_PERHOST=allcores
    LD_LIBRARY_PATH=/opt/intel//impi/5.0.1.035/intel64/lib:/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/mpirt/lib/intel64:/opt/intel/composer_xe_2015.0.090/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2015.0.090/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64:/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64:/opt/intel/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4::/opt/python/lib:/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mpirt/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/ipp/lib/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mkl/lib/mic:/opt/intel/composer_xe_2015.0.090/tbb/lib/mic:/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mpirt/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/ipp/lib/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mkl/lib/mic:/opt/intel/composer_xe_2015.0.090/tbb/lib/mic:/opt/intel//impi/5.0.1.035/intel64/lib:/opt/intel//impi/5.0.1.035/mic/lib
    MKLROOT=/opt/intel/composer_xe_2015.0.090/mkl
    MANPATH=/opt/intel//impi/5.0.1.035/man:/opt/intel/composer_xe_2015.0.090/man/en_US:/opt/intel/composer_xe_2015.0.090/man/en_US:/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64/share/man/:/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64_mic/share/man/:
    BIOROLL=/opt/bio
    HOSTNAME=compute-0-5.local
    IPPROOT=/opt/intel/composer_xe_2015.0.090/ipp
    TERM=xterm
    SHELL=/bin/bash
    HISTSIZE=1000
    I_MPI_MIC=1
    GDBSERVER_MIC=/opt/intel/composer_xe_2015.0.090/debugger/gdb/target/mic/bin/gdbserver
    HMMER_DB=/home/wasim/bio/hmmer/db
    SSH_CLIENT=10.1.1.1 59616 22
    LIBRARY_PATH=/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64:/opt/intel/composer_xe_2015.0.090/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2015.0.090/ipp/lib/intel64:/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64:/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64:/opt/intel/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4
    QTDIR=/usr/lib64/qt-3.3
    OLDPWD=/home/wasim
    QTINC=/usr/lib64/qt-3.3/include
    MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mpirt/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/ipp/lib/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mkl/lib/mic:/opt/intel/composer_xe_2015.0.090/tbb/lib/mic
    SSH_TTY=/dev/pts/0
    ANT_HOME=/opt/rocks
    USER=wasim
    LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
    MIC_LIBRARY_PATH=/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mpirt/lib/mic:/opt/intel/composer_xe_2015.0.090/tbb/lib/mic
    ROCKS_ROOT=/opt/rocks
    CPATH=/opt/intel/composer_xe_2015.0.090/ipp/include:/opt/intel/composer_xe_2015.0.090/mkl/include:/opt/intel/composer_xe_2015.0.090/tbb/include
    MIC=1
    NLSPATH=/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2015.0.090/ipp/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64_mic/share/locale/%l_%t/%N:/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64/share/locale/%l_%t/%N
    MAIL=/var/spool/mail/wasim
    PATH=/opt/intel//impi/5.0.1.035/intel64/bin:/share/apps/Intel/bin:/opt/intel/composer_xe_2015.0.090/bin/intel64:/opt/intel/composer_xe_2015.0.090/mpirt/bin/intel64:/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64_mic/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/bio/ncbi/bin:/opt/bio/mpiblast/bin:/opt/bio/EMBOSS/bin:/opt/bio/clustalw/bin:/opt/bio/tcoffee/bin:/opt/bio/hmmer/bin:/opt/bio/phylip/exe:/opt/bio/mrbayes:/opt/bio/fasta:/opt/bio/glimmer/bin:/opt/bio/glimmer/scripts:/opt/bio/gromacs/bin:/opt/bio/gmap/bin:/opt/bio/tigr/bin:/opt/bio/autodocksuite/bin:/opt/bio/wgs/bin:/opt/ganglia/bin:/opt/ganglia/sbin:/usr/java/latest/bin:/opt/rocks/bin:/opt/rocks/sbin:/home/wasim/bin
    TBBROOT=/opt/intel/composer_xe_2015.0.090/tbb
    PWD=/home/wasim/codebase/grn_hetero_app
    JAVA_HOME=/usr/java/latest
    GDB_CROSS=/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64_mic/bin/gdb-mic
    LANG=en_US.iso885915
    MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles
    LOADEDMODULES=
    BLASTDB=/home/wasim/bio/ncbi/db
    MPM_LAUNCHER=/opt/intel/composer_xe_2015.0.090/debugger/mpm/bin/start_mpm.sh
    SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
    HISTCONTROL=ignoredups
    INTEL_PYTHONHOME=/opt/intel/composer_xe_2015.0.090/debugger/python/intel64/
    SHLVL=1
    HOME=/home/wasim
    LOGNAME=wasim
    QTLIB=/usr/lib64/qt-3.3/lib
    CVS_RSH=ssh
    SSH_CONNECTION=10.1.1.1 59616 10.1.1.248 22
    MODULESHOME=/usr/share/Modules
    LESSOPEN=|/usr/bin/lesspipe.sh %s
    BLASTMAT=/opt/bio/ncbi/data
    INFOPATH=/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64/share/info/:/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64_mic/share/info/
    INCLUDE=/opt/intel/composer_xe_2015.0.090/mkl/include
    G_BROKEN_FILENAMES=1
    I_MPI_ROOT=/opt/intel//impi/5.0.1.035
    module=() {  eval `/usr/bin/modulecmd bash $*`
}
    _=/opt/intel//impi/5.0.1.035/intel64/bin/mpiexec.hydra

  Hydra internal environment:
  ---------------------------
    MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1
    GFORTRAN_UNBUFFERED_PRECONNECTED=y

  User set environment:
  ---------------------
    I_MPI_MIC=1
    I_MPI_HYDRA_DEBUG=1

    Proxy information:
    *********************
      [1] proxy: mic0 (20 cores)
      Exec list: ./exec_mic (20 processes);

==================================================================================================

[mpiexec@compute-0-5.local] Timeout set to -1 (-1 means infinite)
[mpiexec@compute-0-5.local] Got a control port string of 10.1.1.248:43606

Proxy launch args: pmi_proxy --control-port 10.1.1.248:43606 --debug --pmi-connect lazy-cache --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel//impi/5.0.1.035/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 1218988696 --usize -2 --proxy-id

Arguments being passed to proxy 0:
--version 3.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname mic0 --global-core-map 0,20,20 --pmi-id-map 0,0 --global-process-count 20 --auto-cleanup 1 --pmi-kvsname kvs_2556_0 --pmi-process-mapping (vector,(0,1,20)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 58 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/opt/intel//impi/5.0.1.035/intel64/lib:/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/mpirt/lib/intel64:/opt/intel/composer_xe_2015.0.090/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2015.0.090/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64:/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64:/opt/intel/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4::/opt/python/lib:/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mpirt/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/ipp/lib/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mkl/lib/mic:/opt/intel/composer_xe_2015.0.090/tbb/lib/mic:/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mpirt/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/ipp/lib/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mkl/lib/mic:/opt/intel/composer_xe_2015.0.090/tbb/lib/mic:/opt/intel//impi/5.0.1.035/intel64/lib:/opt/intel//impi/5.0.1.035/mic/lib' 'MKLROOT=/opt/intel/composer_xe_2015.0.090/mkl' 'MANPATH=/opt/intel//impi/5.0.1.035/man:/opt/intel/composer_xe_2015.0.090/man/en_US:/opt/intel/composer_xe_2015.0.090/man/en_US:/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64/share/man/:/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64_mic/share/man/:' 'BIOROLL=/opt/bio' 'HOSTNAME=compute-0-5.local' 'IPPROOT=/opt/intel/composer_xe_2015.0.090/ipp' 'TERM=xterm' 'SHELL=/bin/bash' 'HISTSIZE=1000' 'I_MPI_MIC=1' 'GDBSERVER_MIC=/opt/intel/composer_xe_2015.0.090/debugger/gdb/target/mic/bin/gdbserver' 'HMMER_DB=/home/wasim/bio/hmmer/db' 'SSH_CLIENT=10.1.1.1 59616 22' 'LIBRARY_PATH=/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64:/opt/intel/composer_xe_2015.0.090/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2015.0.090/ipp/lib/intel64:/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64:/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64:/opt/intel/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4' 'QTDIR=/usr/lib64/qt-3.3' 'OLDPWD=/home/wasim' 'QTINC=/usr/lib64/qt-3.3/include' 'MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mpirt/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/ipp/lib/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mkl/lib/mic:/opt/intel/composer_xe_2015.0.090/tbb/lib/mic' 'SSH_TTY=/dev/pts/0' 'ANT_HOME=/opt/rocks' 'USER=wasim' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:' 'MIC_LIBRARY_PATH=/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/composer_xe_2015.0.090/mpirt/lib/mic:/opt/intel/composer_xe_2015.0.090/tbb/lib/mic' 'ROCKS_ROOT=/opt/rocks' 'CPATH=/opt/intel/composer_xe_2015.0.090/ipp/include:/opt/intel/composer_xe_2015.0.090/mkl/include:/opt/intel/composer_xe_2015.0.090/tbb/include' 'MIC=1' 'NLSPATH=/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2015.0.090/ipp/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64/locale/%l_%t/%N:/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64_mic/share/locale/%l_%t/%N:/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64/share/locale/%l_%t/%N' 'MAIL=/var/spool/mail/wasim' 'PATH=/opt/intel//impi/5.0.1.035/intel64/bin:/share/apps/Intel/bin:/opt/intel/composer_xe_2015.0.090/bin/intel64:/opt/intel/composer_xe_2015.0.090/mpirt/bin/intel64:/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64_mic/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/bio/ncbi/bin:/opt/bio/mpiblast/bin:/opt/bio/EMBOSS/bin:/opt/bio/clustalw/bin:/opt/bio/tcoffee/bin:/opt/bio/hmmer/bin:/opt/bio/phylip/exe:/opt/bio/mrbayes:/opt/bio/fasta:/opt/bio/glimmer/bin:/opt/bio/glimmer/scripts:/opt/bio/gromacs/bin:/opt/bio/gmap/bin:/opt/bio/tigr/bin:/opt/bio/autodocksuite/bin:/opt/bio/wgs/bin:/opt/ganglia/bin:/opt/ganglia/sbin:/usr/java/latest/bin:/opt/rocks/bin:/opt/rocks/sbin:/home/wasim/bin' 'TBBROOT=/opt/intel/composer_xe_2015.0.090/tbb' 'PWD=/home/wasim/codebase/grn_hetero_app' 'JAVA_HOME=/usr/java/latest' 'GDB_CROSS=/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64_mic/bin/gdb-mic' 'LANG=en_US.iso885915' 'MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles' 'LOADEDMODULES=' 'BLASTDB=/home/wasim/bio/ncbi/db' 'MPM_LAUNCHER=/opt/intel/composer_xe_2015.0.090/debugger/mpm/bin/start_mpm.sh' 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass' 'HISTCONTROL=ignoredups' 'INTEL_PYTHONHOME=/opt/intel/composer_xe_2015.0.090/debugger/python/intel64/' 'SHLVL=1' 'HOME=/home/wasim' 'LOGNAME=wasim' 'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh' 'SSH_CONNECTION=10.1.1.1 59616 10.1.1.248 22' 'MODULESHOME=/usr/share/Modules' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'BLASTMAT=/opt/bio/ncbi/data' 'INFOPATH=/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64/share/info/:/opt/intel/composer_xe_2015.0.090/debugger/gdb/intel64_mic/share/info/' 'INCLUDE=/opt/intel/composer_xe_2015.0.090/mkl/include' 'G_BROKEN_FILENAMES=1' 'I_MPI_ROOT=/opt/intel//impi/5.0.1.035' 'module=() {  eval `/usr/bin/modulecmd bash $*`
}' '_=/opt/intel//impi/5.0.1.035/intel64/bin/mpiexec.hydra' --global-user-env 2 'I_MPI_MIC=1' 'I_MPI_HYDRA_DEBUG=1' --global-system-env 2 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 20 --exec --exec-appnum 0 --exec-proc-count 20 --exec-local-env 0 --exec-wdir /home/wasim/codebase/grn_hetero_app --exec-args 1 ./exec_mic

[mpiexec@compute-0-5.local] Launch arguments: /usr/bin/ssh -x -q mic0 sh -c 'export I_MPI_ROOT="/opt/intel//impi/5.0.1.035" ; export PATH="/opt/intel//impi/5.0.1.035/intel64/bin/:${I_MPI_ROOT}/intel64/bin:${PATH}" ; exec "$0" "$@"' pmi_proxy --control-port 10.1.1.248:43606 --debug --pmi-connect lazy-cache --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel//impi/5.0.1.035/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 1218988696 --usize -2 --proxy-id 0
[mpiexec@compute-0-5.local] STDIN will be redirected to 1 fd(s): 9
pmi_proxy: /opt/intel//impi/5.0.1.035/intel64/bin/pmi_proxy: cannot execute binary file
pmi_proxy: /opt/intel//impi/5.0.1.035/intel64/bin/pmi_proxy: Success

Thanks.

[Artem_R_Intel1 · Accepted solution]

One comment:
It's important to export I_MPI_MIC=1 before the MPI run ('-genv I_MPI_MIC 1' won't work).
But as far as I see I_MPI_MIC=1 was set additionally before.

Could you please provide output of the following commands (from the host and MIC):
hostname -i
hostname -s
hostname -f
Also please provide /etc/hosts content from the host and MIC.

Could you please try the same test scenario with MIC IP address instead of mic0?




 

[Wasim_M_]

Hi,

   Yes, the problem was with /etc/hosts.

   replacing mic0 with ip solved the problem.

   Thanks a ton :).