Intel® MPI Library
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.
2159 Discussions

MPI on two machines with different choices of I_MPI_FABRICS

James_G_
Beginner
‎06-05-2017 05:21 AM
1,341 Views

I am trying to run a simple "hello world" MPI program using two machines.

Here is the MPI program:

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
  MPI_Init(NULL, NULL);

  int world_size;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);

  int world_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int name_len;
  MPI_Get_processor_name(processor_name, &name_len);

  printf("Hello world from processor %s, rank %d out of %d processors\n",
         processor_name, world_rank, world_size);

  MPI_Finalize();
}

A few notes on my setup:

  • I am using Mellanox 3.18-2 OFED distribution, as directed by the Xeon Phi "MPSS user's guide"
  • For simplicity, I'm just trying to run MPI between two PC machines (no Xeon Phi involved)
  • Program is compiled using:
    mpiicpc helloMPI.cc -o helloMPI.XEON
  • Both machines are running bare-metal installations of CentOS 7 (i.e. no VMs)

I get different errors depending on some of the I_MPI_* environment variables.

The only configuration that works without any issues is:

export I_MPI_FABRICS=tcp
export I_MPI_MIC=1   # For some reason, I need this, even though I'm not running on a Xeon Phi
mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 ~/helloMPI.XEON

However, if I attempt to use other settings for I_MPI_FABRICS, then I get errors and it doesn't work.
For example, for the "dapl" fabric choice (which is my eventual desired setting):

export I_MPI_FABRICS=dapl
export I_MPI_MIC=1
mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 ~/helloMPI.XEON


 

host: 192.168.1.111
host: 192.168.1.222

==================================================================================================
mpiexec options:
----------------
  Base path: /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/
  Launcher: ssh
  Debug level: 1
  Enable X: -1

  Global environment:
  -------------------
    I_MPI_PERHOST=allcores
    LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib
    MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl
    MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man
    XDG_SESSION_ID=150
    SPARK_HOME=/home/james/clone/benchmark_spark/spark
    HOSTNAME=xen2
    SELINUX_ROLE_REQUESTED=
    INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses
    IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp
    TERM=xterm-256color
    SHELL=/bin/zsh
    I_MPI_FABRICS=dapl
    HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1
    HISTSIZE=10000
    I_MPI_MIC=1
    KVM=/home/james/phi/modules/linux/custom/kvm
    SSH_CLIENT=10.70.2.94 46738 22
    LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4
    BENCH=/home/james/clone/benchmark_spark
    SELINUX_USE_CURRENT_RANGE=
    COI=/home/james/phi/src/mpss/mpss-coi-3.8.1
    MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic
    SSH_TTY=/dev/pts/0
    VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468
    ZSH=/home/james/.oh-my-zsh
    QT_GRAPHICSSYSTEM_CHECKED=1
    MPSS=/home/james/phi/src/mpss/mpss-3.8.1
    USER=james
    C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot
    LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
    I_MPI_MPIRUN=mpirun
    MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic
    CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include
    PAGER=less
    MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m
    _PHI_ROOT=/home/james/phi
    LSCOLORS=Gxfxcxdxbxegedabagacad
    _INTEL_SOURCE_ME=yes
    NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N
    MAIL=/var/spool/mail/james
    PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin
    FZF_COMPLETION_TRIGGER=##
    TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb
    I_MPI_HYDRA_DEBUG=on
    PHI=/home/james/phi
    PWD=/home/james
    JAVA_HOME=/home/james/java/jdk1.8.0_111
    EDITOR=vim
    HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop
    KERN=/home/james/phi/modules/linux
    LANG=en_CA.UTF-8
    NODE_PATH=/home/james/.jsctags/lib/jsctags/:
    SELINUX_LEVEL_REQUESTED=
    DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal
    HISTCONTROL=ignoredups
    MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1
    C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root
    SHLVL=2
    HOME=/home/james
    GOROOT=/home/james/golang
    I_MPI_DEBUG=6
    PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python
    LESS=-R
    LOGNAME=james
    CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar
    SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22
    LC_CTYPE=en_CA.UTF-8
    GOPATH=/home/james/gopath
    LESSOPEN=||/usr/bin/lesspipe.sh %s
    _PHI_MPSS_SRC=/home/james/phi/src/mpss
    CMAKE_PREFIX_PATH=/home/james/phi/build:
    XDG_RUNTIME_DIR=/run/user/1000
    I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi
    _=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra

  Hydra internal environment:
  ---------------------------
    MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1
    GFORTRAN_UNBUFFERED_PRECONNECTED=y
    I_MPI_HYDRA_UUID=653e0000-4054-4f5d-3551-050001dec0a8
    DAPL_NETWORK_PROCESS_NUM=2

  Intel(R) MPI Library specific variables:
  ----------------------------------------
    I_MPI_PERHOST=allcores
    I_MPI_FABRICS=dapl
    I_MPI_MIC=1
    I_MPI_MPIRUN=mpirun
    I_MPI_HYDRA_DEBUG=on
    I_MPI_DEBUG=6
    I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi
    I_MPI_HYDRA_UUID=653e0000-4054-4f5d-3551-050001dec0a8


    Proxy information:
    *********************
      [1] proxy: 192.168.1.111 (1 cores)
      Exec list: /home/james/helloMPI.XEON (1 processes); 

      [2] proxy: 192.168.1.222 (1 cores)
      Exec list: /home/james/helloMPI.XEON (1 processes); 


==================================================================================================

[mpiexec@xen2] Timeout set to -1 (-1 means infinite)
[mpiexec@xen2] Got a control port string of 192.168.1.222:35852

Proxy launch args: pmi_proxy --control-port 192.168.1.222:35852 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 855354664 --usize -2 --proxy-id 

Arguments being passed to proxy 0:
--version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.1.111 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_15973_0 --pmi-process-mapping (vector,(0,2,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 75 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib' 'MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl' 'MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man' 'XDG_SESSION_ID=150' 'SPARK_HOME=/home/james/clone/benchmark_spark/spark' 'HOSTNAME=xen2' 'SELINUX_ROLE_REQUESTED=' 'INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses' 'IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp' 'TERM=xterm-256color' 'SHELL=/bin/zsh' 'I_MPI_FABRICS=dapl' 'HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1' 'HISTSIZE=10000' 'I_MPI_MIC=1' 'KVM=/home/james/phi/modules/linux/custom/kvm' 'SSH_CLIENT=10.70.2.94 46738 22' 'LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4' 'BENCH=/home/james/clone/benchmark_spark' 'SELINUX_USE_CURRENT_RANGE=' 'COI=/home/james/phi/src/mpss/mpss-coi-3.8.1' 'MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'SSH_TTY=/dev/pts/0' 'VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468' 'ZSH=/home/james/.oh-my-zsh' 'QT_GRAPHICSSYSTEM_CHECKED=1' 'MPSS=/home/james/phi/src/mpss/mpss-3.8.1' 'USER=james' 'C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot' 'LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:' 'I_MPI_MPIRUN=mpirun' 'MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include' 'PAGER=less' 'MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m' '_PHI_ROOT=/home/james/phi' 'LSCOLORS=Gxfxcxdxbxegedabagacad' '_INTEL_SOURCE_ME=yes' 'NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N' 'MAIL=/var/spool/mail/james' 'PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin' 'FZF_COMPLETION_TRIGGER=##' 'TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb' 'I_MPI_HYDRA_DEBUG=on' 'PHI=/home/james/phi' 'PWD=/home/james' 'JAVA_HOME=/home/james/java/jdk1.8.0_111' 'EDITOR=vim' 'HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop' 'KERN=/home/james/phi/modules/linux' 'LANG=en_CA.UTF-8' 'NODE_PATH=/home/james/.jsctags/lib/jsctags/:' 'SELINUX_LEVEL_REQUESTED=' 'DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal' 'HISTCONTROL=ignoredups' 'MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1' 'C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root' 'SHLVL=2' 'HOME=/home/james' 'GOROOT=/home/james/golang' 'I_MPI_DEBUG=6' 'PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python' 'LESS=-R' 'LOGNAME=james' 'CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar' 'SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22' 'LC_CTYPE=en_CA.UTF-8' 'GOPATH=/home/james/gopath' 'LESSOPEN=||/usr/bin/lesspipe.sh %s' '_PHI_MPSS_SRC=/home/james/phi/src/mpss' 'CMAKE_PREFIX_PATH=/home/james/phi/build:' 'XDG_RUNTIME_DIR=/run/user/1000' 'I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi' '_=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 4 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' 'I_MPI_HYDRA_UUID=653e0000-4054-4f5d-3551-050001dec0a8' 'DAPL_NETWORK_PROCESS_NUM=2' --proxy-core-count 1 --mpi-cmd-env mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 /home/james/helloMPI.XEON  --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/james --exec-args 1 /home/james/helloMPI.XEON 

Arguments being passed to proxy 1:
--version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.1.222 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_15973_0 --pmi-process-mapping (vector,(0,2,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 75 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib' 'MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl' 'MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man' 'XDG_SESSION_ID=150' 'SPARK_HOME=/home/james/clone/benchmark_spark/spark' 'HOSTNAME=xen2' 'SELINUX_ROLE_REQUESTED=' 'INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses' 'IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp' 'TERM=xterm-256color' 'SHELL=/bin/zsh' 'I_MPI_FABRICS=dapl' 'HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1' 'HISTSIZE=10000' 'I_MPI_MIC=1' 'KVM=/home/james/phi/modules/linux/custom/kvm' 'SSH_CLIENT=10.70.2.94 46738 22' 'LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4' 'BENCH=/home/james/clone/benchmark_spark' 'SELINUX_USE_CURRENT_RANGE=' 'COI=/home/james/phi/src/mpss/mpss-coi-3.8.1' 'MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'SSH_TTY=/dev/pts/0' 'VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468' 'ZSH=/home/james/.oh-my-zsh' 'QT_GRAPHICSSYSTEM_CHECKED=1' 'MPSS=/home/james/phi/src/mpss/mpss-3.8.1' 'USER=james' 'C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot' 'LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:' 'I_MPI_MPIRUN=mpirun' 'MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include' 'PAGER=less' 'MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m' '_PHI_ROOT=/home/james/phi' 'LSCOLORS=Gxfxcxdxbxegedabagacad' '_INTEL_SOURCE_ME=yes' 'NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N' 'MAIL=/var/spool/mail/james' 'PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin' 'FZF_COMPLETION_TRIGGER=##' 'TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb' 'I_MPI_HYDRA_DEBUG=on' 'PHI=/home/james/phi' 'PWD=/home/james' 'JAVA_HOME=/home/james/java/jdk1.8.0_111' 'EDITOR=vim' 'HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop' 'KERN=/home/james/phi/modules/linux' 'LANG=en_CA.UTF-8' 'NODE_PATH=/home/james/.jsctags/lib/jsctags/:' 'SELINUX_LEVEL_REQUESTED=' 'DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal' 'HISTCONTROL=ignoredups' 'MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1' 'C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root' 'SHLVL=2' 'HOME=/home/james' 'GOROOT=/home/james/golang' 'I_MPI_DEBUG=6' 'PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python' 'LESS=-R' 'LOGNAME=james' 'CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar' 'SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22' 'LC_CTYPE=en_CA.UTF-8' 'GOPATH=/home/james/gopath' 'LESSOPEN=||/usr/bin/lesspipe.sh %s' '_PHI_MPSS_SRC=/home/james/phi/src/mpss' 'CMAKE_PREFIX_PATH=/home/james/phi/build:' 'XDG_RUNTIME_DIR=/run/user/1000' 'I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi' '_=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 4 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' 'I_MPI_HYDRA_UUID=653e0000-4054-4f5d-3551-050001dec0a8' 'DAPL_NETWORK_PROCESS_NUM=2' --proxy-core-count 1 --mpi-cmd-env mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 /home/james/helloMPI.XEON  --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/james --exec-args 1 /home/james/helloMPI.XEON 

[mpiexec@xen2] Launch arguments: /usr/bin/ssh -x -q 192.168.1.111 sh -c 'export I_MPI_ROOT="/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi" ; export PATH="/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/:${I_MPI_ROOT}/intel64/bin:${PATH}" ; exec "$0" "$@"' pmi_proxy --control-port 192.168.1.222:35852 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 855354664 --usize -2 --proxy-id 0 
[mpiexec@xen2] Launch arguments: pmi_proxy --control-port 192.168.1.222:35852 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 855354664 --usize -2 --proxy-id 1 
[proxy:0:1@xen2] Start PMI_proxy 1
[proxy:0:1@xen2] got pmi command (from 9): init
pmi_version=1 pmi_subversion=1 
[proxy:0:1@xen2] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@xen2] got pmi command (from 9): get_maxes

[proxy:0:1@xen2] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[proxy:0:0@xen1] Start PMI_proxy 0
[proxy:0:0@xen1] STDIN will be redirected to 1 fd(s): 9 
[proxy:0:0@xen1] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0@xen1] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@xen1] got pmi command (from 6): get_maxes

[proxy:0:0@xen1] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 6: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 6: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:1@xen2] got pmi command (from 9): get_ranks2hosts

[proxy:0:1@xen2] PMI response: put_ranks2hosts 42 2
13 192.168.1.111 0, 13 192.168.1.222 1, 
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[proxy:0:1@xen2] got pmi command (from 9): get_appnum

[proxy:0:1@xen2] PMI response: cmd=appnum appnum=0
[proxy:0:0@xen1] got pmi command (from 6): get_ranks2hosts

[proxy:0:0@xen1] PMI response: put_ranks2hosts 42 2
13 192.168.1.111 0, 13 192.168.1.222 1, 
[proxy:0:1@xen2] got pmi command (from 9): get_my_kvsname

[proxy:0:1@xen2] PMI response: cmd=my_kvsname kvsname=kvs_15973_0
[proxy:0:0@xen1] got pmi command (from 6): get_appnum

[proxy:0:0@xen1] PMI response: cmd=appnum appnum=0
[proxy:0:0@xen1] got pmi command (from 6): get_my_kvsname

[proxy:0:0@xen1] PMI response: cmd=my_kvsname kvsname=kvs_15973_0
[proxy:0:1@xen2] got pmi command (from 9): get_my_kvsname

[proxy:0:1@xen2] PMI response: cmd=my_kvsname kvsname=kvs_15973_0
[proxy:0:0@xen1] got pmi command (from 6): get_my_kvsname

[proxy:0:0@xen1] PMI response: cmd=my_kvsname kvsname=kvs_15973_0
[0] MPI startup(): Intel(R) MPI Library, Version 2017 Update 2  Build 20170125 (id: 16752)
[0] MPI startup(): Copyright (C) 2003-2017 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded optimized library
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 6: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 6: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[0] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[0] MPI startup(): dapl data transfer mode
[mpiexec@xen2] [pgid: -1] got PMI command: cmd=put kvsname=kvs_15973_0 key=P0-businesscard-0 value=rdma_port0#23708$rdma_host0#0A0000000000029EFE80000000000000E61D2DFFFE129B4000000004$arch_code#6$fabrics_list#dapl$ 
[proxy:0:0@xen1] got pmi command (from 6): put
kvsname=kvs_15973_0 key=P0-businesscard-0 value=rdma_port0#23708$rdma_host0#0A0000000000029EFE80000000000000E61D2DFFFE129B4000000004$arch_code#6$fabrics_list#dapl$ 
[proxy:0:0@xen1] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[1] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-2u
[1] MPI startup(): DAPL provider ofa-v2-mlx4_0-2u
[1] DAPL startup(): trying to open secondary (2) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mcm-2
[1] MPI startup(): secondary DAPL provider ofa-v2-mcm-2
[proxy:0:1@xen2] got pmi command (from 9): put
kvsname=kvs_15973_0 key=P1-businesscard-0 value=rdma_port0#15982$rdma_host0#0A00000000000327FE80000000000000E61D2DFFFE129B1100000004$rdma_port1#15982$rdma_host1#0A00000000000329FE80000000000000E61D2DFFFE129B1100010004$arch_code#8$fabrics_list#dapl$ 
[proxy:0:1@xen2] PMI response: cmd=put_result rc=0 msg=success
[1] MPI startup(): dapl data transfer mode
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: -1] got PMI command: cmd=put kvsname=kvs_15973_0 key=P1-businesscard-0 value=rdma_port0#15982$rdma_host0#0A00000000000327FE80000000000000E61D2DFFFE129B1100000004$rdma_port1#15982$rdma_host1#0A00000000000329FE80000000000000E61D2DFFFE129B1100010004$arch_code#8$fabrics_list#dapl$ 
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 9: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 9: cmd=barrier_out
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): get
kvsname=kvs_15973_0 key=P1-businesscard-0 
[proxy:0:0@xen1] PMI response: cmd=get_result rc=0 msg=success value=rdma_port0#15982$rdma_host0#0A00000000000327FE80000000000000E61D2DFFFE129B1100000004$rdma_port1#15982$rdma_host1#0A00000000000329FE80000000000000E61D2DFFFE129B1100010004$arch_code#8$fabrics_list#dapl$
xen1:UCM:5c9c:194f1740: 21010 us(21010 us):  create_ah: ERR Success
xen1:UCM:5c9c:194f1740: 21019 us(9 us): UCM connect: snd ERR -> cm_lid 0 cm_qpn 327 r_psp 3e6e p_sz=24
libibverbs: GRH is mandatory For RoCE address handle
[0:192.168.1.111][../../src/mpid/ch3/channels/nemesis/netmod/dapl/dapl_conn_rc.c:247] error(0x30000): ofa-v2-mlx4_0-1u: could not connect DAPL endpoints: DAT_INSUFFICIENT_RESOURCES()
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=abort exitcode=68729104
Fatal error in MPI_Init: Internal MPI error!, error stack:
MPIR_Init_thread(805).................: fail failed
MPID_Init(1831).......................: channel initialization failed
MPIDI_CH3_Init(147)...................: fail failed
dapl_rc_setup_all_connections_20(1434): generic failure with errno = 16
(unknown)(): Internal MPI error!
[proxy:0:0@xen1] got pmi command (from 6): abort
exitcode=68729104 
[proxy:0:0@xen1] we don't understand this command abort; forwarding upstream

 

 

The "ofa" fabric also does not work, although it has a different error message:

export I_MPI_FABRICS=ofa
export I_MPI_MIC=1
mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 ~/helloMPI.XEON

 

 

host: 192.168.1.111
host: 192.168.1.222

==================================================================================================
mpiexec options:
----------------
  Base path: /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/
  Launcher: ssh
  Debug level: 1
  Enable X: -1

  Global environment:
  -------------------
    I_MPI_PERHOST=allcores
    LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib
    MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl
    MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man
    XDG_SESSION_ID=150
    SPARK_HOME=/home/james/clone/benchmark_spark/spark
    HOSTNAME=xen2
    SELINUX_ROLE_REQUESTED=
    INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses
    IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp
    TERM=xterm-256color
    SHELL=/bin/zsh
    I_MPI_FABRICS=ofa
    HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1
    HISTSIZE=10000
    I_MPI_MIC=1
    KVM=/home/james/phi/modules/linux/custom/kvm
    SSH_CLIENT=10.70.2.94 46738 22
    LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4
    BENCH=/home/james/clone/benchmark_spark
    SELINUX_USE_CURRENT_RANGE=
    COI=/home/james/phi/src/mpss/mpss-coi-3.8.1
    MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic
    SSH_TTY=/dev/pts/0
    VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468
    ZSH=/home/james/.oh-my-zsh
    QT_GRAPHICSSYSTEM_CHECKED=1
    MPSS=/home/james/phi/src/mpss/mpss-3.8.1
    USER=james
    C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot
    LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
    I_MPI_MPIRUN=mpirun
    MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic
    CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include
    PAGER=less
    MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m
    _PHI_ROOT=/home/james/phi
    LSCOLORS=Gxfxcxdxbxegedabagacad
    _INTEL_SOURCE_ME=yes
    NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N
    MAIL=/var/spool/mail/james
    PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin
    FZF_COMPLETION_TRIGGER=##
    TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb
    I_MPI_HYDRA_DEBUG=on
    PHI=/home/james/phi
    PWD=/home/james
    JAVA_HOME=/home/james/java/jdk1.8.0_111
    EDITOR=vim
    HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop
    KERN=/home/james/phi/modules/linux
    LANG=en_CA.UTF-8
    NODE_PATH=/home/james/.jsctags/lib/jsctags/:
    SELINUX_LEVEL_REQUESTED=
    DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal
    HISTCONTROL=ignoredups
    MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1
    C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root
    SHLVL=2
    HOME=/home/james
    GOROOT=/home/james/golang
    I_MPI_DEBUG=6
    PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python
    LESS=-R
    LOGNAME=james
    CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar
    SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22
    LC_CTYPE=en_CA.UTF-8
    GOPATH=/home/james/gopath
    LESSOPEN=||/usr/bin/lesspipe.sh %s
    _PHI_MPSS_SRC=/home/james/phi/src/mpss
    CMAKE_PREFIX_PATH=/home/james/phi/build:
    XDG_RUNTIME_DIR=/run/user/1000
    I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi
    _=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra

  Hydra internal environment:
  ---------------------------
    MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1
    GFORTRAN_UNBUFFERED_PRECONNECTED=y
    I_MPI_HYDRA_UUID=bc3e0000-f374-a462-3551-050001dec0a8
    DAPL_NETWORK_PROCESS_NUM=2

  Intel(R) MPI Library specific variables:
  ----------------------------------------
    I_MPI_PERHOST=allcores
    I_MPI_FABRICS=ofa
    I_MPI_MIC=1
    I_MPI_MPIRUN=mpirun
    I_MPI_HYDRA_DEBUG=on
    I_MPI_DEBUG=6
    I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi
    I_MPI_HYDRA_UUID=bc3e0000-f374-a462-3551-050001dec0a8


    Proxy information:
    *********************
      [1] proxy: 192.168.1.111 (1 cores)
      Exec list: /home/james/helloMPI.XEON (1 processes); 

      [2] proxy: 192.168.1.222 (1 cores)
      Exec list: /home/james/helloMPI.XEON (1 processes); 


==================================================================================================

[mpiexec@xen2] Timeout set to -1 (-1 means infinite)
[mpiexec@xen2] Got a control port string of 192.168.1.222:36282

Proxy launch args: pmi_proxy --control-port 192.168.1.222:36282 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 629081922 --usize -2 --proxy-id 

Arguments being passed to proxy 0:
--version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.1.111 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_16060_0 --pmi-process-mapping (vector,(0,2,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 75 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib' 'MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl' 'MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man' 'XDG_SESSION_ID=150' 'SPARK_HOME=/home/james/clone/benchmark_spark/spark' 'HOSTNAME=xen2' 'SELINUX_ROLE_REQUESTED=' 'INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses' 'IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp' 'TERM=xterm-256color' 'SHELL=/bin/zsh' 'I_MPI_FABRICS=ofa' 'HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1' 'HISTSIZE=10000' 'I_MPI_MIC=1' 'KVM=/home/james/phi/modules/linux/custom/kvm' 'SSH_CLIENT=10.70.2.94 46738 22' 'LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4' 'BENCH=/home/james/clone/benchmark_spark' 'SELINUX_USE_CURRENT_RANGE=' 'COI=/home/james/phi/src/mpss/mpss-coi-3.8.1' 'MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'SSH_TTY=/dev/pts/0' 'VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468' 'ZSH=/home/james/.oh-my-zsh' 'QT_GRAPHICSSYSTEM_CHECKED=1' 'MPSS=/home/james/phi/src/mpss/mpss-3.8.1' 'USER=james' 'C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot' 'LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:' 'I_MPI_MPIRUN=mpirun' 'MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include' 'PAGER=less' 'MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m' '_PHI_ROOT=/home/james/phi' 'LSCOLORS=Gxfxcxdxbxegedabagacad' '_INTEL_SOURCE_ME=yes' 'NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N' 'MAIL=/var/spool/mail/james' 'PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin' 'FZF_COMPLETION_TRIGGER=##' 'TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb' 'I_MPI_HYDRA_DEBUG=on' 'PHI=/home/james/phi' 'PWD=/home/james' 'JAVA_HOME=/home/james/java/jdk1.8.0_111' 'EDITOR=vim' 'HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop' 'KERN=/home/james/phi/modules/linux' 'LANG=en_CA.UTF-8' 'NODE_PATH=/home/james/.jsctags/lib/jsctags/:' 'SELINUX_LEVEL_REQUESTED=' 'DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal' 'HISTCONTROL=ignoredups' 'MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1' 'C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root' 'SHLVL=2' 'HOME=/home/james' 'GOROOT=/home/james/golang' 'I_MPI_DEBUG=6' 'PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python' 'LESS=-R' 'LOGNAME=james' 'CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar' 'SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22' 'LC_CTYPE=en_CA.UTF-8' 'GOPATH=/home/james/gopath' 'LESSOPEN=||/usr/bin/lesspipe.sh %s' '_PHI_MPSS_SRC=/home/james/phi/src/mpss' 'CMAKE_PREFIX_PATH=/home/james/phi/build:' 'XDG_RUNTIME_DIR=/run/user/1000' 'I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi' '_=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 4 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' 'I_MPI_HYDRA_UUID=bc3e0000-f374-a462-3551-050001dec0a8' 'DAPL_NETWORK_PROCESS_NUM=2' --proxy-core-count 1 --mpi-cmd-env mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 /home/james/helloMPI.XEON  --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/james --exec-args 1 /home/james/helloMPI.XEON 

Arguments being passed to proxy 1:
--version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.1.222 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_16060_0 --pmi-process-mapping (vector,(0,2,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 75 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib' 'MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl' 'MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man' 'XDG_SESSION_ID=150' 'SPARK_HOME=/home/james/clone/benchmark_spark/spark' 'HOSTNAME=xen2' 'SELINUX_ROLE_REQUESTED=' 'INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses' 'IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp' 'TERM=xterm-256color' 'SHELL=/bin/zsh' 'I_MPI_FABRICS=ofa' 'HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1' 'HISTSIZE=10000' 'I_MPI_MIC=1' 'KVM=/home/james/phi/modules/linux/custom/kvm' 'SSH_CLIENT=10.70.2.94 46738 22' 'LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4' 'BENCH=/home/james/clone/benchmark_spark' 'SELINUX_USE_CURRENT_RANGE=' 'COI=/home/james/phi/src/mpss/mpss-coi-3.8.1' 'MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'SSH_TTY=/dev/pts/0' 'VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468' 'ZSH=/home/james/.oh-my-zsh' 'QT_GRAPHICSSYSTEM_CHECKED=1' 'MPSS=/home/james/phi/src/mpss/mpss-3.8.1' 'USER=james' 'C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot' 'LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:' 'I_MPI_MPIRUN=mpirun' 'MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include' 'PAGER=less' 'MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m' '_PHI_ROOT=/home/james/phi' 'LSCOLORS=Gxfxcxdxbxegedabagacad' '_INTEL_SOURCE_ME=yes' 'NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N' 'MAIL=/var/spool/mail/james' 'PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin' 'FZF_COMPLETION_TRIGGER=##' 'TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb' 'I_MPI_HYDRA_DEBUG=on' 'PHI=/home/james/phi' 'PWD=/home/james' 'JAVA_HOME=/home/james/java/jdk1.8.0_111' 'EDITOR=vim' 'HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop' 'KERN=/home/james/phi/modules/linux' 'LANG=en_CA.UTF-8' 'NODE_PATH=/home/james/.jsctags/lib/jsctags/:' 'SELINUX_LEVEL_REQUESTED=' 'DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal' 'HISTCONTROL=ignoredups' 'MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1' 'C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root' 'SHLVL=2' 'HOME=/home/james' 'GOROOT=/home/james/golang' 'I_MPI_DEBUG=6' 'PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python' 'LESS=-R' 'LOGNAME=james' 'CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar' 'SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22' 'LC_CTYPE=en_CA.UTF-8' 'GOPATH=/home/james/gopath' 'LESSOPEN=||/usr/bin/lesspipe.sh %s' '_PHI_MPSS_SRC=/home/james/phi/src/mpss' 'CMAKE_PREFIX_PATH=/home/james/phi/build:' 'XDG_RUNTIME_DIR=/run/user/1000' 'I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi' '_=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 4 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' 'I_MPI_HYDRA_UUID=bc3e0000-f374-a462-3551-050001dec0a8' 'DAPL_NETWORK_PROCESS_NUM=2' --proxy-core-count 1 --mpi-cmd-env mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 /home/james/helloMPI.XEON  --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/james --exec-args 1 /home/james/helloMPI.XEON 

[mpiexec@xen2] Launch arguments: /usr/bin/ssh -x -q 192.168.1.111 sh -c 'export I_MPI_ROOT="/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi" ; export PATH="/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/:${I_MPI_ROOT}/intel64/bin:${PATH}" ; exec "$0" "$@"' pmi_proxy --control-port 192.168.1.222:36282 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 629081922 --usize -2 --proxy-id 0 
[mpiexec@xen2] Launch arguments: pmi_proxy --control-port 192.168.1.222:36282 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 629081922 --usize -2 --proxy-id 1 
[proxy:0:1@xen2] Start PMI_proxy 1
[proxy:0:1@xen2] got pmi command (from 9): init
pmi_version=1 pmi_subversion=1 
[proxy:0:1@xen2] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@xen2] got pmi command (from 9): get_maxes

[proxy:0:1@xen2] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:0@xen1] Start PMI_proxy 0
[proxy:0:0@xen1] STDIN will be redirected to 1 fd(s): 9 
[proxy:0:0@xen1] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0@xen1] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@xen1] got pmi command (from 6): get_maxes

[proxy:0:0@xen1] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 6: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 6: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:1@xen2] got pmi command (from 9): get_ranks2hosts

[proxy:0:1@xen2] PMI response: put_ranks2hosts 42 2
13 192.168.1.111 0, 13 192.168.1.222 1, 
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[proxy:0:1@xen2] got pmi command (from 9): get_appnum

[proxy:0:1@xen2] PMI response: cmd=appnum appnum=0
[proxy:0:0@xen1] got pmi command (from 6): get_ranks2hosts

[proxy:0:0@xen1] PMI response: put_ranks2hosts 42 2
13 192.168.1.111 0, 13 192.168.1.222 1, 
[proxy:0:1@xen2] got pmi command (from 9): get_my_kvsname

[proxy:0:1@xen2] PMI response: cmd=my_kvsname kvsname=kvs_16060_0
[proxy:0:0@xen1] got pmi command (from 6): get_appnum

[proxy:0:0@xen1] PMI response: cmd=appnum appnum=0
[proxy:0:1@xen2] got pmi command (from 9): get_my_kvsname

[proxy:0:0@xen1] got pmi command (from 6): get_my_kvsname

[proxy:0:1@xen2] PMI response: cmd=my_kvsname kvsname=kvs_16060_0
[proxy:0:0@xen1] PMI response: cmd=my_kvsname kvsname=kvs_16060_0
[0] MPI startup(): Intel(R) MPI Library, Version 2017 Update 2  Build 20170125 (id: 16752)
[proxy:0:0@xen1] got pmi command (from 6): get_my_kvsname

[proxy:0:0@xen1] PMI response: cmd=my_kvsname kvsname=kvs_16060_0
[0] MPI startup(): Copyright (C) 2003-2017 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded optimized library
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 6: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 6: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[1] MPI startup(): Found 1 IB devices
[1] MPI startup(): Open 0 IB device: mlx4_0
[proxy:0:1@xen2] got pmi command (from 9): put
kvsname=kvs_16060_0 key=OFA_Init_fail value=1 
[proxy:0:1@xen2] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1@xen2] forwarding command (cmd=put kvsname=kvs_16060_0 key=OFA_Init_fail value=1) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=put kvsname=kvs_16060_0 key=OFA_Init_fail value=1
[1] MPI startup(): ofa fabric is not available and fallback fabric is not enabled

I don't have access to the Intel MPI source code, so sadly I cannot figure out where exactly these errors are manifesting.

Let me know if there's any other information I can provide.
Cheers,
James

0 Kudos

Link Copied

2 Replies
James_S
Employee
‎06-05-2017 11:42 PM
1,341 Views

Dear James, could you please provide the outputs of "env | grep I_MPI" and the debug results when exporting I_MPI_DEBUG=6 under tcp/dapl/ofa fabrics. Thanks.

0 Kudos
Copy link
jimdempseyatthecove
Honored Contributor III
‎06-16-2017 08:12 AM
1,341 Views

Although I do not have a fabric other than tcp, I find that if I omit setting I_MPI_FABRICS, that mpiexec appears to perform an ordered search using fastest fabrics to slowest fabric (tcp being last). Try removing the environment variable I_MPI_FABRICS and see what happens.

Jim Dempsey

0 Kudos
Copy link
Reply

Community support is provided during standard business hours (Monday to Friday 7AM - 5PM PST). Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.