Intel® MPI Library
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.
2218 Discusiones

MPI on two machines with different choices of I_MPI_FABRICS

James_G_
Principiante
‎06-05-2017 05:21 AM
1.878 Vistas

I am trying to run a simple "hello world" MPI program using two machines.

Here is the MPI program:

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
  MPI_Init(NULL, NULL);

  int world_size;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);

  int world_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int name_len;
  MPI_Get_processor_name(processor_name, &name_len);

  printf("Hello world from processor %s, rank %d out of %d processors\n",
         processor_name, world_rank, world_size);

  MPI_Finalize();
}

A few notes on my setup:

  • I am using Mellanox 3.18-2 OFED distribution, as directed by the Xeon Phi "MPSS user's guide"
  • For simplicity, I'm just trying to run MPI between two PC machines (no Xeon Phi involved)
  • Program is compiled using:
    mpiicpc helloMPI.cc -o helloMPI.XEON
  • Both machines are running bare-metal installations of CentOS 7 (i.e. no VMs)

I get different errors depending on some of the I_MPI_* environment variables.

The only configuration that works without any issues is:

export I_MPI_FABRICS=tcp
export I_MPI_MIC=1   # For some reason, I need this, even though I'm not running on a Xeon Phi
mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 ~/helloMPI.XEON

However, if I attempt to use other settings for I_MPI_FABRICS, then I get errors and it doesn't work.
For example, for the "dapl" fabric choice (which is my eventual desired setting):

export I_MPI_FABRICS=dapl
export I_MPI_MIC=1
mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 ~/helloMPI.XEON


 

host: 192.168.1.111
host: 192.168.1.222

==================================================================================================
mpiexec options:
----------------
  Base path: /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/
  Launcher: ssh
  Debug level: 1
  Enable X: -1

  Global environment:
  -------------------
    I_MPI_PERHOST=allcores
    LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib
    MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl
    MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man
    XDG_SESSION_ID=150
    SPARK_HOME=/home/james/clone/benchmark_spark/spark
    HOSTNAME=xen2
    SELINUX_ROLE_REQUESTED=
    INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses
    IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp
    TERM=xterm-256color
    SHELL=/bin/zsh
    I_MPI_FABRICS=dapl
    HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1
    HISTSIZE=10000
    I_MPI_MIC=1
    KVM=/home/james/phi/modules/linux/custom/kvm
    SSH_CLIENT=10.70.2.94 46738 22
    LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4
    BENCH=/home/james/clone/benchmark_spark
    SELINUX_USE_CURRENT_RANGE=
    COI=/home/james/phi/src/mpss/mpss-coi-3.8.1
    MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic
    SSH_TTY=/dev/pts/0
    VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468
    ZSH=/home/james/.oh-my-zsh
    QT_GRAPHICSSYSTEM_CHECKED=1
    MPSS=/home/james/phi/src/mpss/mpss-3.8.1
    USER=james
    C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot
    LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
    I_MPI_MPIRUN=mpirun
    MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic
    CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include
    PAGER=less
    MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m
    _PHI_ROOT=/home/james/phi
    LSCOLORS=Gxfxcxdxbxegedabagacad
    _INTEL_SOURCE_ME=yes
    NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N
    MAIL=/var/spool/mail/james
    PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin
    FZF_COMPLETION_TRIGGER=##
    TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb
    I_MPI_HYDRA_DEBUG=on
    PHI=/home/james/phi
    PWD=/home/james
    JAVA_HOME=/home/james/java/jdk1.8.0_111
    EDITOR=vim
    HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop
    KERN=/home/james/phi/modules/linux
    LANG=en_CA.UTF-8
    NODE_PATH=/home/james/.jsctags/lib/jsctags/:
    SELINUX_LEVEL_REQUESTED=
    DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal
    HISTCONTROL=ignoredups
    MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1
    C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root
    SHLVL=2
    HOME=/home/james
    GOROOT=/home/james/golang
    I_MPI_DEBUG=6
    PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python
    LESS=-R
    LOGNAME=james
    CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar
    SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22
    LC_CTYPE=en_CA.UTF-8
    GOPATH=/home/james/gopath
    LESSOPEN=||/usr/bin/lesspipe.sh %s
    _PHI_MPSS_SRC=/home/james/phi/src/mpss
    CMAKE_PREFIX_PATH=/home/james/phi/build:
    XDG_RUNTIME_DIR=/run/user/1000
    I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi
    _=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra

  Hydra internal environment:
  ---------------------------
    MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1
    GFORTRAN_UNBUFFERED_PRECONNECTED=y
    I_MPI_HYDRA_UUID=653e0000-4054-4f5d-3551-050001dec0a8
    DAPL_NETWORK_PROCESS_NUM=2

  Intel(R) MPI Library specific variables:
  ----------------------------------------
    I_MPI_PERHOST=allcores
    I_MPI_FABRICS=dapl
    I_MPI_MIC=1
    I_MPI_MPIRUN=mpirun
    I_MPI_HYDRA_DEBUG=on
    I_MPI_DEBUG=6
    I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi
    I_MPI_HYDRA_UUID=653e0000-4054-4f5d-3551-050001dec0a8


    Proxy information:
    *********************
      [1] proxy: 192.168.1.111 (1 cores)
      Exec list: /home/james/helloMPI.XEON (1 processes); 

      [2] proxy: 192.168.1.222 (1 cores)
      Exec list: /home/james/helloMPI.XEON (1 processes); 


==================================================================================================

[mpiexec@xen2] Timeout set to -1 (-1 means infinite)
[mpiexec@xen2] Got a control port string of 192.168.1.222:35852

Proxy launch args: pmi_proxy --control-port 192.168.1.222:35852 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 855354664 --usize -2 --proxy-id 

Arguments being passed to proxy 0:
--version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.1.111 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_15973_0 --pmi-process-mapping (vector,(0,2,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 75 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib' 'MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl' 'MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man' 'XDG_SESSION_ID=150' 'SPARK_HOME=/home/james/clone/benchmark_spark/spark' 'HOSTNAME=xen2' 'SELINUX_ROLE_REQUESTED=' 'INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses' 'IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp' 'TERM=xterm-256color' 'SHELL=/bin/zsh' 'I_MPI_FABRICS=dapl' 'HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1' 'HISTSIZE=10000' 'I_MPI_MIC=1' 'KVM=/home/james/phi/modules/linux/custom/kvm' 'SSH_CLIENT=10.70.2.94 46738 22' 'LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4' 'BENCH=/home/james/clone/benchmark_spark' 'SELINUX_USE_CURRENT_RANGE=' 'COI=/home/james/phi/src/mpss/mpss-coi-3.8.1' 'MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'SSH_TTY=/dev/pts/0' 'VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468' 'ZSH=/home/james/.oh-my-zsh' 'QT_GRAPHICSSYSTEM_CHECKED=1' 'MPSS=/home/james/phi/src/mpss/mpss-3.8.1' 'USER=james' 'C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot' 'LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:' 'I_MPI_MPIRUN=mpirun' 'MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include' 'PAGER=less' 'MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m' '_PHI_ROOT=/home/james/phi' 'LSCOLORS=Gxfxcxdxbxegedabagacad' '_INTEL_SOURCE_ME=yes' 'NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N' 'MAIL=/var/spool/mail/james' 'PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin' 'FZF_COMPLETION_TRIGGER=##' 'TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb' 'I_MPI_HYDRA_DEBUG=on' 'PHI=/home/james/phi' 'PWD=/home/james' 'JAVA_HOME=/home/james/java/jdk1.8.0_111' 'EDITOR=vim' 'HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop' 'KERN=/home/james/phi/modules/linux' 'LANG=en_CA.UTF-8' 'NODE_PATH=/home/james/.jsctags/lib/jsctags/:' 'SELINUX_LEVEL_REQUESTED=' 'DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal' 'HISTCONTROL=ignoredups' 'MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1' 'C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root' 'SHLVL=2' 'HOME=/home/james' 'GOROOT=/home/james/golang' 'I_MPI_DEBUG=6' 'PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python' 'LESS=-R' 'LOGNAME=james' 'CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar' 'SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22' 'LC_CTYPE=en_CA.UTF-8' 'GOPATH=/home/james/gopath' 'LESSOPEN=||/usr/bin/lesspipe.sh %s' '_PHI_MPSS_SRC=/home/james/phi/src/mpss' 'CMAKE_PREFIX_PATH=/home/james/phi/build:' 'XDG_RUNTIME_DIR=/run/user/1000' 'I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi' '_=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 4 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' 'I_MPI_HYDRA_UUID=653e0000-4054-4f5d-3551-050001dec0a8' 'DAPL_NETWORK_PROCESS_NUM=2' --proxy-core-count 1 --mpi-cmd-env mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 /home/james/helloMPI.XEON  --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/james --exec-args 1 /home/james/helloMPI.XEON 

Arguments being passed to proxy 1:
--version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.1.222 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_15973_0 --pmi-process-mapping (vector,(0,2,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 75 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib' 'MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl' 'MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man' 'XDG_SESSION_ID=150' 'SPARK_HOME=/home/james/clone/benchmark_spark/spark' 'HOSTNAME=xen2' 'SELINUX_ROLE_REQUESTED=' 'INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses' 'IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp' 'TERM=xterm-256color' 'SHELL=/bin/zsh' 'I_MPI_FABRICS=dapl' 'HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1' 'HISTSIZE=10000' 'I_MPI_MIC=1' 'KVM=/home/james/phi/modules/linux/custom/kvm' 'SSH_CLIENT=10.70.2.94 46738 22' 'LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4' 'BENCH=/home/james/clone/benchmark_spark' 'SELINUX_USE_CURRENT_RANGE=' 'COI=/home/james/phi/src/mpss/mpss-coi-3.8.1' 'MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'SSH_TTY=/dev/pts/0' 'VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468' 'ZSH=/home/james/.oh-my-zsh' 'QT_GRAPHICSSYSTEM_CHECKED=1' 'MPSS=/home/james/phi/src/mpss/mpss-3.8.1' 'USER=james' 'C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot' 'LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:' 'I_MPI_MPIRUN=mpirun' 'MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include' 'PAGER=less' 'MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m' '_PHI_ROOT=/home/james/phi' 'LSCOLORS=Gxfxcxdxbxegedabagacad' '_INTEL_SOURCE_ME=yes' 'NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N' 'MAIL=/var/spool/mail/james' 'PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin' 'FZF_COMPLETION_TRIGGER=##' 'TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb' 'I_MPI_HYDRA_DEBUG=on' 'PHI=/home/james/phi' 'PWD=/home/james' 'JAVA_HOME=/home/james/java/jdk1.8.0_111' 'EDITOR=vim' 'HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop' 'KERN=/home/james/phi/modules/linux' 'LANG=en_CA.UTF-8' 'NODE_PATH=/home/james/.jsctags/lib/jsctags/:' 'SELINUX_LEVEL_REQUESTED=' 'DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal' 'HISTCONTROL=ignoredups' 'MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1' 'C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root' 'SHLVL=2' 'HOME=/home/james' 'GOROOT=/home/james/golang' 'I_MPI_DEBUG=6' 'PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python' 'LESS=-R' 'LOGNAME=james' 'CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar' 'SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22' 'LC_CTYPE=en_CA.UTF-8' 'GOPATH=/home/james/gopath' 'LESSOPEN=||/usr/bin/lesspipe.sh %s' '_PHI_MPSS_SRC=/home/james/phi/src/mpss' 'CMAKE_PREFIX_PATH=/home/james/phi/build:' 'XDG_RUNTIME_DIR=/run/user/1000' 'I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi' '_=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 4 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' 'I_MPI_HYDRA_UUID=653e0000-4054-4f5d-3551-050001dec0a8' 'DAPL_NETWORK_PROCESS_NUM=2' --proxy-core-count 1 --mpi-cmd-env mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 /home/james/helloMPI.XEON  --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/james --exec-args 1 /home/james/helloMPI.XEON 

[mpiexec@xen2] Launch arguments: /usr/bin/ssh -x -q 192.168.1.111 sh -c 'export I_MPI_ROOT="/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi" ; export PATH="/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/:${I_MPI_ROOT}/intel64/bin:${PATH}" ; exec "$0" "$@"' pmi_proxy --control-port 192.168.1.222:35852 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 855354664 --usize -2 --proxy-id 0 
[mpiexec@xen2] Launch arguments: pmi_proxy --control-port 192.168.1.222:35852 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 855354664 --usize -2 --proxy-id 1 
[proxy:0:1@xen2] Start PMI_proxy 1
[proxy:0:1@xen2] got pmi command (from 9): init
pmi_version=1 pmi_subversion=1 
[proxy:0:1@xen2] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@xen2] got pmi command (from 9): get_maxes

[proxy:0:1@xen2] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[proxy:0:0@xen1] Start PMI_proxy 0
[proxy:0:0@xen1] STDIN will be redirected to 1 fd(s): 9 
[proxy:0:0@xen1] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0@xen1] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@xen1] got pmi command (from 6): get_maxes

[proxy:0:0@xen1] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 6: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 6: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:1@xen2] got pmi command (from 9): get_ranks2hosts

[proxy:0:1@xen2] PMI response: put_ranks2hosts 42 2
13 192.168.1.111 0, 13 192.168.1.222 1, 
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[proxy:0:1@xen2] got pmi command (from 9): get_appnum

[proxy:0:1@xen2] PMI response: cmd=appnum appnum=0
[proxy:0:0@xen1] got pmi command (from 6): get_ranks2hosts

[proxy:0:0@xen1] PMI response: put_ranks2hosts 42 2
13 192.168.1.111 0, 13 192.168.1.222 1, 
[proxy:0:1@xen2] got pmi command (from 9): get_my_kvsname

[proxy:0:1@xen2] PMI response: cmd=my_kvsname kvsname=kvs_15973_0
[proxy:0:0@xen1] got pmi command (from 6): get_appnum

[proxy:0:0@xen1] PMI response: cmd=appnum appnum=0
[proxy:0:0@xen1] got pmi command (from 6): get_my_kvsname

[proxy:0:0@xen1] PMI response: cmd=my_kvsname kvsname=kvs_15973_0
[proxy:0:1@xen2] got pmi command (from 9): get_my_kvsname

[proxy:0:1@xen2] PMI response: cmd=my_kvsname kvsname=kvs_15973_0
[proxy:0:0@xen1] got pmi command (from 6): get_my_kvsname

[proxy:0:0@xen1] PMI response: cmd=my_kvsname kvsname=kvs_15973_0
[0] MPI startup(): Intel(R) MPI Library, Version 2017 Update 2  Build 20170125 (id: 16752)
[0] MPI startup(): Copyright (C) 2003-2017 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded optimized library
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 6: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 6: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[0] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[0] MPI startup(): dapl data transfer mode
[mpiexec@xen2] [pgid: -1] got PMI command: cmd=put kvsname=kvs_15973_0 key=P0-businesscard-0 value=rdma_port0#23708$rdma_host0#0A0000000000029EFE80000000000000E61D2DFFFE129B4000000004$arch_code#6$fabrics_list#dapl$ 
[proxy:0:0@xen1] got pmi command (from 6): put
kvsname=kvs_15973_0 key=P0-businesscard-0 value=rdma_port0#23708$rdma_host0#0A0000000000029EFE80000000000000E61D2DFFFE129B4000000004$arch_code#6$fabrics_list#dapl$ 
[proxy:0:0@xen1] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[1] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-2u
[1] MPI startup(): DAPL provider ofa-v2-mlx4_0-2u
[1] DAPL startup(): trying to open secondary (2) DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mcm-2
[1] MPI startup(): secondary DAPL provider ofa-v2-mcm-2
[proxy:0:1@xen2] got pmi command (from 9): put
kvsname=kvs_15973_0 key=P1-businesscard-0 value=rdma_port0#15982$rdma_host0#0A00000000000327FE80000000000000E61D2DFFFE129B1100000004$rdma_port1#15982$rdma_host1#0A00000000000329FE80000000000000E61D2DFFFE129B1100010004$arch_code#8$fabrics_list#dapl$ 
[proxy:0:1@xen2] PMI response: cmd=put_result rc=0 msg=success
[1] MPI startup(): dapl data transfer mode
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: -1] got PMI command: cmd=put kvsname=kvs_15973_0 key=P1-businesscard-0 value=rdma_port0#15982$rdma_host0#0A00000000000327FE80000000000000E61D2DFFFE129B1100000004$rdma_port1#15982$rdma_host1#0A00000000000329FE80000000000000E61D2DFFFE129B1100010004$arch_code#8$fabrics_list#dapl$ 
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 9: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 9: cmd=barrier_out
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): get
kvsname=kvs_15973_0 key=P1-businesscard-0 
[proxy:0:0@xen1] PMI response: cmd=get_result rc=0 msg=success value=rdma_port0#15982$rdma_host0#0A00000000000327FE80000000000000E61D2DFFFE129B1100000004$rdma_port1#15982$rdma_host1#0A00000000000329FE80000000000000E61D2DFFFE129B1100010004$arch_code#8$fabrics_list#dapl$
xen1:UCM:5c9c:194f1740: 21010 us(21010 us):  create_ah: ERR Success
xen1:UCM:5c9c:194f1740: 21019 us(9 us): UCM connect: snd ERR -> cm_lid 0 cm_qpn 327 r_psp 3e6e p_sz=24
libibverbs: GRH is mandatory For RoCE address handle
[0:192.168.1.111][../../src/mpid/ch3/channels/nemesis/netmod/dapl/dapl_conn_rc.c:247] error(0x30000): ofa-v2-mlx4_0-1u: could not connect DAPL endpoints: DAT_INSUFFICIENT_RESOURCES()
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=abort exitcode=68729104
Fatal error in MPI_Init: Internal MPI error!, error stack:
MPIR_Init_thread(805).................: fail failed
MPID_Init(1831).......................: channel initialization failed
MPIDI_CH3_Init(147)...................: fail failed
dapl_rc_setup_all_connections_20(1434): generic failure with errno = 16
(unknown)(): Internal MPI error!
[proxy:0:0@xen1] got pmi command (from 6): abort
exitcode=68729104 
[proxy:0:0@xen1] we don't understand this command abort; forwarding upstream

 

 

The "ofa" fabric also does not work, although it has a different error message:

export I_MPI_FABRICS=ofa
export I_MPI_MIC=1
mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 ~/helloMPI.XEON

 

 

host: 192.168.1.111
host: 192.168.1.222

==================================================================================================
mpiexec options:
----------------
  Base path: /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/
  Launcher: ssh
  Debug level: 1
  Enable X: -1

  Global environment:
  -------------------
    I_MPI_PERHOST=allcores
    LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib
    MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl
    MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man
    XDG_SESSION_ID=150
    SPARK_HOME=/home/james/clone/benchmark_spark/spark
    HOSTNAME=xen2
    SELINUX_ROLE_REQUESTED=
    INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses
    IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp
    TERM=xterm-256color
    SHELL=/bin/zsh
    I_MPI_FABRICS=ofa
    HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1
    HISTSIZE=10000
    I_MPI_MIC=1
    KVM=/home/james/phi/modules/linux/custom/kvm
    SSH_CLIENT=10.70.2.94 46738 22
    LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4
    BENCH=/home/james/clone/benchmark_spark
    SELINUX_USE_CURRENT_RANGE=
    COI=/home/james/phi/src/mpss/mpss-coi-3.8.1
    MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic
    SSH_TTY=/dev/pts/0
    VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468
    ZSH=/home/james/.oh-my-zsh
    QT_GRAPHICSSYSTEM_CHECKED=1
    MPSS=/home/james/phi/src/mpss/mpss-3.8.1
    USER=james
    C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot
    LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
    I_MPI_MPIRUN=mpirun
    MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic
    CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include
    PAGER=less
    MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m
    _PHI_ROOT=/home/james/phi
    LSCOLORS=Gxfxcxdxbxegedabagacad
    _INTEL_SOURCE_ME=yes
    NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N
    MAIL=/var/spool/mail/james
    PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin
    FZF_COMPLETION_TRIGGER=##
    TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb
    I_MPI_HYDRA_DEBUG=on
    PHI=/home/james/phi
    PWD=/home/james
    JAVA_HOME=/home/james/java/jdk1.8.0_111
    EDITOR=vim
    HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop
    KERN=/home/james/phi/modules/linux
    LANG=en_CA.UTF-8
    NODE_PATH=/home/james/.jsctags/lib/jsctags/:
    SELINUX_LEVEL_REQUESTED=
    DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal
    HISTCONTROL=ignoredups
    MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1
    C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root
    SHLVL=2
    HOME=/home/james
    GOROOT=/home/james/golang
    I_MPI_DEBUG=6
    PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python
    LESS=-R
    LOGNAME=james
    CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar
    SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22
    LC_CTYPE=en_CA.UTF-8
    GOPATH=/home/james/gopath
    LESSOPEN=||/usr/bin/lesspipe.sh %s
    _PHI_MPSS_SRC=/home/james/phi/src/mpss
    CMAKE_PREFIX_PATH=/home/james/phi/build:
    XDG_RUNTIME_DIR=/run/user/1000
    I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi
    _=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra

  Hydra internal environment:
  ---------------------------
    MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1
    GFORTRAN_UNBUFFERED_PRECONNECTED=y
    I_MPI_HYDRA_UUID=bc3e0000-f374-a462-3551-050001dec0a8
    DAPL_NETWORK_PROCESS_NUM=2

  Intel(R) MPI Library specific variables:
  ----------------------------------------
    I_MPI_PERHOST=allcores
    I_MPI_FABRICS=ofa
    I_MPI_MIC=1
    I_MPI_MPIRUN=mpirun
    I_MPI_HYDRA_DEBUG=on
    I_MPI_DEBUG=6
    I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi
    I_MPI_HYDRA_UUID=bc3e0000-f374-a462-3551-050001dec0a8


    Proxy information:
    *********************
      [1] proxy: 192.168.1.111 (1 cores)
      Exec list: /home/james/helloMPI.XEON (1 processes); 

      [2] proxy: 192.168.1.222 (1 cores)
      Exec list: /home/james/helloMPI.XEON (1 processes); 


==================================================================================================

[mpiexec@xen2] Timeout set to -1 (-1 means infinite)
[mpiexec@xen2] Got a control port string of 192.168.1.222:36282

Proxy launch args: pmi_proxy --control-port 192.168.1.222:36282 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 629081922 --usize -2 --proxy-id 

Arguments being passed to proxy 0:
--version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.1.111 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_16060_0 --pmi-process-mapping (vector,(0,2,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 75 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib' 'MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl' 'MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man' 'XDG_SESSION_ID=150' 'SPARK_HOME=/home/james/clone/benchmark_spark/spark' 'HOSTNAME=xen2' 'SELINUX_ROLE_REQUESTED=' 'INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses' 'IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp' 'TERM=xterm-256color' 'SHELL=/bin/zsh' 'I_MPI_FABRICS=ofa' 'HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1' 'HISTSIZE=10000' 'I_MPI_MIC=1' 'KVM=/home/james/phi/modules/linux/custom/kvm' 'SSH_CLIENT=10.70.2.94 46738 22' 'LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4' 'BENCH=/home/james/clone/benchmark_spark' 'SELINUX_USE_CURRENT_RANGE=' 'COI=/home/james/phi/src/mpss/mpss-coi-3.8.1' 'MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'SSH_TTY=/dev/pts/0' 'VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468' 'ZSH=/home/james/.oh-my-zsh' 'QT_GRAPHICSSYSTEM_CHECKED=1' 'MPSS=/home/james/phi/src/mpss/mpss-3.8.1' 'USER=james' 'C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot' 'LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:' 'I_MPI_MPIRUN=mpirun' 'MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include' 'PAGER=less' 'MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m' '_PHI_ROOT=/home/james/phi' 'LSCOLORS=Gxfxcxdxbxegedabagacad' '_INTEL_SOURCE_ME=yes' 'NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N' 'MAIL=/var/spool/mail/james' 'PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin' 'FZF_COMPLETION_TRIGGER=##' 'TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb' 'I_MPI_HYDRA_DEBUG=on' 'PHI=/home/james/phi' 'PWD=/home/james' 'JAVA_HOME=/home/james/java/jdk1.8.0_111' 'EDITOR=vim' 'HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop' 'KERN=/home/james/phi/modules/linux' 'LANG=en_CA.UTF-8' 'NODE_PATH=/home/james/.jsctags/lib/jsctags/:' 'SELINUX_LEVEL_REQUESTED=' 'DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal' 'HISTCONTROL=ignoredups' 'MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1' 'C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root' 'SHLVL=2' 'HOME=/home/james' 'GOROOT=/home/james/golang' 'I_MPI_DEBUG=6' 'PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python' 'LESS=-R' 'LOGNAME=james' 'CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar' 'SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22' 'LC_CTYPE=en_CA.UTF-8' 'GOPATH=/home/james/gopath' 'LESSOPEN=||/usr/bin/lesspipe.sh %s' '_PHI_MPSS_SRC=/home/james/phi/src/mpss' 'CMAKE_PREFIX_PATH=/home/james/phi/build:' 'XDG_RUNTIME_DIR=/run/user/1000' 'I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi' '_=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 4 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' 'I_MPI_HYDRA_UUID=bc3e0000-f374-a462-3551-050001dec0a8' 'DAPL_NETWORK_PROCESS_NUM=2' --proxy-core-count 1 --mpi-cmd-env mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 /home/james/helloMPI.XEON  --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/james --exec-args 1 /home/james/helloMPI.XEON 

Arguments being passed to proxy 1:
--version 3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.1.222 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_16060_0 --pmi-process-mapping (vector,(0,2,1)) --topolib ipl --ckpointlib blcr --ckpoint-prefix /tmp --ckpoint-preserve 1 --ckpoint off --ckpoint-num -1 --global-inherited-env 75 'I_MPI_PERHOST=allcores' 'LD_LIBRARY_PATH=/home/james/phi/build/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib' 'MKLROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl' 'MANPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/man:/usr/local/share/man:/usr/share/man:/opt/ibutils/share/man:/home/james/.fzf/man' 'XDG_SESSION_ID=150' 'SPARK_HOME=/home/james/clone/benchmark_spark/spark' 'HOSTNAME=xen2' 'SELINUX_ROLE_REQUESTED=' 'INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2017.3.191/linux/licenses:/opt/intel/licenses:/home/james/intel/licenses' 'IPPROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp' 'TERM=xterm-256color' 'SHELL=/bin/zsh' 'I_MPI_FABRICS=ofa' 'HADOOP_HOME=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1' 'HISTSIZE=10000' 'I_MPI_MIC=1' 'KVM=/home/james/phi/modules/linux/custom/kvm' 'SSH_CLIENT=10.70.2.94 46738 22' 'LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/../tbb/lib/intel64_lin/gcc4.4' 'BENCH=/home/james/clone/benchmark_spark' 'SELINUX_USE_CURRENT_RANGE=' 'COI=/home/james/phi/src/mpss/mpss-coi-3.8.1' 'MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'SSH_TTY=/dev/pts/0' 'VTUNE_AMPLIFIER_XE_2017_DIR=/opt/intel/vtune_amplifier_xe_2017.0.2.478468' 'ZSH=/home/james/.oh-my-zsh' 'QT_GRAPHICSSYSTEM_CHECKED=1' 'MPSS=/home/james/phi/src/mpss/mpss-3.8.1' 'USER=james' 'C_BOOT=/home/james/phi/modules/linux/custom/kvm/centos_boot' 'LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:' 'I_MPI_MPIRUN=mpirun' 'MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin_mic:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/lib/mic' 'CPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/ipp/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb/include:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/include' 'PAGER=less' 'MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m' '_PHI_ROOT=/home/james/phi' 'LSCOLORS=Gxfxcxdxbxegedabagacad' '_INTEL_SOURCE_ME=yes' 'NLSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/compiler/lib/intel64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2017.3.191/linux/mkl/lib/intel64_lin/locale/%l_%t/%N' 'MAIL=/var/spool/mail/james' 'PATH=/opt/intel/vtune_amplifier_xe_2017.0.2.478468/bin64:/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/bin:/home/james/clone/benchmark_spark/spark/bin:/home/james/maven/apache-maven-3.3.9/bin:/home/james/java/jdk1.8.0_111/bin:/home/james/bin:/home/james/local/bin:/home/james/.fzf:/home/james/phi/src/python:/home/james/phi/src/sh:/home/james/phi/build/bin:/home/james/phi/modules/linux/custom/custom_scripts:/opt/intel/compilers_and_libraries_2017.3.191/linux/bin/intel64:/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin:/usr/local/bin:/usr/bin:/home/james/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/home/james/.rvm/bin:/home/james/gopath/bin:/home/james/.fzf/bin' 'FZF_COMPLETION_TRIGGER=##' 'TBBROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/tbb' 'I_MPI_HYDRA_DEBUG=on' 'PHI=/home/james/phi' 'PWD=/home/james' 'JAVA_HOME=/home/james/java/jdk1.8.0_111' 'EDITOR=vim' 'HADOOP_CONF_DIR=/home/james/clone/benchmark_spark/dist/hadoop-3.0.0-alpha1/etc/hadoop' 'KERN=/home/james/phi/modules/linux' 'LANG=en_CA.UTF-8' 'NODE_PATH=/home/james/.jsctags/lib/jsctags/:' 'SELINUX_LEVEL_REQUESTED=' 'DAALROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/daal' 'HISTCONTROL=ignoredups' 'MOD=/home/james/phi/src/mpss/mpss-modules-srpm-3.8.1' 'C_MNT=/home/james/phi/modules/linux/custom/kvm/centos_root' 'SHLVL=2' 'HOME=/home/james' 'GOROOT=/home/james/golang' 'I_MPI_DEBUG=6' 'PYTHONPATH=.:/home/james/python:/home/james/.vim/src/python:/home/james/phi/build/local/lib/python2.7/site-packages:/home/james/phi/src/python' 'LESS=-R' 'LOGNAME=james' 'CLASSPATH=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/lib/mpi.jar:/opt/intel/compilers_and_libraries_2017.3.191/linux/daal/lib/daal.jar' 'SSH_CONNECTION=10.70.2.94 46738 10.70.2.83 22' 'LC_CTYPE=en_CA.UTF-8' 'GOPATH=/home/james/gopath' 'LESSOPEN=||/usr/bin/lesspipe.sh %s' '_PHI_MPSS_SRC=/home/james/phi/src/mpss' 'CMAKE_PREFIX_PATH=/home/james/phi/build:' 'XDG_RUNTIME_DIR=/run/user/1000' 'I_MPI_ROOT=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi' '_=/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/mpiexec.hydra' --global-user-env 0 --global-system-env 4 'MPIR_CVAR_NEMESIS_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' 'I_MPI_HYDRA_UUID=bc3e0000-f374-a462-3551-050001dec0a8' 'DAPL_NETWORK_PROCESS_NUM=2' --proxy-core-count 1 --mpi-cmd-env mpirun -ppn 1 -host 192.168.1.111,192.168.1.222 -np 2 /home/james/helloMPI.XEON  --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/james --exec-args 1 /home/james/helloMPI.XEON 

[mpiexec@xen2] Launch arguments: /usr/bin/ssh -x -q 192.168.1.111 sh -c 'export I_MPI_ROOT="/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi" ; export PATH="/opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/:${I_MPI_ROOT}/intel64/bin:${PATH}" ; exec "$0" "$@"' pmi_proxy --control-port 192.168.1.222:36282 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 629081922 --usize -2 --proxy-id 0 
[mpiexec@xen2] Launch arguments: pmi_proxy --control-port 192.168.1.222:36282 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --enable-mic --i_mpi_base_path /opt/intel/compilers_and_libraries_2017.3.191/linux/mpi/intel64/bin/ --i_mpi_base_arch 0 --rmk user --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 629081922 --usize -2 --proxy-id 1 
[proxy:0:1@xen2] Start PMI_proxy 1
[proxy:0:1@xen2] got pmi command (from 9): init
pmi_version=1 pmi_subversion=1 
[proxy:0:1@xen2] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@xen2] got pmi command (from 9): get_maxes

[proxy:0:1@xen2] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:0@xen1] Start PMI_proxy 0
[proxy:0:0@xen1] STDIN will be redirected to 1 fd(s): 9 
[proxy:0:0@xen1] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0@xen1] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@xen1] got pmi command (from 6): get_maxes

[proxy:0:0@xen1] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 6: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 6: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:1@xen2] got pmi command (from 9): get_ranks2hosts

[proxy:0:1@xen2] PMI response: put_ranks2hosts 42 2
13 192.168.1.111 0, 13 192.168.1.222 1, 
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[proxy:0:1@xen2] got pmi command (from 9): get_appnum

[proxy:0:1@xen2] PMI response: cmd=appnum appnum=0
[proxy:0:0@xen1] got pmi command (from 6): get_ranks2hosts

[proxy:0:0@xen1] PMI response: put_ranks2hosts 42 2
13 192.168.1.111 0, 13 192.168.1.222 1, 
[proxy:0:1@xen2] got pmi command (from 9): get_my_kvsname

[proxy:0:1@xen2] PMI response: cmd=my_kvsname kvsname=kvs_16060_0
[proxy:0:0@xen1] got pmi command (from 6): get_appnum

[proxy:0:0@xen1] PMI response: cmd=appnum appnum=0
[proxy:0:1@xen2] got pmi command (from 9): get_my_kvsname

[proxy:0:0@xen1] got pmi command (from 6): get_my_kvsname

[proxy:0:1@xen2] PMI response: cmd=my_kvsname kvsname=kvs_16060_0
[proxy:0:0@xen1] PMI response: cmd=my_kvsname kvsname=kvs_16060_0
[0] MPI startup(): Intel(R) MPI Library, Version 2017 Update 2  Build 20170125 (id: 16752)
[proxy:0:0@xen1] got pmi command (from 6): get_my_kvsname

[proxy:0:0@xen1] PMI response: cmd=my_kvsname kvsname=kvs_16060_0
[0] MPI startup(): Copyright (C) 2003-2017 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded optimized library
[proxy:0:1@xen2] got pmi command (from 9): barrier_in

[proxy:0:1@xen2] forwarding command (cmd=barrier_in) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@xen2] PMI response to fd 11 pid 6: cmd=barrier_out
[mpiexec@xen2] PMI response to fd 8 pid 6: cmd=barrier_out
[proxy:0:0@xen1] got pmi command (from 6): barrier_in

[proxy:0:0@xen1] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@xen2] PMI response: cmd=barrier_out
[proxy:0:0@xen1] PMI response: cmd=barrier_out
[1] MPI startup(): Found 1 IB devices
[1] MPI startup(): Open 0 IB device: mlx4_0
[proxy:0:1@xen2] got pmi command (from 9): put
kvsname=kvs_16060_0 key=OFA_Init_fail value=1 
[proxy:0:1@xen2] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1@xen2] forwarding command (cmd=put kvsname=kvs_16060_0 key=OFA_Init_fail value=1) upstream
[mpiexec@xen2] [pgid: 0] got PMI command: cmd=put kvsname=kvs_16060_0 key=OFA_Init_fail value=1
[1] MPI startup(): ofa fabric is not available and fallback fabric is not enabled

I don't have access to the Intel MPI source code, so sadly I cannot figure out where exactly these errors are manifesting.

Let me know if there's any other information I can provide.
Cheers,
James

0 kudos

Enlace copiado

2 Respuestas
James_S
Empleados
‎06-05-2017 11:42 PM
1.878 Vistas

Dear James, could you please provide the outputs of "env | grep I_MPI" and the debug results when exporting I_MPI_DEBUG=6 under tcp/dapl/ofa fabrics. Thanks.

0 kudos
Copiar enlace
jimdempseyatthecove
Colaborador Distinguido III
‎06-16-2017 08:12 AM
1.878 Vistas

Although I do not have a fabric other than tcp, I find that if I omit setting I_MPI_FABRICS, that mpiexec appears to perform an ordered search using fastest fabrics to slowest fabric (tcp being last). Try removing the environment variable I_MPI_FABRICS and see what happens.

Jim Dempsey

0 kudos
Copiar enlace
Responder

La asistencia de la comunidad se proporciona de lunes a viernes. Existen otros métodos de contacto disponibles aquí.

Intel no verifica todas las soluciones, como las transferencias de archivos, entre otras, que puedan aparecer en esta comunidad. Por consiguiente, Intel renuncia a todas las garantías expresas e implícitas, incluidas, entre otras, las garantías implícitas de comerciabilidad, idoneidad para un propósito particular y cumplimiento, así como cualquier garantía derivada del transcurso de la ejecución, de la negociación o del uso en el comercio.

Para obtener información más completa sobre las optimizaciones del compilador, consulte nuestro Aviso de optimización.