Performance issues with Omni Path

silvio_stanzani · ‎02-02-2018

Hi all,

I installed two Omni Path Fabric cards on two Xeon Servers.

Following the instructions present in this web site: https://software.intel.com/en-us/articles/using-intel-omni-path-architecture ;

The performance tests in this link shows that the network achieved 100 Gb/s - (4194304 10 360.39 360.39 360.39 23276.25)

I the network i deployed i achieved half of this performance ( 4194304 10 661.40 661.40 661.40 12683.17):

Is there some configuration needed to achieve 100 Gb/s using Omni Path?

Here is the complete output of benchmark execute:

mpirun -PSM2 -host 10.0.0.3 -n 1 /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv : -host 10.0.0.1 -n 1 /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv

[silvio@phi03 ~]$ mpirun -PSM2 -host 10.0.0.3 -n 1 /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv : -host 10.0.0.1 -n 1 /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv
#------------------------------------------------------------
# Intel (R) MPI Benchmarks 2018 Update 1, MPI-1 part
#------------------------------------------------------------
# Date : Fri Feb 2 11:14:01 2018
# Machine : x86_64
# System : Linux
# Release : 3.10.0-693.17.1.el7.x86_64
# Version : #1 SMP Thu Jan 25 20:13:58 UTC 2018
# MPI Version : 3.1
# MPI Thread Environment:

# Calling sequence was:

# /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv

# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#

# List of Benchmarks to run:

# Sendrecv

#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# #processes = 2
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 1.92 1.92 1.92 0.00
1 1000 1.85 1.85 1.85 1.08
2 1000 1.84 1.84 1.84 2.17
4 1000 1.84 1.84 1.84 4.35
8 1000 1.76 1.76 1.76 9.10
16 1000 2.07 2.07 2.07 15.44
32 1000 2.06 2.07 2.07 30.98
64 1000 2.02 2.02 2.02 63.46
128 1000 2.08 2.08 2.08 123.26
256 1000 2.11 2.11 2.11 242.41
512 1000 2.25 2.25 2.25 454.30
1024 1000 3.56 3.56 3.56 575.46
2048 1000 4.19 4.19 4.19 976.91
4096 1000 5.16 5.16 5.16 1586.69
8192 1000 7.15 7.15 7.15 2290.80
16384 1000 14.32 14.32 14.32 2288.44
32768 1000 20.77 20.77 20.77 3154.69
65536 640 26.08 26.09 26.09 5024.04
131072 320 34.77 34.77 34.77 7538.32
262144 160 53.03 53.03 53.03 9886.58
524288 80 93.55 93.55 93.55 11208.78
1048576 40 172.25 172.28 172.26 12173.26
2097152 20 355.15 355.21 355.18 11808.02
4194304 10 661.40 661.40 661.40 12683.17

# All processes entering MPI_Finalize

Thanks in advance!

Silvio

Mikhail_S_Intel · ‎02-13-2018

Hello Silvio,

From your results it looks like that you use Xeon Phi nodes but not Xeon.
In general to improve Omni-Path bandwidth numbers on Xeon Phi you need to use more that 1 core or use BKMs described in OPA tuning guide (https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Intel_OP_Performance_Tuning_UG_H93143_v10_0.pdf, for example at section "9.1 Mapping from MPI Processes to SDMA Engines")

Sendrecv benchmark does isend and irecv on each iteration so it is bidirectional benchmark and bidirectional bandwidth limit for Omni-Path is 25 Gbytes/sec. To be closer to this number I would suggest to use new thread-split model which is available with IMPI 2019 (https://software.intel.com/en-us/articles/intel-mpi-library-2019-technical-preview). To get more information about thread-split mode read section "4. Multiple Endpoints Support " from Developer Reference (should be placed at <install_path>/compilers_and_libraries_2018.1.163/linux/mpi_2019/doc/Developer_Reference.pdf).

Here is example of usage on IMB-MT (it is suite of multi-threaded benchmarks which can employ thread-split model):

source <install_path>/compilers_and_libraries_2018.1.163/linux/mpi_2019/intel64/bin/mpivars.sh release_mt

I_MPI_THREAD_RUNTIME=openmp OMP_NUM_THREADS=4 I_MPI_THREAD_SPLIT=1 mpiexec.hydra -n 2 -ppn 1 -hosts host1,host2 IMB-MT sendrecvmt -count 1000000 -thread_level multiple

#-----------------------------------------------------------------------------

# Benchmarking SendRecvMT

# #processes = 2 (threads: 4)

#-----------------------------------------------------------------------------

#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec

16000000 1000 1231.05 1344.16 1298.92 24635.81

silvio_stanzani · ‎02-15-2018

dear Mikhail Shiryaev,

Thanks a lot for answering this!

Yes. I am performing tests using two xeon phi nodes.

The execution of IMB-MT finished with errors on my cluster.

[silvio@phi05 mpi-benchmarks]$ source /opt/intel/parallel_studio_xe_2018/compilers_and_libraries_2018/linux/mpi_2019/intel64/bin/mpivars.sh release_mt

[silvio@phi05 mpi-benchmarks]$ I_MPI_THREAD_RUNTIME=openmp OMP_NUM_THREADS=4 I_MPI_THREAD_SPLIT=1 mpiexec.hydra -n 2 -ppn 1 -hosts 10.0.0.5,10.0.0.6 IMB-MT sendrecvmt -count 1000000 -thread_level multiple

IMB-MT: /usr/lib64/libfabric.so.1: version `FABRIC_1.1' not found (required by /opt/intel/compilers_and_libraries_2018.1.163/linux/mpi_2019/intel64/lib/release_mt/libmpi.so.12)
[mpiexec@phi05] HYDU_sock_write (../../utils/sock/sock.c:418): write error (Bad file descriptor)
[mpiexec@phi05] HYD_pmcd_pmiserv_send_signal (../../pm/pmiserv/pmiserv_cb.c:253): unable to write data to proxy
IMB-MT: /usr/lib64/libfabric.so.1: version `FABRIC_1.1' not found (required by /opt/intel/compilers_and_libraries_2018.1.163/linux/mpi_2019/intel64/lib/release_mt/libmpi.so.12)

The operation system i am using is centos 7 which provides libfabric version 4.

Do i need to perform downgrade of libfabric.so?

I tried to install the version 1.1 but it misses infiniband/driver.h that i cound not find in any package

Mikhail_S_Intel · ‎02-15-2018

Hi Silvio,

You can check your libfabric version using "fi_info --version".
If you have old libfabric then you can download and build it manually (on node with OPA software stack, for example on worker node):

git clone https://github.com/ofiwg/libfabric.git
cd ./libfabric
./autogen.sh
./configure --prefix=<libfabric_install_path> --enable-psm2
make clean && make all && make install
source <...>/mpivars.sh release_mt
export LD_LIBRARY_PATH=<libfabric_install_path>/lib/:${LD_LIBRARY_PATH}
mpiexec.hydra ...

4f0drlp7eyj3 · ‎07-01-2019

Hi Mikhail,

Adding to this thread rather than starting over because it seems most germane to my issue. I am trying to enable Intel MPI 2019 on a system without a libfabric, and without any Omni-Path packages (since it's a Mellanox-based network). So, I built dependencies (because I get unresolved symbols from psm and psm2 at runtime despite disabling them during the libfabric build) from fresh clones of intel/psm and intel/opa-psm2, using the Centos 7.4 GCC (v4.8.5). I then built libfabric 1.8.0 with

./configure --prefix=/nopt/nrel/apps/centos/7.4 --enable-psm=no --enable-psm2=no --enable-sockets=yes --enable-verbs=yes --enable-mlx=/opt/mellanox/mxm --enable-udp=yes --enable-tcp=yes --enable-rxm=no --enable-mrail=no --enable-rxd=no --enable-bgq=no --enable-shm=yes --enable-rstream=no --enable-perf=no

Build goes fine, but then when I try a little MPI test, I get

srun --nodes=2 --ntasks=4 --time=5:00 --account=hpcapps ./test

/home/cchang/tests/IMPI/./test: /nopt/nrel/apps/centos/7.4/lib64/libfabric.so.1: version `FABRIC_1.1' not found (required by /nopt/nrel/apps/compilers/2018-11-19/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/intel-parallel-studio-cluster.2019.1-nn7isnm3kmcdwixuelyzaedqcyisum4j/impi/2019.1.144/intel64/lib/release/libmpi.so.12)

/home/cchang/tests/IMPI/./test: /nopt/nrel/apps/centos/7.4/lib64/libfabric.so.1: version `FABRIC_1.1' not found (required by /nopt/nrel/apps/compilers/2018-11-19/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/intel-parallel-studio-cluster.2019.1-nn7isnm3kmcdwixuelyzaedqcyisum4j/impi/2019.1.144/intel64/lib/release/libmpi.so.12)

/home/cchang/tests/IMPI/./test: /nopt/nrel/apps/centos/7.4/lib64/libfabric.so.1: version `FABRIC_1.1' not found (required by /nopt/nrel/apps/compilers/2018-11-19/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/intel-parallel-studio-cluster.2019.1-nn7isnm3kmcdwixuelyzaedqcyisum4j/impi/2019.1.144/intel64/lib/release/libmpi.so.12)

/home/cchang/tests/IMPI/./test: /nopt/nrel/apps/centos/7.4/lib64/libfabric.so.1: version `FABRIC_1.1' not found (required by /nopt/nrel/apps/compilers/2018-11-19/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/intel-parallel-studio-cluster.2019.1-nn7isnm3kmcdwixuelyzaedqcyisum4j/impi/2019.1.144/intel64/lib/release/libmpi.so.12)

Sure enough, I only see 1.0 in the libfabric binary:

[cchang@el2 IMPI]$ readelf -a /nopt/nrel/apps/centos/7.4/lib64/libfabric.so.1.2.3 | grep FABRIC
   272: 0000000000011da0   330 FUNC    GLOBAL DEFAULT   12 fi_log@@FABRIC_1.0
   273: 0000000000011d60    62 FUNC    GLOBAL DEFAULT   12 fi_log_enabled@@FABRIC_1.0
   274: 0000000000012350   763 FUNC    GLOBAL DEFAULT   12 fi_param_get@@FABRIC_1.0
   275: 000000000000f070   166 FUNC    GLOBAL DEFAULT   12 fi_freeinfo@@FABRIC_1.0
   276: 000000000000f120  1031 FUNC    GLOBAL DEFAULT   12 fi_getinfo@@FABRIC_1.0
   277: 0000000000011f20   299 FUNC    GLOBAL DEFAULT   12 fi_getparams@@FABRIC_1.0
   278: 0000000000012110   563 FUNC    GLOBAL DEFAULT   12 fi_param_define@@FABRIC_1.0
   279: 000000000000fa20   127 FUNC    GLOBAL DEFAULT   12 fi_fabric@@FABRIC_1.0
   280: 00000000000111c0  2518 FUNC    GLOBAL DEFAULT   12 fi_tostr@@FABRIC_1.0
   281: 000000000000fab0    53 FUNC    GLOBAL DEFAULT   12 fi_strerror@@FABRIC_1.0
   282: 0000000000012050    83 FUNC    GLOBAL DEFAULT   12 fi_freeparams@@FABRIC_1.0
   283: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  ABS FABRIC_1.0
   284: 000000000000f530  1254 FUNC    GLOBAL DEFAULT   12 fi_dupinfo@@FABRIC_1.0
   285: 000000000000faa0     6 FUNC    GLOBAL DEFAULT   12 fi_version@@FABRIC_1.0
  110:   2 (FABRIC_1.0)    2 (FABRIC_1.0)    2 (FABRIC_1.0)    2 (FABRIC_1.0) 
  114:   2 (FABRIC_1.0)    2 (FABRIC_1.0)    2 (FABRIC_1.0)    2 (FABRIC_1.0) 
  118:   2 (FABRIC_1.0)    2 (FABRIC_1.0)    2 (FABRIC_1.0)    2 (FABRIC_1.0) 
  11c:   2 (FABRIC_1.0)    2 (FABRIC_1.0) 
  0x001c: Rev: 1  Flags: none  Index: 2  Cnt: 1  Name: FABRIC_1.0

This is the latest libfabric release, so how would I go about getting symbols compatible with Intel MPI 2019?

Thanks; Chris

4f0drlp7eyj3 · ‎07-01-2019

OK, never mind, I was looking at an older file installed from a Centos RPM.