I am a newbie on HPC world, and I know a little about MPI and Infiniband.
I want to execute the IMB 2017 and HPCC 1.5.0 benchmarks on our HPC to be sure that all is configured correctly.
we have 32 compute nodes, each one with 16 cores and 32GB of memory. each node have a qlogic infiniband card with one port at 40Gb/s.
the OS used is RHEL 6.5 with IBM Platform HPC 4.2.
Ofed used is the : IntelIB-OFED.RHEL6-x86_64.3.5.2.34
GCC : gcc version 4.4.7
I managed to compile IMB and HPCC with both IBM platform MPI (PMPI) and OpenMPI 2.0.1 (OMPI)
1) IMB When executing IMB benchmark with both PMPI and OMPI on infiniband links I get at most
-----------------------------------------------------
Benchmarking PingPong
# processes = 2
---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 1.51 0.00
1 1000 1.51 0.63
... ... ... ...
2097152 20 675.20 2962.09
4194304 10 1320.45 3029.26
3029 MB/s of throughput, I expected more, something near 4000 Mb/s is this result correct?
2) HPCC I used this site http://www.advancedclustering.com/act-kb/tune-hpl-dat-file/ to generate the test profile. When executing the benchmark, using less than 25 nodes, the test goes without problem, I didn't wait for the test to complete, but my problem is when I start the benchmark on all nodes, after 2 to 5 seconds I get this error message : compute014.6359Exhausted 1048576 MQ irecv request descriptors, which usually indicates a user program error or insufficient request descriptors (PSM_MQ_RECVREQS_MAX=1048576)
and the benchmark exits and is killed, it's not the same node, each time it's another node. any idea?
This are the commands used to start the hpcc benchmark : OMPI : mpirun -np 512 --display-allocation --mca btl self,sm --mca mtl psm --hostfile hosts32 /shared/build/hpcc-150-blas-ompi-201/hpcc hpccinf.txt
PMPI : mpirun -np 512 -PSM -hostfile hosts32 /shared/build/hpcc-150-blas-pmpi/hpcc hpccinf.txt
If you need more information, let me know.
Regards
if you need more information, let me know.
Regards.
