topic Performance issues with Omni Path in Intel® MPI Library https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132679#M5683 <P>Hi all,</P> <P>I installed two Omni Path Fabric cards on two Xeon Servers.&nbsp;</P> <P>Following the instructions present in this web site: <A href="https://software.intel.com/en-us/articles/using-intel-omni-path-architecture&nbsp;" target="_blank">https://software.intel.com/en-us/articles/using-intel-omni-path-architecture&nbsp;</A>;</P> <P>The performance tests in this link shows that the network achieved 100 Gb/s - (4194304 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 10 &nbsp; &nbsp; &nbsp; 360.39 &nbsp; &nbsp; &nbsp; 360.39 &nbsp; &nbsp; &nbsp; 360.39 &nbsp; &nbsp; 23276.25)</P> <P>I the network i deployed i achieved half of this performance (<SPAN style="font-size: 13.008px;">&nbsp; &nbsp; &nbsp;4194304 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 10 &nbsp; &nbsp; &nbsp; 661.40 &nbsp; &nbsp; &nbsp; 661.40 &nbsp; &nbsp; &nbsp; 661.40 &nbsp; &nbsp; 12683.17</SPAN><SPAN style="font-size: 1em;">): &nbsp;</SPAN></P> <P><SPAN style="font-size: 1em;">Is there some configuration needed to achieve 100 Gb/s using Omni Path?</SPAN></P> <P>&nbsp;</P> <P><SPAN style="font-size: 1em;">Here is the complete output of benchmark execute:</SPAN></P> <P>mpirun -PSM2 -host 10.0.0.3 -n 1 /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv : -host 10.0.0.1 -n 1 /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv</P> <P>[silvio@phi03 ~]$ mpirun -PSM2 -host 10.0.0.3 -n 1 /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv : -host 10.0.0.1 -n 1 /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv<BR /> #------------------------------------------------------------<BR /> # &nbsp; &nbsp;Intel (R) MPI Benchmarks 2018 Update 1, MPI-1 part &nbsp; &nbsp;<BR /> #------------------------------------------------------------<BR /> # Date &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;: Fri Feb &nbsp;2 11:14:01 2018<BR /> # Machine &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : x86_64<BR /> # System &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;: Linux<BR /> # Release &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : 3.10.0-693.17.1.el7.x86_64<BR /> # Version &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : #1 SMP Thu Jan 25 20:13:58 UTC 2018<BR /> # MPI Version &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : 3.1<BR /> # MPI Thread Environment:&nbsp;</P> <P><BR /> # Calling sequence was:&nbsp;</P> <P># /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv</P> <P># Minimum message length in bytes: &nbsp; 0<BR /> # Maximum message length in bytes: &nbsp; 4194304<BR /> #<BR /> # MPI_Datatype &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : &nbsp; MPI_BYTE&nbsp;<BR /> # MPI_Datatype for reductions &nbsp; &nbsp;: &nbsp; MPI_FLOAT<BR /> # MPI_Op &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : &nbsp; MPI_SUM &nbsp;<BR /> #<BR /> #</P> <P># List of Benchmarks to run:</P> <P># Sendrecv</P> <P>#-----------------------------------------------------------------------------<BR /> # Benchmarking Sendrecv&nbsp;<BR /> # #processes = 2&nbsp;<BR /> #-----------------------------------------------------------------------------<BR /> &nbsp; &nbsp; &nbsp; &nbsp;#bytes #repetitions &nbsp;t_min[usec] &nbsp;t_max[usec] &nbsp;t_avg[usec] &nbsp; Mbytes/sec<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 1.92 &nbsp; &nbsp; &nbsp; &nbsp; 1.92 &nbsp; &nbsp; &nbsp; &nbsp; 1.92 &nbsp; &nbsp; &nbsp; &nbsp; 0.00<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 1.85 &nbsp; &nbsp; &nbsp; &nbsp; 1.85 &nbsp; &nbsp; &nbsp; &nbsp; 1.85 &nbsp; &nbsp; &nbsp; &nbsp; 1.08<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 1.84 &nbsp; &nbsp; &nbsp; &nbsp; 1.84 &nbsp; &nbsp; &nbsp; &nbsp; 1.84 &nbsp; &nbsp; &nbsp; &nbsp; 2.17<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 1.84 &nbsp; &nbsp; &nbsp; &nbsp; 1.84 &nbsp; &nbsp; &nbsp; &nbsp; 1.84 &nbsp; &nbsp; &nbsp; &nbsp; 4.35<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 1.76 &nbsp; &nbsp; &nbsp; &nbsp; 1.76 &nbsp; &nbsp; &nbsp; &nbsp; 1.76 &nbsp; &nbsp; &nbsp; &nbsp; 9.10<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 2.07 &nbsp; &nbsp; &nbsp; &nbsp; 2.07 &nbsp; &nbsp; &nbsp; &nbsp; 2.07 &nbsp; &nbsp; &nbsp; &nbsp;15.44<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 2.06 &nbsp; &nbsp; &nbsp; &nbsp; 2.07 &nbsp; &nbsp; &nbsp; &nbsp; 2.07 &nbsp; &nbsp; &nbsp; &nbsp;30.98<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;64 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 2.02 &nbsp; &nbsp; &nbsp; &nbsp; 2.02 &nbsp; &nbsp; &nbsp; &nbsp; 2.02 &nbsp; &nbsp; &nbsp; &nbsp;63.46<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 128 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 2.08 &nbsp; &nbsp; &nbsp; &nbsp; 2.08 &nbsp; &nbsp; &nbsp; &nbsp; 2.08 &nbsp; &nbsp; &nbsp; 123.26<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 256 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 2.11 &nbsp; &nbsp; &nbsp; &nbsp; 2.11 &nbsp; &nbsp; &nbsp; &nbsp; 2.11 &nbsp; &nbsp; &nbsp; 242.41<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 512 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 2.25 &nbsp; &nbsp; &nbsp; &nbsp; 2.25 &nbsp; &nbsp; &nbsp; &nbsp; 2.25 &nbsp; &nbsp; &nbsp; 454.30<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1024 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 3.56 &nbsp; &nbsp; &nbsp; &nbsp; 3.56 &nbsp; &nbsp; &nbsp; &nbsp; 3.56 &nbsp; &nbsp; &nbsp; 575.46<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;2048 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 4.19 &nbsp; &nbsp; &nbsp; &nbsp; 4.19 &nbsp; &nbsp; &nbsp; &nbsp; 4.19 &nbsp; &nbsp; &nbsp; 976.91<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4096 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 5.16 &nbsp; &nbsp; &nbsp; &nbsp; 5.16 &nbsp; &nbsp; &nbsp; &nbsp; 5.16 &nbsp; &nbsp; &nbsp;1586.69<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;8192 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 7.15 &nbsp; &nbsp; &nbsp; &nbsp; 7.15 &nbsp; &nbsp; &nbsp; &nbsp; 7.15 &nbsp; &nbsp; &nbsp;2290.80<BR /> &nbsp; &nbsp; &nbsp; &nbsp; 16384 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp;14.32 &nbsp; &nbsp; &nbsp; &nbsp;14.32 &nbsp; &nbsp; &nbsp; &nbsp;14.32 &nbsp; &nbsp; &nbsp;2288.44<BR /> &nbsp; &nbsp; &nbsp; &nbsp; 32768 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp;20.77 &nbsp; &nbsp; &nbsp; &nbsp;20.77 &nbsp; &nbsp; &nbsp; &nbsp;20.77 &nbsp; &nbsp; &nbsp;3154.69<BR /> &nbsp; &nbsp; &nbsp; &nbsp; 65536 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;640 &nbsp; &nbsp; &nbsp; &nbsp;26.08 &nbsp; &nbsp; &nbsp; &nbsp;26.09 &nbsp; &nbsp; &nbsp; &nbsp;26.09 &nbsp; &nbsp; &nbsp;5024.04<BR /> &nbsp; &nbsp; &nbsp; &nbsp;131072 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;320 &nbsp; &nbsp; &nbsp; &nbsp;34.77 &nbsp; &nbsp; &nbsp; &nbsp;34.77 &nbsp; &nbsp; &nbsp; &nbsp;34.77 &nbsp; &nbsp; &nbsp;7538.32<BR /> &nbsp; &nbsp; &nbsp; &nbsp;262144 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;160 &nbsp; &nbsp; &nbsp; &nbsp;53.03 &nbsp; &nbsp; &nbsp; &nbsp;53.03 &nbsp; &nbsp; &nbsp; &nbsp;53.03 &nbsp; &nbsp; &nbsp;9886.58<BR /> &nbsp; &nbsp; &nbsp; &nbsp;524288 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 80 &nbsp; &nbsp; &nbsp; &nbsp;93.55 &nbsp; &nbsp; &nbsp; &nbsp;93.55 &nbsp; &nbsp; &nbsp; &nbsp;93.55 &nbsp; &nbsp; 11208.78<BR /> &nbsp; &nbsp; &nbsp; 1048576 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 40 &nbsp; &nbsp; &nbsp; 172.25 &nbsp; &nbsp; &nbsp; 172.28 &nbsp; &nbsp; &nbsp; 172.26 &nbsp; &nbsp; 12173.26<BR /> &nbsp; &nbsp; &nbsp; 2097152 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 20 &nbsp; &nbsp; &nbsp; 355.15 &nbsp; &nbsp; &nbsp; 355.21 &nbsp; &nbsp; &nbsp; 355.18 &nbsp; &nbsp; 11808.02<BR /> &nbsp; &nbsp; &nbsp; 4194304 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 10 &nbsp; &nbsp; &nbsp; 661.40 &nbsp; &nbsp; &nbsp; 661.40 &nbsp; &nbsp; &nbsp; 661.40 &nbsp; &nbsp; 12683.17</P> <P><BR /> # All processes entering MPI_Finalize</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Thanks in advance!</P> <P>Silvio</P> Fri, 02 Feb 2018 13:19:57 GMT silvio_stanzani 2018-02-02T13:19:57Z Performance issues with Omni Path https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132679#M5683 <P>Hi all,</P> <P>I installed two Omni Path Fabric cards on two Xeon Servers.&nbsp;</P> <P>Following the instructions present in this web site: <A href="https://software.intel.com/en-us/articles/using-intel-omni-path-architecture&nbsp;" target="_blank">https://software.intel.com/en-us/articles/using-intel-omni-path-architecture&nbsp;</A>;</P> <P>The performance tests in this link shows that the network achieved 100 Gb/s - (4194304 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 10 &nbsp; &nbsp; &nbsp; 360.39 &nbsp; &nbsp; &nbsp; 360.39 &nbsp; &nbsp; &nbsp; 360.39 &nbsp; &nbsp; 23276.25)</P> <P>I the network i deployed i achieved half of this performance (<SPAN style="font-size: 13.008px;">&nbsp; &nbsp; &nbsp;4194304 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 10 &nbsp; &nbsp; &nbsp; 661.40 &nbsp; &nbsp; &nbsp; 661.40 &nbsp; &nbsp; &nbsp; 661.40 &nbsp; &nbsp; 12683.17</SPAN><SPAN style="font-size: 1em;">): &nbsp;</SPAN></P> <P><SPAN style="font-size: 1em;">Is there some configuration needed to achieve 100 Gb/s using Omni Path?</SPAN></P> <P>&nbsp;</P> <P><SPAN style="font-size: 1em;">Here is the complete output of benchmark execute:</SPAN></P> <P>mpirun -PSM2 -host 10.0.0.3 -n 1 /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv : -host 10.0.0.1 -n 1 /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv</P> <P>[silvio@phi03 ~]$ mpirun -PSM2 -host 10.0.0.3 -n 1 /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv : -host 10.0.0.1 -n 1 /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv<BR /> #------------------------------------------------------------<BR /> # &nbsp; &nbsp;Intel (R) MPI Benchmarks 2018 Update 1, MPI-1 part &nbsp; &nbsp;<BR /> #------------------------------------------------------------<BR /> # Date &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;: Fri Feb &nbsp;2 11:14:01 2018<BR /> # Machine &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : x86_64<BR /> # System &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;: Linux<BR /> # Release &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : 3.10.0-693.17.1.el7.x86_64<BR /> # Version &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : #1 SMP Thu Jan 25 20:13:58 UTC 2018<BR /> # MPI Version &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : 3.1<BR /> # MPI Thread Environment:&nbsp;</P> <P><BR /> # Calling sequence was:&nbsp;</P> <P># /opt/intel/impi/2018.1.163/bin64/IMB-MPI1 Sendrecv</P> <P># Minimum message length in bytes: &nbsp; 0<BR /> # Maximum message length in bytes: &nbsp; 4194304<BR /> #<BR /> # MPI_Datatype &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : &nbsp; MPI_BYTE&nbsp;<BR /> # MPI_Datatype for reductions &nbsp; &nbsp;: &nbsp; MPI_FLOAT<BR /> # MPI_Op &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : &nbsp; MPI_SUM &nbsp;<BR /> #<BR /> #</P> <P># List of Benchmarks to run:</P> <P># Sendrecv</P> <P>#-----------------------------------------------------------------------------<BR /> # Benchmarking Sendrecv&nbsp;<BR /> # #processes = 2&nbsp;<BR /> #-----------------------------------------------------------------------------<BR /> &nbsp; &nbsp; &nbsp; &nbsp;#bytes #repetitions &nbsp;t_min[usec] &nbsp;t_max[usec] &nbsp;t_avg[usec] &nbsp; Mbytes/sec<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 1.92 &nbsp; &nbsp; &nbsp; &nbsp; 1.92 &nbsp; &nbsp; &nbsp; &nbsp; 1.92 &nbsp; &nbsp; &nbsp; &nbsp; 0.00<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 1.85 &nbsp; &nbsp; &nbsp; &nbsp; 1.85 &nbsp; &nbsp; &nbsp; &nbsp; 1.85 &nbsp; &nbsp; &nbsp; &nbsp; 1.08<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 1.84 &nbsp; &nbsp; &nbsp; &nbsp; 1.84 &nbsp; &nbsp; &nbsp; &nbsp; 1.84 &nbsp; &nbsp; &nbsp; &nbsp; 2.17<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 1.84 &nbsp; &nbsp; &nbsp; &nbsp; 1.84 &nbsp; &nbsp; &nbsp; &nbsp; 1.84 &nbsp; &nbsp; &nbsp; &nbsp; 4.35<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 1.76 &nbsp; &nbsp; &nbsp; &nbsp; 1.76 &nbsp; &nbsp; &nbsp; &nbsp; 1.76 &nbsp; &nbsp; &nbsp; &nbsp; 9.10<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 2.07 &nbsp; &nbsp; &nbsp; &nbsp; 2.07 &nbsp; &nbsp; &nbsp; &nbsp; 2.07 &nbsp; &nbsp; &nbsp; &nbsp;15.44<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 2.06 &nbsp; &nbsp; &nbsp; &nbsp; 2.07 &nbsp; &nbsp; &nbsp; &nbsp; 2.07 &nbsp; &nbsp; &nbsp; &nbsp;30.98<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;64 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 2.02 &nbsp; &nbsp; &nbsp; &nbsp; 2.02 &nbsp; &nbsp; &nbsp; &nbsp; 2.02 &nbsp; &nbsp; &nbsp; &nbsp;63.46<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 128 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 2.08 &nbsp; &nbsp; &nbsp; &nbsp; 2.08 &nbsp; &nbsp; &nbsp; &nbsp; 2.08 &nbsp; &nbsp; &nbsp; 123.26<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 256 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 2.11 &nbsp; &nbsp; &nbsp; &nbsp; 2.11 &nbsp; &nbsp; &nbsp; &nbsp; 2.11 &nbsp; &nbsp; &nbsp; 242.41<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 512 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 2.25 &nbsp; &nbsp; &nbsp; &nbsp; 2.25 &nbsp; &nbsp; &nbsp; &nbsp; 2.25 &nbsp; &nbsp; &nbsp; 454.30<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1024 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 3.56 &nbsp; &nbsp; &nbsp; &nbsp; 3.56 &nbsp; &nbsp; &nbsp; &nbsp; 3.56 &nbsp; &nbsp; &nbsp; 575.46<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;2048 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 4.19 &nbsp; &nbsp; &nbsp; &nbsp; 4.19 &nbsp; &nbsp; &nbsp; &nbsp; 4.19 &nbsp; &nbsp; &nbsp; 976.91<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4096 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 5.16 &nbsp; &nbsp; &nbsp; &nbsp; 5.16 &nbsp; &nbsp; &nbsp; &nbsp; 5.16 &nbsp; &nbsp; &nbsp;1586.69<BR /> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;8192 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp; 7.15 &nbsp; &nbsp; &nbsp; &nbsp; 7.15 &nbsp; &nbsp; &nbsp; &nbsp; 7.15 &nbsp; &nbsp; &nbsp;2290.80<BR /> &nbsp; &nbsp; &nbsp; &nbsp; 16384 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp;14.32 &nbsp; &nbsp; &nbsp; &nbsp;14.32 &nbsp; &nbsp; &nbsp; &nbsp;14.32 &nbsp; &nbsp; &nbsp;2288.44<BR /> &nbsp; &nbsp; &nbsp; &nbsp; 32768 &nbsp; &nbsp; &nbsp; &nbsp; 1000 &nbsp; &nbsp; &nbsp; &nbsp;20.77 &nbsp; &nbsp; &nbsp; &nbsp;20.77 &nbsp; &nbsp; &nbsp; &nbsp;20.77 &nbsp; &nbsp; &nbsp;3154.69<BR /> &nbsp; &nbsp; &nbsp; &nbsp; 65536 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;640 &nbsp; &nbsp; &nbsp; &nbsp;26.08 &nbsp; &nbsp; &nbsp; &nbsp;26.09 &nbsp; &nbsp; &nbsp; &nbsp;26.09 &nbsp; &nbsp; &nbsp;5024.04<BR /> &nbsp; &nbsp; &nbsp; &nbsp;131072 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;320 &nbsp; &nbsp; &nbsp; &nbsp;34.77 &nbsp; &nbsp; &nbsp; &nbsp;34.77 &nbsp; &nbsp; &nbsp; &nbsp;34.77 &nbsp; &nbsp; &nbsp;7538.32<BR /> &nbsp; &nbsp; &nbsp; &nbsp;262144 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;160 &nbsp; &nbsp; &nbsp; &nbsp;53.03 &nbsp; &nbsp; &nbsp; &nbsp;53.03 &nbsp; &nbsp; &nbsp; &nbsp;53.03 &nbsp; &nbsp; &nbsp;9886.58<BR /> &nbsp; &nbsp; &nbsp; &nbsp;524288 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 80 &nbsp; &nbsp; &nbsp; &nbsp;93.55 &nbsp; &nbsp; &nbsp; &nbsp;93.55 &nbsp; &nbsp; &nbsp; &nbsp;93.55 &nbsp; &nbsp; 11208.78<BR /> &nbsp; &nbsp; &nbsp; 1048576 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 40 &nbsp; &nbsp; &nbsp; 172.25 &nbsp; &nbsp; &nbsp; 172.28 &nbsp; &nbsp; &nbsp; 172.26 &nbsp; &nbsp; 12173.26<BR /> &nbsp; &nbsp; &nbsp; 2097152 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 20 &nbsp; &nbsp; &nbsp; 355.15 &nbsp; &nbsp; &nbsp; 355.21 &nbsp; &nbsp; &nbsp; 355.18 &nbsp; &nbsp; 11808.02<BR /> &nbsp; &nbsp; &nbsp; 4194304 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 10 &nbsp; &nbsp; &nbsp; 661.40 &nbsp; &nbsp; &nbsp; 661.40 &nbsp; &nbsp; &nbsp; 661.40 &nbsp; &nbsp; 12683.17</P> <P><BR /> # All processes entering MPI_Finalize</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Thanks in advance!</P> <P>Silvio</P> Fri, 02 Feb 2018 13:19:57 GMT https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132679#M5683 silvio_stanzani 2018-02-02T13:19:57Z Hello Silvio, https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132680#M5684 <P>Hello Silvio,</P> <P>From your results it looks like that you use Xeon Phi nodes but not Xeon.<BR /> In general to improve Omni-Path bandwidth numbers on&nbsp;Xeon Phi you need to use more that 1 core or use BKMs described in OPA tuning guide (<SPAN style="font-size: 13.008px;"><A href="https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Intel_OP_Performance_Tuning_UG_H93143_v10_0.pdf" target="_blank">https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Intel_OP_Performance_Tuning_UG_H93143_v10_0.pdf</A>, for example at section "9.1&nbsp;</SPAN>Mapping from MPI Processes to SDMA Engines")</P> <P>Sendrecv benchmark does isend and irecv on each iteration so it is bidirectional benchmark and bidirectional bandwidth limit for Omni-Path is 25 Gbytes/sec. To be closer to this number I would suggest to use new thread-split model which is available with IMPI 2019 (<SPAN style="font-size: 13.008px;"><A href="https://software.intel.com/en-us/articles/intel-mpi-library-2019-technical-preview" target="_blank">https://software.intel.com/en-us/articles/intel-mpi-library-2019-technical-preview</A></SPAN>). To get more information about thread-split mode read section "<SPAN class="fontstyle0">4. Multiple Endpoints Support&nbsp;</SPAN>" from Developer Reference (should be placed at &lt;install_path&gt;/<SPAN style="font-size: 13.008px;">compilers_and_libraries_2018.1.163/linux/mpi_2019/doc/Developer_Reference.pdf</SPAN>).</P> <P>Here is example of usage on IMB-MT (it is suite of multi-threaded benchmarks which can employ thread-split model):</P> <P><SPAN style="font-size: 13.008px;">source </SPAN>&lt;install_path&gt;/compilers_and_libraries_2018.1.163/linux/mpi_2019<SPAN style="font-size: 13.008px;">/intel64/bin/mpivars.sh release_mt</SPAN></P> <P><SPAN style="font-size: 13.008px;">I_MPI_THREAD_RUNTIME=openmp OMP_NUM_THREADS=4 I_MPI_THREAD_SPLIT=1 mpiexec.hydra -n 2 -ppn 1 -hosts host1,host2 IMB-MT sendrecvmt -count 1000000 -thread_level multiple</SPAN></P> <DIV><SPAN style="font-size: 13.008px;">#-----------------------------------------------------------------------------</SPAN></DIV> <DIV><SPAN style="font-size: 13.008px;"># Benchmarking SendRecvMT</SPAN></DIV> <DIV><SPAN style="font-size: 13.008px;"># #processes = 2 (threads: 4)</SPAN></DIV> <DIV><SPAN style="font-size: 13.008px;">#-----------------------------------------------------------------------------</SPAN></DIV> <DIV><SPAN style="font-size: 13.008px;">&nbsp; &nbsp; &nbsp; &nbsp;#bytes #repetitions&nbsp; t_min[usec]&nbsp; t_max[usec]&nbsp; t_avg[usec]&nbsp; &nbsp;Mbytes/sec</SPAN></DIV> <DIV><SPAN style="font-size: 13.008px;">&nbsp; &nbsp; &nbsp;16000000&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1000&nbsp; &nbsp; &nbsp; 1231.05&nbsp; &nbsp; &nbsp; 1344.16&nbsp; &nbsp; &nbsp; 1298.92&nbsp; &nbsp; &nbsp;<STRONG>24635.81</STRONG></SPAN></DIV> <DIV>&nbsp;</DIV> Tue, 13 Feb 2018 22:58:00 GMT https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132680#M5684 Mikhail_S_Intel 2018-02-13T22:58:00Z dear Mikhail Shiryaev, https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132681#M5685 <P>dear&nbsp;<A href="https://software.intel.com/en-us/user/1121654" style="font-size: 11px; background-color: rgb(238, 238, 238);">Mikhail Shiryaev</A>,</P> <P>Thanks a lot for answering this!</P> <P>Yes. I am performing tests using two xeon phi nodes.</P> <P>The execution of <SPAN style="font-size: 12px;">IMB-MT finished with errors on my cluster.</SPAN></P> <P><STRONG>[silvio@phi05 mpi-benchmarks]$ source &nbsp;/opt/intel/parallel_studio_xe_2018/compilers_and_libraries_2018/linux/mpi_2019/intel64/bin/mpivars.sh release_mt</STRONG></P> <P><STRONG>[silvio@phi05 mpi-benchmarks]$ I_MPI_THREAD_RUNTIME=openmp OMP_NUM_THREADS=4 I_MPI_THREAD_SPLIT=1 mpiexec.hydra -n 2 -ppn 1 -hosts 10.0.0.5,10.0.0.6 IMB-MT sendrecvmt -count 1000000 -thread_level multiple</STRONG></P> <P><BR /> IMB-MT: /usr/lib64/libfabric.so.1: version `FABRIC_1.1' not found (required by /opt/intel/compilers_and_libraries_2018.1.163/linux/mpi_2019/intel64/lib/release_mt/libmpi.so.12)<BR /> [mpiexec@phi05] HYDU_sock_write (../../utils/sock/sock.c:418): write error (Bad file descriptor)<BR /> [mpiexec@phi05] HYD_pmcd_pmiserv_send_signal (../../pm/pmiserv/pmiserv_cb.c:253): unable to write data to proxy<BR /> IMB-MT: /usr/lib64/libfabric.so.1: <STRONG>version `FABRIC_1.1' not found</STRONG> (required by /opt/intel/compilers_and_libraries_2018.1.163/linux/mpi_2019/intel64/lib/release_mt/libmpi.so.12)</P> <P>&nbsp;</P> <P>The operation system i am using is centos 7 which provides libfabric version 4.</P> <P><STRONG>Do i need to perform downgrade of&nbsp;<SPAN style="font-size: 13.008px;">libfabric.so?</SPAN></STRONG></P> <P><B>I tried to install the version 1.1 but it misses infiniband/driver.h that i cound not find in any package</B></P> Thu, 15 Feb 2018 17:55:05 GMT https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132681#M5685 silvio_stanzani 2018-02-15T17:55:05Z Hi Silvio, https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132682#M5686 <P>Hi Silvio,</P> <P>You can check your libfabric version using "<SPAN style="font-size: 13.008px;">fi_info --version</SPAN>".<BR /> If you have old libfabric then you can download and build it manually (on node with OPA software stack, for example on worker node):</P> <OL> <LI>git clone&nbsp;<SPAN style="font-size: 13.008px;"><A href="https://github.com/ofiwg/libfabric.git" target="_blank">https://github.com/ofiwg/libfabric.git</A></SPAN></LI> <LI><SPAN style="font-size: 13.008px;">cd ./libfabric</SPAN></LI> <LI><SPAN style="font-size: 13.008px;">./autogen.sh</SPAN></LI> <LI><SPAN style="font-size: 13.008px;">./configure --prefix=&lt;libfabric_install_path&gt; --enable-psm2</SPAN></LI> <LI><SPAN style="font-size: 13.008px;">make clean &amp;&amp; make all &amp;&amp; make install</SPAN></LI> <LI><SPAN style="font-size: 13.008px;">source &lt;...&gt;/mpivars.sh release_mt</SPAN></LI> <LI><SPAN style="font-size: 13.008px;">export LD_LIBRARY_PATH=</SPAN>&lt;libfabric_install_path&gt;<SPAN style="font-size: 13.008px;">/lib/:${LD_LIBRARY_PATH}</SPAN></LI> <LI><SPAN style="font-size: 13.008px;">mpiexec.hydra ...</SPAN></LI> </OL> Thu, 15 Feb 2018 18:28:15 GMT https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132682#M5686 Mikhail_S_Intel 2018-02-15T18:28:15Z Hi Mikhail, https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132683#M5687 <P>Hi Mikhail,</P><P>&nbsp; &nbsp;Adding to this thread rather than starting over because it seems most germane to my issue. I am trying to enable Intel MPI 2019 on a system without a libfabric, and without any Omni-Path packages (since it's a Mellanox-based network). So, I built dependencies (because I get unresolved symbols from psm and psm2 at runtime despite disabling them during the libfabric build) from fresh clones of intel/psm and intel/opa-psm2, using the Centos 7.4&nbsp;GCC (v4.8.5). I then built libfabric 1.8.0 with</P> <PRE class="brush:plain; class-name:dark;">./configure --prefix=/nopt/nrel/apps/centos/7.4 --enable-psm=no --enable-psm2=no --enable-sockets=yes --enable-verbs=yes --enable-mlx=/opt/mellanox/mxm --enable-udp=yes --enable-tcp=yes --enable-rxm=no --enable-mrail=no --enable-rxd=no --enable-bgq=no --enable-shm=yes --enable-rstream=no --enable-perf=no</PRE> <P>Build goes fine, but then when I try a little MPI test, I get</P> <PRE class="brush:plain; class-name:dark;">srun --nodes=2 --ntasks=4 --time=5:00 --account=hpcapps ./test /home/cchang/tests/IMPI/./test: /nopt/nrel/apps/centos/7.4/lib64/libfabric.so.1: version `FABRIC_1.1' not found (required by /nopt/nrel/apps/compilers/2018-11-19/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/intel-parallel-studio-cluster.2019.1-nn7isnm3kmcdwixuelyzaedqcyisum4j/impi/2019.1.144/intel64/lib/release/libmpi.so.12) /home/cchang/tests/IMPI/./test: /nopt/nrel/apps/centos/7.4/lib64/libfabric.so.1: version `FABRIC_1.1' not found (required by /nopt/nrel/apps/compilers/2018-11-19/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/intel-parallel-studio-cluster.2019.1-nn7isnm3kmcdwixuelyzaedqcyisum4j/impi/2019.1.144/intel64/lib/release/libmpi.so.12) /home/cchang/tests/IMPI/./test: /nopt/nrel/apps/centos/7.4/lib64/libfabric.so.1: version `FABRIC_1.1' not found (required by /nopt/nrel/apps/compilers/2018-11-19/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/intel-parallel-studio-cluster.2019.1-nn7isnm3kmcdwixuelyzaedqcyisum4j/impi/2019.1.144/intel64/lib/release/libmpi.so.12) /home/cchang/tests/IMPI/./test: /nopt/nrel/apps/centos/7.4/lib64/libfabric.so.1: version `FABRIC_1.1' not found (required by /nopt/nrel/apps/compilers/2018-11-19/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/intel-parallel-studio-cluster.2019.1-nn7isnm3kmcdwixuelyzaedqcyisum4j/impi/2019.1.144/intel64/lib/release/libmpi.so.12) </PRE> <P>Sure&nbsp;enough, I only see 1.0 in the libfabric binary:</P> <PRE class="brush:plain; class-name:dark;">[cchang@el2 IMPI]$ readelf -a /nopt/nrel/apps/centos/7.4/lib64/libfabric.so.1.2.3 | grep FABRIC 272: 0000000000011da0 330 FUNC GLOBAL DEFAULT 12 fi_log@@FABRIC_1.0 273: 0000000000011d60 62 FUNC GLOBAL DEFAULT 12 fi_log_enabled@@FABRIC_1.0 274: 0000000000012350 763 FUNC GLOBAL DEFAULT 12 fi_param_get@@FABRIC_1.0 275: 000000000000f070 166 FUNC GLOBAL DEFAULT 12 fi_freeinfo@@FABRIC_1.0 276: 000000000000f120 1031 FUNC GLOBAL DEFAULT 12 fi_getinfo@@FABRIC_1.0 277: 0000000000011f20 299 FUNC GLOBAL DEFAULT 12 fi_getparams@@FABRIC_1.0 278: 0000000000012110 563 FUNC GLOBAL DEFAULT 12 fi_param_define@@FABRIC_1.0 279: 000000000000fa20 127 FUNC GLOBAL DEFAULT 12 fi_fabric@@FABRIC_1.0 280: 00000000000111c0 2518 FUNC GLOBAL DEFAULT 12 fi_tostr@@FABRIC_1.0 281: 000000000000fab0 53 FUNC GLOBAL DEFAULT 12 fi_strerror@@FABRIC_1.0 282: 0000000000012050 83 FUNC GLOBAL DEFAULT 12 fi_freeparams@@FABRIC_1.0 283: 0000000000000000 0 OBJECT GLOBAL DEFAULT ABS FABRIC_1.0 284: 000000000000f530 1254 FUNC GLOBAL DEFAULT 12 fi_dupinfo@@FABRIC_1.0 285: 000000000000faa0 6 FUNC GLOBAL DEFAULT 12 fi_version@@FABRIC_1.0 110: 2 (FABRIC_1.0) 2 (FABRIC_1.0) 2 (FABRIC_1.0) 2 (FABRIC_1.0) 114: 2 (FABRIC_1.0) 2 (FABRIC_1.0) 2 (FABRIC_1.0) 2 (FABRIC_1.0) 118: 2 (FABRIC_1.0) 2 (FABRIC_1.0) 2 (FABRIC_1.0) 2 (FABRIC_1.0) 11c: 2 (FABRIC_1.0) 2 (FABRIC_1.0) 0x001c: Rev: 1 Flags: none Index: 2 Cnt: 1 Name: FABRIC_1.0 </PRE> <P>This is the latest libfabric release, so how would I go about getting symbols compatible with Intel MPI 2019?</P> <P>Thanks; Chris</P> Mon, 01 Jul 2019 19:10:48 GMT https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132683#M5687 4f0drlp7eyj3 2019-07-01T19:10:48Z OK, never mind, appears GCC 4 https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132684#M5688 <P>OK, never mind, I was looking at an older file installed from a Centos&nbsp;RPM.</P> Mon, 01 Jul 2019 19:22:00 GMT https://community.intel.com/t5/Intel-MPI-Library/Performance-issues-with-Omni-Path/m-p/1132684#M5688 4f0drlp7eyj3 2019-07-01T19:22:00Z