https://community.intel.com/t5/Intel-Moderncode-for-Parallel/MPI-errors-on-large-OPA-fabric/m-p/1117226#M7491 <P>Hello,</P> <P>We're getting MPI communication errors using Intel MPI on our cluster using omnipath.&nbsp; This is a job using 931 nodes, smaller runs using 600 nodes execute properly.</P> <P>Other details:</P> <P>We're using Intel Parallel Studio 2017 update 4 (compilers_and_libraries_2017.4.196).</P> <P>There are 1024 total nodes on the fabric, we would like to run jobs utilizing the entire cluster.</P> <P>This is an HPL run using Intel l_mklb_p_2017.3.017.</P> <P>This is an example of the errors we see - what is interesting is the buffer and target size is the same, however the error states it is truncated.&nbsp; Is there normally a header the target buffer needs to have space for?</P> <P>Fatal error in MPI_Recv: Message truncated, error stack:<BR /> MPI_Recv(224)................: MPI_Recv(buf=0x2b1ee8401840, count=1455, MPI_DOUBLE, src=17, tag=10001, comm=0x84000002, status=0x7ffef5ddfe50) failed<BR /> MPID_nem_tmi_handle_rreq(738): Message from rank 17 and tag 10001 truncated; 11640 bytes received but buffer size is 11640<BR /> Fatal error in MPI_Sendrecv: Message truncated, error stack:<BR /> MPI_Sendrecv(259)............: MPI_Sendrecv(sbuf=0x2b93ba000000, scount=1164, MPI_DOUBLE, dest=13, stag=10001, rbuf=0x2b93ba002460, rcount=1746, MPI_DOUBLE, src=13, rtag=10001, comm=0x84000002, status=0x7ffcec3f3f50) failed<BR /> MPID_nem_tmi_handle_rreq(738): Message from rank 13 and tag 10001 truncated; 13968 bytes received but buffer size is 13968<BR /> Fatal error in MPI_Sendrecv: Message truncated, error stack:<BR /> MPI_Sendrecv(259)............: MPI_Sendrecv(sbuf=0x2b30f5880808, scount=24576, MPI_DOUBLE, dest=16, stag=10001, rbuf=0x2b30ef400000, rcount=1164, MPI_DOUBLE, src=16, rtag=10001, comm=0x84000002, status=0x7ffc4278ec10) failed</P> <P>&nbsp;</P> Sat, 27 May 2017 14:41:03 GMT mcs-systems 2017-05-27T14:41:03Z https://community.intel.com/t5/Intel-Moderncode-for-Parallel/MPI-errors-on-large-OPA-fabric/m-p/1117226#M7491 <P>Hello,</P> <P>We're getting MPI communication errors using Intel MPI on our cluster using omnipath.&nbsp; This is a job using 931 nodes, smaller runs using 600 nodes execute properly.</P> <P>Other details:</P> <P>We're using Intel Parallel Studio 2017 update 4 (compilers_and_libraries_2017.4.196).</P> <P>There are 1024 total nodes on the fabric, we would like to run jobs utilizing the entire cluster.</P> <P>This is an HPL run using Intel l_mklb_p_2017.3.017.</P> <P>This is an example of the errors we see - what is interesting is the buffer and target size is the same, however the error states it is truncated.&nbsp; Is there normally a header the target buffer needs to have space for?</P> <P>Fatal error in MPI_Recv: Message truncated, error stack:<BR /> MPI_Recv(224)................: MPI_Recv(buf=0x2b1ee8401840, count=1455, MPI_DOUBLE, src=17, tag=10001, comm=0x84000002, status=0x7ffef5ddfe50) failed<BR /> MPID_nem_tmi_handle_rreq(738): Message from rank 17 and tag 10001 truncated; 11640 bytes received but buffer size is 11640<BR /> Fatal error in MPI_Sendrecv: Message truncated, error stack:<BR /> MPI_Sendrecv(259)............: MPI_Sendrecv(sbuf=0x2b93ba000000, scount=1164, MPI_DOUBLE, dest=13, stag=10001, rbuf=0x2b93ba002460, rcount=1746, MPI_DOUBLE, src=13, rtag=10001, comm=0x84000002, status=0x7ffcec3f3f50) failed<BR /> MPID_nem_tmi_handle_rreq(738): Message from rank 13 and tag 10001 truncated; 13968 bytes received but buffer size is 13968<BR /> Fatal error in MPI_Sendrecv: Message truncated, error stack:<BR /> MPI_Sendrecv(259)............: MPI_Sendrecv(sbuf=0x2b30f5880808, scount=24576, MPI_DOUBLE, dest=16, stag=10001, rbuf=0x2b30ef400000, rcount=1164, MPI_DOUBLE, src=16, rtag=10001, comm=0x84000002, status=0x7ffc4278ec10) failed</P> <P>&nbsp;</P> Sat, 27 May 2017 14:41:03 GMT https://community.intel.com/t5/Intel-Moderncode-for-Parallel/MPI-errors-on-large-OPA-fabric/m-p/1117226#M7491 mcs-systems 2017-05-27T14:41:03Z https://community.intel.com/t5/Intel-Moderncode-for-Parallel/MPI-errors-on-large-OPA-fabric/m-p/1117227#M7492 <P>The question seems more appropriate to the cluster hpc forum, if you could quote intel cluster checker diagnoses.</P> Sat, 27 May 2017 18:16:27 GMT https://community.intel.com/t5/Intel-Moderncode-for-Parallel/MPI-errors-on-large-OPA-fabric/m-p/1117227#M7492 TimP 2017-05-27T18:16:27Z