https://community.intel.com/t5/Intel-MPI-Library/Scalability-on-Intel-quad-CPU/m-p/1119482#M5451 <P>A customer is asking for more references on parallel scalability on 4 CPU machines such as E5-4620 (4x8 core).&nbsp; I noted on his box that the sibling logicals under hyperthread are numbered 32 apart under linux.&nbsp; At least up to 8 threads, I don't run into any issue with spreading threads across consecutive cores, although there seems to be unexpected HT speedup for 1 thread.&nbsp; I could get at best a 20% performance gain with his application by using all cores under OpenMP on 2 CPUs vs. 1 CPU, which doesn't entirely surprise me. We have a 3x performance gain from 1 to 8 threads, on top of the best the production ifort can do with AVX vectorization.&nbsp; Needless to say, he didn't hear about this when deciding on the purchase (and maybe the sales force doesn't care to know this).</P> <P>I pointed out that crossing among CPUs involves possible double remote memory hop, increasing latency and reducing bandwidth, and this might be alleviated by using MPI ranks pinned to individual CPUs.&nbsp; Are there other likely methods for gaining parallel performance?</P> <P>We found that ifort 16.0.3 needs -fno-inline-functions to avoid hidden race conditions (which are flagged indirectly by turning on subscript range check).&nbsp;&nbsp; The application passes range check (and incurs none of the 16.0 crashes) with all flags on with 15.0 and 17.0 compilers.&nbsp; Needless to say, I used Inspector to fix OpenMP to its satisfaction.</P> <P>On a Westmere 2x6 core linux box (HT disabled) it scaled well to 8 threads, no further.&nbsp; None of my usual hypotheses about pinning to favorite cores (there are 2 varieties of cache connection) made any difference. On a back to back Windows vs. linux comparison on a single Nehalem CPU, linux gave a 25% performance increase, which I attribute to Transparent Huge Pages.&nbsp; Haswell page prefetch seems to reduce but not eliminate the importance of THP.</P> Fri, 06 May 2016 20:30:56 GMT TimP 2016-05-06T20:30:56Z https://community.intel.com/t5/Intel-MPI-Library/Scalability-on-Intel-quad-CPU/m-p/1119482#M5451 <P>A customer is asking for more references on parallel scalability on 4 CPU machines such as E5-4620 (4x8 core).&nbsp; I noted on his box that the sibling logicals under hyperthread are numbered 32 apart under linux.&nbsp; At least up to 8 threads, I don't run into any issue with spreading threads across consecutive cores, although there seems to be unexpected HT speedup for 1 thread.&nbsp; I could get at best a 20% performance gain with his application by using all cores under OpenMP on 2 CPUs vs. 1 CPU, which doesn't entirely surprise me. We have a 3x performance gain from 1 to 8 threads, on top of the best the production ifort can do with AVX vectorization.&nbsp; Needless to say, he didn't hear about this when deciding on the purchase (and maybe the sales force doesn't care to know this).</P> <P>I pointed out that crossing among CPUs involves possible double remote memory hop, increasing latency and reducing bandwidth, and this might be alleviated by using MPI ranks pinned to individual CPUs.&nbsp; Are there other likely methods for gaining parallel performance?</P> <P>We found that ifort 16.0.3 needs -fno-inline-functions to avoid hidden race conditions (which are flagged indirectly by turning on subscript range check).&nbsp;&nbsp; The application passes range check (and incurs none of the 16.0 crashes) with all flags on with 15.0 and 17.0 compilers.&nbsp; Needless to say, I used Inspector to fix OpenMP to its satisfaction.</P> <P>On a Westmere 2x6 core linux box (HT disabled) it scaled well to 8 threads, no further.&nbsp; None of my usual hypotheses about pinning to favorite cores (there are 2 varieties of cache connection) made any difference. On a back to back Windows vs. linux comparison on a single Nehalem CPU, linux gave a 25% performance increase, which I attribute to Transparent Huge Pages.&nbsp; Haswell page prefetch seems to reduce but not eliminate the importance of THP.</P> Fri, 06 May 2016 20:30:56 GMT https://community.intel.com/t5/Intel-MPI-Library/Scalability-on-Intel-quad-CPU/m-p/1119482#M5451 TimP 2016-05-06T20:30:56Z