ScaLAPACK arrays are distributed in a block-cyclic fashion over the process "grid". ScaLAPACK then uses the PBLAS and BLACS to perform BLAS-like operations, but in a distributed SPMD fashion, which become a mix of communication between processes, and BLAS operations within the processes, more-or-less.
So the size of the block is going to affect the performance of the communication and the BLAS calls, but the degree to which it does depends on the implementation. The MKL implementation is a black-box to the end user (me). And I don't have an ATLAS-like search tool to point me in the right direction toward what block size I should be using, especially when the parameters are things like { Gig-ethernet vs 10G infiniband vs ....} and {westmere vs sandy/ivy-bridge vs haswell } etc.
So... are there any guidelines for choice of block size when using MKL ScaLAPACK, LAPACK, and BLAS ?
- E.g. is it important for the ScaLAPACK block size to be a cache-friendly size (e.g no larger than 1/2 of L1 or L2, etc)?
- Or alternatively does the ScaLAPACK block size affect primarily the load balancing as an operation that works on successively smaller areas of a matrix as many of the algorithms do? But are not relevant to the efficiency of block-matrix multiplies at the BLAS level?
- Perhaps the MKL Level-3 BLAS calls are themselves made less-sensitive to large block sizes? (E.g. because there is re-blocking within gemm() etc, anyway where threads are exploited by OpenMP etc ... maybe the MKL BLAS is already subdividing (re-blocking) to be as effcient as it can given that it gets a large enough block?
- If I want to avoid such hypothesized re-blocking, because for some reason there are places where I can manage this block size "for free" as a side-effect of the way my code is structured, is there an optimal block size for MKL level-3 BLAS calls?
