Assign specific mpi tasks to specific IB interfaces
I have a system with 16 cores and 2 IB interfaces. I need to use both rails, so have been using
But this has the undesirable effect of sending some of the data from the second 8 cores to the IB on the first eight cores and vice-versa.
But this assigns process 0 to interface 1, process 1 to interface 2, process 3 to interface 1, etc. This sends alot of data from one set of cores to the opposing interface. Furthermore, the code is written so that it assumes that process 0 and process 1 are in the same cpu so it tries to do a cyclic data movement.
I need to assign process 0 - 7 to interface 1 and process 8 - 15 to interface 2.
mpirun -n 8 -env I_MPI_OFA_ADAPTER_NAME adap1 : -n 8 -env I_MPI_OFA_ADAPTER_NAME adap2 [/bash] This should set the first 8 processes to use the first adapter, and the next 8 to use the second adapter.
Sincerely, James Tullos Technical Consulting Engineer Intel Cluster Tools
I need to make some clarifications. If you use I_MPI_FABRICS=shm:ofa it means that 'shm' will be used for INTRA-node communication and 'ofa' will be used for INTER-node communication. Since you are going to use OFA for intra-node communication you need to set I_MPI_FABRICS to 'ofa' or 'ofa:ofa'. And all other paratemers as James mentioned. Please give it a try and compare results with default settings. It would be nice if you could share with us results of different runs.