Intel® MPI Library
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.

IMB Allreduce and Reduce message size confusion

Pavel_Mezentsev
Beginner
410 Views

I've been doing some tuning and noticed a confusing thing.

I've tried different types of Allreduce and Reduce for particular message size and noticed that in the output my setting influenced the message size 4 times bigger than what I've specified. For exaple if I say use this type for message size 128 actually message size 512 would be affected. For other collectives everything works as expected.

Documentation states that MINMSGLOG for Allreduce and Reduce is sizeof(float) which is 4 in my case instead of 1 for other cases by default. So perhaps the actual message size starts from 1 byte but the value printed is just 4 times bigger. Is it possible?

0 Kudos
3 Replies
James_T_Intel
Moderator
410 Views
Hi Pavel, What command line are you using? If I use [plain]mpirun -n 2 ./IMB-MPI1 -msglog 4 allreduce reduce sendrecv[/plain] I get a maximum message size of 16 (2^4) for each of the operations. Sincerely, James Tullos Technical Consulting Engineer Intel® Cluster Tools
0 Kudos
Pavel_Mezentsev
Beginner
410 Views
Hello, James. My command looks like this: mpiexec.hydra -np $ NP ... ./IMB-MPI1 -npmin $NP -mem 4G $COLL The maximum size is the same, but Allreduce skips sizes 1 and 2, which should be fine according to the documentation. I'm doing some FCA testing with OpenMPI and in particular I'm trying to tune the FCA parameters in the fca_mpi_spec.ini file. That's where I got confused. For example I set the following rules: [rule-allreduce-2] msg_size_min = 64 msg_size_max = 64 comm_size_min = 80 comm_size_max = 80 offload_type = none fca_enable = 1 ## Sample bcast rules [rule-bcast-1] msg_size_min = 64 msg_size_max = 64 comm_size_min = 80 comm_size_max = 80 offload_type = none fca_enable = 1 And here are the results I get: #---------------------------------------------------------------- # Benchmarking Allreduce # #processes = 80 #---------------------------------------------------------------- #bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] 0 1000 0.02 0.02 0.02 4 1000 3.29 3.30 3.30 8 1000 3.24 3.24 3.24 16 1000 3.26 3.26 3.26 32 1000 3.34 3.35 3.34 64 1000 4.11 4.12 4.12 128 1000 4.35 4.35 4.35 256 1000 24.48 24.50 24.49 512 1000 6.42 6.43 6.42 1024 1000 8.16 8.16 8.16 2048 1000 13.55 13.56 13.55 ........ #---------------------------------------------------------------- # Benchmarking Bcast # #processes = 80 #---------------------------------------------------------------- #bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] 0 1000 0.02 0.02 0.02 1 1000 2.60 2.61 2.61 2 1000 1.55 1.57 1.56 4 1000 1.51 1.52 1.51 8 1000 1.50 1.51 1.51 16 1000 1.50 1.51 1.51 32 1000 1.54 1.56 1.55 64 1000 7.91 7.95 7.93 128 1000 1.72 1.73 1.73 256 1000 1.74 1.75 1.74 512 1000 1.75 1.77 1.76 1024 1000 1.95 1.96 1.95 2048 1000 2.92 2.93 2.93 ....... In case of Allreduce the affected message size is four times bigger than the one specified in the FCA rules. I'm trying to determine why the results are so weird. Is there a bug in IMB or in FCA or in OpenMPI or it's just something that I don't understand and everything should be the way it is :)
0 Kudos
James_T_Intel
Moderator
410 Views
Hi Pavel, I think the problem may be a difference in how "message size" is defined. For the Intel® MPI Benchmarks (and the Intel® MPI Library), the message size is defined by the number of bytes in the message. It is possible that the message size defined in fca_mpi_spec.ini is the number of elements. If so, the allreduce benchmark would show a difference at 4 times the message size. If you are using the standard code for IMB, all reduction operations use MPI_FLOAT (4 bytes), and everything else uses MPI_BYTE (1 byte). I would check with the Open MPI team about this. Sincerely, James Tullos Technical Consulting Engineer Intel® Cluster Tools
0 Kudos
Reply