- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I've been doing some tuning and noticed a confusing thing.
I've tried different types of Allreduce and Reduce for particular message size and noticed that in the output my setting influenced the message size 4 times bigger than what I've specified. For exaple if I say use this type for message size 128 actually message size 512 would be affected. For other collectives everything works as expected.
Documentation states that MINMSGLOG for Allreduce and Reduce is sizeof(float) which is 4 in my case instead of 1 for other cases by default. So perhaps the actual message size starts from 1 byte but the value printed is just 4 times bigger. Is it possible?
Link Copied
3 Replies
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Pavel,
What command line are you using? If I use
[plain]mpirun -n 2 ./IMB-MPI1 -msglog 4 allreduce reduce sendrecv[/plain]
I get a maximum message size of 16 (2^4) for each of the operations.
Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello, James.
My command looks like this:
mpiexec.hydra -np $ NP ... ./IMB-MPI1 -npmin $NP -mem 4G $COLL
The maximum size is the same, but Allreduce skips sizes 1 and 2, which should be fine according to the documentation.
I'm doing some FCA testing with OpenMPI and in particular I'm trying to tune the FCA parameters in the fca_mpi_spec.ini file. That's where I got confused. For example I set the following rules:
[rule-allreduce-2]
msg_size_min = 64
msg_size_max = 64
comm_size_min = 80
comm_size_max = 80
offload_type = none
fca_enable = 1
## Sample bcast rules
[rule-bcast-1]
msg_size_min = 64
msg_size_max = 64
comm_size_min = 80
comm_size_max = 80
offload_type = none
fca_enable = 1
And here are the results I get:
#----------------------------------------------------------------
# Benchmarking Allreduce
# #processes = 80
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.02 0.02 0.02
4 1000 3.29 3.30 3.30
8 1000 3.24 3.24 3.24
16 1000 3.26 3.26 3.26
32 1000 3.34 3.35 3.34
64 1000 4.11 4.12 4.12
128 1000 4.35 4.35 4.35
256 1000 24.48 24.50 24.49
512 1000 6.42 6.43 6.42
1024 1000 8.16 8.16 8.16
2048 1000 13.55 13.56 13.55
........
#----------------------------------------------------------------
# Benchmarking Bcast
# #processes = 80
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.02 0.02 0.02
1 1000 2.60 2.61 2.61
2 1000 1.55 1.57 1.56
4 1000 1.51 1.52 1.51
8 1000 1.50 1.51 1.51
16 1000 1.50 1.51 1.51
32 1000 1.54 1.56 1.55
64 1000 7.91 7.95 7.93
128 1000 1.72 1.73 1.73
256 1000 1.74 1.75 1.74
512 1000 1.75 1.77 1.76
1024 1000 1.95 1.96 1.95
2048 1000 2.92 2.93 2.93
.......
In case of Allreduce the affected message size is four times bigger than the one specified in the FCA rules.
I'm trying to determine why the results are so weird. Is there a bug in IMB or in FCA or in OpenMPI or it's just something that I don't understand and everything should be the way it is :)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Pavel,
I think the problem may be a difference in how "message size" is defined. For the Intel® MPI Benchmarks (and the Intel® MPI Library), the message size is defined by the number of bytes in the message. It is possible that the message size defined in fca_mpi_spec.ini is the number of elements. If so, the allreduce benchmark would show a difference at 4 times the message size. If you are using the standard code for IMB, all reduction operations use MPI_FLOAT (4 bytes), and everything else uses MPI_BYTE (1 byte). I would check with the Open MPI team about this.
Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools
Reply
Topic Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page