Hi Rene,

René_O_ · ‎04-01-2015

Hello,

there is a problem with the latest MPI library and multi-rail support over OFA:

Version 5.0 Update 2 runs just fine:

$ mpirun -env I_MPI_DEBUG 6 -hosts node1,node2 -np 2 -ppn 1 -genv I_MPI_OFA_ADAPTER_NAME mlx5_0 -genv I_MPI_OFA_NUM_PORTS 2 -genv I_MPI_FABRICS shm:ofa -genv MV2_USE_APM 0 /exports/bin/osu-micro-benchmarks-4.3_impi/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw
[0] MPI startup(): Intel(R) MPI Library, Version 5.0 Update 2  Build 20141030 (build id: 10994)
[0] MPI startup(): Copyright (C) 2003-2014 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded debug library
[0] MPI startup(): Found 2 IB devices
[1] MPI startup(): Found 2 IB devices
[0] MPI startup(): Open 0 IB device: mlx5_0
[1] MPI startup(): Open 0 IB device: mlx5_0
[0] MPI startup(): Start 2 ports per adapter
[1] MPI startup(): Start 2 ports per adapter
[0] MPID_nem_ofacm_init(): Init
[1] MPID_nem_ofacm_init(): Init
[0] MPI startup(): shm and ofa data transfer modes
[1] MPI startup(): shm and ofa data transfer modes
[0] MPI startup(): Device_reset_idx=3
[0] MPI startup(): Allgather: 3: 0-1 & 0-4
[0] MPI startup(): Allgather: 1: 1-1 & 0-4
[0] MPI startup(): Allgather: 4: 2-4 & 0-4
[0] MPI startup(): Allgather: 1: 5-92 & 0-4
[0] MPI startup(): Allgather: 4: 93-161 & 0-4
[0] MPI startup(): Allgather: 1: 162-33532 & 0-4
[0] MPI startup(): Allgather: 4: 33533-102618 & 0-4
[0] MPI startup(): Allgather: 1: 102619-546855 & 0-4
[0] MPI startup(): Allgather: 4: 0-2147483647 & 0-4
[0] MPI startup(): Allgather: 4: 1-839 & 5-8
[0] MPI startup(): Allgather: 2: 840-4170 & 5-8
[0] MPI startup(): Allgather: 4: 0-2147483647 & 5-8
[0] MPI startup(): Allgather: 1: 1-5 & 9-16
[0] MPI startup(): Allgather: 4: 6-8 & 9-16
[0] MPI startup(): Allgather: 1: 9-440 & 9-16
[0] MPI startup(): Allgather: 2: 441-2083 & 9-16
[0] MPI startup(): Allgather: 4: 2084-9050 & 9-16
[0] MPI startup(): Allgather: 1: 9050-65530 & 9-16
[0] MPI startup(): Allgather: 4: 65530-218986 & 9-16
[0] MPI startup(): Allgather: 1: 0-2147483647 & 9-16
[0] MPI startup(): Allgather: 1: 1-247 & 17-32
[0] MPI startup(): Allgather: 2: 248-1043 & 17-32
[0] MPI startup(): Allgather: 4: 1044-3132 & 17-32
[0] MPI startup(): Allgather: 1: 3133-3010585 & 17-32
[0] MPI startup(): Allgather: 3: 0-2147483647 & 17-32
[0] MPI startup(): Allgather: 1: 1-176 & 33-2147483647
[0] MPI startup(): Allgather: 4: 177-1818 & 33-2147483647
[0] MPI startup(): Allgather: 1: 1819-2278 & 33-2147483647
[0] MPI startup(): Allgather: 4: 2279-4129 & 33-2147483647
[0] MPI startup(): Allgather: 1: 4130-1555968 & 33-2147483647
[0] MPI startup(): Allgather: 4: 0-2147483647 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 0-261835 & 0-2
[0] MPI startup(): Allgatherv: 4: 261835-611669 & 0-2
[0] MPI startup(): Allgatherv: 3: 611669-3196878 & 0-2
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 0-2
[0] MPI startup(): Allgatherv: 1: 0-370 & 3-4
[0] MPI startup(): Allgatherv: 2: 370-838 & 3-4
[0] MPI startup(): Allgatherv: 1: 838-3403 & 3-4
[0] MPI startup(): Allgatherv: 4: 3403-4808 & 3-4
[0] MPI startup(): Allgatherv: 1: 4808-8294 & 3-4
[0] MPI startup(): Allgatherv: 3: 8294-130946 & 3-4
[0] MPI startup(): Allgatherv: 4: 130946-208466 & 3-4
[0] MPI startup(): Allgatherv: 3: 208466-393216 & 3-4
[0] MPI startup(): Allgatherv: 4: 393216-617063 & 3-4
[0] MPI startup(): Allgatherv: 3: 617063-1398101 & 3-4
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 3-4
[0] MPI startup(): Allgatherv: 1: 0-25 & 5-8
[0] MPI startup(): Allgatherv: 4: 25-46 & 5-8
[0] MPI startup(): Allgatherv: 1: 46-1403 & 5-8
[0] MPI startup(): Allgatherv: 2: 1403-4296 & 5-8
[0] MPI startup(): Allgatherv: 3: 4296-88097 & 5-8
[0] MPI startup(): Allgatherv: 4: 88097-345405 & 5-8
[0] MPI startup(): Allgatherv: 3: 345405-2801646 & 5-8
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 5-8
[0] MPI startup(): Allgatherv: 4: 0-3 & 9-16
[0] MPI startup(): Allgatherv: 1: 3-23 & 9-16
[0] MPI startup(): Allgatherv: 4: 23-106 & 9-16
[0] MPI startup(): Allgatherv: 1: 106-797 & 9-16
[0] MPI startup(): Allgatherv: 2: 797-3141 & 9-16
[0] MPI startup(): Allgatherv: 3: 3141-5874 & 9-16
[0] MPI startup(): Allgatherv: 2: 5874-9274 & 9-16
[0] MPI startup(): Allgatherv: 1: 9274-17847 & 9-16
[0] MPI startup(): Allgatherv: 3: 17847-84012 & 9-16
[0] MPI startup(): Allgatherv: 4: 84012-225929 & 9-16
[0] MPI startup(): Allgatherv: 3: 225929-450203 & 9-16
[0] MPI startup(): Allgatherv: 4: 450203-545471 & 9-16
[0] MPI startup(): Allgatherv: 3: 545471-1767416 & 9-16
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 9-16
[0] MPI startup(): Allgatherv: 1: 0-25 & 17-32
[0] MPI startup(): Allgatherv: 4: 25-46 & 17-32
[0] MPI startup(): Allgatherv: 1: 46-1403 & 17-32
[0] MPI startup(): Allgatherv: 2: 1403-4296 & 17-32
[0] MPI startup(): Allgatherv: 3: 4296-88097 & 17-32
[0] MPI startup(): Allgatherv: 4: 88097-345405 & 17-32
[0] MPI startup(): Allgatherv: 3: 345405-2801646 & 17-32
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 17-32
[0] MPI startup(): Allgatherv: 4: 0-1 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 1-3 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 3-6 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 6-96 & 33-2147483647
[0] MPI startup(): Allgatherv: 2: 96-535 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 535-1255 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 1255-3148 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 3148-5455 & 33-2147483647
[0] MPI startup(): Allgatherv: 2: 5455-17721 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 17721-41571 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 41571-110427 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 110427-596874 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 596874-1273594 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 1273594-2420103 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 0-2147483647 & 33-2147483647
[0] MPI startup(): Allreduce: 7: 0-20 & 0-2
[0] MPI startup(): Allreduce: 1: 20-905 & 0-2
[0] MPI startup(): Allreduce: 4: 905-2058 & 0-2
[0] MPI startup(): Allreduce: 1: 2058-17240 & 0-2
[0] MPI startup(): Allreduce: 8: 17240-36706 & 0-2
[0] MPI startup(): Allreduce: 3: 36706-777395 & 0-2
[0] MPI startup(): Allreduce: 8: 777395-1247501 & 0-2
[0] MPI startup(): Allreduce: 2: 1247501-3149905 & 0-2
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 0-2
[0] MPI startup(): Allreduce: 1: 0-597 & 3-4
[0] MPI startup(): Allreduce: 7: 597-1755 & 3-4
[0] MPI startup(): Allreduce: 1: 1755-16932 & 3-4
[0] MPI startup(): Allreduce: 6: 16932-34438 & 3-4
[0] MPI startup(): Allreduce: 8: 34438-81725 & 3-4
[0] MPI startup(): Allreduce: 3: 81725-396925 & 3-4
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 3-4
[0] MPI startup(): Allreduce: 1: 0-55 & 5-8
[0] MPI startup(): Allreduce: 6: 55-74 & 5-8
[0] MPI startup(): Allreduce: 4: 74-213 & 5-8
[0] MPI startup(): Allreduce: 1: 213-1765 & 5-8
[0] MPI startup(): Allreduce: 4: 1765-2065 & 5-8
[0] MPI startup(): Allreduce: 1: 2065-16561 & 5-8
[0] MPI startup(): Allreduce: 2: 16561-39707 & 5-8
[0] MPI startup(): Allreduce: 8: 39707-168698 & 5-8
[0] MPI startup(): Allreduce: 7: 168698-570628 & 5-8
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 5-8
[0] MPI startup(): Allreduce: 6: 0-5 & 9-16
[0] MPI startup(): Allreduce: 1: 5-935 & 9-16
[0] MPI startup(): Allreduce: 4: 935-1105 & 9-16
[0] MPI startup(): Allreduce: 1: 1105-17349 & 9-16
[0] MPI startup(): Allreduce: 4: 17349-53440 & 9-16
[0] MPI startup(): Allreduce: 8: 53440-315615 & 9-16
[0] MPI startup(): Allreduce: 7: 315615-1140547 & 9-16
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 9-16
[0] MPI startup(): Allreduce: 6: 0-74 & 17-32
[0] MPI startup(): Allreduce: 4: 74-199 & 17-32
[0] MPI startup(): Allreduce: 6: 199-392 & 17-32
[0] MPI startup(): Allreduce: 4: 392-750 & 17-32
[0] MPI startup(): Allreduce: 6: 750-2087 & 17-32
[0] MPI startup(): Allreduce: 1: 2087-14841 & 17-32
[0] MPI startup(): Allreduce: 2: 14841-20338 & 17-32
[0] MPI startup(): Allreduce: 6: 20338-124787 & 17-32
[1] MPI startup(): Recognition=2 Platform(code=8 ippn=0 dev=8) Fabric(intra=1 inter=2 flags=0x0)
[0] MPI startup(): Allreduce: 8: 124787-646663 & 17-32
[0] MPI startup(): Allreduce: 7: 646663-2129006 & 17-32
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 17-32
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 0-245 & 0-2
[0] MPI startup(): Alltoall: 2: 245-276 & 0-2
[0] MPI startup(): Alltoall: 3: 0-2147483647 & 0-2
[0] MPI startup(): Alltoall: 2: 0-3878 & 3-4
[0] MPI startup(): Alltoall: 3: 3879-6492 & 3-4
[0] MPI startup(): Alltoall: 4: 6493-9680 & 3-4
[0] MPI startup(): Alltoall: 3: 9681-24681 & 3-4
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 3-4
[0] MPI startup(): Alltoall: 2: 0-3082 & 5-8
[0] MPI startup(): Alltoall: 3: 3083-20124 & 5-8
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 5-8
[0] MPI startup(): Alltoall: 2: 0-2235 & 9-16
[0] MPI startup(): Alltoall: 3: 2236-26946 & 9-16
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 9-16
[0] MPI startup(): Alltoall: 1: 0-329 & 17-32
[0] MPI startup(): Alltoall: 2: 330-4661 & 17-32
[0] MPI startup(): Alltoall: 3: 4662-282206 & 17-32
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 17-32
[0] MPI startup(): Alltoall: 1: 0-217 & 33-2147483647
[0] MPI startup(): Alltoall: 4: 218-409 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 410-4415 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 4416-32356 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 32357-36297 & 33-2147483647
[0] MPI startup(): Alltoall: 4: 36298-65861 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 65862-866313 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Alltoallv: 1: 0-2147483647 & 0-2147483647
[0] MPI startup(): Alltoallw: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Barrier: 6: 0-2147483647 & 0-2147483647
[0] MPI startup(): Bcast: 2: 0-1 & 0-2
[0] MPI startup(): Bcast: 7: 1-402 & 0-2
[0] MPI startup(): Bcast: 5: 402-682 & 0-2
[0] MPI startup(): Bcast: 4: 682-1433 & 0-2
[0] MPI startup(): Bcast: 7: 1433-5734 & 0-2
[0] MPI startup(): Bcast: 1: 5734-21845 & 0-2
[0] MPI startup(): Bcast: 6: 21845-95963 & 0-2
[0] MPI startup(): Bcast: 5: 95963-409897 & 0-2
[0] MPI startup(): Bcast: 1: 0-2147483647 & 0-2
[0] MPI startup(): Bcast: 7: 0-2147483647 & 3-4
[0] MPI startup(): Bcast: 7: 1-1 & 5-8
[0] MPI startup(): Bcast: 1: 2-254 & 5-8
[0] MPI startup(): Bcast: 5: 255-261 & 5-8
[0] MPI startup(): Bcast: 1: 262-752 & 5-8
[0] MPI startup(): Bcast: 6: 753-1254 & 5-8
[0] MPI startup(): Bcast: 1: 1255-4032 & 5-8
[0] MPI startup(): Bcast: 4: 4033-19764 & 5-8
[0] MPI startup(): Bcast: 1: 19765-121346 & 5-8
[0] MPI startup(): Bcast: 6: 121347-201114 & 5-8
[0] MPI startup(): Bcast: 1: 201115-267790 & 5-8
[0] MPI startup(): Bcast: 2: 267791-588703 & 5-8
[0] MPI startup(): Bcast: 3: 588704-1341590 & 5-8
[0] MPI startup(): Bcast: 5: 1341591-2139116 & 5-8
[0] MPI startup(): Bcast: 6: 0-2147483647 & 5-8
[0] MPI startup(): Bcast: 7: 1-84 & 9-16
[0] MPI startup(): Bcast: 1: 85-137 & 9-16
[0] MPI startup(): Bcast: 6: 138-475 & 9-16
[0] MPI startup(): Bcast: 1: 476-1496 & 9-16
[0] MPI startup(): Bcast: 6: 1497-10273 & 9-16
[0] MPI startup(): Bcast: 1: 10274-106199 & 9-16
[0] MPI startup(): Bcast: 6: 106200-323972 & 9-16
[0] MPI startup(): Bcast: 2: 0-2147483647 & 9-16
[0] MPI startup(): Bcast: 7: 1-103 & 17-32
[0] MPI startup(): Bcast: 5: 104-180 & 17-32
[0] MPI startup(): Bcast: 7: 181-590 & 17-32
[0] MPI startup(): Bcast: 5: 591-1170 & 17-32
[0] MPI startup(): Bcast: 6: 1171-3459 & 17-32
[0] MPI startup(): Bcast: 1: 3460-52532 & 17-32
[0] MPI startup(): Bcast: 2: 52533-239896 & 17-32
[0] MPI startup(): Bcast: 5: 239897-262474 & 17-32
[0] MPI startup(): Bcast: 2: 0-2147483647 & 17-32
[0] MPI startup(): Bcast: 7: 1-935 & 33-2147483647
[0] MPI startup(): Bcast: 6: 936-1185 & 33-2147483647
[0] MPI startup(): Bcast: 1: 1186-4873 & 33-2147483647
[0] MPI startup(): Bcast: 6: 4874-9664 & 33-2147483647
[0] MPI startup(): Bcast: 1: 9665-85765 & 33-2147483647
[0] MPI startup(): Bcast: 2: 85766-250963 & 33-2147483647
[0] MPI startup(): Bcast: 5: 250964-262409 & 33-2147483647
[0] MPI startup(): Bcast: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Exscan: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Gather: 3: 1-8 & 0-4
[0] MPI startup(): Gather: 1: 9-29 & 0-4
[0] MPI startup(): Gather: 3: 30-25688 & 0-4
[0] MPI startup(): Gather: 2: 25689-56274 & 0-4
[0] MPI startup(): Gather: 3: 56275-68784 & 0-4
[0] MPI startup(): Gather: 2: 68785-1304326 & 0-4
[0] MPI startup(): Gather: 3: 1304327-2684607 & 0-4
[0] MPI startup(): Gather: 2: 0-2147483647 & 0-4
[0] MPI startup(): Gather: 3: 1-10278 & 5-8
[0] MPI startup(): Gather: 2: 10279-22432 & 5-8
[0] MPI startup(): Gather: 3: 22433-51123 & 5-8
[0] MPI startup(): Gather: 2: 51124-93434 & 5-8
[0] MPI startup(): Gather: 3: 93435-667907 & 5-8
[0] MPI startup(): Gather: 2: 667908-1262190 & 5-8
[0] MPI startup(): Gather: 3: 1262191-2879271 & 5-8
[0] MPI startup(): Gather: 2: 0-2147483647 & 5-8
[0] MPI startup(): Gather: 3: 1-4004 & 9-16
[0] MPI startup(): Gather: 2: 4005-4168 & 9-16
[0] MPI startup(): Gather: 3: 4169-20254 & 9-16
[0] MPI startup(): Gather: 2: 20255-41713 & 9-16
[0] MPI startup(): Gather: 3: 41714-793099 & 9-16
[0] MPI startup(): Gather: 2: 793100-1272717 & 9-16
[0] MPI startup(): Gather: 3: 0-2147483647 & 9-16
[0] MPI startup(): Gather: 3: 1-2047 & 17-32
[0] MPI startup(): Gather: 2: 2048-16131 & 17-32
[0] MPI startup(): Gather: 3: 16132-25688 & 17-32
[0] MPI startup(): Gather: 2: 25689-56274 & 17-32
[0] MPI startup(): Gather: 3: 56275-68784 & 17-32
[0] MPI startup(): Gather: 2: 68785-1304326 & 17-32
[0] MPI startup(): Gather: 3: 1304327-2684607 & 17-32
[0] MPI startup(): Gather: 2: 0-2147483647 & 17-32
[0] MPI startup(): Gather: 3: 1-51845 & 33-2147483647
[0] MPI startup(): Gather: 2: 51846-76744 & 33-2147483647
[0] MPI startup(): Gather: 3: 76745-721936 & 33-2147483647
[0] MPI startup(): Gather: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Gatherv: 2: 0-2147483647 & 0-2147483647
[0] MPI startup(): Reduce_scatter: 4: 0-5 & 0-2
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 0-2
[0] MPI startup(): Reduce_scatter: 1: 0-29 & 3-8
[0] MPI startup(): Reduce_scatter: 5: 29-48 & 3-8
[0] MPI startup(): Reduce_scatter: 1: 48-2805 & 3-8
[0] MPI startup(): Reduce_scatter: 3: 2805-5556 & 3-8
[0] MPI startup(): Reduce_scatter: 5: 5556-11732 & 3-8
[0] MPI startup(): Reduce_scatter: 3: 11732-35932 & 3-8
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 3-8
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 9-16
[0] MPI startup(): Reduce_scatter: 1: 4-2807 & 9-16
[0] MPI startup(): Reduce_scatter: 3: 2807-6883 & 9-16
[0] MPI startup(): Reduce_scatter: 5: 6883-39653 & 9-16
[0] MPI startup(): Reduce_scatter: 2: 39653-513184 & 9-16
[0] MPI startup(): Reduce_scatter: 5: 513184-966175 & 9-16
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 9-16
[0] MPI startup(): Reduce_scatter: 3: 0-0 & 17-32
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 17-32
[0] MPI startup(): Reduce_scatter: 1: 4-14959 & 17-32
[0] MPI startup(): Reduce_scatter: 5: 14959-19362 & 17-32
[0] MPI startup(): Reduce_scatter: 1: 19362-49970 & 17-32
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 17-32
[0] MPI startup(): Reduce_scatter: 3: 0-0 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 1: 4-79521 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 2: 79521-520875 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 5: 520875-1101846 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Reduce: 1: 0-2147483647 & 0-2147483647
[0] MPI startup(): Scan: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Scatter: 3: 1-8 & 0-4
[0] MPI startup(): Scatter: 1: 9-31 & 0-4
[0] MPI startup(): Scatter: 3: 32-4867 & 0-4
[0] MPI startup(): Scatter: 2: 4868-18877 & 0-4
[0] MPI startup(): Scatter: 3: 18878-854395 & 0-4
[0] MPI startup(): Scatter: 2: 854396-3121304 & 0-4
[0] MPI startup(): Scatter: 3: 0-2147483647 & 0-4
[0] MPI startup(): Scatter: 3: 1-1 & 5-16
[0] MPI startup(): Scatter: 2: 2-3 & 5-16
[0] MPI startup(): Scatter: 3: 4-8 & 5-16
[0] MPI startup(): Scatter: 1: 9-31 & 5-16
[0] MPI startup(): Scatter: 3: 32-1902 & 5-16
[0] MPI startup(): Scatter: 2: 1903-4895 & 5-16
[0] MPI startup(): Scatter: 3: 4896-27566 & 5-16
[0] MPI startup(): Scatter: 2: 27567-39885 & 5-16
[0] MPI startup(): Scatter: 3: 39886-126420 & 5-16
[0] MPI startup(): Scatter: 2: 126421-1096865 & 5-16
[0] MPI startup(): Scatter: 3: 0-2147483647 & 5-16
[0] MPI startup(): Scatter: 3: 1-71026 & 17-32
[0] MPI startup(): Scatter: 2: 71027-339781 & 17-32
[0] MPI startup(): Scatter: 3: 339782-1290592 & 17-32
[0] MPI startup(): Scatter: 2: 1290593-3878829 & 17-32
[0] MPI startup(): Scatter: 3: 0-2147483647 & 17-32
[0] MPI startup(): Scatter: 3: 1-1 & 33-2147483647
[0] MPI startup(): Scatter: 2: 2-3 & 33-2147483647
[0] MPI startup(): Scatter: 3: 4-8 & 33-2147483647
[0] MPI startup(): Scatter: 1: 9-31 & 33-2147483647
[0] MPI startup(): Scatter: 3: 32-1902 & 33-2147483647
[0] MPI startup(): Scatter: 2: 1903-4895 & 33-2147483647
[0] MPI startup(): Scatter: 3: 4896-27566 & 33-2147483647
[0] MPI startup(): Scatter: 2: 27567-39885 & 33-2147483647
[0] MPI startup(): Scatter: 3: 39886-126420 & 33-2147483647
[0] MPI startup(): Scatter: 2: 126421-1096865 & 33-2147483647
[0] MPI startup(): Scatter: 3: 0-2147483647 & 33-2147483647
[0] MPI startup(): Scatterv: 2: 0-2147483647 & 0-2147483647
[0] MPI startup(): Rank    Pid      Node name  Pin cpu
[0] MPI startup(): 0       6281     e5-2670-1  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
[0] MPI startup(): 1       1972     e5-2670-2  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
[0] MPI startup(): Recognition=2 Platform(code=8 ippn=0 dev=8) Fabric(intra=1 inter=2 flags=0x0)
[0] MPI startup(): I_MPI_DEBUG=6
[0] MPI startup(): I_MPI_FABRICS=shm:ofa
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,21,10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=mlx4_0:1,mlx5_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=2
[0] MPI startup(): I_MPI_OFA_ADAPTER_NAME=mlx5_0
[0] MPI startup(): I_MPI_OFA_NUM_PORTS=2
[0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0
[0] MPI startup(): PMI_RANK=0
# OSU MPI Bandwidth Test v4.3
# Size      Bandwidth (MB/s)
1                       0.99
2                       2.02
4                       4.06
8                       8.28
16                     16.53
32                     31.99
64                     63.87
128                   128.85
256                   254.65
512                   495.51
1024                  949.66
2048                 1720.14
4096                 3043.21
8192                 4893.90
16384                5793.38
32768                7280.63
65536               10894.34
131072              11713.48
262144              12230.60
524288              12462.42
1048576             12613.03
2097152             12653.87
4194304             12701.80

Version 5.0 Update 3 doesn't run:

mpirun -env I_MPI_DEBUG 6 -hosts node1,node2 -np 2 -ppn 1 -genv I_MPI_OFA_ADAPTER_NAME mlx5_0 -genv I_MPI_OFA_NUM_PORTS 2 -genv I_MPI_FABRICS shm:ofa -genv MV2_USE_APM 0 /exports/bin/osu-micro-benchmarks-4.3_impi/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw
[0] MPI startup(): Intel(R) MPI Library, Version 5.0 Update 3  Build 20150128 (build id: 11250)
[0] MPI startup(): Copyright (C) 2003-2015 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded debug library
[0] MPI startup(): Found 2 IB devices
[1] MPI startup(): Found 2 IB devices
[1] MPI startup(): Open 0 IB device: mlx5_0
[0] MPI startup(): Open 0 IB device: mlx5_0
[0] MPI startup(): Start 2 ports per adapter
[1] MPI startup(): Start 2 ports per adapter
[0] MPI startup(): ofa fabric is not available and fallback fabric is not enabled
[1] MPI startup(): ofa fabric is not available and fallback fabric is not enabled

Single-rail with OFA is not a problem for both versions:

$ mpirun -env I_MPI_DEBUG 6 -hosts node1,node2 -np 2 -ppn 1 -genv I_MPI_OFA_ADAPTER_NAME mlx5_0 -genv I_MPI_OFA_NUM_PORTS 1 -genv I_MPI_FABRICS shm:ofa -genv MV2_USE_APM 0 /exports/bin/osu-micro-benchmarks-4.3_impi/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw
[0] MPI startup(): Intel(R) MPI Library, Version 5.0 Update 2  Build 20141030 (build id: 10994)
[0] MPI startup(): Copyright (C) 2003-2014 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded debug library
[0] MPI startup(): Found 2 IB devices
[1] MPI startup(): Found 2 IB devices
[0] MPI startup(): Open 0 IB device: mlx5_0
[1] MPI startup(): Open 0 IB device: mlx5_0
[0] MPI startup(): Start 1 ports per adapter
[1] MPI startup(): Start 1 ports per adapter
[0] MPID_nem_ofacm_init(): Init
[1] MPID_nem_ofacm_init(): Init
[0] MPI startup(): shm and ofa data transfer modes
[1] MPI startup(): shm and ofa data transfer modes
[0] MPI startup(): Device_reset_idx=3
[0] MPI startup(): Allgather: 3: 0-1 & 0-4
[0] MPI startup(): Allgather: 1: 1-1 & 0-4
[0] MPI startup(): Allgather: 4: 2-4 & 0-4
[0] MPI startup(): Allgather: 1: 5-92 & 0-4
[0] MPI startup(): Allgather: 4: 93-161 & 0-4
[0] MPI startup(): Allgather: 1: 162-33532 & 0-4
[0] MPI startup(): Allgather: 4: 33533-102618 & 0-4
[0] MPI startup(): Allgather: 1: 102619-546855 & 0-4
[0] MPI startup(): Allgather: 4: 0-2147483647 & 0-4
[0] MPI startup(): Allgather: 4: 1-839 & 5-8
[0] MPI startup(): Allgather: 2: 840-4170 & 5-8
[0] MPI startup(): Allgather: 4: 0-2147483647 & 5-8
[0] MPI startup(): Allgather: 1: 1-5 & 9-16
[0] MPI startup(): Allgather: 4: 6-8 & 9-16
[0] MPI startup(): Allgather: 1: 9-440 & 9-16
[0] MPI startup(): Allgather: 2: 441-2083 & 9-16
[0] MPI startup(): Allgather: 4: 2084-9050 & 9-16
[0] MPI startup(): Allgather: 1: 9050-65530 & 9-16
[0] MPI startup(): Allgather: 4: 65530-218986 & 9-16
[0] MPI startup(): Allgather: 1: 0-2147483647 & 9-16
[0] MPI startup(): Allgather: 1: 1-247 & 17-32
[0] MPI startup(): Allgather: 2: 248-1043 & 17-32
[0] MPI startup(): Allgather: 4: 1044-3132 & 17-32
[0] MPI startup(): Allgather: 1: 3133-3010585 & 17-32
[0] MPI startup(): Allgather: 3: 0-2147483647 & 17-32
[0] MPI startup(): Allgather: 1: 1-176 & 33-2147483647
[0] MPI startup(): Allgather: 4: 177-1818 & 33-2147483647
[0] MPI startup(): Allgather: 1: 1819-2278 & 33-2147483647
[0] MPI startup(): Allgather: 4: 2279-4129 & 33-2147483647
[0] MPI startup(): Allgather: 1: 4130-1555968 & 33-2147483647
[0] MPI startup(): Allgather: 4: 0-2147483647 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 0-261835 & 0-2
[0] MPI startup(): Allgatherv: 4: 261835-611669 & 0-2
[0] MPI startup(): Allgatherv: 3: 611669-3196878 & 0-2
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 0-2
[0] MPI startup(): Allgatherv: 1: 0-370 & 3-4
[0] MPI startup(): Allgatherv: 2: 370-838 & 3-4
[0] MPI startup(): Allgatherv: 1: 838-3403 & 3-4
[0] MPI startup(): Allgatherv: 4: 3403-4808 & 3-4
[0] MPI startup(): Allgatherv: 1: 4808-8294 & 3-4
[0] MPI startup(): Allgatherv: 3: 8294-130946 & 3-4
[0] MPI startup(): Allgatherv: 4: 130946-208466 & 3-4
[0] MPI startup(): Allgatherv: 3: 208466-393216 & 3-4
[0] MPI startup(): Allgatherv: 4: 393216-617063 & 3-4
[0] MPI startup(): Allgatherv: 3: 617063-1398101 & 3-4
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 3-4
[0] MPI startup(): Allgatherv: 1: 0-25 & 5-8
[0] MPI startup(): Allgatherv: 4: 25-46 & 5-8
[0] MPI startup(): Allgatherv: 1: 46-1403 & 5-8
[0] MPI startup(): Allgatherv: 2: 1403-4296 & 5-8
[0] MPI startup(): Allgatherv: 3: 4296-88097 & 5-8
[0] MPI startup(): Allgatherv: 4: 88097-345405 & 5-8
[0] MPI startup(): Allgatherv: 3: 345405-2801646 & 5-8
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 5-8
[0] MPI startup(): Allgatherv: 4: 0-3 & 9-16
[0] MPI startup(): Allgatherv: 1: 3-23 & 9-16
[0] MPI startup(): Allgatherv: 4: 23-106 & 9-16
[0] MPI startup(): Allgatherv: 1: 106-797 & 9-16
[0] MPI startup(): Allgatherv: 2: 797-3141 & 9-16
[0] MPI startup(): Allgatherv: 3: 3141-5874 & 9-16
[0] MPI startup(): Allgatherv: 2: 5874-9274 & 9-16
[0] MPI startup(): Allgatherv: 1: 9274-17847 & 9-16
[0] MPI startup(): Allgatherv: 3: 17847-84012 & 9-16
[0] MPI startup(): Allgatherv: 4: 84012-225929 & 9-16
[0] MPI startup(): Allgatherv: 3: 225929-450203 & 9-16
[0] MPI startup(): Allgatherv: 4: 450203-545471 & 9-16
[0] MPI startup(): Allgatherv: 3: 545471-1767416 & 9-16
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 9-16
[0] MPI startup(): Allgatherv: 1: 0-25 & 17-32
[0] MPI startup(): Allgatherv: 4: 25-46 & 17-32
[0] MPI startup(): Allgatherv: 1: 46-1403 & 17-32
[0] MPI startup(): Allgatherv: 2: 1403-4296 & 17-32
[0] MPI startup(): Allgatherv: 3: 4296-88097 & 17-32
[0] MPI startup(): Allgatherv: 4: 88097-345405 & 17-32
[0] MPI startup(): Allgatherv: 3: 345405-2801646 & 17-32
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 17-32
[0] MPI startup(): Allgatherv: 4: 0-1 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 1-3 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 3-6 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 6-96 & 33-2147483647
[0] MPI startup(): Allgatherv: 2: 96-535 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 535-1255 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 1255-3148 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 3148-5455 & 33-2147483647
[0] MPI startup(): Allgatherv: 2: 5455-17721 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 17721-41571 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 41571-110427 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 110427-596874 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 596874-1273594 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 1273594-2420103 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 0-2147483647 & 33-2147483647
[0] MPI startup(): Allreduce: 7: 0-20 & 0-2
[0] MPI startup(): Allreduce: 1: 20-905 & 0-2
[0] MPI startup(): Allreduce: 4: 905-2058 & 0-2
[0] MPI startup(): Allreduce: 1: 2058-17240 & 0-2
[0] MPI startup(): Allreduce: 8: 17240-36706 & 0-2
[0] MPI startup(): Allreduce: 3: 36706-777395 & 0-2
[0] MPI startup(): Allreduce: 8: 777395-1247501 & 0-2
[0] MPI startup(): Allreduce: 2: 1247501-3149905 & 0-2
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 0-2
[0] MPI startup(): Allreduce: 1: 0-597 & 3-4
[0] MPI startup(): Allreduce: 7: 597-1755 & 3-4
[0] MPI startup(): Allreduce: 1: 1755-16932 & 3-4
[0] MPI startup(): Allreduce: 6: 16932-34438 & 3-4
[0] MPI startup(): Allreduce: 8: 34438-81725 & 3-4
[0] MPI startup(): Allreduce: 3: 81725-396925 & 3-4
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 3-4
[0] MPI startup(): Allreduce: 1: 0-55 & 5-8
[0] MPI startup(): Allreduce: 6: 55-74 & 5-8
[0] MPI startup(): Allreduce: 4: 74-213 & 5-8
[0] MPI startup(): Allreduce: 1: 213-1765 & 5-8
[0] MPI startup(): Allreduce: 4: 1765-2065 & 5-8
[0] MPI startup(): Allreduce: 1: 2065-16561 & 5-8
[0] MPI startup(): Allreduce: 2: 16561-39707 & 5-8
[0] MPI startup(): Allreduce: 8: 39707-168698 & 5-8
[0] MPI startup(): Allreduce: 7: 168698-570628 & 5-8
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 5-8
[0] MPI startup(): Allreduce: 6: 0-5 & 9-16
[0] MPI startup(): Allreduce: 1: 5-935 & 9-16
[0] MPI startup(): Allreduce: 4: 935-1105 & 9-16
[0] MPI startup(): Allreduce: 1: 1105-17349 & 9-16
[0] MPI startup(): Allreduce: 4: 17349-53440 & 9-16
[0] MPI startup(): Allreduce: 8: 53440-315615 & 9-16
[0] MPI startup(): Allreduce: 7: 315615-1140547 & 9-16
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 9-16
[0] MPI startup(): Allreduce: 6: 0-74 & 17-32
[0] MPI startup(): Allreduce: 4: 74-199 & 17-32
[0] MPI startup(): Allreduce: 6: 199-392 & 17-32
[0] MPI startup(): Allreduce: 4: 392-750 & 17-32
[0] MPI startup(): Allreduce: 6: 750-2087 & 17-32
[0] MPI startup(): Allreduce: 1: 2087-14841 & 17-32
[0] MPI startup(): Allreduce: 2: 14841-20338 & 17-32
[0] MPI startup(): Allreduce: 6: 20338-124787 & 17-32
[0] MPI startup(): Allreduce: 8: 124787-646663 & 17-32
[0] MPI startup(): Allreduce: 7: 646663-2129006 & 17-32
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 17-32
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 33-2147483647
[1] MPI startup(): Recognition=2 Platform(code=8 ippn=0 dev=8) Fabric(intra=1 inter=2 flags=0x0)
[0] MPI startup(): Alltoall: 3: 0-245 & 0-2
[0] MPI startup(): Alltoall: 2: 245-276 & 0-2
[0] MPI startup(): Alltoall: 3: 0-2147483647 & 0-2
[0] MPI startup(): Alltoall: 2: 0-3878 & 3-4
[0] MPI startup(): Alltoall: 3: 3879-6492 & 3-4
[0] MPI startup(): Alltoall: 4: 6493-9680 & 3-4
[0] MPI startup(): Alltoall: 3: 9681-24681 & 3-4
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 3-4
[0] MPI startup(): Alltoall: 2: 0-3082 & 5-8
[0] MPI startup(): Alltoall: 3: 3083-20124 & 5-8
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 5-8
[0] MPI startup(): Alltoall: 2: 0-2235 & 9-16
[0] MPI startup(): Alltoall: 3: 2236-26946 & 9-16
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 9-16
[0] MPI startup(): Alltoall: 1: 0-329 & 17-32
[0] MPI startup(): Alltoall: 2: 330-4661 & 17-32
[0] MPI startup(): Alltoall: 3: 4662-282206 & 17-32
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 17-32
[0] MPI startup(): Alltoall: 1: 0-217 & 33-2147483647
[0] MPI startup(): Alltoall: 4: 218-409 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 410-4415 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 4416-32356 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 32357-36297 & 33-2147483647
[0] MPI startup(): Alltoall: 4: 36298-65861 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 65862-866313 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Alltoallv: 1: 0-2147483647 & 0-2147483647
[0] MPI startup(): Alltoallw: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Barrier: 6: 0-2147483647 & 0-2147483647
[0] MPI startup(): Bcast: 2: 0-1 & 0-2
[0] MPI startup(): Bcast: 7: 1-402 & 0-2
[0] MPI startup(): Bcast: 5: 402-682 & 0-2
[0] MPI startup(): Bcast: 4: 682-1433 & 0-2
[0] MPI startup(): Bcast: 7: 1433-5734 & 0-2
[0] MPI startup(): Bcast: 1: 5734-21845 & 0-2
[0] MPI startup(): Bcast: 6: 21845-95963 & 0-2
[0] MPI startup(): Bcast: 5: 95963-409897 & 0-2
[0] MPI startup(): Bcast: 1: 0-2147483647 & 0-2
[0] MPI startup(): Bcast: 7: 0-2147483647 & 3-4
[0] MPI startup(): Bcast: 7: 1-1 & 5-8
[0] MPI startup(): Bcast: 1: 2-254 & 5-8
[0] MPI startup(): Bcast: 5: 255-261 & 5-8
[0] MPI startup(): Bcast: 1: 262-752 & 5-8
[0] MPI startup(): Bcast: 6: 753-1254 & 5-8
[0] MPI startup(): Bcast: 1: 1255-4032 & 5-8
[0] MPI startup(): Bcast: 4: 4033-19764 & 5-8
[0] MPI startup(): Bcast: 1: 19765-121346 & 5-8
[0] MPI startup(): Bcast: 6: 121347-201114 & 5-8
[0] MPI startup(): Bcast: 1: 201115-267790 & 5-8
[0] MPI startup(): Bcast: 2: 267791-588703 & 5-8
[0] MPI startup(): Bcast: 3: 588704-1341590 & 5-8
[0] MPI startup(): Bcast: 5: 1341591-2139116 & 5-8
[0] MPI startup(): Bcast: 6: 0-2147483647 & 5-8
[0] MPI startup(): Bcast: 7: 1-84 & 9-16
[0] MPI startup(): Bcast: 1: 85-137 & 9-16
[0] MPI startup(): Bcast: 6: 138-475 & 9-16
[0] MPI startup(): Bcast: 1: 476-1496 & 9-16
[0] MPI startup(): Bcast: 6: 1497-10273 & 9-16
[0] MPI startup(): Bcast: 1: 10274-106199 & 9-16
[0] MPI startup(): Bcast: 6: 106200-323972 & 9-16
[0] MPI startup(): Bcast: 2: 0-2147483647 & 9-16
[0] MPI startup(): Bcast: 7: 1-103 & 17-32
[0] MPI startup(): Bcast: 5: 104-180 & 17-32
[0] MPI startup(): Bcast: 7: 181-590 & 17-32
[0] MPI startup(): Bcast: 5: 591-1170 & 17-32
[0] MPI startup(): Bcast: 6: 1171-3459 & 17-32
[0] MPI startup(): Bcast: 1: 3460-52532 & 17-32
[0] MPI startup(): Bcast: 2: 52533-239896 & 17-32
[0] MPI startup(): Bcast: 5: 239897-262474 & 17-32
[0] MPI startup(): Bcast: 2: 0-2147483647 & 17-32
[0] MPI startup(): Bcast: 7: 1-935 & 33-2147483647
[0] MPI startup(): Bcast: 6: 936-1185 & 33-2147483647
[0] MPI startup(): Bcast: 1: 1186-4873 & 33-2147483647
[0] MPI startup(): Bcast: 6: 4874-9664 & 33-2147483647
[0] MPI startup(): Bcast: 1: 9665-85765 & 33-2147483647
[0] MPI startup(): Bcast: 2: 85766-250963 & 33-2147483647
[0] MPI startup(): Bcast: 5: 250964-262409 & 33-2147483647
[0] MPI startup(): Bcast: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Exscan: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Gather: 3: 1-8 & 0-4
[0] MPI startup(): Gather: 1: 9-29 & 0-4
[0] MPI startup(): Gather: 3: 30-25688 & 0-4
[0] MPI startup(): Gather: 2: 25689-56274 & 0-4
[0] MPI startup(): Gather: 3: 56275-68784 & 0-4
[0] MPI startup(): Gather: 2: 68785-1304326 & 0-4
[0] MPI startup(): Gather: 3: 1304327-2684607 & 0-4
[0] MPI startup(): Gather: 2: 0-2147483647 & 0-4
[0] MPI startup(): Gather: 3: 1-10278 & 5-8
[0] MPI startup(): Gather: 2: 10279-22432 & 5-8
[0] MPI startup(): Gather: 3: 22433-51123 & 5-8
[0] MPI startup(): Gather: 2: 51124-93434 & 5-8
[0] MPI startup(): Gather: 3: 93435-667907 & 5-8
[0] MPI startup(): Gather: 2: 667908-1262190 & 5-8
[0] MPI startup(): Gather: 3: 1262191-2879271 & 5-8
[0] MPI startup(): Gather: 2: 0-2147483647 & 5-8
[0] MPI startup(): Gather: 3: 1-4004 & 9-16
[0] MPI startup(): Gather: 2: 4005-4168 & 9-16
[0] MPI startup(): Gather: 3: 4169-20254 & 9-16
[0] MPI startup(): Gather: 2: 20255-41713 & 9-16
[0] MPI startup(): Gather: 3: 41714-793099 & 9-16
[0] MPI startup(): Gather: 2: 793100-1272717 & 9-16
[0] MPI startup(): Gather: 3: 0-2147483647 & 9-16
[0] MPI startup(): Gather: 3: 1-2047 & 17-32
[0] MPI startup(): Gather: 2: 2048-16131 & 17-32
[0] MPI startup(): Gather: 3: 16132-25688 & 17-32
[0] MPI startup(): Gather: 2: 25689-56274 & 17-32
[0] MPI startup(): Gather: 3: 56275-68784 & 17-32
[0] MPI startup(): Gather: 2: 68785-1304326 & 17-32
[0] MPI startup(): Gather: 3: 1304327-2684607 & 17-32
[0] MPI startup(): Gather: 2: 0-2147483647 & 17-32
[0] MPI startup(): Gather: 3: 1-51845 & 33-2147483647
[0] MPI startup(): Gather: 2: 51846-76744 & 33-2147483647
[0] MPI startup(): Gather: 3: 76745-721936 & 33-2147483647
[0] MPI startup(): Gather: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Gatherv: 2: 0-2147483647 & 0-2147483647
[0] MPI startup(): Reduce_scatter: 4: 0-5 & 0-2
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 0-2
[0] MPI startup(): Reduce_scatter: 1: 0-29 & 3-8
[0] MPI startup(): Reduce_scatter: 5: 29-48 & 3-8
[0] MPI startup(): Reduce_scatter: 1: 48-2805 & 3-8
[0] MPI startup(): Reduce_scatter: 3: 2805-5556 & 3-8
[0] MPI startup(): Reduce_scatter: 5: 5556-11732 & 3-8
[0] MPI startup(): Reduce_scatter: 3: 11732-35932 & 3-8
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 3-8
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 9-16
[0] MPI startup(): Reduce_scatter: 1: 4-2807 & 9-16
[0] MPI startup(): Reduce_scatter: 3: 2807-6883 & 9-16
[0] MPI startup(): Reduce_scatter: 5: 6883-39653 & 9-16
[0] MPI startup(): Reduce_scatter: 2: 39653-513184 & 9-16
[0] MPI startup(): Reduce_scatter: 5: 513184-966175 & 9-16
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 9-16
[0] MPI startup(): Reduce_scatter: 3: 0-0 & 17-32
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 17-32
[0] MPI startup(): Reduce_scatter: 1: 4-14959 & 17-32
[0] MPI startup(): Reduce_scatter: 5: 14959-19362 & 17-32
[0] MPI startup(): Reduce_scatter: 1: 19362-49970 & 17-32
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 17-32
[0] MPI startup(): Reduce_scatter: 3: 0-0 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 1: 4-79521 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 2: 79521-520875 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 5: 520875-1101846 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Reduce: 1: 0-2147483647 & 0-2147483647
[0] MPI startup(): Scan: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Scatter: 3: 1-8 & 0-4
[0] MPI startup(): Scatter: 1: 9-31 & 0-4
[0] MPI startup(): Scatter: 3: 32-4867 & 0-4
[0] MPI startup(): Scatter: 2: 4868-18877 & 0-4
[0] MPI startup(): Scatter: 3: 18878-854395 & 0-4
[0] MPI startup(): Scatter: 2: 854396-3121304 & 0-4
[0] MPI startup(): Scatter: 3: 0-2147483647 & 0-4
[0] MPI startup(): Scatter: 3: 1-1 & 5-16
[0] MPI startup(): Scatter: 2: 2-3 & 5-16
[0] MPI startup(): Scatter: 3: 4-8 & 5-16
[0] MPI startup(): Scatter: 1: 9-31 & 5-16
[0] MPI startup(): Scatter: 3: 32-1902 & 5-16
[0] MPI startup(): Scatter: 2: 1903-4895 & 5-16
[0] MPI startup(): Scatter: 3: 4896-27566 & 5-16
[0] MPI startup(): Scatter: 2: 27567-39885 & 5-16
[0] MPI startup(): Scatter: 3: 39886-126420 & 5-16
[0] MPI startup(): Scatter: 2: 126421-1096865 & 5-16
[0] MPI startup(): Scatter: 3: 0-2147483647 & 5-16
[0] MPI startup(): Scatter: 3: 1-71026 & 17-32
[0] MPI startup(): Scatter: 2: 71027-339781 & 17-32
[0] MPI startup(): Scatter: 3: 339782-1290592 & 17-32
[0] MPI startup(): Scatter: 2: 1290593-3878829 & 17-32
[0] MPI startup(): Scatter: 3: 0-2147483647 & 17-32
[0] MPI startup(): Scatter: 3: 1-1 & 33-2147483647
[0] MPI startup(): Scatter: 2: 2-3 & 33-2147483647
[0] MPI startup(): Scatter: 3: 4-8 & 33-2147483647
[0] MPI startup(): Scatter: 1: 9-31 & 33-2147483647
[0] MPI startup(): Scatter: 3: 32-1902 & 33-2147483647
[0] MPI startup(): Scatter: 2: 1903-4895 & 33-2147483647
[0] MPI startup(): Scatter: 3: 4896-27566 & 33-2147483647
[0] MPI startup(): Scatter: 2: 27567-39885 & 33-2147483647
[0] MPI startup(): Scatter: 3: 39886-126420 & 33-2147483647
[0] MPI startup(): Scatter: 2: 126421-1096865 & 33-2147483647
[0] MPI startup(): Scatter: 3: 0-2147483647 & 33-2147483647
[0] MPI startup(): Scatterv: 2: 0-2147483647 & 0-2147483647
[0] MPI startup(): Rank    Pid      Node name  Pin cpu
[0] MPI startup(): 0       6303     e5-2670-1  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
[0] MPI startup(): 1       2013     e5-2670-2  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
[0] MPI startup(): Recognition=2 Platform(code=8 ippn=0 dev=8) Fabric(intra=1 inter=2 flags=0x0)
[0] MPI startup(): I_MPI_DEBUG=6
[0] MPI startup(): I_MPI_FABRICS=shm:ofa
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,21,10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=mlx4_0:1,mlx5_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=2
[0] MPI startup(): I_MPI_OFA_ADAPTER_NAME=mlx5_0
[0] MPI startup(): I_MPI_OFA_NUM_PORTS=1
[0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0
[0] MPI startup(): PMI_RANK=0
# OSU MPI Bandwidth Test v4.3
# Size      Bandwidth (MB/s)
1                       0.97
2                       1.98
4                       4.06
8                       8.06
16                     16.28
32                     31.57
64                     62.85
128                   126.21
256                   250.36
512                   489.73
1024                  939.56
2048                 1705.57
4096                 3024.29
8192                 4840.15
16384                5495.84
32768                5529.70
65536                5952.25
131072               6190.62
262144               6300.36
524288               6362.90
1048576              6390.01
2097152              6405.34
4194304              6414.40

$ mpirun -env I_MPI_DEBUG 6 -hosts node1,node2 -np 2 -ppn 1 -genv I_MPI_OFA_ADAPTER_NAME mlx5_0 -genv I_MPI_OFA_NUM_PORTS 1 -genv I_MPI_FABRICS shm:ofa -genv MV2_USE_APM 0 /exports/bin/osu-micro-benchmarks-4.3_impi/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw
[0] MPI startup(): Intel(R) MPI Library, Version 5.0 Update 3  Build 20150128 (build id: 11250)
[0] MPI startup(): Copyright (C) 2003-2015 Intel Corporation.  All rights reserved.
[0] MPI startup(): Multi-threaded debug library
[0] MPI startup(): Found 2 IB devices
[1] MPI startup(): Found 2 IB devices
[0] MPI startup(): Open 0 IB device: mlx5_0
[1] MPI startup(): Open 0 IB device: mlx5_0
[0] MPI startup(): Start 1 ports per adapter
[1] MPI startup(): Start 1 ports per adapter
[0] MPID_nem_ofacm_init(): Init
[1] MPID_nem_ofacm_init(): Init
[0] MPI startup(): shm and ofa data transfer modes
[1] MPI startup(): shm and ofa data transfer modes
[0] MPI startup(): Device_reset_idx=3
[0] MPI startup(): Allgather: 3: 0-1 & 0-4
[0] MPI startup(): Allgather: 1: 1-1 & 0-4
[0] MPI startup(): Allgather: 4: 2-4 & 0-4
[0] MPI startup(): Allgather: 1: 5-92 & 0-4
[0] MPI startup(): Allgather: 4: 93-161 & 0-4
[0] MPI startup(): Allgather: 1: 162-33532 & 0-4
[0] MPI startup(): Allgather: 4: 33533-102618 & 0-4
[0] MPI startup(): Allgather: 1: 102619-546855 & 0-4
[0] MPI startup(): Allgather: 4: 0-2147483647 & 0-4
[0] MPI startup(): Allgather: 4: 1-839 & 5-8
[0] MPI startup(): Allgather: 2: 840-4170 & 5-8
[0] MPI startup(): Allgather: 4: 0-2147483647 & 5-8
[0] MPI startup(): Allgather: 1: 1-5 & 9-16
[0] MPI startup(): Allgather: 4: 6-8 & 9-16
[0] MPI startup(): Allgather: 1: 9-440 & 9-16
[0] MPI startup(): Allgather: 2: 441-2083 & 9-16
[0] MPI startup(): Allgather: 4: 2084-9050 & 9-16
[0] MPI startup(): Allgather: 1: 9050-65530 & 9-16
[0] MPI startup(): Allgather: 4: 65530-218986 & 9-16
[0] MPI startup(): Allgather: 1: 0-2147483647 & 9-16
[0] MPI startup(): Allgather: 1: 1-247 & 17-32
[0] MPI startup(): Allgather: 2: 248-1043 & 17-32
[0] MPI startup(): Allgather: 4: 1044-3132 & 17-32
[0] MPI startup(): Allgather: 1: 3133-3010585 & 17-32
[0] MPI startup(): Allgather: 3: 0-2147483647 & 17-32
[0] MPI startup(): Allgather: 1: 1-176 & 33-2147483647
[0] MPI startup(): Allgather: 4: 177-1818 & 33-2147483647
[0] MPI startup(): Allgather: 1: 1819-2278 & 33-2147483647
[0] MPI startup(): Allgather: 4: 2279-4129 & 33-2147483647
[0] MPI startup(): Allgather: 1: 4130-1555968 & 33-2147483647
[0] MPI startup(): Allgather: 4: 0-2147483647 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 0-261835 & 0-2
[0] MPI startup(): Allgatherv: 4: 261835-611669 & 0-2
[0] MPI startup(): Allgatherv: 3: 611669-3196878 & 0-2
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 0-2
[0] MPI startup(): Allgatherv: 1: 0-370 & 3-4
[0] MPI startup(): Allgatherv: 2: 370-838 & 3-4
[0] MPI startup(): Allgatherv: 1: 838-3403 & 3-4
[0] MPI startup(): Allgatherv: 4: 3403-4808 & 3-4
[0] MPI startup(): Allgatherv: 1: 4808-8294 & 3-4
[0] MPI startup(): Allgatherv: 3: 8294-130946 & 3-4
[0] MPI startup(): Allgatherv: 4: 130946-208466 & 3-4
[0] MPI startup(): Allgatherv: 3: 208466-393216 & 3-4
[0] MPI startup(): Allgatherv: 4: 393216-617063 & 3-4
[0] MPI startup(): Allgatherv: 3: 617063-1398101 & 3-4
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 3-4
[0] MPI startup(): Allgatherv: 1: 0-25 & 5-8
[0] MPI startup(): Allgatherv: 4: 25-46 & 5-8
[0] MPI startup(): Allgatherv: 1: 46-1403 & 5-8
[0] MPI startup(): Allgatherv: 2: 1403-4296 & 5-8
[0] MPI startup(): Allgatherv: 3: 4296-88097 & 5-8
[0] MPI startup(): Allgatherv: 4: 88097-345405 & 5-8
[0] MPI startup(): Allgatherv: 3: 345405-2801646 & 5-8
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 5-8
[0] MPI startup(): Allgatherv: 4: 0-3 & 9-16
[0] MPI startup(): Allgatherv: 1: 3-23 & 9-16
[0] MPI startup(): Allgatherv: 4: 23-106 & 9-16
[0] MPI startup(): Allgatherv: 1: 106-797 & 9-16
[0] MPI startup(): Allgatherv: 2: 797-3141 & 9-16
[0] MPI startup(): Allgatherv: 3: 3141-5874 & 9-16
[0] MPI startup(): Allgatherv: 2: 5874-9274 & 9-16
[0] MPI startup(): Allgatherv: 1: 9274-17847 & 9-16
[0] MPI startup(): Allgatherv: 3: 17847-84012 & 9-16
[0] MPI startup(): Allgatherv: 4: 84012-225929 & 9-16
[0] MPI startup(): Allgatherv: 3: 225929-450203 & 9-16
[0] MPI startup(): Allgatherv: 4: 450203-545471 & 9-16
[0] MPI startup(): Allgatherv: 3: 545471-1767416 & 9-16
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 9-16
[0] MPI startup(): Allgatherv: 1: 0-25 & 17-32
[0] MPI startup(): Allgatherv: 4: 25-46 & 17-32
[0] MPI startup(): Allgatherv: 1: 46-1403 & 17-32
[0] MPI startup(): Allgatherv: 2: 1403-4296 & 17-32
[0] MPI startup(): Allgatherv: 3: 4296-88097 & 17-32
[0] MPI startup(): Allgatherv: 4: 88097-345405 & 17-32
[0] MPI startup(): Allgatherv: 3: 345405-2801646 & 17-32
[0] MPI startup(): Allgatherv: 4: 0-2147483647 & 17-32
[0] MPI startup(): Allgatherv: 4: 0-1 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 1-3 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 3-6 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 6-96 & 33-2147483647
[0] MPI startup(): Allgatherv: 2: 96-535 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 535-1255 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 1255-3148 & 33-2147483647
[0] MPI startup(): Allgatherv: 1: 3148-5455 & 33-2147483647
[0] MPI startup(): Allgatherv: 2: 5455-17721 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 17721-41571 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 41571-110427 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 110427-596874 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 596874-1273594 & 33-2147483647
[0] MPI startup(): Allgatherv: 4: 1273594-2420103 & 33-2147483647
[0] MPI startup(): Allgatherv: 3: 0-2147483647 & 33-2147483647
[0] MPI startup(): Allreduce: 7: 0-20 & 0-2
[0] MPI startup(): Allreduce: 1: 20-905 & 0-2
[0] MPI startup(): Allreduce: 4: 905-2058 & 0-2
[0] MPI startup(): Allreduce: 1: 2058-17240 & 0-2
[0] MPI startup(): Allreduce: 8: 17240-36706 & 0-2
[0] MPI startup(): Allreduce: 3: 36706-777395 & 0-2
[0] MPI startup(): Allreduce: 8: 777395-1247501 & 0-2
[0] MPI startup(): Allreduce: 2: 1247501-3149905 & 0-2
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 0-2
[0] MPI startup(): Allreduce: 1: 0-597 & 3-4
[0] MPI startup(): Allreduce: 7: 597-1755 & 3-4
[0] MPI startup(): Allreduce: 1: 1755-16932 & 3-4
[0] MPI startup(): Allreduce: 6: 16932-34438 & 3-4
[0] MPI startup(): Allreduce: 8: 34438-81725 & 3-4
[0] MPI startup(): Allreduce: 3: 81725-396925 & 3-4
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 3-4
[0] MPI startup(): Allreduce: 1: 0-55 & 5-8
[0] MPI startup(): Allreduce: 6: 55-74 & 5-8
[0] MPI startup(): Allreduce: 4: 74-213 & 5-8
[0] MPI startup(): Allreduce: 1: 213-1765 & 5-8
[0] MPI startup(): Allreduce: 4: 1765-2065 & 5-8
[0] MPI startup(): Allreduce: 1: 2065-16561 & 5-8
[0] MPI startup(): Allreduce: 2: 16561-39707 & 5-8
[0] MPI startup(): Allreduce: 8: 39707-168698 & 5-8
[0] MPI startup(): Allreduce: 7: 168698-570628 & 5-8
[0] MPI startup(): Allreduce: 8: 0-2147483647 & 5-8
[0] MPI startup(): Allreduce: 6: 0-5 & 9-16
[0] MPI startup(): Allreduce: 1: 5-935 & 9-16
[0] MPI startup(): Allreduce: 4: 935-1105 & 9-16
[0] MPI startup(): Allreduce: 1: 1105-17349 & 9-16
[0] MPI startup(): Allreduce: 4: 17349-53440 & 9-16
[0] MPI startup(): Allreduce: 8: 53440-315615 & 9-16
[0] MPI startup(): Allreduce: 7: 315615-1140547 & 9-16
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 9-16
[0] MPI startup(): Allreduce: 6: 0-74 & 17-32
[0] MPI startup(): Allreduce: 4: 74-199 & 17-32
[0] MPI startup(): Allreduce: 6: 199-392 & 17-32
[0] MPI startup(): Allreduce: 4: 392-750 & 17-32
[0] MPI startup(): Allreduce: 6: 750-2087 & 17-32
[0] MPI startup(): Allreduce: 1: 2087-14841 & 17-32
[0] MPI startup(): Allreduce: 2: 14841-20338 & 17-32
[0] MPI startup(): Allreduce: 6: 20338-124787 & 17-32
[0] MPI startup(): Allreduce: 8: 124787-646663 & 17-32
[0] MPI startup(): Allreduce: 7: 646663-2129006 & 17-32
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 17-32
[0] MPI startup(): Allreduce: 4: 0-2147483647 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 0-245 & 0-2
[0] MPI startup(): Alltoall: 2: 245-276 & 0-2
[0] MPI startup(): Alltoall: 3: 0-2147483647 & 0-2
[0] MPI startup(): Alltoall: 2: 0-3878 & 3-4
[0] MPI startup(): Alltoall: 3: 3879-6492 & 3-4
[0] MPI startup(): Alltoall: 4: 6493-9680 & 3-4
[0] MPI startup(): Alltoall: 3: 9681-24681 & 3-4
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 3-4
[0] MPI startup(): Alltoall: 2: 0-3082 & 5-8
[0] MPI startup(): Alltoall: 3: 3083-20124 & 5-8
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 5-8
[0] MPI startup(): Alltoall: 2: 0-2235 & 9-16
[0] MPI startup(): Alltoall: 3: 2236-26946 & 9-16
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 9-16
[0] MPI startup(): Alltoall: 1: 0-329 & 17-32
[0] MPI startup(): Alltoall: 2: 330-4661 & 17-32
[0] MPI startup(): Alltoall: 3: 4662-282206 & 17-32
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 17-32
[0] MPI startup(): Alltoall: 1: 0-217 & 33-2147483647
[0] MPI startup(): Alltoall: 4: 218-409 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 410-4415 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 4416-32356 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 32357-36297 & 33-2147483647
[0] MPI startup(): Alltoall: 4: 36298-65861 & 33-2147483647
[0] MPI startup(): Alltoall: 3: 65862-866313 & 33-2147483647
[0] MPI startup(): Alltoall: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Alltoallv: 1: 0-2147483647 & 0-2147483647
[0] MPI startup(): Alltoallw: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Barrier: 6: 0-2147483647 & 0-2147483647
[0] MPI startup(): Bcast: 2: 0-1 & 0-2
[0] MPI startup(): Bcast: 7: 1-402 & 0-2
[0] MPI startup(): Bcast: 5: 402-682 & 0-2
[0] MPI startup(): Bcast: 4: 682-1433 & 0-2
[0] MPI startup(): Bcast: 7: 1433-5734 & 0-2
[0] MPI startup(): Bcast: 1: 5734-21845 & 0-2
[0] MPI startup(): Bcast: 6: 21845-95963 & 0-2
[0] MPI startup(): Bcast: 5: 95963-409897 & 0-2
[0] MPI startup(): Bcast: 1: 0-2147483647 & 0-2
[0] MPI startup(): Bcast: 7: 0-2147483647 & 3-4
[0] MPI startup(): Bcast: 7: 1-1 & 5-8
[0] MPI startup(): Bcast: 1: 2-254 & 5-8
[0] MPI startup(): Bcast: 5: 255-261 & 5-8
[0] MPI startup(): Bcast: 1: 262-752 & 5-8
[0] MPI startup(): Bcast: 6: 753-1254 & 5-8
[0] MPI startup(): Bcast: 1: 1255-4032 & 5-8
[0] MPI startup(): Bcast: 4: 4033-19764 & 5-8
[0] MPI startup(): Bcast: 1: 19765-121346 & 5-8
[0] MPI startup(): Bcast: 6: 121347-201114 & 5-8
[0] MPI startup(): Bcast: 1: 201115-267790 & 5-8
[0] MPI startup(): Bcast: 2: 267791-588703 & 5-8
[0] MPI startup(): Bcast: 3: 588704-1341590 & 5-8
[0] MPI startup(): Bcast: 5: 1341591-2139116 & 5-8
[0] MPI startup(): Bcast: 6: 0-2147483647 & 5-8
[0] MPI startup(): Bcast: 7: 1-84 & 9-16
[0] MPI startup(): Bcast: 1: 85-137 & 9-16
[0] MPI startup(): Bcast: 6: 138-475 & 9-16
[0] MPI startup(): Bcast: 1: 476-1496 & 9-16
[0] MPI startup(): Bcast: 6: 1497-10273 & 9-16
[0] MPI startup(): Bcast: 1: 10274-106199 & 9-16
[0] MPI startup(): Bcast: 6: 106200-323972 & 9-16
[0] MPI startup(): Bcast: 2: 0-2147483647 & 9-16
[0] MPI startup(): Bcast: 7: 1-103 & 17-32
[0] MPI startup(): Bcast: 5: 104-180 & 17-32
[0] MPI startup(): Bcast: 7: 181-590 & 17-32
[0] MPI startup(): Bcast: 5: 591-1170 & 17-32
[0] MPI startup(): Bcast: 6: 1171-3459 & 17-32
[0] MPI startup(): Bcast: 1: 3460-52532 & 17-32
[0] MPI startup(): Bcast: 2: 52533-239896 & 17-32
[0] MPI startup(): Bcast: 5: 239897-262474 & 17-32
[0] MPI startup(): Bcast: 2: 0-2147483647 & 17-32
[0] MPI startup(): Bcast: 7: 1-935 & 33-2147483647
[0] MPI startup(): Bcast: 6: 936-1185 & 33-2147483647
[0] MPI startup(): Bcast: 1: 1186-4873 & 33-2147483647
[0] MPI startup(): Bcast: 6: 4874-9664 & 33-2147483647
[0] MPI startup(): Bcast: 1: 9665-85765 & 33-2147483647
[0] MPI startup(): Bcast: 2: 85766-250963 & 33-2147483647
[0] MPI startup(): Bcast: 5: 250964-262409 & 33-2147483647
[0] MPI startup(): Bcast: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Exscan: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Gather: 3: 1-8 & 0-4
[0] MPI startup(): Gather: 1: 9-29 & 0-4
[0] MPI startup(): Gather: 3: 30-25688 & 0-4
[0] MPI startup(): Gather: 2: 25689-56274 & 0-4
[0] MPI startup(): Gather: 3: 56275-68784 & 0-4
[0] MPI startup(): Gather: 2: 68785-1304326 & 0-4
[0] MPI startup(): Gather: 3: 1304327-2684607 & 0-4
[0] MPI startup(): Gather: 2: 0-2147483647 & 0-4
[0] MPI startup(): Gather: 3: 1-10278 & 5-8
[0] MPI startup(): Gather: 2: 10279-22432 & 5-8
[0] MPI startup(): Gather: 3: 22433-51123 & 5-8
[0] MPI startup(): Gather: 2: 51124-93434 & 5-8
[0] MPI startup(): Gather: 3: 93435-667907 & 5-8
[0] MPI startup(): Gather: 2: 667908-1262190 & 5-8
[0] MPI startup(): Gather: 3: 1262191-2879271 & 5-8
[0] MPI startup(): Gather: 2: 0-2147483647 & 5-8
[0] MPI startup(): Gather: 3: 1-4004 & 9-16
[0] MPI startup(): Gather: 2: 4005-4168 & 9-16
[0] MPI startup(): Gather: 3: 4169-20254 & 9-16
[0] MPI startup(): Gather: 2: 20255-41713 & 9-16
[0] MPI startup(): Gather: 3: 41714-793099 & 9-16
[0] MPI startup(): Gather: 2: 793100-1272717 & 9-16
[0] MPI startup(): Gather: 3: 0-2147483647 & 9-16
[0] MPI startup(): Gather: 3: 1-2047 & 17-32
[0] MPI startup(): Gather: 2: 2048-16131 & 17-32
[0] MPI startup(): Gather: 3: 16132-25688 & 17-32
[0] MPI startup(): Gather: 2: 25689-56274 & 17-32
[0] MPI startup(): Gather: 3: 56275-68784 & 17-32
[0] MPI startup(): Gather: 2: 68785-1304326 & 17-32
[0] MPI startup(): Gather: 3: 1304327-2684607 & 17-32
[0] MPI startup(): Gather: 2: 0-2147483647 & 17-32
[0] MPI startup(): Gather: 3: 1-51845 & 33-2147483647
[0] MPI startup(): Gather: 2: 51846-76744 & 33-2147483647
[0] MPI startup(): Gather: 3: 76745-721936 & 33-2147483647
[0] MPI startup(): Gather: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Gatherv: 2: 0-2147483647 & 0-2147483647
[0] MPI startup(): Reduce_scatter: 4: 0-5 & 0-2
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 0-2
[0] MPI startup(): Reduce_scatter: 1: 0-29 & 3-8
[0] MPI startup(): Reduce_scatter: 5: 29-48 & 3-8
[0] MPI startup(): Reduce_scatter: 1: 48-2805 & 3-8
[0] MPI startup(): Reduce_scatter: 3: 2805-5556 & 3-8
[0] MPI startup(): Reduce_scatter: 5: 5556-11732 & 3-8
[0] MPI startup(): Reduce_scatter: 3: 11732-35932 & 3-8
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 3-8
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 9-16
[0] MPI startup(): Reduce_scatter: 1: 4-2807 & 9-16
[0] MPI startup(): Reduce_scatter: 3: 2807-6883 & 9-16
[0] MPI startup(): Reduce_scatter: 5: 6883-39653 & 9-16
[0] MPI startup(): Reduce_scatter: 2: 39653-513184 & 9-16
[0] MPI startup(): Reduce_scatter: 5: 513184-966175 & 9-16
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 9-16
[0] MPI startup(): Reduce_scatter: 3: 0-0 & 17-32
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 17-32
[0] MPI startup(): Reduce_scatter: 1: 4-14959 & 17-32
[0] MPI startup(): Reduce_scatter: 5: 14959-19362 & 17-32
[0] MPI startup(): Reduce_scatter: 1: 19362-49970 & 17-32
[1] MPI startup(): Recognition=2 Platform(code=8 ippn=0 dev=8) Fabric(intra=1 inter=2 flags=0x0)
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 17-32
[0] MPI startup(): Reduce_scatter: 3: 0-0 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 4: 0-4 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 1: 4-79521 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 2: 79521-520875 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 5: 520875-1101846 & 33-2147483647
[0] MPI startup(): Reduce_scatter: 2: 0-2147483647 & 33-2147483647
[0] MPI startup(): Reduce: 1: 0-2147483647 & 0-2147483647
[0] MPI startup(): Scan: 0: 0-2147483647 & 0-2147483647
[0] MPI startup(): Scatter: 3: 1-8 & 0-4
[0] MPI startup(): Scatter: 1: 9-31 & 0-4
[0] MPI startup(): Scatter: 3: 32-4867 & 0-4
[0] MPI startup(): Scatter: 2: 4868-18877 & 0-4
[0] MPI startup(): Scatter: 3: 18878-854395 & 0-4
[0] MPI startup(): Scatter: 2: 854396-3121304 & 0-4
[0] MPI startup(): Scatter: 3: 0-2147483647 & 0-4
[0] MPI startup(): Scatter: 3: 1-1 & 5-16
[0] MPI startup(): Scatter: 2: 2-3 & 5-16
[0] MPI startup(): Scatter: 3: 4-8 & 5-16
[0] MPI startup(): Scatter: 1: 9-31 & 5-16
[0] MPI startup(): Scatter: 3: 32-1902 & 5-16
[0] MPI startup(): Scatter: 2: 1903-4895 & 5-16
[0] MPI startup(): Scatter: 3: 4896-27566 & 5-16
[0] MPI startup(): Scatter: 2: 27567-39885 & 5-16
[0] MPI startup(): Scatter: 3: 39886-126420 & 5-16
[0] MPI startup(): Scatter: 2: 126421-1096865 & 5-16
[0] MPI startup(): Scatter: 3: 0-2147483647 & 5-16
[0] MPI startup(): Scatter: 3: 1-71026 & 17-32
[0] MPI startup(): Scatter: 2: 71027-339781 & 17-32
[0] MPI startup(): Scatter: 3: 339782-1290592 & 17-32
[0] MPI startup(): Scatter: 2: 1290593-3878829 & 17-32
[0] MPI startup(): Scatter: 3: 0-2147483647 & 17-32
[0] MPI startup(): Scatter: 3: 1-1 & 33-2147483647
[0] MPI startup(): Scatter: 2: 2-3 & 33-2147483647
[0] MPI startup(): Scatter: 3: 4-8 & 33-2147483647
[0] MPI startup(): Scatter: 1: 9-31 & 33-2147483647
[0] MPI startup(): Scatter: 3: 32-1902 & 33-2147483647
[0] MPI startup(): Scatter: 2: 1903-4895 & 33-2147483647
[0] MPI startup(): Scatter: 3: 4896-27566 & 33-2147483647
[0] MPI startup(): Scatter: 2: 27567-39885 & 33-2147483647
[0] MPI startup(): Scatter: 3: 39886-126420 & 33-2147483647
[0] MPI startup(): Scatter: 2: 126421-1096865 & 33-2147483647
[0] MPI startup(): Scatter: 3: 0-2147483647 & 33-2147483647
[0] MPI startup(): Scatterv: 2: 0-2147483647 & 0-2147483647
[0] MPI startup(): Rank    Pid      Node name  Pin cpu
[0] MPI startup(): 0       6405     e5-2670-1  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
[0] MPI startup(): 1       2108     e5-2670-2  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
[0] MPI startup(): Recognition=2 Platform(code=8 ippn=0 dev=8) Fabric(intra=1 inter=2 flags=0x0)
[0] MPI startup(): I_MPI_DEBUG=6
[0] MPI startup(): I_MPI_FABRICS=shm:ofa
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,21,10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=mlx4_0:1,mlx5_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=2
[0] MPI startup(): I_MPI_OFA_ADAPTER_NAME=mlx5_0
[0] MPI startup(): I_MPI_OFA_NUM_PORTS=1
[0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0
[0] MPI startup(): PMI_RANK=0
# OSU MPI Bandwidth Test v4.3
# Size      Bandwidth (MB/s)
1                       0.97
2                       1.94
4                       3.99
8                       8.08
16                     16.05
32                     29.25
64                     58.31
128                   116.07
256                   229.98
512                   446.80
1024                  873.71
2048                 1564.83
4096                 2819.04
8192                 4638.51
16384                5488.30
32768                5518.08
65536                5942.60
131072               6175.84
262144               6291.72
524288               6358.98
1048576              6390.17
2097152              6406.38
4194304              6413.95

We are running CentOS 6.6 with Mellanox OFED 2.4-1.0.4 and Mellanox Connect-IB MCB194A-FCAT Dual-Port FDR PCIe 3.0 x16.

Best regards,

René

Gergana_S_Intel · ‎04-01-2015

Hi Rene,

Thanks for getting in touch. We have seen this issue before and a fix is planned to be released in our upcoming Intel MPI Library 5.1 Beta version - slated to go out next week.

I'll let you know here when you can register for the Beta so you can test it out.

All the best,
~Gergana

Multi-rail ofa fabric not available (Intel MPI Library Version 5.0 Update 3)