- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi everyone,
On a node with single E5-2667v3 (8 phys. cores), single ConnectX-4 board (81:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]), CentOS 7, UCX version 1.6.0, I'm experiencing an unexpected program action with I_MPI_ASYNC_PROGRESS=1 with "release_mt" and "debug_mt" library kind.
Intel MPI library version is 2019 Update 5, also re-checked with Update 9.
The program code is a simple MPI_init()/MPI_Finalize() pair:
#include <iostream>
#include <mpi.h>
#include <assert.h>
int main(int argc, char **argv)
{
int thrlev;
MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &thrlev);
assert(thrlev == MPI_THREAD_MULTIPLE);
MPI_Finalize();
return 0;
}
Built simply with: mpiicpc -o myapp myapp.cpp
Ran it as:
. /xxx/compilers_and_libraries_2020.0.166/linux/bin/compilervars.sh intel64
. /xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/bin/mpivars.sh debug_mt
module load gcc/7.2.0
export I_MPI_DEBUG=1000
export FI_PROVIDER=verbs
export I_MPI_ASYNC_PROGRESS=1
mpiexec.hydra -machinefile hostfile -np 2 -ppn 2 --errfile-pattern=err.%r --outfile-pattern=out.%r ./myapp
I have in "out.0":
MPI startup(): Imported environment partly inaccesible. Map=0 Info=0
[0] MPI startup(): libfabric version: 1.9.0a1-impi
[0] MPI startup(): libfabric provider: verbs;ofi_rxm
[0] MPI startup(): max_ch4_vcis: 1, max_reg_eps 64, enable_sep 0, enable_shared_ctxs 0, do_av_insert 1
[0] MPI startup(): addrname_len: 16, addrname_firstlen: 16
[0] MPI startup(): val_max: 4096, part_len: 4095, bc_len: 38, num_parts: 1
[0] MPI startup(): selected platform: unknown
[0] MPI startup(): Platform was not configured (platform=auto will be ignored).
[0] MPI startup(): Load tuning file: "/xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/etc/tuning_skx_ofi.dat"
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALL
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLW
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: EXSCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER_BLOCK
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALL
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLW
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IEXSCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE_SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCATTERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE_SCATTER_BLOCK
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALL
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLW
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: EXSCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER_BLOCK
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALL
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLW
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IEXSCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE_SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ISCATTERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: FLAT_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IREDUCE_SCATTER_BLOCK
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALL
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLV
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLTOALLW
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: EXSCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 1
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 8
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: GATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE_SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCAN
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 1
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTER
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: SCATTERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALL
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLTOALLV
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IBCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: COMPOSITION
MPIU_Selection_find_entry(): Layer: COMM_KIND, Key: INTRA_COMM
MPIU_Selection_find_entry(): Layer: COMM_HIERARCHY, Key: TOPO_COMM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 1
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 2
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: IALLGATHERV
MPIU_Selection_find_entry(): Layer: PPN, Key: 2
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 4
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: NETWORK
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: ALLREDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BARRIER
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: BCAST
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: COLLECTIVE, Key: REDUCE
MPIU_Selection_find_entry(): Layer: PPN, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMMSIZE, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: COMM_ID, Key: 18446744073709551615
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
MPIU_Selection_find_entry(): Layer: DIRECTORY, Key: SHM
MPIU_Selection_find_entry(): Layer: UNKNOWN, Key: 16787456
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 30047 n944011 {8,9}
[0] MPI startup(): 1 30048 n944011 {8,9}
[0] MPI startup(): I_MPI_ROOT=/xxx/compilers_and_libraries_2020.0.166/linux/mpi
[0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc
[0] MPI startup(): I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0
[0] MPI startup(): I_MPI_PIN_UNIT=-1
[0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default
[0] MPI startup(): I_MPI_ASYNC_PROGRESS=1
[0] MPI startup(): I_MPI_DEBUG=1000
[0] MPI startup(): global_rank 0, local_rank 0, local_size 2, threads_per_node 2
[0] MPI startup(): threading: thread: 0, processor: 15
[0] MPI startup(): threading: thread: 1, processor: 14
without "export FI_PROVIDER=verbs" the library defaults at "mlx" provider, and in this case I have only this difference in output, the action of program is the same:
@@ -1,9 +1,10 @@
MPI startup(): Imported environment partly inaccesible. Map=0 Info=0
[0] MPI startup(): libfabric version: 1.9.0a1-impi
-[0] MPI startup(): libfabric provider: verbs;ofi_rxm
+[0] MPI startup(): libfabric provider: mlx
+[0] MPI startup(): detected mlx provider, set device name to "mlx"
[0] MPI startup(): max_ch4_vcis: 1, max_reg_eps 64, enable_sep 0, enable_shared_ctxs 0, do_av_insert 1
-[0] MPI startup(): addrname_len: 16, addrname_firstlen: 16
-[0] MPI startup(): val_max: 4096, part_len: 4095, bc_len: 38, num_parts: 1
+[0] MPI startup(): addrname_len: 512, addrname_firstlen: 512
+[0] MPI startup(): val_max: 4096, part_len: 4095, bc_len: 1030, num_parts: 1
[0] MPI startup(): selected platform: unknown
[0] MPI startup(): Platform was not configured (platform=auto will be ignored).
[0] MPI startup(): Load tuning file: "/xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/etc/tuning_skx_ofi.dat"
The program is checked on another system with IMPI 2019u9, and it works there fine.
Could you suggest any actions to do to isolate and fix this?
--
Regards,
Alexey
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Alexey,
Could you please elaborate the unexpected program action that you are referring to? We haven't observed any error messages in the output of both the versions.
Regards
Prasanth
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Prasanth,
as I noticed with a debugger attached, program exits inside MPI_Init_thread(), doesn't seem to get any signal, didn't elaborate any further details yet.
I changed the reproducer code like this:
#include <iostream>
#include <mpi.h>
#include <assert.h>
int main(int argc, char **argv)
{
int thrlev;
MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &thrlev);
assert(thrlev == MPI_THREAD_MULTIPLE);
std::cout << "Hello" << std::endl;
MPI_Finalize();
return 0;
}
The problem is: it doesn't print "Hello".
--
Regards,
Alexey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
With I_MPI_HYDRA_DEBUG=1 i've got the output:
[mpiexec@n944004] Launch arguments: /xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/bin//hyd
ra_bstrap_proxy --upstream-host n944004 --upstream-port 44997 --pgid 0 --launcher ssh --launcher-number 0 --base-pa
th /xxx/compilers_and_libraries_2020.0.166/linux/mpi/intel64/bin/ --tree-width 16 --tree-level 1 --tim
e-left -1 --collective-launch 1 --debug --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 7 /xxx
/compilers_and_libraries_2020.0.166/linux/mpi/intel64/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-sign
al 15
[proxy:0:0@n944004] Warning - oversubscription detected: 2 processes will be placed on 0 cores
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@n944004] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@n944004] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_maxes
[proxy:0:0@n944004] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_maxes
[proxy:0:0@n944004] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_appnum
[proxy:0:0@n944004] PMI response: cmd=appnum appnum=0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_appnum
[proxy:0:0@n944004] PMI response: cmd=appnum appnum=0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get kvsname=kvs_20759_0 key=PMI_process_mapping
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,2))
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_20759_0 key=PMI_process_mapping
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,2))
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=barrier_in
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_20759_0 key=sharedFilename[0]-0 value=/dev/shm/Intel_MPI_RRkf1r
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=barrier_in
[proxy:0:0@n944004] PMI response: cmd=barrier_out
[proxy:0:0@n944004] PMI response: cmd=barrier_out
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_20759_0 key=sharedFilename[0]-0
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/Intel_MPI_RRkf1r
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_20759_0 key=bc-0 value=mpi#D28698A54D5AC3504008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090EE000000225100000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009001800200000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300880178DA82679CE83A2383215700035C77CC2B33EF483850DB2A31351EDF0B00854A130022020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=put kvsname=kvs_20759_0 key=bc-1 value=mpi#4C326D72FEDB027F4008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090F6000000275100000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009000000200000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300888B5A953BF42AC57F2383215700035C77CC2B33EF483850DB2A31351EDF0B008549130027020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_20759_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=barrier_in
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_20759_0 key=sharedFilename[0]-1 value=/dev/shm/Intel_MPI_O5oWwX
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=barrier_in
[proxy:0:0@n944004] PMI response: cmd=barrier_out
[proxy:0:0@n944004] PMI response: cmd=barrier_out
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_20759_0 key=sharedFilename[0]-1
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/Intel_MPI_O5oWwX
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get kvsname=kvs_20759_0 key=bc-0
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=mpi#D28698A54D5AC3504008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090EE000000225100000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009001800200000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300880178DA82679CE83A2383215700035C77CC2B33EF483850DB2A31351EDF0B00854A130022020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_20759_0 key=bc-1
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=mpi#4C326D72FEDB027F4008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090F6000000275100000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009000000200000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300888B5A953BF42AC57F2383215700035C77CC2B33EF483850DB2A31351EDF0B008549130027020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=abort exitcode=604200719
--
Regards,
Alexey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
With I_MPI_HYDRA_DEBUG=1, the last lines of output are:
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_23009_0 key=bc-0 value=mpi#2530F7487EEB3DF84008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090EE000000F25900000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009000000800000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F030088F688CAA4827ADE152383215700035C77CC2B33EF483850DB2A31351EDF0B00855D1400F2020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=put kvsname=kvs_23009_0 key=bc-1 value=mpi#B28B5B2DF39D97F64008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090F6000000EB5900000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009001800800000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300883FEE98C455AA100A2383215700035C77CC2B33EF483850DB2A31351EDF0B00855E1400EB020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_23009_0
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=barrier_in
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get_my_kvsname
[proxy:0:0@n944004] PMI response: cmd=my_kvsname kvsname=kvs_23009_0
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=put kvsname=kvs_23009_0 key=sharedFilename[0]-1 value=/dev/shm/Intel_MPI_SIF710
[proxy:0:0@n944004] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=barrier_in
[proxy:0:0@n944004] PMI response: cmd=barrier_out
[proxy:0:0@n944004] PMI response: cmd=barrier_out
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_23009_0 key=sharedFilename[0]-1
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/Intel_MPI_SIF710
[proxy:0:0@n944004] pmi cmd from fd 6: cmd=get kvsname=kvs_23009_0 key=bc-0
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=mpi#2530F7487EEB3DF84008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090EE000000F25900000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009000000800000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F030088F688CAA4827ADE152383215700035C77CC2B33EF483850DB2A31351EDF0B00855D1400F2020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=get kvsname=kvs_23009_0 key=bc-1
[proxy:0:0@n944004] PMI response: cmd=get_result rc=0 msg=success value=mpi#B28B5B2DF39D97F64008E055B2E5855E4F777DB477CC2B32004C3E5077CCAB33004F130090F6000000EB5900000030C8AAAA2A0000410816F2E70F7D4949A67DB477CC2B32004C3E5077CCAB33004F13009001800800000000000060C8AAAA2A00002200B43977CC2B3200F8D74F00000000004F0300883FEE98C455AA100A2383215700035C77CC2B33EF483850DB2A31351EDF0B00855E1400EB020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$
[proxy:0:0@n944004] pmi cmd from fd 10: cmd=abort exitcode=604200719
With attached gdb, made a breakpoint on MPI_Init_thread(), reduced I_MPI_DEBUG=5, going step-by-step with gdb's "next" command:
Breakpoint 3, PMPI_Init_thread (argc=0x7fffffffc1b0, argv=0x7fffffffc1b8, required=3, provided=0x7fffffffc1a0) at ../../src/mpi/init/initthread.c:893
900 in ../../src/mpi/init/initthread.c
901 in ../../src/mpi/init/initthread.c
902 in ../../src/mpi/init/initthread.c
910 in ../../src/mpi/init/initthread.c
917 in ../../src/mpi/init/initthread.c
938 in ../../src/mpi/init/initthread.c
941 in ../../src/mpi/init/initthread.c
944 in ../../src/mpi/init/initthread.c
945 in ../../src/mpi/init/initthread.c
946 in ../../src/mpi/init/initthread.c
949 in ../../src/mpi/init/initthread.c
955 in ../../src/mpi/init/initthread.c
956 in ../../src/mpi/init/initthread.c
958 in ../../src/mpi/init/initthread.c
MPI startup(): Imported environment partly inaccesible. Map=0 Info=0
[0] MPI startup(): libfabric version: 1.9.0a1-impi
[0] MPI startup(): libfabric provider: mlx
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 23026 n944004 {10,11}
[0] MPI startup(): 1 23019 n944004 {10,11}
[0] MPI startup(): I_MPI_ROOT=/uls/7/intel-2020/compilers_and_libraries_2020.0.166/linux/mpi
[0] MPI startup(): I_MPI_HYDRA_DEBUG=1
[0] MPI startup(): I_MPI_JOB_ABORT_SIGNAL=15
[0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc
[0] MPI startup(): I_MPI_PIN_UNIT=-1
[0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default
[0] MPI startup(): I_MPI_ASYNC_PROGRESS=1
[0] MPI startup(): I_MPI_DEBUG=5
[New Thread 0x2aaad4488700 (LWP 23061)]
959 in ../../src/mpi/init/initthread.c
962 in ../../src/mpi/init/initthread.c
967 in ../../src/mpi/init/initthread.c
968 in ../../src/mpi/init/initthread.c
[0] MPI startup(): global_rank 0, local_rank 0, local_size 2, threads_per_node 2
[0] MPI startup(): threading: thread: 0, processor: 15
[0] MPI startup(): threading: thread: 1, processor: 14
[New Thread 0x2aaad58e5700 (LWP 23063)]
969 in ../../src/mpi/init/initthread.c
1000 in ../../src/mpi/init/initthread.c
Hope this helps a bit.
--
Regards,
Alexey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Prasanth,
1) last lines of output with I_MPI_HYDRA_DEBUG=1 is here in the attached file "async_issue_hydra.txt.gz"
2) I've attached gdb to both ranks, set breakpoint at MPI_Init_thread(), then went on with "next" gdb commands until the program stopped. (debug_mt library kind, I_MPI_DEBUG=5). The output from gdb (rank=0) is here in the attached file "async_issue_gdb.txt.gz". For rank=1 reported program lines are the same.
Hope this would help.
--
Regards,
Alexey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Alexey,
Could you please try with mlx provider? For InfiniBand, mlx is recommended over verbs.
You can set the provider using -: export FI_PROVIDER=mlx to use mlx provider.
I have tried with mlx in my environment on InfiniBand and found no errors (i can see the stdout messages)
Let us know if you observe any error with mlx too.
Regards
Prasanth
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Prasanth,
please look at my original post, where I wrote:
>> without "export FI_PROVIDER=verbs" the library defaults at "mlx" provider, and in this case I have only
>> this difference in output, the action of program is the same:
So: yes, I can confirm that the error persists on this system with "mlx" as well. I don't see such an error on another machine where these modes (release_mt and I_MPI_ASYNC_PROGRESS) work fine on 2019u9, so the problem seems to be machine/environment specific. The question is: how to debug this and how to find the root cause?
--
Regards,
Alexey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Alexey,
Sorry for the late response.
I suggested mlx since I thought you were using mlx only in 2019u9.
I am escalating this issue to the Subject Matter experts for better support.
Could you please provide the output of the following command:
ucx_info -d | grep Transport
Regards
Prasanth
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Prasanth,
I'm currently on vacation, will return wirg this info on the next week. Thanks.
--
Regards,
Alexey
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page