cannot enable coredumping: Operation not permitted cannot enable coredumping: Operation not permitted cannot enable coredumping: Operation not permitted cannot enable coredumping: Operation not permitted [0] MPI startup(): Multi-threaded optimized library [87] MPID_nem_ofacm_init(): Init [87] MPI startup(): shm and ofa data transfer modes [88] MPID_nem_ofacm_init(): Init [88] MPI startup(): shm and ofa data transfer modes [81] MPID_nem_ofacm_init(): Init [78] MPID_nem_ofacm_init(): Init [82] MPID_nem_ofacm_init(): Init [89] MPID_nem_ofacm_init(): Init [84] MPID_nem_ofacm_init(): Init [85] MPID_nem_ofacm_init(): Init [73] MPID_nem_ofacm_init(): Init [72] MPID_nem_ofacm_init(): Init [77] MPID_nem_ofacm_init(): Init [76] MPID_nem_ofacm_init(): Init [86] MPID_nem_ofacm_init(): Init [80] MPID_nem_ofacm_init(): Init [83] MPID_nem_ofacm_init(): Init [74] MPID_nem_ofacm_init(): Init [79] MPID_nem_ofacm_init(): Init [81] MPI startup(): shm and ofa data transfer modes [75] MPID_nem_ofacm_init(): Init [78] MPI startup(): shm and ofa data transfer modes [91] MPID_nem_ofacm_init(): Init [92] MPID_nem_ofacm_init(): Init [94] MPID_nem_ofacm_init(): Init [95] MPID_nem_ofacm_init(): Init [93] MPID_nem_ofacm_init(): Init [90] MPID_nem_ofacm_init(): Init [82] MPI startup(): shm and ofa data transfer modes [89] MPI startup(): shm and ofa data transfer modes [84] MPI startup(): shm and ofa data transfer modes [85] MPI startup(): shm and ofa data transfer modes [73] MPI startup(): shm and ofa data transfer modes [77] MPI startup(): shm and ofa data transfer modes [83] MPI startup(): shm and ofa data transfer modes [72] MPI startup(): shm and ofa data transfer modes [74] MPI startup(): shm and ofa data transfer modes [76] MPI startup(): shm and ofa data transfer modes [80] MPI startup(): shm and ofa data transfer modes [86] MPI startup(): shm and ofa data transfer modes [75] MPI startup(): shm and ofa data transfer modes [79] MPI startup(): shm and ofa data transfer modes [92] MPI startup(): shm and ofa data transfer modes [94] MPI startup(): shm and ofa data transfer modes [95] MPI startup(): shm and ofa data transfer modes [91] MPI startup(): shm and ofa data transfer modes [90] MPI startup(): shm and ofa data transfer modes [93] MPI startup(): shm and ofa data transfer modes [0] MPID_nem_ofacm_init(): Init [0] MPI startup(): shm and ofa data transfer modes [14] MPID_nem_ofacm_init(): Init [44] MPID_nem_ofacm_init(): Init [14] MPI startup(): shm and ofa data transfer modes [2] MPID_nem_ofacm_init(): Init [36] MPID_nem_ofacm_init(): Init [15] MPID_nem_ofacm_init(): Init [44] MPI startup(): shm and ofa data transfer modes [1] MPID_nem_ofacm_init(): Init [2] MPI startup(): shm and ofa data transfer modes [50] MPID_nem_ofacm_init(): Init [12] MPID_nem_ofacm_init(): Init [53] MPID_nem_ofacm_init(): Init [36] MPI startup(): shm and ofa data transfer modes [45] MPID_nem_ofacm_init(): Init [46] MPID_nem_ofacm_init(): Init [27] MPID_nem_ofacm_init(): Init [52] MPID_nem_ofacm_init(): Init [43] MPID_nem_ofacm_init(): Init [69] MPID_nem_ofacm_init(): Init [19] MPID_nem_ofacm_init(): Init [22] MPID_nem_ofacm_init(): Init [33] MPID_nem_ofacm_init(): Init [39] MPID_nem_ofacm_init(): Init [49] MPID_nem_ofacm_init(): Init [15] MPI startup(): shm and ofa data transfer modes [42] MPID_nem_ofacm_init(): Init [71] MPID_nem_ofacm_init(): Init [38] MPID_nem_ofacm_init(): Init [6] MPID_nem_ofacm_init(): Init [18] MPID_nem_ofacm_init(): Init [28] MPID_nem_ofacm_init(): Init [51] MPID_nem_ofacm_init(): Init [47] MPID_nem_ofacm_init(): Init [16] MPID_nem_ofacm_init(): Init [8] MPID_nem_ofacm_init(): Init [1] MPI startup(): shm and ofa data transfer modes [35] MPID_nem_ofacm_init(): Init [63] MPID_nem_ofacm_init(): Init [59] MPID_nem_ofacm_init(): Init [68] MPID_nem_ofacm_init(): Init [24] MPID_nem_ofacm_init(): Init [70] MPID_nem_ofacm_init(): Init [50] MPI startup(): shm and ofa data transfer modes [66] MPID_nem_ofacm_init(): Init [20] MPID_nem_ofacm_init(): Init [12] MPI startup(): shm and ofa data transfer modes [57] MPID_nem_ofacm_init(): Init [61] MPID_nem_ofacm_init(): Init [23] MPID_nem_ofacm_init(): Init [40] MPID_nem_ofacm_init(): Init [9] MPID_nem_ofacm_init(): Init [4] MPID_nem_ofacm_init(): Init [53] MPI startup(): shm and ofa data transfer modes [5] MPID_nem_ofacm_init(): Init [25] MPID_nem_ofacm_init(): Init [27] MPI startup(): shm and ofa data transfer modes [13] MPID_nem_ofacm_init(): Init [48] MPID_nem_ofacm_init(): Init [31] MPID_nem_ofacm_init(): Init [60] MPID_nem_ofacm_init(): Init [69] MPI startup(): shm and ofa data transfer modes [43] MPI startup(): shm and ofa data transfer modes [45] MPI startup(): shm and ofa data transfer modes [46] MPI startup(): shm and ofa data transfer modes [67] MPID_nem_ofacm_init(): Init [52] MPI startup(): shm and ofa data transfer modes [34] MPID_nem_ofacm_init(): Init [10] MPID_nem_ofacm_init(): Init [19] MPI startup(): shm and ofa data transfer modes [62] MPID_nem_ofacm_init(): Init [64] MPID_nem_ofacm_init(): Init [22] MPI startup(): shm and ofa data transfer modes [29] MPID_nem_ofacm_init(): Init [65] MPID_nem_ofacm_init(): Init [26] MPID_nem_ofacm_init(): Init [7] MPID_nem_ofacm_init(): Init [33] MPI startup(): shm and ofa data transfer modes [49] MPI startup(): shm and ofa data transfer modes [39] MPI startup(): shm and ofa data transfer modes [21] MPID_nem_ofacm_init(): Init [32] MPID_nem_ofacm_init(): Init [17] MPID_nem_ofacm_init(): Init [41] MPID_nem_ofacm_init(): Init [42] MPI startup(): shm and ofa data transfer modes [51] MPI startup(): shm and ofa data transfer modes [30] MPID_nem_ofacm_init(): Init [11] MPID_nem_ofacm_init(): Init [3] MPID_nem_ofacm_init(): Init [6] MPI startup(): shm and ofa data transfer modes [18] MPI startup(): shm and ofa data transfer modes [28] MPI startup(): shm and ofa data transfer modes [8] MPI startup(): shm and ofa data transfer modes [16] MPI startup(): shm and ofa data transfer modes [35] MPI startup(): shm and ofa data transfer modes [24] MPI startup(): shm and ofa data transfer modes [20] MPI startup(): shm and ofa data transfer modes [37] MPID_nem_ofacm_init(): Init [38] MPI startup(): shm and ofa data transfer modes [47] MPI startup(): shm and ofa data transfer modes [54] MPID_nem_ofacm_init(): Init [55] MPID_nem_ofacm_init(): Init [56] MPID_nem_ofacm_init(): Init [57] MPI startup(): shm and ofa data transfer modes [58] MPID_nem_ofacm_init(): Init [59] MPI startup(): shm and ofa data transfer modes [61] MPI startup(): shm and ofa data transfer modes [23] MPI startup(): shm and ofa data transfer modes [63] MPI startup(): shm and ofa data transfer modes [66] MPI startup(): shm and ofa data transfer modes [68] MPI startup(): shm and ofa data transfer modes [70] MPI startup(): shm and ofa data transfer modes [71] MPI startup(): shm and ofa data transfer modes [40] MPI startup(): shm and ofa data transfer modes [48] MPI startup(): shm and ofa data transfer modes [4] MPI startup(): shm and ofa data transfer modes [9] MPI startup(): shm and ofa data transfer modes [13] MPI startup(): shm and ofa data transfer modes [25] MPI startup(): shm and ofa data transfer modes [60] MPI startup(): shm and ofa data transfer modes [64] MPI startup(): shm and ofa data transfer modes [67] MPI startup(): shm and ofa data transfer modes [41] MPI startup(): shm and ofa data transfer modes [62] MPI startup(): shm and ofa data transfer modes [65] MPI startup(): shm and ofa data transfer modes [5] MPI startup(): shm and ofa data transfer modes [10] MPI startup(): shm and ofa data transfer modes [31] MPI startup(): shm and ofa data transfer modes [54] MPI startup(): shm and ofa data transfer modes [56] MPI startup(): shm and ofa data transfer modes [3] MPI startup(): shm and ofa data transfer modes [7] MPI startup(): shm and ofa data transfer modes [11] MPI startup(): shm and ofa data transfer modes [17] MPI startup(): shm and ofa data transfer modes [21] MPI startup(): shm and ofa data transfer modes [26] MPI startup(): shm and ofa data transfer modes [29] MPI startup(): shm and ofa data transfer modes [30] MPI startup(): shm and ofa data transfer modes [32] MPI startup(): shm and ofa data transfer modes [34] MPI startup(): shm and ofa data transfer modes [55] MPI startup(): shm and ofa data transfer modes [37] MPI startup(): shm and ofa data transfer modes [58] MPI startup(): shm and ofa data transfer modes [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 267545 n140 0 [0] MPI startup(): 1 267546 n140 4 [0] MPI startup(): 2 267547 n140 8 [0] MPI startup(): 3 267548 n140 6 [0] MPI startup(): 4 267549 n140 2 [0] MPI startup(): 5 267550 n140 12 [0] MPI startup(): 6 267551 n140 16 [0] MPI startup(): 7 267552 n140 14 [0] MPI startup(): 8 267553 n140 10 [0] MPI startup(): 9 267554 n140 20 [0] MPI startup(): 10 267555 n140 24 [0] MPI startup(): 11 267556 n140 26 [0] MPI startup(): 12 267557 n140 22 [0] MPI startup(): 13 267558 n140 18 [0] MPI startup(): 14 267559 n140 28 [0] MPI startup(): 15 267560 n140 32 [0] MPI startup(): 16 267561 n140 34 [0] MPI startup(): 17 267562 n140 30 [0] MPI startup(): 18 267563 n140 1 [0] MPI startup(): 19 267564 n140 5 [0] MPI startup(): 20 267565 n140 9 [0] MPI startup(): 21 267566 n140 7 [0] MPI startup(): 22 267567 n140 3 [0] MPI startup(): 23 267568 n140 13 [0] MPI startup(): 24 267569 n140 17 [0] MPI startup(): 25 267570 n140 15 [0] MPI startup(): 26 267571 n140 11 [0] MPI startup(): 27 267572 n140 21 [0] MPI startup(): 28 267573 n140 25 [0] MPI startup(): 29 267574 n140 27 [0] MPI startup(): 30 267575 n140 23 [0] MPI startup(): 31 267576 n140 19 [0] MPI startup(): 32 267577 n140 29 [0] MPI startup(): 33 267578 n140 33 [0] MPI startup(): 34 267579 n140 35 [0] MPI startup(): 35 267580 n140 31 [0] MPI startup(): 36 265667 n139 0 [0] MPI startup(): 37 265668 n139 4 [0] MPI startup(): 38 265669 n139 8 [0] MPI startup(): 39 265670 n139 6 [0] MPI startup(): 40 265671 n139 2 [0] MPI startup(): 41 265672 n139 12 [0] MPI startup(): 42 265673 n139 16 [0] MPI startup(): 43 265674 n139 14 [0] MPI startup(): 44 265675 n139 10 [0] MPI startup(): 45 265676 n139 20 [0] MPI startup(): 46 265677 n139 24 [0] MPI startup(): 47 265678 n139 26 [0] MPI startup(): 48 265679 n139 22 [0] MPI startup(): 49 265680 n139 18 [0] MPI startup(): 50 265681 n139 28 [0] MPI startup(): 51 265682 n139 32 [0] MPI startup(): 52 265683 n139 34 [0] MPI startup(): 53 265684 n139 30 [0] MPI startup(): 54 265685 n139 1 [0] MPI startup(): 55 265686 n139 5 [0] MPI startup(): 56 265687 n139 9 [0] MPI startup(): 57 265688 n139 7 [0] MPI startup(): 58 265689 n139 3 [0] MPI startup(): 59 265690 n139 13 [0] MPI startup(): 60 265691 n139 17 [0] MPI startup(): 61 265692 n139 15 [0] MPI startup(): 62 265693 n139 11 [0] MPI startup(): 63 265694 n139 21 [0] MPI startup(): 64 265695 n139 25 [0] MPI startup(): 65 265696 n139 27 [0] MPI startup(): 66 265697 n139 23 [0] MPI startup(): 67 265698 n139 19 [0] MPI startup(): 68 265699 n139 29 [0] MPI startup(): 69 265700 n139 33 [0] MPI startup(): 70 265701 n139 35 [0] MPI startup(): 71 265702 n139 31 [0] MPI startup(): 72 257644 n138 0 [0] MPI startup(): 73 257645 n138 4 [0] MPI startup(): 74 257646 n138 8 [0] MPI startup(): 75 257647 n138 6 [0] MPI startup(): 76 257648 n138 2 [0] MPI startup(): 77 257649 n138 12 [0] MPI startup(): 78 257650 n138 16 [0] MPI startup(): 79 257651 n138 14 [0] MPI startup(): 80 257652 n138 10 [0] MPI startup(): 81 257653 n138 20 [0] MPI startup(): 82 257654 n138 24 [0] MPI startup(): 83 257655 n138 26 [0] MPI startup(): 84 257656 n138 22 [0] MPI startup(): 85 257657 n138 18 [0] MPI startup(): 86 257658 n138 28 [0] MPI startup(): 87 257659 n138 32 [0] MPI startup(): 88 257660 n138 34 [0] MPI startup(): 89 257661 n138 30 [0] MPI startup(): 90 257662 n138 1 [0] MPI startup(): 91 257663 n138 5 [0] MPI startup(): 92 257664 n138 9 [0] MPI startup(): 93 257665 n138 7 [0] MPI startup(): 94 257666 n138 3 [0] MPI startup(): 95 257667 n138 11 [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_FABRICS=shm:ofa [0] MPI startup(): I_MPI_FALLBACK=0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,21,10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=mlx5_0:0,mlx5_1:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=2 [0] MPI startup(): I_MPI_OFA_NUM_ADAPTERS=2 [0] MPI startup(): I_MPI_OFA_NUM_PORTS=1 [0] MPI startup(): I_MPI_PIN_MAPPING=36:0 0,1 4,2 8,3 6,4 2,5 12,6 16,7 14,8 10,9 20,10 24,11 26,12 22,13 18,14 28,15 32,16 34,17 30,18 1,19 5,20 9,21 7,22 3,23 13,24 17,25 15,26 11,27 21,28 25,29 27,30 23,31 19,32 29,33 33,34 35,35 31 [0] MPI startup(): I_MPI_PLATFORM=auto [0] MPI startup(): I_MPI_STATS=ipm [0] MPI startup(): I_MPI_STATS_FILE=stats.ipm