- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi All -
I'm getting an error and not quite sure where to begin tracking it down. I'm running a model known to run on our system using:
Intel(R) MPI Library for Linux* OS, Version 2019 Update 6 Build 20191024 (id: 082ae5608)
The code runs for certain core counts(generally smaller processor counts) but errors for some counts with:
Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) 8 Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) 9 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f01f66321d4] 10 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f01f5dba031] 11 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f01f5f34c5d] 12 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f01f5e465e4] 13 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f01f5e1ad8e] 14 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f01f5f376a7] 15 /apps/applications/development/compilers/intel/1 16 Abort(1) on node 2: Internal error 17 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f983aed31d4] 18 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f983a65b031] 19 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f983a7d5c5d] 20 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f983a6e75e4] 21 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f983a6bbd8e] 22 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f983a7d86a7] 23 /apps/applications/development/compilers/intel/1 24 Abort(1) on node 3: Internal error 25 Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) 26 Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) 27 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f9ede95a1d4] 28 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f9ede0e2031] 29 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f9ede25cc5d] 30 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f9ede16e5e4] 31 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f9ede142d8e] 32 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f9ede25f6a7] 33 /apps/applications/development/compilers/intel/1 34 Abort(1) on node 1: Internal error 35 Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) 36 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7ff6ff7ae1d4] 37 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7ff6fef36031] 38 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7ff6ff0b0c5d] 39 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7ff6fefc25e4] 40 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7ff6fef96d8e] 41 /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7ff6ff0b36a7] 42 /apps/applications/development/compilers/intel/1 43 Abort(1) on node 84: Internal error
The code does run correctly with this core configuration when started through SLURM using "srun --mpi=pmi2". Can you provide any guidance?
The machine this is running on is a dual-socket AMD Epyc 7702 with hyperthreading disabled and Ubuntu 18.04 server
Thanks
- Tags:
- Cluster Computing
- General Support
- Intel® Cluster Ready
- Message Passing Interface (MPI)
- Parallel Computing
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Zach,
Can you please provide the following:
1) The command line you are following and the output.
2)The output with hostname as parameter(mpirun/mpiexec -np <processes from which error starts showing> hostname)
3)The log report after running the mpi command with I_MPI_DEBUG=5 (I_MPI_DEBUG=5 mpirun/mpiexec -np<> ./Executable)
We need this information to debug the issue.
Thanks
Prasanth
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Prasanth -
Thanks for your response.
(1) The script which starts the code is (note the memory request @ 64G is far beyond what is required, but I've cranked this up to ensure it is not part of the issue):
#!/bin/bash #SBATCH -p research #SBATCH -A research #SBATCH -t 12:00:00 #SBATCH --ntasks=128 #SBATCH --mem=64G module load adcirc/54.00/intel/19.1 export I_MPI_DEBUG=5 date > run.begin #srun --mpi=pmi2 -n $SLURM_NTASKS padcirc -W 1 mpirun -np $SLURM_NTASKS padcirc -W 1 date > run.end
(2) The output from the hostname command is:
Loading adcirc/54.00/intel/19.1 Loading requirement: compilers/intel/19.1 mpi/impi/19.1/intel/19.1 szip/2.1.1/intel/19.1 hdf5/1.8.18/intel/19.1/serial netcdf/4.5.0/intel/19.1/serial roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01 roux-rome-01
(3) The output from the run with verbose debugging is:
Loading adcirc/54.00/intel/19.1 Loading requirement: compilers/intel/19.1 mpi/impi/19.1/intel/19.1 szip/2.1.1/intel/19.1 hdf5/1.8.18/intel/19.1/serial netcdf/4.5.0/intel/19.1/serial [0] MPI startup(): libfabric version: 1.9.0a1-impi [0] MPI startup(): libfabric provider: tcp;ofi_rxm [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 7450 roux-rome-01 0 [0] MPI startup(): 1 7451 roux-rome-01 1 [0] MPI startup(): 2 7452 roux-rome-01 2 [0] MPI startup(): 3 7453 roux-rome-01 3 [0] MPI startup(): 4 7454 roux-rome-01 4 [0] MPI startup(): 5 7455 roux-rome-01 5 [0] MPI startup(): 6 7456 roux-rome-01 6 [0] MPI startup(): 7 7457 roux-rome-01 7 [0] MPI startup(): 8 7458 roux-rome-01 8 [0] MPI startup(): 9 7459 roux-rome-01 9 [0] MPI startup(): 10 7460 roux-rome-01 10 [0] MPI startup(): 11 7461 roux-rome-01 11 [0] MPI startup(): 12 7462 roux-rome-01 12 [0] MPI startup(): 13 7463 roux-rome-01 13 [0] MPI startup(): 14 7464 roux-rome-01 14 [0] MPI startup(): 15 7465 roux-rome-01 15 [0] MPI startup(): 16 7466 roux-rome-01 16 [0] MPI startup(): 17 7467 roux-rome-01 17 [0] MPI startup(): 18 7468 roux-rome-01 18 [0] MPI startup(): 19 7469 roux-rome-01 19 [0] MPI startup(): 20 7471 roux-rome-01 20 [0] MPI startup(): 21 7472 roux-rome-01 21 [0] MPI startup(): 22 7473 roux-rome-01 22 [0] MPI startup(): 23 7474 roux-rome-01 23 [0] MPI startup(): 24 7475 roux-rome-01 24 [0] MPI startup(): 25 7476 roux-rome-01 25 [0] MPI startup(): 26 7477 roux-rome-01 26 [0] MPI startup(): 27 7478 roux-rome-01 27 [0] MPI startup(): 28 7479 roux-rome-01 28 [0] MPI startup(): 29 7480 roux-rome-01 29 [0] MPI startup(): 30 7481 roux-rome-01 30 [0] MPI startup(): 31 7482 roux-rome-01 31 [0] MPI startup(): 32 7483 roux-rome-01 32 [0] MPI startup(): 33 7484 roux-rome-01 33 [0] MPI startup(): 34 7485 roux-rome-01 34 [0] MPI startup(): 35 7486 roux-rome-01 35 [0] MPI startup(): 36 7487 roux-rome-01 36 [0] MPI startup(): 37 7488 roux-rome-01 37 [0] MPI startup(): 38 7489 roux-rome-01 38 [0] MPI startup(): 39 7490 roux-rome-01 39 [0] MPI startup(): 40 7491 roux-rome-01 40 [0] MPI startup(): 41 7492 roux-rome-01 41 [0] MPI startup(): 42 7494 roux-rome-01 42 [0] MPI startup(): 43 7495 roux-rome-01 43 [0] MPI startup(): 44 7497 roux-rome-01 44 [0] MPI startup(): 45 7498 roux-rome-01 45 [0] MPI startup(): 46 7499 roux-rome-01 46 [0] MPI startup(): 47 7500 roux-rome-01 47 [0] MPI startup(): 48 7501 roux-rome-01 48 [0] MPI startup(): 49 7502 roux-rome-01 49 [0] MPI startup(): 50 7503 roux-rome-01 50 [0] MPI startup(): 51 7504 roux-rome-01 51 [0] MPI startup(): 52 7505 roux-rome-01 52 [0] MPI startup(): 53 7506 roux-rome-01 53 [0] MPI startup(): 54 7507 roux-rome-01 54 [0] MPI startup(): 55 7508 roux-rome-01 55 [0] MPI startup(): 56 7509 roux-rome-01 56 [0] MPI startup(): 57 7510 roux-rome-01 57 [0] MPI startup(): 58 7511 roux-rome-01 58 [0] MPI startup(): 59 7512 roux-rome-01 59 [0] MPI startup(): 60 7513 roux-rome-01 60 [0] MPI startup(): 61 7514 roux-rome-01 61 [0] MPI startup(): 62 7515 roux-rome-01 62 [0] MPI startup(): 63 7516 roux-rome-01 63 [0] MPI startup(): 64 7517 roux-rome-01 64 [0] MPI startup(): 65 7518 roux-rome-01 65 [0] MPI startup(): 66 7519 roux-rome-01 66 [0] MPI startup(): 67 7520 roux-rome-01 67 [0] MPI startup(): 68 7521 roux-rome-01 68 [0] MPI startup(): 69 7522 roux-rome-01 69 [0] MPI startup(): 70 7523 roux-rome-01 70 [0] MPI startup(): 71 7524 roux-rome-01 71 [0] MPI startup(): 72 7525 roux-rome-01 72 [0] MPI startup(): 73 7526 roux-rome-01 73 [0] MPI startup(): 74 7527 roux-rome-01 74 [0] MPI startup(): 75 7528 roux-rome-01 75 [0] MPI startup(): 76 7529 roux-rome-01 76 [0] MPI startup(): 77 7530 roux-rome-01 77 [0] MPI startup(): 78 7531 roux-rome-01 78 [0] MPI startup(): 79 7532 roux-rome-01 79 [0] MPI startup(): 80 7533 roux-rome-01 80 [0] MPI startup(): 81 7534 roux-rome-01 81 [0] MPI startup(): 82 7535 roux-rome-01 82 [0] MPI startup(): 83 7536 roux-rome-01 83 [0] MPI startup(): 84 7538 roux-rome-01 84 [0] MPI startup(): 85 7539 roux-rome-01 85 [0] MPI startup(): 86 7540 roux-rome-01 86 [0] MPI startup(): 87 7541 roux-rome-01 87 [0] MPI startup(): 88 7542 roux-rome-01 88 [0] MPI startup(): 89 7543 roux-rome-01 89 [0] MPI startup(): 90 7544 roux-rome-01 90 [0] MPI startup(): 91 7545 roux-rome-01 91 [0] MPI startup(): 92 7546 roux-rome-01 92 [0] MPI startup(): 93 7547 roux-rome-01 93 [0] MPI startup(): 94 7548 roux-rome-01 94 [0] MPI startup(): 95 7549 roux-rome-01 95 [0] MPI startup(): 96 7550 roux-rome-01 96 [0] MPI startup(): 97 7551 roux-rome-01 97 [0] MPI startup(): 98 7552 roux-rome-01 98 [0] MPI startup(): 99 7553 roux-rome-01 99 [0] MPI startup(): 100 7554 roux-rome-01 100 [0] MPI startup(): 101 7555 roux-rome-01 101 [0] MPI startup(): 102 7556 roux-rome-01 102 [0] MPI startup(): 103 7557 roux-rome-01 103 [0] MPI startup(): 104 7558 roux-rome-01 104 [0] MPI startup(): 105 7559 roux-rome-01 105 [0] MPI startup(): 106 7560 roux-rome-01 106 [0] MPI startup(): 107 7561 roux-rome-01 107 [0] MPI startup(): 108 7562 roux-rome-01 108 [0] MPI startup(): 109 7563 roux-rome-01 109 [0] MPI startup(): 110 7564 roux-rome-01 110 [0] MPI startup(): 111 7565 roux-rome-01 111 [0] MPI startup(): 112 7566 roux-rome-01 112 [0] MPI startup(): 113 7567 roux-rome-01 113 [0] MPI startup(): 114 7568 roux-rome-01 114 [0] MPI startup(): 115 7569 roux-rome-01 115 [0] MPI startup(): 116 7570 roux-rome-01 116 [0] MPI startup(): 117 7571 roux-rome-01 117 [0] MPI startup(): 118 7572 roux-rome-01 118 [0] MPI startup(): 119 7573 roux-rome-01 119 [0] MPI startup(): 120 7574 roux-rome-01 120 [0] MPI startup(): 121 7575 roux-rome-01 121 [0] MPI startup(): 122 7576 roux-rome-01 122 [0] MPI startup(): 123 7577 roux-rome-01 123 [0] MPI startup(): 124 7578 roux-rome-01 124 [0] MPI startup(): 125 7579 roux-rome-01 125 [0] MPI startup(): 126 7580 roux-rome-01 126 [0] MPI startup(): 127 7581 roux-rome-01 127 [0] MPI startup(): I_MPI_ROOT=/apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi [0] MPI startup(): I_MPI_MPIRUN=mpirun [0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc [0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP=slurm [0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default [0] MPI startup(): I_MPI_DEBUG=5 ECHO: Application launched with: padcirc -W INFO: Searching for ADCIRC subdomain directories: INFO: Looking for './PE0000/fort.14' ... INFO: File './PE0000/fort.14' was found! INFO: The search for the subdomain directory was completed successfully. Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f08459e11d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f0845169031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f08452e3c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f08451f55e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f08451c9d8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f08452e66a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 2: Internal error Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f4fd11c51d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f4fd094d031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f4fd0ac7c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f4fd09d95e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f4fd09add8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f4fd0aca6a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 23: Internal error /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f4eaa9b81d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f4eaa140031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f4eaa2bac5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f4eaa1cc5e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f4eaa1a0d8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f4eaa2bd6a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 3: Internal error /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7fa0bb4c81d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7fa0bac50031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7fa0badcac5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7fa0bacdc5e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7fa0bacb0d8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7fa0badcd6a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 20: Internal error Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f7fac68f1d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f7fabe17031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f7fabf91c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f7fabea35e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f7fabe77d8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f7fabf946a7] /apps/applications/development/compilers/intel/1 Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) Abort(1) on node 22: Internal error /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f6f8e2161d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f6f8d99e031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f6f8db18c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f6f8da2a5e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f6f8d9fed8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f6f8db1b6a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 71: Internal error Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f98266891d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f9825e11031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f9825f8bc5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f9825e9d5e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f9825e71d8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f9825f8e6a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 70: Internal error Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f579685d1d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f5795fe5031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f579615fc5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f57960715e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f5796045d8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f57961626a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 76: Internal error /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f38293241d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f3828aac031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f3828c26c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f3828b385e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f3828b0cd8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f3828c296a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 77: Internal error Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f53aff911d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f53af719031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f53af893c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f53af7a55e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f53af779d8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f53af8966a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 86: Internal error /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f36ae8f11d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f36ae079031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f36ae1f3c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f36ae1055e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f36ae0d9d8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f36ae1f66a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 68: Internal error /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7ff79ed621d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7ff79e4ea031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7ff79e664c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7ff79e5765e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7ff79e54ad8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7ff79e6676a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 1: Internal error Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7fc3d6f0a1d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7fc3d6692031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7fc3d680cc5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7fc3d671e5e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7fc3d66f2d8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7fc3d680f6a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 84: Internal error Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f244dc141d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f244d39c031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f244d516c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f244d4285e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f244d3fcd8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f244d5196a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 79: Internal error /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f36308f51d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f363007d031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f36301f7c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f36301095e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f36300ddd8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f36301fa6a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 87: Internal error Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7ff7b2ba61d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7ff7b232e031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7ff7b24a8c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7ff7b23ba5e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7ff7b238ed8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7ff7b24ab6a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 85: Internal error Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f41a45a81d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f41a3d30031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f41a3eaac5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f41a3dbc5e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f41a3d90d8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f41a3ead6a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 74: Internal error Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f7a5dc331d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f7a5d3bb031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f7a5d535c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f7a5d4475e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f7a5d41bd8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f7a5d5386a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 72: Internal error /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f78081c61d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f780794e031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f7807ac8c5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f78079da5e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f78079aed8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f7807acb6a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 73: Internal error Assertion failed in file ../../src/mpid/ch4/src/intel/ch4_shm_coll.c at line 2101: node_info->numa_num <= ((MPIDI_SHMGR_SYNCPAGE_SIZE / MPIDI_SHMGR_FLAG_SPACE) - 1) /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPL_backtrace_show+0x34) [0x7f24fe79a1d4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(MPIR_Assert_fail+0x21) [0x7f24fdf22031] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x285c5d) [0x7f24fe09cc5d] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x1975e4) [0x7f24fdfae5e4] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x16bd8e) [0x7f24fdf82d8e] /apps/applications/development/compilers/intel/19.1/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12(+0x2886a7) [0x7f24fe09f6a7] /apps/applications/development/compilers/intel/1 Abort(1) on node 21: Internal error =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 0 PID 7450 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 4 PID 7454 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 5 PID 7455 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 6 PID 7456 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 7 PID 7457 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 8 PID 7458 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 9 PID 7459 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 10 PID 7460 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 11 PID 7461 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 12 PID 7462 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 13 PID 7463 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 14 PID 7464 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 15 PID 7465 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 16 PID 7466 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 17 PID 7467 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 18 PID 7468 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 19 PID 7469 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 21 PID 7472 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 24 PID 7475 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 25 PID 7476 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 26 PID 7477 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 27 PID 7478 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 28 PID 7479 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 29 PID 7480 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 30 PID 7481 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 31 PID 7482 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 32 PID 7483 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 33 PID 7484 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 34 PID 7485 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 35 PID 7486 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 36 PID 7487 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 37 PID 7488 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 38 PID 7489 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 39 PID 7490 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 40 PID 7491 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 41 PID 7492 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 42 PID 7494 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 43 PID 7495 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 44 PID 7497 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 45 PID 7498 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 46 PID 7499 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 47 PID 7500 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 48 PID 7501 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 49 PID 7502 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 50 PID 7503 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 51 PID 7504 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 52 PID 7505 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 53 PID 7506 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 54 PID 7507 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 55 PID 7508 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 56 PID 7509 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 57 PID 7510 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 58 PID 7511 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 59 PID 7512 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 60 PID 7513 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 61 PID 7514 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 62 PID 7515 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 63 PID 7516 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 64 PID 7517 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 65 PID 7518 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 66 PID 7519 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 67 PID 7520 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 69 PID 7522 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 75 PID 7528 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 78 PID 7531 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 80 PID 7533 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 81 PID 7534 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 82 PID 7535 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 83 PID 7536 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 88 PID 7542 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 89 PID 7543 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 90 PID 7544 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 91 PID 7545 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 92 PID 7546 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 93 PID 7547 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 94 PID 7548 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 95 PID 7549 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 96 PID 7550 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 97 PID 7551 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 98 PID 7552 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 99 PID 7553 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 100 PID 7554 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 101 PID 7555 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 102 PID 7556 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 103 PID 7557 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 104 PID 7558 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 105 PID 7559 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 106 PID 7560 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 107 PID 7561 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 108 PID 7562 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 109 PID 7563 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 110 PID 7564 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 111 PID 7565 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 112 PID 7566 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 113 PID 7567 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 114 PID 7568 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 115 PID 7569 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 116 PID 7570 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 117 PID 7571 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 118 PID 7572 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 119 PID 7573 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 120 PID 7574 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 121 PID 7575 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 122 PID 7576 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 123 PID 7577 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 124 PID 7578 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 125 PID 7579 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 126 PID 7580 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 127 PID 7581 RUNNING AT roux-rome-01 = KILLED BY SIGNAL: 9 (Killed) ===================================================================================
Thanks for your help
Zach
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Zach,
We are forwarding this issue to the concerned engineering team.
We will get back to you soon.
Regards
Prasanth
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We have released an update to the oneAPI beta HPC toolkit with better MPI support. Can you try the latest update and see if the problem is resolved?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Any luck trying the more recent version of IMPI? OneAPI has been officially released and IMPI is part of the HPC toolkit.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
As we have not heard back from you, we are considering that your issue has been resolved and we have answered all your queries. So we will no longer respond to this thread. If you require additional assistance from Intel, please start a new thread. Any further interaction in this thread will be considered community only
Have a Good day!
Thanks & Regards
Goutham

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page