+ mpif90 -I/usr/local/other/SLES11.3/ncl/gcc-4.3.4/6.3.0-static/include -I/usr/local/other/SLES11.3/silo/4.10.2/include -extend-source -r8 -c 3dpic_full_mpi.f + mpif90 -o 3dpic_full_mpi.exe 3dpic_full_mpi.o -L/usr/local/other/SLES11.3/ncl/gcc-4.3.4/6.3.0-static/lib -L/usr/local/other/SLES11.3/silo/4.10.2/lib -lsiloh5 -lhdf5_hl -lhdf5 -lsz -lz -lm -lrt -ldl -lstdc++ rm: cannot remove `pcrimth.dat': No such file or directory forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source 3dpic_full_mpi.ex 0000000000869189 Unknown Unknown Unknown 3dpic_full_mpi.ex 0000000000867A5E Unknown Unknown Unknown 3dpic_full_mpi.ex 0000000000825B72 Unknown Unknown Unknown 3dpic_full_mpi.ex 00000000007F2633 Unknown Unknown Unknown 3dpic_full_mpi.ex 00000000007F621B Unknown Unknown Unknown libpthread.so.0 00002AAAAB669810 Unknown Unknown Unknown libc.so.6 00002AAAAC126C52 Unknown Unknown Unknown 3dpic_full_mpi.ex 00000000005389A2 Unknown Unknown Unknown 3dpic_full_mpi.ex 00000000004A6643 Unknown Unknown Unknown 3dpic_full_mpi.ex 0000000000462106 Unknown Unknown Unknown 3dpic_full_mpi.ex 000000000041B72F Unknown Unknown Unknown 3dpic_full_mpi.ex 00000000004165C6 Unknown Unknown Unknown libc.so.6 00002AAAAC02FC36 Unknown Unknown Unknown 3dpic_full_mpi.ex 00000000004164B9 Unknown Unknown Unknown srun.slurm: error: borgo015: task 0: Exited with exit code 174 MPT ERROR: borgo021 has had continuous IB fabric problems for 10 (MPI_WATCHDOG_TIMER) minutes trying to reach borgo015. Aborting. MPT ERROR: borgo020 has had continuous IB fabric problems for 10 (MPI_WATCHDOG_TIMER) minutes trying to reach borgo015. Aborting. MPT: Global rank 32 is aborting with error code 0. Process ID: 12240, Host: borgo021, Program: /gpfsm/dnb32/gbrambil/Kcode/pulsarSILOF/3dpic_full_mpi.exe MPT: --------stack traceback------- MPT: Global rank 16 is aborting with error code 0. Process ID: 5232, Host: borgo020, Program: /gpfsm/dnb32/gbrambil/Kcode/pulsarSILOF/3dpic_full_mpi.exe MPT: --------stack traceback------- MPT ERROR: borgo033 has had continuous IB fabric problems for 10 (MPI_WATCHDOG_TIMER) minutes trying to reach borgo015. Aborting. MPT: Global rank 128 is aborting with error code 0. Process ID: 12467, Host: borgo033, Program: /gpfsm/dnb32/gbrambil/Kcode/pulsarSILOF/3dpic_full_mpi.exe MPT: --------stack traceback------- MPT ERROR: borgo023 has had continuous IB fabric problems for 10 (MPI_WATCHDOG_TIMER) minutes trying to reach borgo015. Aborting. MPT: Global rank 64 is aborting with error code 0. Process ID: 31976, Host: borgo023, Program: /gpfsm/dnb32/gbrambil/Kcode/pulsarSILOF/3dpic_full_mpi.exe MPT: --------stack traceback------- MPT ERROR: borgo100 has had continuous IB fabric problems for 10 (MPI_WATCHDOG_TIMER) minutes trying to reach borgo015. Aborting. MPT: Global rank 256 is aborting with error code 0. Process ID: 15596, Host: borgo100, Program: /gpfsm/dnb32/gbrambil/Kcode/pulsarSILOF/3dpic_full_mpi.exe MPT: --------stack traceback------- MPT ERROR: borgo191 has had continuous IB fabric problems for 10 (MPI_WATCHDOG_TIMER) minutes trying to reach borgo015. Aborting. MPT: Global rank 512 is aborting with error code 0. Process ID: 18777, Host: borgo191, Program: /gpfsm/dnb32/gbrambil/Kcode/pulsarSILOF/3dpic_full_mpi.exe MPT: --------stack traceback------- MPT ERROR: borgq079 has had continuous IB fabric problems for 10 (MPI_WATCHDOG_TIMER) minutes trying to reach borgo015. Aborting. MPT: Global rank 1024 is aborting with error code 0. Process ID: 24838, Host: borgq079, Program: /gpfsm/dnb32/gbrambil/Kcode/pulsarSILOF/3dpic_full_mpi.exe MPT: --------stack traceback------- MPT: Attaching to program: /proc/18777/exe, process 18777 MPT: Try: zypper install -C "debuginfo(build-id)=d4191084441e39a7b480fc4b41f67083812e9811" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=682e4f7a27ee294a58f17249a0717861db546f2d" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=32c1c7a7a20b54ac3af6b2f436b3375ffeb12f0b" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=2f51a06469a025d507534fe292dcf4e02235bd18" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=91334fd8105f0b62c0bdbbec14b45a9fd043f4c3" MPT: (no debugging symbols found)...done. MPT: [Thread debugging using libthread_db enabled] MPT: Using host libthread_db library "/lib64/libthread_db.so.1". MPT: Try: zypper install -C "debuginfo(build-id)=4aee0c3923838575483ebd16be6db85ecb6f0b75" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=3b149eccd897f1f37dce50ad22614043eba757a2" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=a95c0ce9f9752baf052d3b55b3b8ce19f662d2eb" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=719375f80fd84b85b905db2c20ec70e8805b36e5" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=4f3dc8efbe18b50a6abe70d8b2f862ce185542d6" MPT: (no debugging symbols found)...done. MPT: 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: (gdb) #0 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: #1 0x00002aaaabd6f32c in mpi_sgi_system (command=) at sig.c:99 MPT: #2 MPI_SGI_stacktraceback (header=) at sig.c:319 MPT: #3 0x00002aaaabcc4a4a in print_traceback (ecode=0) at abort.c:197 MPT: #4 0x00002aaaabcc4b3e in MPI_SGI_abort () at abort.c:85 MPT: #5 0x00002aaaabd7ff4f in try_repush (gap=, MPT: now=2886902.8844944318) at ud.c:1599 MPT: #6 0x00002aaaabd80eb6 in MPI_SGI_ud_progress () at ud.c:1769 MPT: #7 0x00002aaaabd57f62 in MPI_SGI_progress_devices () at progress.c:118 MPT: #8 MPI_SGI_progress () at progress.c:241 MPT: #9 0x00002aaaabd690d5 in MPI_SGI_request_wait (request=0x7fffffffa35c, MPT: status=0x2aaaac010970 , set=0x7fffffffa364, MPT: gen_rc=0x7fffffffa360) at req.c:1703 MPT: #10 0x00002aaaabcd6d32 in MPI_SGI_barrier_basic (comm=6) at barrier.c:61 MPT: #11 0x00002aaaabcd6e65 in MPI_SGI_barrier (comm=6) at barrier.c:204 MPT: #12 0x00002aaaabcd70f5 in MPI_SGI_barrier (comm=) MPT: at barrier.c:166 MPT: #13 0x00002aaaabcd7193 in PMPI_Barrier (comm=1) at barrier.c:344 MPT: #14 0x00002aaaabcd72df in pmpi_barrier__ () MPT: from /usr/local/sgi/mpi/mpt-2.12/opt/sgi/mpt/mpt-2.12/lib/libmpi.so MPT: #15 0x000000000041bd07 in MAIN__ () MPT: #16 0x00000000004165c6 in main () MPT: (gdb) A debugging session is active. MPT: MPT: Inferior 1 [process 18777] will be detached. MPT: MPT: Quit anyway? (y or n) [answered Y; input not from terminal] MPT: Detaching from program: /proc/18777/exe, process 18777 MPT: Attaching to program: /proc/5232/exe, process 5232 MPT: Try: zypper install -C "debuginfo(build-id)=d4191084441e39a7b480fc4b41f67083812e9811" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=682e4f7a27ee294a58f17249a0717861db546f2d" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=32c1c7a7a20b54ac3af6b2f436b3375ffeb12f0b" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=2f51a06469a025d507534fe292dcf4e02235bd18" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=91334fd8105f0b62c0bdbbec14b45a9fd043f4c3" MPT: (no debugging symbols found)...done. MPT: [Thread debugging using libthread_db enabled] MPT: Using host libthread_db library "/lib64/libthread_db.so.1". MPT: Try: zypper install -C "debuginfo(build-id)=4aee0c3923838575483ebd16be6db85ecb6f0b75" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=3b149eccd897f1f37dce50ad22614043eba757a2" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=a95c0ce9f9752baf052d3b55b3b8ce19f662d2eb" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=719375f80fd84b85b905db2c20ec70e8805b36e5" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=4f3dc8efbe18b50a6abe70d8b2f862ce185542d6" MPT: (no debugging symbols found)...done. MPT: 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: (gdb) #0 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: #1 0x00002aaaabd6f32c in mpi_sgi_system (command=) at sig.c:99 MPT: #2 MPI_SGI_stacktraceback (header=) at sig.c:319 MPT: #3 0x00002aaaabcc4a4a in print_traceback (ecode=0) at abort.c:197 MPT: #4 0x00002aaaabcc4b3e in MPI_SGI_abort () at abort.c:85 MPT: #5 0x00002aaaabd7ff4f in try_repush (gap=, MPT: now=2886904.607016339) at ud.c:1599 MPT: #6 0x00002aaaabd80eb6 in MPI_SGI_ud_progress () at ud.c:1769 MPT: #7 0x00002aaaabd57f62 in MPI_SGI_progress_devices () at progress.c:118 MPT: #8 MPI_SGI_progress () at progress.c:241 MPT: #9 0x00002aaaabd690d5 in MPI_SGI_request_wait (request=0x7fffffffa3dc, MPT: status=0x2aaaac010970 , set=0x7fffffffa3e4, MPT: gen_rc=0x7fffffffa3e0) at req.c:1703 MPT: #10 0x00002aaaabcd6d32 in MPI_SGI_barrier_basic (comm=6) at barrier.c:61 MPT: #11 0x00002aaaabcd6e65 in MPI_SGI_barrier (comm=6) at barrier.c:204 MPT: #12 0x00002aaaabcd70f5 in MPI_SGI_barrier (comm=) MPT: at barrier.c:166 MPT: #13 0x00002aaaabcd7193 in PMPI_Barrier (comm=1) at barrier.c:344 MPT: #14 0x00002aaaabcd72df in pmpi_barrier__ () MPT: from /usr/local/sgi/mpi/mpt-2.12/opt/sgi/mpt/mpt-2.12/lib/libmpi.so MPT: #15 0x000000000041bd07 in MAIN__ () MPT: #16 0x00000000004165c6 in main () MPT: (gdb) A debugging session is active. MPT: MPT: Inferior 1 [process 5232] will be detached. MPT: MPT: Quit anyway? (y or n) [answered Y; input not from terminal] MPT: Detaching from program: /proc/5232/exe, process 5232 MPT: Attaching to program: /proc/31976/exe, process 31976 MPT: Try: zypper install -C "debuginfo(build-id)=d4191084441e39a7b480fc4b41f67083812e9811" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=682e4f7a27ee294a58f17249a0717861db546f2d" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=32c1c7a7a20b54ac3af6b2f436b3375ffeb12f0b" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=2f51a06469a025d507534fe292dcf4e02235bd18" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=91334fd8105f0b62c0bdbbec14b45a9fd043f4c3" MPT: (no debugging symbols found)...done. MPT: [Thread debugging using libthread_db enabled] MPT: Using host libthread_db library "/lib64/libthread_db.so.1". MPT: Try: zypper install -C "debuginfo(build-id)=4aee0c3923838575483ebd16be6db85ecb6f0b75" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=3b149eccd897f1f37dce50ad22614043eba757a2" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=a95c0ce9f9752baf052d3b55b3b8ce19f662d2eb" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=719375f80fd84b85b905db2c20ec70e8805b36e5" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=4f3dc8efbe18b50a6abe70d8b2f862ce185542d6" MPT: (no debugging symbols found)...done. MPT: 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: (gdb) #0 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: #1 0x00002aaaabd6f32c in mpi_sgi_system (command=) at sig.c:99 MPT: #2 MPI_SGI_stacktraceback (header=) at sig.c:319 MPT: #3 0x00002aaaabcc4a4a in print_traceback (ecode=0) at abort.c:197 MPT: #4 0x00002aaaabcc4b3e in MPI_SGI_abort () at abort.c:85 MPT: #5 0x00002aaaabd7ff4f in try_repush (gap=, MPT: now=2886904.7364443871) at ud.c:1599 MPT: #6 0x00002aaaabd80eb6 in MPI_SGI_ud_progress () at ud.c:1769 MPT: #7 0x00002aaaabd57f62 in MPI_SGI_progress_devices () at progress.c:118 MPT: #8 MPI_SGI_progress () at progress.c:241 MPT: #9 0x00002aaaabd690d5 in MPI_SGI_request_wait (request=0x7fffffffa3dc, MPT: status=0x2aaaac010970 , set=0x7fffffffa3e4, MPT: gen_rc=0x7fffffffa3e0) at req.c:1703 MPT: #10 0x00002aaaabcd6d32 in MPI_SGI_barrier_basic (comm=6) at barrier.c:61 MPT: #11 0x00002aaaabcd6e65 in MPI_SGI_barrier (comm=6) at barrier.c:204 MPT: #12 0x00002aaaabcd70f5 in MPI_SGI_barrier (comm=) MPT: at barrier.c:166 MPT: #13 0x00002aaaabcd7193 in PMPI_Barrier (comm=1) at barrier.c:344 MPT: #14 0x00002aaaabcd72df in pmpi_barrier__ () MPT: from /usr/local/sgi/mpi/mpt-2.12/opt/sgi/mpt/mpt-2.12/lib/libmpi.so MPT: #15 0x000000000041bd07 in MAIN__ () MPT: #16 0x00000000004165c6 in main () MPT: (gdb) A debugging session is active. MPT: MPT: Inferior 1 [process 31976] will be detached. MPT: MPT: Quit anyway? (y or n) [answered Y; input not from terminal] MPT: Detaching from program: /proc/31976/exe, process 31976 MPT: Attaching to program: /proc/24838/exe, process 24838 MPT: Try: zypper install -C "debuginfo(build-id)=d4191084441e39a7b480fc4b41f67083812e9811" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=682e4f7a27ee294a58f17249a0717861db546f2d" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=32c1c7a7a20b54ac3af6b2f436b3375ffeb12f0b" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=2f51a06469a025d507534fe292dcf4e02235bd18" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=91334fd8105f0b62c0bdbbec14b45a9fd043f4c3" MPT: (no debugging symbols found)...done. MPT: [Thread debugging using libthread_db enabled] MPT: Using host libthread_db library "/lib64/libthread_db.so.1". MPT: Try: zypper install -C "debuginfo(build-id)=4aee0c3923838575483ebd16be6db85ecb6f0b75" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=3b149eccd897f1f37dce50ad22614043eba757a2" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=a95c0ce9f9752baf052d3b55b3b8ce19f662d2eb" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=719375f80fd84b85b905db2c20ec70e8805b36e5" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=4f3dc8efbe18b50a6abe70d8b2f862ce185542d6" MPT: (no debugging symbols found)...done. MPT: 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: (gdb) #0 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: #1 0x00002aaaabd6f32c in mpi_sgi_system (command=) at sig.c:99 MPT: #2 MPI_SGI_stacktraceback (header=) at sig.c:319 MPT: #3 0x00002aaaabcc4a4a in print_traceback (ecode=0) at abort.c:197 MPT: #4 0x00002aaaabcc4b3e in MPI_SGI_abort () at abort.c:85 MPT: #5 0x00002aaaabd7ff4f in try_repush (gap=, MPT: now=2886899.4397963542) at ud.c:1599 MPT: #6 0x00002aaaabd80eb6 in MPI_SGI_ud_progress () at ud.c:1769 MPT: #7 0x00002aaaabd57f62 in MPI_SGI_progress_devices () at progress.c:118 MPT: #8 MPI_SGI_progress () at progress.c:241 MPT: #9 0x00002aaaabd690d5 in MPI_SGI_request_wait (request=0x7fffffffa324, MPT: status=0x2aaaac010970 , set=0x7fffffffa320, MPT: gen_rc=0x7fffffffa31c) at req.c:1703 MPT: #10 0x00002aaaabd7324d in MPI_SGI_recv (buf=, MPT: count=, type=, des=, MPT: tag=, comm=, MPT: status=0x2aaaac010970 ) at sugar.c:40 MPT: #11 0x00002aaaabcd6df3 in MPI_SGI_barrier_basic (comm=6) at barrier.c:74 MPT: #12 0x00002aaaabcd6e65 in MPI_SGI_barrier (comm=6) at barrier.c:204 MPT: #13 0x00002aaaabcd70f5 in MPI_SGI_barrier (comm=) MPT: at barrier.c:166 MPT: #14 0x00002aaaabcd7193 in PMPI_Barrier (comm=1) at barrier.c:344 MPT: #15 0x00002aaaabcd72df in pmpi_barrier__ () MPT: from /usr/local/sgi/mpi/mpt-2.12/opt/sgi/mpt/mpt-2.12/lib/libmpi.so MPT: #16 0x000000000041bd07 in MAIN__ () MPT: #17 0x00000000004165c6 in main () MPT: (gdb) A debugging session is active. MPT: MPT: Inferior 1 [process 24838] will be detached. MPT: MPT: Quit anyway? (y or n) [answered Y; input not from terminal] MPT: Detaching from program: /proc/24838/exe, process 24838 MPT: Attaching to program: /proc/12467/exe, process 12467 MPT: Try: zypper install -C "debuginfo(build-id)=d4191084441e39a7b480fc4b41f67083812e9811" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=682e4f7a27ee294a58f17249a0717861db546f2d" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=32c1c7a7a20b54ac3af6b2f436b3375ffeb12f0b" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=2f51a06469a025d507534fe292dcf4e02235bd18" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=91334fd8105f0b62c0bdbbec14b45a9fd043f4c3" MPT: (no debugging symbols found)...done. MPT: [Thread debugging using libthread_db enabled] MPT: Using host libthread_db library "/lib64/libthread_db.so.1". MPT: Try: zypper install -C "debuginfo(build-id)=4aee0c3923838575483ebd16be6db85ecb6f0b75" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=3b149eccd897f1f37dce50ad22614043eba757a2" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=a95c0ce9f9752baf052d3b55b3b8ce19f662d2eb" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=719375f80fd84b85b905db2c20ec70e8805b36e5" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=4f3dc8efbe18b50a6abe70d8b2f862ce185542d6" MPT: (no debugging symbols found)...done. MPT: 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: (gdb) #0 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: #1 0x00002aaaabd6f32c in mpi_sgi_system (command=) at sig.c:99 MPT: #2 MPI_SGI_stacktraceback (header=) at sig.c:319 MPT: #3 0x00002aaaabcc4a4a in print_traceback (ecode=0) at abort.c:197 MPT: #4 0x00002aaaabcc4b3e in MPI_SGI_abort () at abort.c:85 MPT: #5 0x00002aaaabd7ff4f in try_repush (gap=, MPT: now=2886904.2856587609) at ud.c:1599 MPT: #6 0x00002aaaabd80eb6 in MPI_SGI_ud_progress () at ud.c:1769 MPT: #7 0x00002aaaabd57f62 in MPI_SGI_progress_devices () at progress.c:118 MPT: #8 MPI_SGI_progress () at progress.c:241 MPT: #9 0x00002aaaabd690d5 in MPI_SGI_request_wait (request=0x7fffffffa35c, MPT: status=0x2aaaac010970 , set=0x7fffffffa364, MPT: gen_rc=0x7fffffffa360) at req.c:1703 MPT: #10 0x00002aaaabcd6d32 in MPI_SGI_barrier_basic (comm=6) at barrier.c:61 MPT: #11 0x00002aaaabcd6e65 in MPI_SGI_barrier (comm=6) at barrier.c:204 MPT: #12 0x00002aaaabcd70f5 in MPI_SGI_barrier (comm=) MPT: at barrier.c:166 MPT: #13 0x00002aaaabcd7193 in PMPI_Barrier (comm=1) at barrier.c:344 MPT: #14 0x00002aaaabcd72df in pmpi_barrier__ () MPT: from /usr/local/sgi/mpi/mpt-2.12/opt/sgi/mpt/mpt-2.12/lib/libmpi.so MPT: #15 0x000000000041bd07 in MAIN__ () MPT: #16 0x00000000004165c6 in main () MPT: (gdb) A debugging session is active. MPT: MPT: Inferior 1 [process 12467] will be detached. MPT: MPT: Quit anyway? (y or n) [answered Y; input not from terminal] MPT: Detaching from program: /proc/12467/exe, process 12467 MPT: Attaching to program: /proc/12240/exe, process 12240 MPT: Try: zypper install -C "debuginfo(build-id)=d4191084441e39a7b480fc4b41f67083812e9811" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=682e4f7a27ee294a58f17249a0717861db546f2d" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=32c1c7a7a20b54ac3af6b2f436b3375ffeb12f0b" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=2f51a06469a025d507534fe292dcf4e02235bd18" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=91334fd8105f0b62c0bdbbec14b45a9fd043f4c3" MPT: (no debugging symbols found)...done. MPT: [Thread debugging using libthread_db enabled] MPT: Using host libthread_db library "/lib64/libthread_db.so.1". MPT: Try: zypper install -C "debuginfo(build-id)=4aee0c3923838575483ebd16be6db85ecb6f0b75" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=3b149eccd897f1f37dce50ad22614043eba757a2" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=a95c0ce9f9752baf052d3b55b3b8ce19f662d2eb" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=719375f80fd84b85b905db2c20ec70e8805b36e5" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=4f3dc8efbe18b50a6abe70d8b2f862ce185542d6" MPT: (no debugging symbols found)...done. MPT: 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: (gdb) #0 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: #1 0x00002aaaabd6f32c in mpi_sgi_system (command=) at sig.c:99 MPT: #2 MPI_SGI_stacktraceback (header=) at sig.c:319 MPT: #3 0x00002aaaabcc4a4a in print_traceback (ecode=0) at abort.c:197 MPT: #4 0x00002aaaabcc4b3e in MPI_SGI_abort () at abort.c:85 MPT: #5 0x00002aaaabd7ff4f in try_repush (gap=, MPT: now=2886905.1025811359) at ud.c:1599 MPT: #6 0x00002aaaabd80eb6 in MPI_SGI_ud_progress () at ud.c:1769 MPT: #7 0x00002aaaabd57f62 in MPI_SGI_progress_devices () at progress.c:118 MPT: #8 MPI_SGI_progress () at progress.c:241 MPT: #9 0x00002aaaabd690d5 in MPI_SGI_request_wait (request=0x7fffffffa3dc, MPT: status=0x2aaaac010970 , set=0x7fffffffa3e4, MPT: gen_rc=0x7fffffffa3e0) at req.c:1703 MPT: #10 0x00002aaaabcd6d32 in MPI_SGI_barrier_basic (comm=6) at barrier.c:61 MPT: #11 0x00002aaaabcd6e65 in MPI_SGI_barrier (comm=6) at barrier.c:204 MPT: #12 0x00002aaaabcd70f5 in MPI_SGI_barrier (comm=) MPT: at barrier.c:166 MPT: #13 0x00002aaaabcd7193 in PMPI_Barrier (comm=1) at barrier.c:344 MPT: #14 0x00002aaaabcd72df in pmpi_barrier__ () MPT: from /usr/local/sgi/mpi/mpt-2.12/opt/sgi/mpt/mpt-2.12/lib/libmpi.so MPT: #15 0x000000000041bd07 in MAIN__ () MPT: #16 0x00000000004165c6 in main () MPT: (gdb) A debugging session is active. MPT: MPT: Inferior 1 [process 12240] will be detached. MPT: MPT: Quit anyway? (y or n) [answered Y; input not from terminal] MPT: Detaching from program: /proc/12240/exe, process 12240 MPT: Attaching to program: /proc/15596/exe, process 15596 MPT: Try: zypper install -C "debuginfo(build-id)=d4191084441e39a7b480fc4b41f67083812e9811" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=682e4f7a27ee294a58f17249a0717861db546f2d" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=32c1c7a7a20b54ac3af6b2f436b3375ffeb12f0b" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=2f51a06469a025d507534fe292dcf4e02235bd18" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=91334fd8105f0b62c0bdbbec14b45a9fd043f4c3" MPT: (no debugging symbols found)...done. MPT: [Thread debugging using libthread_db enabled] MPT: Using host libthread_db library "/lib64/libthread_db.so.1". MPT: Try: zypper install -C "debuginfo(build-id)=4aee0c3923838575483ebd16be6db85ecb6f0b75" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=3b149eccd897f1f37dce50ad22614043eba757a2" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=a95c0ce9f9752baf052d3b55b3b8ce19f662d2eb" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=719375f80fd84b85b905db2c20ec70e8805b36e5" MPT: (no debugging symbols found)...done. MPT: Try: zypper install -C "debuginfo(build-id)=4f3dc8efbe18b50a6abe70d8b2f862ce185542d6" MPT: (no debugging symbols found)...done. MPT: 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: (gdb) #0 0x00002aaaab66937f in waitpid () from /lib64/libpthread.so.0 MPT: #1 0x00002aaaabd6f32c in mpi_sgi_system (command=) at sig.c:99 MPT: #2 MPI_SGI_stacktraceback (header=) at sig.c:319 MPT: #3 0x00002aaaabcc4a4a in print_traceback (ecode=0) at abort.c:197 MPT: #4 0x00002aaaabcc4b3e in MPI_SGI_abort () at abort.c:85 MPT: #5 0x00002aaaabd7ff4f in try_repush (gap=, MPT: now=2886904.6996221012) at ud.c:1599 MPT: #6 0x00002aaaabd80eb6 in MPI_SGI_ud_progress () at ud.c:1769 MPT: #7 0x00002aaaabd57f62 in MPI_SGI_progress_devices () at progress.c:118 MPT: #8 MPI_SGI_progress () at progress.c:241 MPT: #9 0x00002aaaabd690d5 in MPI_SGI_request_wait (request=0x7fffffffa35c, MPT: status=0x2aaaac010970 , set=0x7fffffffa364, MPT: gen_rc=0x7fffffffa360) at req.c:1703 MPT: #10 0x00002aaaabcd6d32 in MPI_SGI_barrier_basic (comm=6) at barrier.c:61 MPT: #11 0x00002aaaabcd6e65 in MPI_SGI_barrier (comm=6) at barrier.c:204 MPT: #12 0x00002aaaabcd70f5 in MPI_SGI_barrier (comm=) MPT: at barrier.c:166 MPT: #13 0x00002aaaabcd7193 in PMPI_Barrier (comm=1) at barrier.c:344 MPT: #14 0x00002aaaabcd72df in pmpi_barrier__ () MPT: from /usr/local/sgi/mpi/mpt-2.12/opt/sgi/mpt/mpt-2.12/lib/libmpi.so MPT: #15 0x000000000041bd07 in MAIN__ () MPT: #16 0x00000000004165c6 in main () MPT: (gdb) A debugging session is active. MPT: MPT: Inferior 1 [process 15596] will be detached. MPT: MPT: Quit anyway? (y or n) [answered Y; input not from terminal] MPT: Detaching from program: /proc/15596/exe, process 15596 MPT: -----stack traceback ends----- MPT: -----stack traceback ends----- MPT: -----stack traceback ends----- MPT: -----stack traceback ends----- MPT: -----stack traceback ends----- MPT: -----stack traceback ends----- MPT: -----stack traceback ends----- slurmstepd-borgo015: *** STEP 5991137.0 CANCELLED AT 2015-11-17T19:18:23 *** on borgo015 srun.slurm: Job step aborted: Waiting up to 2 seconds for job step to finish. srun.slurm: error: borgo191: task 512: Exited with exit code 255 srun.slurm: error: borgo021: task 32: Exited with exit code 255 srun.slurm: error: borgo020: task 16: Exited with exit code 255 srun.slurm: error: borgo021: tasks 37,41: Killed srun.slurm: error: borgo171: tasks 375,383: Killed srun.slurm: error: borgq079: task 1024: Exited with exit code 255 srun.slurm: error: borgo020: task 23: Killed srun.slurm: error: borgq204: tasks 1616-1626,1628-1631: Killed srun.slurm: error: borgo033: task 128: Exited with exit code 255 srun.slurm: error: borgq101: task 1199: Killed srun.slurm: error: borgo031: task 127: Killed srun.slurm: error: borgp102: tasks 736-743,745-751: Killed srun.slurm: error: borgq006: tasks 912-921,923-927: Killed srun.slurm: error: borgq188: tasks 1424-1429,1431-1439: Killed srun.slurm: error: borgo172: tasks 384-397,399: Killed srun.slurm: error: borgq073: task 1009: Killed srun.slurm: error: borgp019: tasks 561-575: Killed srun.slurm: error: borgo198: tasks 544-545,547-559: Killed srun.slurm: error: borgq189: tasks 1440,1442-1455: Killed srun.slurm: error: borgo022: tasks 48-55,57-63: Killed srun.slurm: error: borgo084: tasks 209-223: Killed srun.slurm: error: borgo048: tasks 160,162-175: Killed srun.slurm: error: borgq203: tasks 1600-1606,1608-1615: Killed srun.slurm: error: borgq170: tasks 1408-1416,1418-1423: Killed srun.slurm: error: borgq198: tasks 1568-1571,1573-1583: Killed srun.slurm: error: borgq202: tasks 1584-1592,1594-1599: Killed srun.slurm: error: borgq084: tasks 1088-1097,1099-1103: Killed srun.slurm: error: borgo077: tasks 192-195,197-207: Killed srun.slurm: error: borgq148: tasks 1248-1254,1256-1263: Killed srun.slurm: error: borgq169: tasks 1393-1407: Killed srun.slurm: error: borgq083: tasks 1073-1087: Killed srun.slurm: error: borgo015: tasks 1-5,7-15: Killed srun.slurm: error: borgq005: tasks 896-901,903-911: Killed srun.slurm: error: borgq080: tasks 1040-1041,1043-1055: Killed srun.slurm: error: borgp081: tasks 704-705,707-719: Killed srun.slurm: error: borgq099: tasks 1152-1153,1164: Killed srun.slurm: error: borgq208: task 1661: Killed srun.slurm: error: borgo100: task 256: Exited with exit code 255 srun.slurm: error: borgp080: tasks 688-689,691-703: Killed srun.slurm: error: borgq104: tasks 1232-1239,1241-1247: Killed srun.slurm: error: borgp084: tasks 720-734: Killed srun.slurm: error: borgq003: tasks 865-879: Killed srun.slurm: error: borgo197: tasks 528-531,533-543: Killed srun.slurm: error: borgo049: tasks 176-187,189-191: Killed srun.slurm: error: borgo167: tasks 352-364,366-367: Killed srun.slurm: error: borgq149: tasks 1265-1279: Killed srun.slurm: error: borgq004: tasks 880-893,895: Killed srun.slurm: error: borgq103: tasks 1217-1231: Killed srun.slurm: error: borgq195: tasks 1536-1539,1541-1551: Killed srun.slurm: error: borgp079: tasks 672-681,683-687: Killed srun.slurm: error: borgq212: tasks 1712-1713,1715-1727: Killed srun.slurm: error: borgq081: tasks 1056,1058-1071: Killed srun.slurm: error: borgo034: tasks 144-153,155-159: Killed srun.slurm: error: borgq196: tasks 1552-1557,1559-1567: Killed srun.slurm: error: borgq167: tasks 1376-1384,1386-1391: Killed srun.slurm: error: borgo021: tasks 33-34,36,38-40,42-47: Killed srun.slurm: error: borgo139: tasks 320-333,335: Killed srun.slurm: error: borgq002: tasks 848-860,862-863: Killed srun.slurm: error: borgq102: tasks 1200-1205,1207-1215: Killed srun.slurm: error: borgo166: tasks 336-350: Killed srun.slurm: error: borgq001: tasks 832-838,840-847: Killed srun.slurm: error: borgq100: tasks 1168-1177,1179-1183: Killed srun.slurm: error: borgq211: tasks 1696-1704,1706-1711: Killed srun.slurm: error: borgp075: tasks 656-668,670-671: Killed srun.slurm: error: borgq166: tasks 1360,1362-1375: Killed srun.slurm: error: borgo108: tasks 304-315,317-319: Killed srun.slurm: error: borgq071: tasks 992-999,1001-1007: Killed srun.slurm: error: borgo191: tasks 513-522,524-527: Killed srun.slurm: error: borgp072: tasks 640-652,654-655: Killed srun.slurm: error: borgp200: tasks 816-825,827-831: Killed srun.slurm: error: borgq192: tasks 1488-1497,1499-1503: Killed srun.slurm: error: borgo189: tasks 496-508,510-511: Killed srun.slurm: error: borgq210: tasks 1680-1690,1692-1695: Killed srun.slurm: error: borgq194: tasks 1520-1529,1531-1535: Killed srun.slurm: error: borgo171: tasks 369-374,376-382: Killed srun.slurm: error: borgo106: tasks 288-300,302-303: Killed srun.slurm: error: borgo188: tasks 480,482-495: Killed srun.slurm: error: borgq209: tasks 1664-1669,1671-1679: Killed srun.slurm: error: borgo099: tasks 240-242,244-255: Killed srun.slurm: error: borgo020: tasks 17-22,24-30: Killed srun.slurm: error: borgp034: tasks 624-637,639: Killed srun.slurm: error: borgp197: tasks 784,786-799: Killed srun.slurm: error: borgq069: tasks 976-981,983-991: Killed srun.slurm: error: borgq193: tasks 1504-1517,1519: Killed srun.slurm: error: borgo024: tasks 80-89,91-95: Killed srun.slurm: error: borgq205: tasks 1632-1634,1636-1647: Killed srun.slurm: error: borgo186: tasks 464-467,469-479: Killed srun.slurm: error: borgo030: tasks 96-107,109-111: Killed srun.slurm: error: borgp033: tasks 608-612,614-623: Killed srun.slurm: error: borgq165: tasks 1344-1350,1352-1359: Killed srun.slurm: error: borgp199: tasks 800-809,811-815: Killed srun.slurm: error: borgo102: tasks 272-285,287: Killed srun.slurm: error: borgq019: tasks 945-959: Killed srun.slurm: error: borgo023: task 64: Exited with exit code 255 srun.slurm: error: borgp028: tasks 592-606: Killed srun.slurm: error: borgq023: tasks 960-974: Killed srun.slurm: error: borgq157: tasks 1312-1325,1327: Killed srun.slurm: error: borgo178: tasks 416-417,419-431: Killed srun.slurm: error: borgq190: tasks 1456-1460,1462-1471: Killed srun.slurm: error: borgq085: tasks 1104-1106,1108-1119: Killed srun.slurm: error: borgq097: tasks 1136-1141,1143-1151: Killed srun.slurm: error: borgo181: tasks 432-437,439-447: Killed srun.slurm: error: borgq164: tasks 1329-1343: Killed srun.slurm: error: borgq079: tasks 1025-1033,1035-1039: Killed srun.slurm: error: borgo183: tasks 448-454,456-463: Killed srun.slurm: error: borgq086: tasks 1120-1132,1134-1135: Killed srun.slurm: error: borgq156: tasks 1296-1301,1303-1311: Killed srun.slurm: error: borgq154: tasks 1280-1292,1294-1295: Killed srun.slurm: error: borgp022: tasks 577-591: Killed srun.slurm: error: borgq191: tasks 1472-1482,1484-1487: Killed srun.slurm: error: borgo098: tasks 224-226,228-239: Killed srun.slurm: error: borgp141: tasks 768-769,771-783: Killed srun.slurm: error: borgq008: tasks 928-938,940-943: Killed srun.slurm: error: borgp113: tasks 753-767: Killed srun.slurm: error: borgo173: tasks 400-414: Killed srun.slurm: error: borgo100: task 257: Killed srun.slurm: error: borgq101: tasks 1184-1196,1198: Killed srun.slurm: error: Timed out waiting for job step to complete srun.slurm: error: borgo033: tasks 129-130,132-143: Killed