pbs_demux: error while loading shared libraries: libtorque.so.2: cannot open shared object file: No such file or directory
Use of uninitialized value in scalar assignment at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 130.
Use of uninitialized value in substitution (s///) at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in scalar assignment at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 136.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
Use of uninitialized value in substitution iterator at /home/somanath/UM_bench/dvd1/Scripts/run_um.pl line 133.
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
libc.so.6          0000003D13230215  Unknown               Unknown  Unknown
libc.so.6          0000003D13231CC0  Unknown               Unknown  Unknown
N216L85.exe        000000000040A416  Unknown               Unknown  Unknown
N216L85.exe        000000000044D172  Unknown               Unknown  Unknown
libmpi.so.4        00002AEF364CD7FA  Unknown               Unknown  Unknown
libmpi.so.4        00002AEF363F1661  Unknown               Unknown  Unknown
libmpigf.so.4      00002AEF369A8279  Unknown               Unknown  Unknown
N216L85.exe        000000000175EADD  Unknown               Unknown  Unknown
N216L85.exe        0000000001058602  Unknown               Unknown  Unknown
N216L85.exe        000000000106DFEA  Unknown               Unknown  Unknown
N216L85.exe        0000000000B91CFD  Unknown               Unknown  Unknown
N216L85.exe        000000000089B3FA  Unknown               Unknown  Unknown
N216L85.exe        00000000004E1F04  Unknown               Unknown  Unknown
N216L85.exe        000000000048548F  Unknown               Unknown  Unknown
N216L85.exe        000000000040D76B  Unknown               Unknown  Unknown
N216L85.exe        0000000000404C7C  Unknown               Unknown  Unknown
N216L85.exe        0000000000404BAC  Unknown               Unknown  Unknown
libc.so.6          0000003D1321D974  Unknown               Unknown  Unknown
N216L85.exe        0000000000404AB9  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
libc.so.6          0000003B46430215  Unknown               Unknown  Unknown
libc.so.6          0000003B46431CC0  Unknown               Unknown  Unknown
N216L85.exe        000000000040A416  Unknown               Unknown  Unknown
N216L85.exe        000000000044D172  Unknown               Unknown  Unknown
libmpi.so.4        00002B96850F07FA  Unknown               Unknown  Unknown
libmpi.so.4        00002B9685014661  Unknown               Unknown  Unknown
libmpigf.so.4      00002B96855CB279  Unknown               Unknown  Unknown
N216L85.exe        000000000175EADD  Unknown               Unknown  Unknown
N216L85.exe        0000000001059A61  Unknown               Unknown  Unknown
N216L85.exe        000000000106DFEA  Unknown               Unknown  Unknown
N216L85.exe        0000000000B91CFD  Unknown               Unknown  Unknown
N216L85.exe        000000000089B3FA  Unknown               Unknown  Unknown
N216L85.exe        00000000004E1F04  Unknown               Unknown  Unknown
N216L85.exe        000000000048548F  Unknown               Unknown  Unknown
N216L85.exe        000000000040D76B  Unknown               Unknown  Unknown
N216L85.exe        0000000000404C7C  Unknown               Unknown  Unknown
N216L85.exe        0000000000404BAC  Unknown               Unknown  Unknown
libc.so.6          0000003B4641D974  Unknown               Unknown  Unknown
N216L85.exe        0000000000404AB9  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
libc.so.6          0000003B46430215  Unknown               Unknown  Unknown
libc.so.6          0000003B46431CC0  Unknown               Unknown  Unknown
N216L85.exe        000000000040A416  Unknown               Unknown  Unknown
N216L85.exe        000000000044D172  Unknown               Unknown  Unknown
libmpi.so.4        00002B32A994F7FA  Unknown               Unknown  Unknown
libmpi.so.4        00002B32A9873661  Unknown               Unknown  Unknown
libmpigf.so.4      00002B32A9E2A279  Unknown               Unknown  Unknown
N216L85.exe        000000000175EADD  Unknown               Unknown  Unknown
N216L85.exe        0000000001059A61  Unknown               Unknown  Unknown
N216L85.exe        000000000106DFEA  Unknown               Unknown  Unknown
N216L85.exe        0000000000B91CFD  Unknown               Unknown  Unknown
N216L85.exe        000000000089B3FA  Unknown               Unknown  Unknown
N216L85.exe        00000000004E1F04  Unknown               Unknown  Unknown
N216L85.exe        000000000048548F  Unknown               Unknown  Unknown
N216L85.exe        000000000040D76B  Unknown               Unknown  Unknown
N216L85.exe        0000000000404C7C  Unknown               Unknown  Unknown
N216L85.exe        0000000000404BAC  Unknown               Unknown  Unknown
libc.so.6          0000003B4641D974  Unknown               Unknown  Unknown
N216L85.exe        0000000000404AB9  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
libc.so.6          0000003488A30215  Unknown               Unknown  Unknown
libc.so.6          0000003488A31CC0  Unknown               Unknown  Unknown
N216L85.exe        000000000040A416  Unknown               Unknown  Unknown
N216L85.exe        000000000044D172  Unknown               Unknown  Unknown
libmpi.so.4        00002B8DB34627FA  Unknown               Unknown  Unknown
libmpi.so.4        00002B8DB3386661  Unknown               Unknown  Unknown
libmpigf.so.4      00002B8DB393D279  Unknown               Unknown  Unknown
N216L85.exe        000000000175EADD  Unknown               Unknown  Unknown
N216L85.exe        0000000001058602  Unknown               Unknown  Unknown
N216L85.exe        000000000106DFEA  Unknown               Unknown  Unknown
N216L85.exe        0000000000B91CFD  Unknown               Unknown  Unknown
N216L85.exe        000000000089B3FA  Unknown               Unknown  Unknown
N216L85.exe        00000000004E1F04  Unknown               Unknown  Unknown
N216L85.exe        000000000048548F  Unknown               Unknown  Unknown
N216L85.exe        000000000040D76B  Unknown               Unknown  Unknown
N216L85.exe        0000000000404C7C  Unknown               Unknown  Unknown
N216L85.exe        0000000000404BAC  Unknown               Unknown  Unknown
libc.so.6          0000003488A1D974  Unknown               Unknown  Unknown
N216L85.exe        0000000000404AB9  Unknown               Unknown  Unknown
send desc error
send desc error
[11] Abort: Got completion with error 12, vendor code=81, dest rank=
 at line 870 in file ../../ofa_poll.c
[9] Abort: Got completion with error 12, vendor code=81, dest rank=
 at line 870 in file ../../ofa_poll.c
=>> PBS: job killed: walltime 126034 exceeded limit 126000
[mpiexec@compute-0-26.local] HYD_pmcd_pmiserv_send_signal (./pm/pmiserv/pmiserv_cb.c:221): assert (!closed) failed
[mpiexec@compute-0-26.local] ui_cmd_cb (./pm/pmiserv/pmiserv_pmci.c:128): unable to send SIGUSR1 downstream
[mpiexec@compute-0-26.local] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[mpiexec@compute-0-26.local] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:388): error waiting for event
[mpiexec@compute-0-26.local] main (./ui/mpich/mpiexec.c:718): process manager error waiting for completion