Job submitted via qsub getting killed.

JThom29 · ‎07-31-2018

When I submit jobs using "qsub" command, it gets terminated after some time. It seems to be due to walltime getting exceeded. How can we increase walltime so that job will get completed.

Below is the error log.

=>> PBS: job killed: walltime 21642 exceeded limit 21600

*** Aborted at 1532971520 (unix time) try "date -d @1532971520" if you are using GNU date ***

PC: @ 0x2b3084a63adb mkldnn::impl::cpu::ref_lrn_bwd_t<>::execute_backward()

*** SIGTERM (@0x6f4) received by PID 80152 (TID 0x2b3083449980) from PID 1780; stack trace: ***

@ 0x2b3089e0f5e0 (unknown)

@ 0x2b3084a63adb mkldnn::impl::cpu::ref_lrn_bwd_t<>::execute_backward()

@ 0x2b30898fbac3 __kmp_invoke_microtask

@ 0x2b30898ca257 __kmp_invoke_task_func

@ 0x2b30898cb498 __kmp_fork_call

@ 0x2b30898a27de __kmpc_fork_call

@ 0x2b3084a634ec mkldnn::impl::cpu::ref_lrn_bwd_t<>::execute_backwa

16364@c002-n003 ~]$ git status | more

fatal: Not a git repository (or any parent up to mount point /home)

Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

[u16364@c002-n003 ~]$ cd git_repo/cpp_crack_app/

[u16364@c002-n003 cpp_crack_app]$ git status | more

# On branch master

# Changes not staged for commit:

# (use "git add ..." to update what will be committed)

# (use "git checkout -- ..." to discard changes in working directory)

#

# modified: build/Makefile

# modified: src/crack.cpp

# modified: src/main.cpp

#

JThom29 · ‎07-31-2018

I have tried with "# PBS -l walltime=48:00:00" option. Still it didn't resolved.

Please do correct me if this option wrong.

idata · ‎07-31-2018

Hi Joseph,

Thanks for reaching out to us.

You can increase the walltime by adding it in bash script before running in qsub mode as shown below:

# PBS -l walltime=24:00:00

Then run "qsub "

Please note that the max walltime is 24 hours.

Hope this helps.

Regards,

Deepthi

jKuma1 · ‎07-31-2018

Will this work even with a job triggered via Jupyter terminal and extend the walltime to 24hours?

idata · ‎08-01-2018

Hi Joseph,

For a Jupyter terminal, the maximum session time is 4 hrs.

We will run a qsub job that requires more time from the jupyter terminal and get back to you on the results.

Regards,

Deepthi

idata · ‎08-01-2018

Hi Joseph,

It is observed that post the maximum session time (4 hrs), the log files are not generated. So it is not ideal to submit a qsub job via Jupyter terminal which takes more than the session time.

Hope that helps.

Regards,

Deepthi

idata · ‎08-02-2018

Hi Joseph,

Could you please confirm if the queries got cleared.

Regards,

Deepthi Raj

idata · ‎08-05-2018

Hi Joseph,

Hope the queries are cleared. We are closing this thread. Please raise a new thread if you are facing any other issues.

Regards,

Deepthi Raj.