topic Re: Job submitted via qsub getting killed. in AI Tools from Intel

Job submitted via qsub getting killed.

JThom29 — Tue, 31 Jul 2018 09:48:25 GMT

When I submit jobs using "qsub" command, it gets terminated after some time. It seems to be due to walltime getting exceeded. How can we increase walltime so that job will get completed.

Below is the error log.

=>> PBS: job killed: walltime 21642 exceeded limit 21600

*** Aborted at 1532971520 (unix time) try "date -d @1532971520" if you are using GNU date ***

PC: @ 0x2b3084a63adb mkldnn::impl::cpu::ref_lrn_bwd_t<>::execute_backward()

*** SIGTERM (@0x6f4) received by PID 80152 (TID 0x2b3083449980) from PID 1780; stack trace: ***

@ 0x2b3089e0f5e0 (unknown)

@ 0x2b3084a63adb mkldnn::impl::cpu::ref_lrn_bwd_t<>::execute_backward()

@ 0x2b30898fbac3 __kmp_invoke_microtask

@ 0x2b30898ca257 __kmp_invoke_task_func

@ 0x2b30898cb498 __kmp_fork_call

@ 0x2b30898a27de __kmpc_fork_call

@ 0x2b3084a634ec mkldnn::impl::cpu::ref_lrn_bwd_t<>::execute_backwa

16364@c002-n003 ~]$ git status | more

fatal: Not a git repository (or any parent up to mount point /home)

Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

[u16364@c002-n003 ~]$ cd git_repo/cpp_crack_app/

[u16364@c002-n003 cpp_crack_app]$ git status | more

# On branch master

# Changes not staged for commit:

# (use "git add ..." to update what will be committed)

# (use "git checkout -- ..." to discard changes in working directory)

# modified: build/Makefile

# modified: src/crack.cpp

# modified: src/main.cpp

Re: Job submitted via qsub getting killed.

JThom29 — Tue, 31 Jul 2018 09:58:25 GMT

I have tried with "# PBS -l walltime=48:00:00" option. Still it didn't resolved.

Please do correct me if this option wrong.

Re: Job submitted via qsub getting killed.

idata — Tue, 31 Jul 2018 10:26:40 GMT

Hi Joseph,

Thanks for reaching out to us.

You can increase the walltime by adding it in bash script before running in qsub mode as shown below:

# PBS -l walltime=24:00:00

Then run "qsub "

Please note that the max walltime is 24 hours.

Hope this helps.

Regards,

Deepthi

Re: Job submitted via qsub getting killed.

jKuma1 — Tue, 31 Jul 2018 14:23:15 GMT

Will this work even with a job triggered via Jupyter terminal and extend the walltime to 24hours?

Re: Job submitted via qsub getting killed.

idata — Wed, 01 Aug 2018 12:55:05 GMT

Hi Joseph,

For a Jupyter terminal, the maximum session time is 4 hrs.

We will run a qsub job that requires more time from the jupyter terminal and get back to you on the results.

Regards,

Deepthi

Re: Job submitted via qsub getting killed.

idata — Thu, 02 Aug 2018 05:11:54 GMT

Hi Joseph,

It is observed that post the maximum session time (4 hrs), the log files are not generated. So it is not ideal to submit a qsub job via Jupyter terminal which takes more than the session time.

Hope that helps.

Regards,

Deepthi

Re: Job submitted via qsub getting killed.

idata — Fri, 03 Aug 2018 04:54:37 GMT

Hi Joseph,

Could you please confirm if the queries got cleared.

Regards,

Deepthi Raj

Re: Job submitted via qsub getting killed.

idata — Mon, 06 Aug 2018 05:46:20 GMT

Hi Joseph,

Hope the queries are cleared. We are closing this thread. Please raise a new thread if you are facing any other issues.

Regards,

Deepthi Raj.