Getting kicked from interactive nodes after approx 5-60 seconds

[DanDaMan]

I can no longer access interactive nodes for any reasonable amount of time. I get kicked after approximately 5-60 seconds.

Previously this was not an issue when I was last active about a month ago.

uXXXXXX@login-2:~$ time qsub -I -l nodes=ppn=2 -d .
qsub: waiting for job 1967836.v-qsvr-1.aidevcloud to start
qsub: job 1967836.v-qsvr-1.aidevcloud ready


########################################################################
#      Date:           Sun 07 Aug 2022 07:13:36 AM PDT
#    Job ID:           1967836.v-qsvr-1.aidevcloud
#      User:           uXXXXXX
# Resources:           neednodes=ppn=2,nodes=ppn=2,walltime=06:00:00
########################################################################

uXXXXXX@s001-n002:~$ tmux
[terminated]
uXXXXXX@s001-n002:~$
qsub: job 1967836.v-qsvr-1.aidevcloud completed

real    0m28.562s
user    0m0.065s
sys     0m0.386s
uXXXXXX@login-2:~$

[JaideepK_Intel]

Hi,

Thank you for posting in Intel communities.

We are working on this internally and we will get back to you with an update.

Thanks,

Jaideep

[DanDaMan]

Hi Jaideep,

Just following up to check if there is any update on this? I can confirm it is still an issue and haven't received any update in email beyond the initial request for my user id.

Thanks, Dan

[JaideepK_Intel]

Hi,

Sorry for the delay. We are working on your issue internally and will get back to you with an update.

Thanks and Regards,

Jaideep

[DanDaMan]

Just following up since it's been two weeks since I first posted this issue. I tried again and the issue remains. In this example, the session was terminated before it even reached the shell prompt.

$ time qsub -I -l nodes=ppn=2 -d .
qsub: waiting for job 1974798.v-qsvr-1.aidevcloud to start
qsub: job 1974798.v-qsvr-1.aidevcloud ready


########################################################################
#      Date:           Tue 23 Aug 2022 11:56:01 AM PDT
#    Job ID:           1974798.v-qsvr-1.aidevcloud
#      User:           uXXXXXX
# Resources:           neednodes=ppn=2,nodes=ppn=2,walltime=06:00:00
########################################################################


qsub: job 1974798.v-qsvr-1.aidevcloud completed

real    0m3.210s
user    0m0.066s
sys     0m0.408s

[JaideepK_Intel]

Hi,

Sorry for the delay, Can you follow the below steps?

• module load tmux

• tmux new -s test

After running the above commands, the shell looks like below. From here you can access interactive sessions.

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issue. Thank you!

Regards,

Jaideep

[DanDaMan]

The commands above didn't help. I don't think the issue is related to tmux, the initial example used tmux just to keep the output short.

I am kicked out regardless of command. I am kicked out even if I enter no command (example below - I did not type logout). As I said in my latest reply, sometimes I am even kicked out before I even I arrive a the shell prompt. 

$time qsub -I -l nodes=ppn=2 -d .
qsub: waiting for job 1975327.v-qsvr-1.aidevcloud to start
qsub: job 1975327.v-qsvr-1.aidevcloud ready


########################################################################
#      Date:           Thu 25 Aug 2022 06:33:54 AM PDT
#    Job ID:           1975327.v-qsvr-1.aidevcloud
#      User:           uXXXXXX
# Resources:           neednodes=ppn=2,nodes=ppn=2,walltime=06:00:00
########################################################################

uXXXXXX@s001-n005:~$ logout

qsub: job 1975327.v-qsvr-1.aidevcloud completed

real    0m8.446s
user    0m0.044s
sys     0m0.472s

[JaideepK_Intel]

Hi,

Sorry for the delay.

We are still diagnosing the issue. Could you please try the following command and let us know what results you get?

time qsub -l nodes=1:ppn=2

Thanks,

Jaideep

[DanDaMan]

It just hangs until I hit Ctrl-C. qstat (in another terminal) shows nothing...

uXXXXXX@login-2:~$ time qsub -l nodes=1:ppn=2
^C

real    4m3.319s
user    0m0.018s
sys     0m0.039s
uXXXXXX@login-2:~$

uXXXXXX@login-2:~$ qstat
uXXXXXXlogin-2:~$

I can verify interactive jobs are still getting kicked on this account in about 60 seconds or less:

uXXXXXX@s001-n012:~$ time qsub -I -l nodes=ppn=2 -d .
qsub: waiting for job 1983903.v-qsvr-1.aidevcloud to start
qsub: job 1983903.v-qsvr-1.aidevcloud ready


########################################################################
#      Date:           Mon 12 Sep 2022 07:11:29 AM PDT
#    Job ID:           1983903.v-qsvr-1.aidevcloud
#      User:           uXXXXXX
# Resources:           neednodes=ppn=2,nodes=ppn=2,walltime=06:00:00
########################################################################

uXXXXXX@s001-n012:~$ logout

qsub: job 1983903.v-qsvr-1.aidevcloud completed

real    0m34.682s
user    0m0.080s
sys     0m0.415s

[RonaldFry]

Whatever the order, I'm expelled. Even if I don't enter any commands, I get banned (example below - I did not type logout). As I mentioned in my most recent response, on occasion I get expelled even before I reach the shell prompt.

[JaideepK_Intel · Accepted solution]

Hi,

From the above reply, I can see you haven't typed the command properly.

time qsub -I -l nodes=1:ppn=2

i.e

time qsub –(capital i) –(lowercase L) nodes=1:ppn=2

Please share the output after trying the above command.

Thanks,

Jaideep

[DanDaMan]

"From the above reply, I can see you haven't typed the command properly."

Please note I copy and pasted directly from your reply, which did not include the "-I".

In any case, the problem does appear to be that I had somehow stopped including the "1:" in the "nodes=" argument. Most likely an unconcious over optimization on my part. 🤦

time qsub -I -l nodes=1:ppn=2

appears to work correctly (with and without the 'time' command).

Thanks for your help and patience, and sorry for submitting what turned out to be such a trivial issue!

[JaideepK_Intel]

Hi,

Glad to know that your issue is resolved. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel. Have a great day ahead.

Thanks,

Jaideep