Intel® DevCloud
Help for those needing help starting or connecting to the Intel® DevCloud
1660 Discussions

Getting kicked from interactive nodes after approx 5-60 seconds

DanDaMan
New Contributor I
2,142 Views

I can no longer access interactive nodes for any reasonable amount of time. I get kicked after approximately 5-60 seconds.

Previously this was not an issue when I was last active about a month ago.

 

 

 

 

 

uXXXXXX@login-2:~$ time qsub -I -l nodes=ppn=2 -d .
qsub: waiting for job 1967836.v-qsvr-1.aidevcloud to start
qsub: job 1967836.v-qsvr-1.aidevcloud ready


########################################################################
#      Date:           Sun 07 Aug 2022 07:13:36 AM PDT
#    Job ID:           1967836.v-qsvr-1.aidevcloud
#      User:           uXXXXXX
# Resources:           neednodes=ppn=2,nodes=ppn=2,walltime=06:00:00
########################################################################

uXXXXXX@s001-n002:~$ tmux
[terminated]
uXXXXXX@s001-n002:~$
qsub: job 1967836.v-qsvr-1.aidevcloud completed

real    0m28.562s
user    0m0.065s
sys     0m0.386s
uXXXXXX@login-2:~$

 

 

 

 

 

 

0 Kudos
1 Solution
JaideepK_Intel
Moderator
1,845 Views

Hi,

 

From the above reply, I can see you haven't typed the command properly.

time qsub -I -l nodes=1:ppn=2

i.e

time qsub –(capital i) –(lowercase L) nodes=1:ppn=2

Please share the output after trying the above command.

 

Thanks,

Jaideep

 

View solution in original post

0 Kudos
12 Replies
JaideepK_Intel
Moderator
2,110 Views

Hi,


Thank you for posting in Intel communities.


We are working on this internally and we will get back to you with an update.


Thanks,

Jaideep



0 Kudos
DanDaMan
New Contributor I
2,074 Views

Hi Jaideep,

Just following up to check if there is any update on this? I can confirm it is still an issue and haven't received any update in email beyond the initial request for my user id.

Thanks, Dan

0 Kudos
JaideepK_Intel
Moderator
2,057 Views

Hi,


Sorry for the delay. We are working on your issue internally and will get back to you with an update.


Thanks and Regards,

Jaideep


0 Kudos
DanDaMan
New Contributor I
2,034 Views

Just following up since it's been two weeks since I first posted this issue. I tried again and the issue remains. In this example, the session was terminated before it even reached the shell prompt.

$ time qsub -I -l nodes=ppn=2 -d .
qsub: waiting for job 1974798.v-qsvr-1.aidevcloud to start
qsub: job 1974798.v-qsvr-1.aidevcloud ready


########################################################################
#      Date:           Tue 23 Aug 2022 11:56:01 AM PDT
#    Job ID:           1974798.v-qsvr-1.aidevcloud
#      User:           uXXXXXX
# Resources:           neednodes=ppn=2,nodes=ppn=2,walltime=06:00:00
########################################################################


qsub: job 1974798.v-qsvr-1.aidevcloud completed

real    0m3.210s
user    0m0.066s
sys     0m0.408s
0 Kudos
JaideepK_Intel
Moderator
2,024 Views

Hi,

 

Sorry for the delay, Can you follow the below steps?

  • module load tmux
  • tmux new -s test

After running the above commands, the shell looks like below. From here you can access interactive sessions.

JaideepK_Intel_0-1661351442162.png

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issue. Thank you!

 

Regards,

Jaideep

 

0 Kudos
DanDaMan
New Contributor I
1,995 Views

The commands above didn't help. I don't think the issue is related to tmux, the initial example used tmux just to keep the output short.

I am kicked out regardless of command. I am kicked out even if I enter no command (example below - I did not type logout). As I said in my latest reply, sometimes I am even kicked out before I even I arrive a the shell prompt. 

$time qsub -I -l nodes=ppn=2 -d .
qsub: waiting for job 1975327.v-qsvr-1.aidevcloud to start
qsub: job 1975327.v-qsvr-1.aidevcloud ready


########################################################################
#      Date:           Thu 25 Aug 2022 06:33:54 AM PDT
#    Job ID:           1975327.v-qsvr-1.aidevcloud
#      User:           uXXXXXX
# Resources:           neednodes=ppn=2,nodes=ppn=2,walltime=06:00:00
########################################################################

uXXXXXX@s001-n005:~$ logout

qsub: job 1975327.v-qsvr-1.aidevcloud completed

real    0m8.446s
user    0m0.044s
sys     0m0.472s



0 Kudos
JaideepK_Intel
Moderator
1,885 Views

Hi,

 

Sorry for the delay.

We are still diagnosing the issue. Could you please try the following command and let us know what results you get?

time qsub -l nodes=1:ppn=2

 

Thanks,

Jaideep

 

0 Kudos
DanDaMan
New Contributor I
1,878 Views

It just hangs until I hit Ctrl-C. qstat (in another terminal) shows nothing...

uXXXXXX@login-2:~$ time qsub -l nodes=1:ppn=2
^C

real    4m3.319s
user    0m0.018s
sys     0m0.039s
uXXXXXX@login-2:~$
uXXXXXX@login-2:~$ qstat
uXXXXXXlogin-2:~$

I can verify interactive jobs are still getting kicked on this account in about 60 seconds or less:

uXXXXXX@s001-n012:~$ time qsub -I -l nodes=ppn=2 -d .
qsub: waiting for job 1983903.v-qsvr-1.aidevcloud to start
qsub: job 1983903.v-qsvr-1.aidevcloud ready


########################################################################
#      Date:           Mon 12 Sep 2022 07:11:29 AM PDT
#    Job ID:           1983903.v-qsvr-1.aidevcloud
#      User:           uXXXXXX
# Resources:           neednodes=ppn=2,nodes=ppn=2,walltime=06:00:00
########################################################################

uXXXXXX@s001-n012:~$ logout

qsub: job 1983903.v-qsvr-1.aidevcloud completed

real    0m34.682s
user    0m0.080s
sys     0m0.415s

 

0 Kudos
RonaldFry
Beginner
1,873 Views

Whatever the order, I'm expelled. Even if I don't enter any commands, I get banned (example below - I did not type logout). As I mentioned in my most recent response, on occasion I get expelled even before I reach the shell prompt.

0 Kudos
JaideepK_Intel
Moderator
1,846 Views

Hi,

 

From the above reply, I can see you haven't typed the command properly.

time qsub -I -l nodes=1:ppn=2

i.e

time qsub –(capital i) –(lowercase L) nodes=1:ppn=2

Please share the output after trying the above command.

 

Thanks,

Jaideep

 

0 Kudos
DanDaMan
New Contributor I
1,818 Views

"From the above reply, I can see you haven't typed the command properly."

Please note I copy and pasted directly from your reply, which did not include the "-I".

In any case, the problem does appear to be that I had somehow stopped including the "1:" in the "nodes=" argument. Most likely an unconcious over optimization on my part. 🤦

time qsub -I -l nodes=1:ppn=2


appears to work correctly (with and without the 'time' command).

Thanks for your help and patience, and sorry for submitting what turned out to be such a trivial issue!

0 Kudos
JaideepK_Intel
Moderator
1,806 Views

Hi,


Glad to know that your issue is resolved. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel. Have a great day ahead.


Thanks,

Jaideep


0 Kudos
Reply