Re: Cannot access "Stratix 10 PAC Compilation and Programming - RTL AFU, OpenCL"

neolide · ‎10-15-2022

Hi,
I was trying to login into one of the Stratix 10 PAC nodes but was unable to enter. Here is the output of devcloud_login

$ devcloud_login

You are selecting an interactive compute server sesssion. Please consider using batch mode submission using
devcloud_login -b to not tie up compute servers with idle sessions.
See the help menu using devcloud_login -h for more details.

What are you trying to use the Devcloud for?

1) Arria 10 PAC Compilation and Programming - RTL AFU, OpenCL
2) Arria 10 - OneAPI, OpenVINO
3) Stratix 10 PAC Compilation and Programming - RTL AFU, OpenCL
4) Stratix 10 - OneAPI, OpenVINO
5) Compilation (Command Line) Only
6) Enter Specific Node Number

Number: 3

--------------------------------------------------------------------------------------
For X2GO tunneling access. If connected to intel firewall, copy and paste the following text in a new mobaxterm terminal:

ssh -L 4002:s005-n008:22 colfax-intel

For X2GO tunneling access. For users NOT connected to intel firewall, copy and paste the following text in a new mobaxterm terminal:

ssh -L 4002:s005-n008:22 devcloud
--------------------------------------------------------------------------------------

running: qsub -q batch@v-qsvr-fpga -I -l nodes=s005-n008:ppn=2
qsub: waiting for job 37178.v-qsvr-fpga.aidevcloud to start
qsub: job 37178.v-qsvr-fpga.aidevcloud apparently deleted

As you can see, The qsub command shows that the job was deleted apparently.

And I can access all nodes (including "Stratix 10 - OneAPI, OpenVINO") except "Stratix 10 - PAC".

BoonBengT_Intel · ‎10-20-2022

Hi @neolide,

Thank you for posting in Intel community forum and hope all is well.

My guess there might be some intermediate issues on the platform.

As I tried on my end I was able to get to the mention node s005-n008.

Hence would recommend to try end enter the option 6 to select the desire node and connect via the method.

Best Wishes

BB

neolide · ‎10-20-2022

Thanks for your advice. I tried but still got this error.

$ devcloud_login

You are selecting an interactive compute server sesssion. Please consider using batch mode submission using
devcloud_login -b to not tie up compute servers with idle sessions.
See the help menu using devcloud_login -h for more details.

What are you trying to use the Devcloud for?

1) Arria 10 PAC Compilation and Programming - RTL AFU, OpenCL
2) Arria 10 - OneAPI, OpenVINO
3) Stratix 10 PAC Compilation and Programming - RTL AFU, OpenCL
4) Stratix 10 - OneAPI, OpenVINO
5) Compilation (Command Line) Only
6) Enter Specific Node Number

Number: 6
Showing available nodes below: (34 available/63 total)
--------------------------------------------------------------------------------------
Nodes with no attached hardware: (14 available/35 total)
s001-n059 s001-n064 s001-n065 s001-n066 s001-n067 s001-n068 s001-n069 s001-n070 s001-n072 s001-n075 s001-n076 s001-n077 s001-n078 s001-n079
--------------------------------------------------------------------------------------
Nodes with Arria 10: (7 available/8 total)
Release 1.2:

Release 1.2.1:
s001-n137 s001-n138 s001-n139 s005-n001 s005-n002 s005-n003 s005-n004
--------------------------------------------------------------------------------------
Nodes with Arria 10 OneAPI: (7 available/12 total)
s001-n081 s001-n082 s001-n083 s001-n084 s001-n085 s001-n087 s001-n088
--------------------------------------------------------------------------------------
Nodes with Stratix 10: (3 available/5 total)
s005-n006 s005-n008 s005-n009
--------------------------------------------------------------------------------------
Nodes with Stratix 10 OneAPI: (2 available/3 total)
s001-n143 s001-n144
--------------------------------------------------------------------------------------

What node would you like to use?

Node: s005-n008

--------------------------------------------------------------------------------------
For X2GO tunneling access. If connected to intel firewall, copy and paste the following text in a new mobaxterm terminal:

ssh -L 4002:s005-n008:22 colfax-intel

For X2GO tunneling access. For users NOT connected to intel firewall, copy and paste the following text in a new mobaxterm terminal:

ssh -L 4002:s005-n008:22 devcloud 
--------------------------------------------------------------------------------------

running: qsub -q batch@v-qsvr-fpga -I -l nodes=s005-n008:ppn=2
qsub: waiting for job 37322.v-qsvr-fpga.aidevcloud to start
qsub: job 37322.v-qsvr-fpga.aidevcloud apparently deleted

BoonBengT_Intel · ‎10-20-2022

Hi @neolide,

That's weird, I also managed to check on the nodes state/power they seems to be running.

It is happening the same for both 2 other nodes (s005-n006 & s005-n009) when you tried to access them by specifying the node?

I did managed to find another node with the s005-n010 which is having the similar spec.

Can you try with the below qsub command to see if you are able to connect to the nodes?

- qsub -q batch@v-qsvr-fpga -I -l nodes=s005-n010:ppn=2

Best Wishes

BB

neolide · ‎10-20-2022

I tried and again I got the same error .

$ qsub -q batch@v-qsvr-fpga -I -l nodes=s005-n010:ppn=2
qsub: waiting for job 37342.v-qsvr-fpga.aidevcloud to start
qsub: job 37342.v-qsvr-fpga.aidevcloud apparently deleted

$ qsub -q batch@v-qsvr-fpga -I -l nodes=s005-n008:ppn=2
qsub: waiting for job 37343.v-qsvr-fpga.aidevcloud to start
qsub: job 37343.v-qsvr-fpga.aidevcloud apparently deleted

$ qsub -q batch@v-qsvr-fpga -I -l nodes=s005-n009:ppn=2
qsub: waiting for job 37344.v-qsvr-fpga.aidevcloud to start
qsub: job 37344.v-qsvr-fpga.aidevcloud apparently deleted

$ qsub -q batch@v-qsvr-fpga -I -l nodes=s005-n006:ppn=2
qsub: waiting for job 37345.v-qsvr-fpga.aidevcloud to start
qsub: job 37345.v-qsvr-fpga.aidevcloud apparently deleted

BoonBengT_Intel · ‎10-20-2022

Hi @neolide,

Suspecting that other jobs might be interfering with this workload.

Can you please run a qstat to check on the jobs details, and followed by 'qdel <job-id>' or 'qdel all' if all jobs are not relevant, and try again for the node connection.

While at the same time I would checking this internally with the devcloud platform team to get a better view on the nodes.

Thank you for your patients.

Best Wishes

BB

neolide · ‎10-20-2022

Here is the output of qstat (I am not running any tasks so there is no output)

$ qstat
$ qstat -s
$ qstat -s batch@v-qsvr-fpga

And I tried qdel all but got this error

$ qdel all
qdel: Unauthorized Request MSG=operation not permitted all

BoonBengT_Intel · ‎10-20-2022

Hi @neolide,

Thanks for the quick actions, managed to get in touch with the relevant platform team with the right access to assist you further on this.

Apologies for the inconvenient cause, they should be contacting you soon.

Best Wishes

BB