Job exceeds queue resource limits error on job submission, devcloud_login says the FPGA is available

[MajdiAS]

Hello,

I am using Devcloud for FPGA, and when I check devcloud_login SNN, it says the Stratix 10 (not OneAPI) nodes are available, but when I submit the job, it cancels after a few seconds. I used the qsub command and it gives me this error: 

qsub: submit error (Job exceeds queue resource limits MSG=cannot locate feasible nodes (nodes file is empty, all systems are busy, or no nodes have the requested feature))

I have not changed my job submission resource requirements so I do not expect resources to be an issue.

I checked the pbsnodes command and only the s001 and s002 nodes seem to show up. My guess is that the devices are available but the host CPUs for those devices are busy.

[AthiraM_Intel]

Hi,

Thank you for posting in Intel Community.

We are able to reproduce your issue. We are checking on this internally, will let you know the updates soon.

Thanks

[AthiraM_Intel]

Hi,

We are sorry for the delay.

qsub command needs to include the "-q batch@v-qsvr-fpga" in order to access the specific node requested. Please see an example below:

qsub -q batch@v-qsvr-fpga -I -l nodes=s005-n006:ppn=2

Please let us know if you face any further issue.

Thanks

[AthiraM_Intel]

Hi,

We have not heard back from you. Could you please give us an update?

Thanks

[MajdiAS]

Hello,

I just tried the command you gave me, it becomes stuck with the following message:

qsub: waiting for job xxxxx.v-qsvr-fpga.aidevcloud to start 

The node I selected was said to be available when checking 

pbsnodes -l free -s v-qsvr-fpga

[AthiraM_Intel]

Hi,

Could you please try to access the node again. We are able to access node by using the below command:

qsub -q batch@v-qsvr-fpga -I -l nodes=<node number>:ppn=2

Please find the below screenshot:

If you face the same issue again, please let us know.

Thanks

[MajdiAS]

Hello,

I am able to access the Arria 10 boards, including the example s005-n007 instance. I do not think I ever had a problem with Arria 10 batch submissions. I had to port my design over to be compatible with Arria 10, which eventually worked.

However, accessing the Stratix 10 instances still seems to be an issue. I tried logging in to s001-n142, a Stratix 10 OneAPI instance, as the non-OneAPI instances were not available. I was receiving the same message as my original post, despite being that instance being listed as available according to devcloud_login. I will check when the regular Stratix 10 instances are available.

Let me know if you are able to reproduce the issue with Stratix 10 (non and OneAPI variants).

Best,

Majdi

[AthiraM_Intel]

Hi,

Could you please try to access Stratix 10 nodes now. We are able to access without any issue.

Please find the below screenshot:

If you are still facing the issue, please let us know.

Thanks

[AthiraM_Intel]

Hi,

We have not heard back from you. Could you please give us an update?

Thanks

[MajdiAS]

Hello,

I can confirm that I can log in to Stratix 10 OneAPI now, but I could not verify Stratix 10 PAC as they have not been available.

Best,

Majdi

[AthiraM_Intel · Accepted solution]

Hi,

Thanks for the confirmation. Since there is very limited number of Stratix 10 PAC nodes, it is difficult to get free nodes.

We are able to use the Stratix 10 PAC nodes. Please find the below screenshots:

If there is no free nodes available, please try to access the nodes after some time.

Since the issue is resolved, we are closing this case. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Thanks