Running more than 100 jobs in parallel

YOren · ‎07-05-2018

Hey,

I submitted more than 120 jobs on Intel AI DevCloud at once. I saw that only 6 were running in parallel at each time point.

Moreover, after 42 jobs were complicated all other jobs were in Q mode and none transition to R mode.

My questions:

1. Why am I limited to running at most 6 jobs in parallel?

2. Why are the rest of the jobs waiting and not running if I have no job running?

Thanks,

Yaron

idata · ‎07-05-2018

Hi Yaron,

Please find the answers below:

1. DevCloud is used by students from around the globe. A limit has been imposed on the number of jobs that could run in parallel per user. This is to ensure a fair utilization of cluster resources by all the users of DevCloud.

2. The jobs submitted, wait in a queue, till it is picked up by job scheduler for running. If the cluster resources are already completely occupied, with the jobs running from other DevCloud users, the new jobs will have to wait in the queue, till the running jobs are completed. Your jobs will eventually be picked up.

Regards,

Anju

idata · ‎07-09-2018

Hi,

Hope the solution provided clarifies your query. Please confirm.

Regards,

Dona

YOren · ‎07-10-2018

Yes, thanks!

idata · ‎07-10-2018

Hi,

Thank you for the confirmation. We are closing the thread. Feel free to open new thread if you face any issues.

Regards,

Dona