I submitted more than 120 jobs on Intel AI DevCloud at once. I saw that only 6 were running in parallel at each time point.
Moreover, after 42 jobs were complicated all other jobs were in Q mode and none transition to R mode.
1. Why am I limited to running at most 6 jobs in parallel?
2. Why are the rest of the jobs waiting and not running if I have no job running?
Please find the answers below:
1. DevCloud is used by students from around the globe. A limit has been imposed on the number of jobs that could run in parallel per user. This is to ensure a fair utilization of cluster resources by all the users of DevCloud.
2. The jobs submitted, wait in a queue, till it is picked up by job scheduler for running. If the cluster resources are already completely occupied, with the jobs running from other DevCloud users, the new jobs will have to wait in the queue, till the running jobs are completed. Your jobs will eventually be picked up.