topic CPU usage 100% during MPI blocking communication waiting in Intel® MPI Library

CPU usage 100% during MPI blocking communication waiting

yinyin — Fri, 11 Jun 2021 13:53:47 GMT

In my program, according to the user input, the master process will broadcast data to other worker processes to do computation, after one time computation, the worker processes will wait for new data once again and do the corresponding computation.

Worker processes use a do-while loop to keep receiving data from master like this,

int master_rank =0
int mycmd;
if(my_rank==master_rank )
{
  //wait for user input
  //accept input command
  MPI_Bcast(&mycmd, 1, MPI_INT, master_rank, MPI_COMM_WORLD);  
  //do some work
}
else
{
  MPI_Bcast(&mycmd, 1, MPI_INT, master_rank, MPI_COMM_WORLD);
  switch (mycmd)
  {
    case  1:
       // do some work
       break;
    case 2:
       // do some work
       break;
    default:
       std::cout << "myrank:" << my_rank << "Unknown command:" << mycmd << std::endl;
       break;

  }
}

However, even during waiting for user input, the worker processes will keep 100% cpu running, which make PC too hot.

After research in web, I found the following parameters can decrease cpu usage ratio,

I_MPI_WAIT_MODE=0
I_MPI_THREAD_YIELD=3
I_MPI_THREAD_SLEEP=10

But this also make computation itself become low performance, that is absolutely not what I want .

Does intel MPI library provide some equivalent function to control the MPI behavior during running? just like openmp's omp_set_xxx function.

Thank you for your help.

Re:CPU usage 100% during MPI blocking communication waiting

SantoshY_Intel — Mon, 14 Jun 2021 11:53:22 GMT

Hi,

Thanks for reaching out to us.

We are working on your issue and we will get back to you soon.

Thanks & regards,

Santosh

Re:CPU usage 100% during MPI blocking communication waiting

SantoshY_Intel — Wed, 16 Jun 2021 13:23:02 GMT

Hi,

Could you please provide us the sample reproducer code along with the commands you used to compile and run?

>>"I_MPI_THREAD_SLEEP=10"

As the sleep time increases, there will be a drop in performance. So try to reduce sleep time in order to enhance the performance.

>>"Does intel MPI library provide some equivalent function to control the MPI behavior during running? just like openmp's omp_set_xxx function."

Could you please elaborate on what kind of MPI behavior to control?

Thanks,

Santosh

Re: Re:CPU usage 100% during MPI blocking communication waiting

yinyin — Wed, 16 Jun 2021 18:19:19 GMT

Dear Santosh

Maybe my description was not clear.

I want to make an emphasize that

- I DO want to drop/decrease response performance during waiting procedure by setting I_MPI_THREAD_SLEEP a big value.

- But once the worker processes receive the command(which is broadcast from master process), these worker processes should run the computing procedure at their full power, where a zero I_MPI_THREAD_SLEEP is wanted.

I wonder if I can adjust I_MPI_THREAD_SLEEP during program running.

Expecting your reply.

Re: CPU usage 100% during MPI blocking communication waiting

SantoshY_Intel — Thu, 09 Sep 2021 06:49:21 GMT

Hi,

>>"I wonder if I can adjust I_MPI_THREAD_SLEEP during program running"
We can not adjust I_MPI_THREAD_SLEEP during program running. Please find more information regarding usage of I_MPI_THREAD_SLEEP from the below link:

https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-reference-linux/top/environment-variable-reference/other-environment-variables.html#other-environment-variables_GUID-A0217433-9A3A-42BC-A27E-C5F555026F23

We need more information to investigate your issue.

So, Could you please provide us the sample reproducer code along with the commands you used to compile and run?"

>>"Does intel MPI library provide some equivalent function to control the MPI behavior during running? just like openmp's omp_set_xxx function."

Could you please elaborate on what kind of MPI behavior to control?

Thanks & Regards,

Santosh

Re: CPU usage 100% during MPI blocking communication waiting

SantoshY_Intel — Thu, 09 Sep 2021 05:51:23 GMT

Hi,

We have not heard back from you. Could you please provide the details that have been asked in my previous post?

Thanks,

Santosh

Re: CPU usage 100% during MPI blocking communication waiting

yinyin — Fri, 25 Jun 2021 21:04:08 GMT

Dear Santosh

I haven't solved this. Sorry for my late response.

I will give a minimal program to show this issue in several days.

Please keep tracking my issue.

B.R.

YIN

Re: CPU usage 100% during MPI blocking communication waiting

SantoshY_Intel — Thu, 09 Sep 2021 05:53:31 GMT

Hi,

We have not heard back from you. Could you please provide us the sample reproducer code along with the commands you used to compile and run?

Thanks,

Santosh

Re:CPU usage 100% during MPI blocking communication waiting

SantoshY_Intel — Fri, 09 Jul 2021 12:09:24 GMT

Hi,

>>"I will give a minimal program to show this issue in several days.Please keep tracking my issue."

We have been waiting for your response. But we haven't heard back from you. Can you provide the reproducer code along with the commands you used to encounter the issue.

Thanks & Regards,

Santosh

Re: CPU usage 100% during MPI blocking communication waiting

SantoshY_Intel — Thu, 29 Jul 2021 11:35:31 GMT

Hi,

As we haven't heard back from you, we are closing this thread for now. However, feel free to raise a new query whenever you can provide us a sample reproducer as this thread will no longer be monitored.

Have a Good day!

Thanks,

Santosh