CPU usage 100% during MPI blocking communication waiting

[yinyin]

In my program, according to the user input, the master process will broadcast data to other worker processes to do computation, after one time computation, the worker processes will wait for new data once again and do the corresponding computation.

Worker processes  use a do-while loop to keep receiving data from master like this,

int master_rank =0
int mycmd;
if(my_rank==master_rank )
{
  //wait for user input
  //accept input command
  MPI_Bcast(&mycmd, 1, MPI_INT, master_rank, MPI_COMM_WORLD);  ​
  //do some work
}
else
{
  MPI_Bcast(&mycmd, 1, MPI_INT, master_rank, MPI_COMM_WORLD);
  switch (mycmd)
  {
    case  1:
       // do some work
       break;
    case 2:
       // do some work
       break;
    default:
       std::cout << "myrank:" << my_rank << "Unknown command:" << mycmd << std::endl;
       break;

  }
} 

However, even during waiting for user input, the worker processes will keep 100% cpu running, which make  PC too hot.

After research in web, I found the following parameters can decrease cpu usage ratio,

I_MPI_WAIT_MODE=0
I_MPI_THREAD_YIELD=3
I_MPI_THREAD_SLEEP=10

But this also make computation itself become low performance, that is absolutely not what I want .

Does intel MPI library provide some equivalent function to control the MPI behavior during running? just like openmp's   omp_set_xxx function.

Thank you for your help. 

[SantoshY_Intel]

Hi,

Thanks for reaching out to us.

We are working on your issue and we will get back to you soon.

Thanks & regards,

Santosh

[SantoshY_Intel]

Hi,

Could you please provide us the sample reproducer code along with the commands you used to compile and run?

>>"I_MPI_THREAD_SLEEP=10"

As the sleep time increases, there will be a drop in performance. So try to reduce sleep time in order to enhance the performance.

>>"Does intel MPI library provide some equivalent function to control the MPI behavior during running? just like openmp's  omp_set_xxx function."

Could you please elaborate on what kind of MPI behavior to control?

Thanks,

Santosh

[yinyin]

Dear Santosh

 Maybe my description was not clear.

  I want to make an emphasize that

 - I DO want to drop/decrease response performance during waiting procedure  by setting I_MPI_THREAD_SLEEP a big value. 

 - But once the worker processes receive the command(which is broadcast from master process), these worker processes should run  the computing procedure at their full power, where a zero   I_MPI_THREAD_SLEEP is wanted.

I wonder if I can adjust  I_MPI_THREAD_SLEEP during program running.

Expecting  your reply.

[SantoshY_Intel]

Hi,

>>"I wonder if I can adjust  I_MPI_THREAD_SLEEP during program running"
We can not adjust I_MPI_THREAD_SLEEP during program running. Please find more information regarding usage of I_MPI_THREAD_SLEEP from the below link:

https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-reference-linux/top/environment-variable-reference/other-environment-variables.html#other-environment-variables_GUID-A0217433-9A3A-42BC-A27E-C5F555026F23

We need more information to investigate your issue.

So, Could you please provide us the sample reproducer code along with the commands you used to compile and run?"

>>"Does intel MPI library provide some equivalent function to control the MPI behavior during running? just like openmp's  omp_set_xxx function."

Could you please elaborate on what kind of MPI behavior to control?

Thanks & Regards,

Santosh

[SantoshY_Intel]

Hi,

We have not heard back from you. Could you please provide the details that have been asked in my previous post?

Thanks,

Santosh

[yinyin]

Dear Santosh

 I haven't solved this. Sorry for my late response.

I will give a minimal program to show this issue in several days.

Please keep tracking my issue.

B.R.

YIN

[SantoshY_Intel]

Hi,

We have not heard back from you. Could you please provide us the sample reproducer code along with the commands you used to compile and run?

Thanks,

Santosh

[SantoshY_Intel]

Hi,

>>"I will give a minimal program to show this issue in several days.Please keep tracking my issue."

We have been waiting for your response. But we haven't heard back from you. Can you provide the reproducer code along with the commands you used to encounter the issue.

Thanks & Regards,

Santosh

[SantoshY_Intel]

Hi,

As we haven't heard back from you, we are closing this thread for now. However, feel free to raise a new query whenever you can provide us a sample reproducer as this thread will no longer be monitored. 

Have a Good day!

Thanks,

Santosh