What could be a cause of the MPI program to run slower on multinode cluster?

[ArthurRatz]

Hello everyone,

I've got a question about MPI program performance: I've developed an MPI program that processes large amounts of data (about 10^9) elements, and running this program I've noticed that as many processes I create using mpiexec utility as longer the duration of the program execution. What could be a cause of the following issue ?? When I run this program in a single computational node, it works faster rather running that using two computational nodes. Please, help.

Regards, Arthur.

[ArthurRatz]

Normally I use the following computational platform: 2 x Intel Core i7 - 4970 4.00 GHZ, 32GB RAM, Network: 1 Gbps.

[ArthurRatz]

My MPI program actually sorts a huge array containing 10^9 elements by splitting the entire array into chunks sorted by each process created by mpiexec utility. The actual sorting is performed using tbb::parallel_sort routine which is a part of Threading Building Blocks (TBB).

[ArthurRatz]

If anyone who is going to answer my question needs an executable to test it on his side, I'm ready provide one.

[Gregg_S_Intel]

Have a look at this MPI tool.   https://software.intel.com/en-us/intel-trace-analyzer 

[ArthurRatz]

Thanks for your reply. I'll take a look at.