performance issue about pdgesv function in scalapack

[Hongda_H_]

     I am suffering from a problem about pdgesv function in scalapack(C++).I have two computing nodes,each with 16 cores.The time spent solveing the linear Equation with pdgesv in two noodes is almost fifth times larger than running in single node using the same number of processes.

    For example,A matrix is 1165*1165,and b matrix is 1165*30000.The job on single node with two processes takes almost 16 seconds while the job on two nodes each with one process takes more than 70 seconds.

    I know the reason should be the communication between these two nodes,but why the gap is so big and how to narrow the gap?Could someone give me some suggestions?

[Gennady_F_Intel · Accepted solution]

This probably the expected gap in the case if 100Mbit Ethernet has been used. Can you give us the test and some additional configurations for checking how it will work on our side?

[Hongda_H_]

Gennady Fedorov (Intel) wrote:

This probably the expected gap in the case if 100Mbit Ethernet has been used. Can you give us the test and some additional configurations for checking how it will work on our side?

Thank you for your reply!! I think I have solved this problem.I just used intelmpi instead of mpich3,then this big gap disappeard.This may be the reason that intel mkl is more suitable to intelmpi.