Use of Scalapack PDGESV with C++ with non square matrices

[mfactor]

Hello all,

I'm trying to use the PDGESV routine from Scalapack. I successfully ran the examples, and now I'm trying to implement my own matrix distribution. The problem lies when the blocks are not squared and in the fact that to call Scalapack routines from C++ it is needed to transpose the matrix.

If the block is a multiple of the number of processes and the size of the matrix, the distribution works, and the result agree with the lapack one. But if in the combination of block size, number of processes and the size of the matrix are not multiple, the transpose of the resulting distributed matrix is not correct. For example, take the matrix

Code: Select all
0 1 2 3 4 5
6 7 8 9 10 11
12 13 14 15 16 17
18 19 20 21 22 23
24 25 26 27 28 29
30 31 32 33 34 35



If I transpose the matrix after being distributed (transposing only the local matrices, not the total one), assuming nprow = 3, npcol = 2, block = 2,

nprocs: 6 nprow 3 npcol 2

Local Matrix myrow 0 mycol 0 locR 2 locC 4 rank 0
0 6
1 7
4 10
5 11
++++++++
Local Matrix myrow 0 mycol 1 locR 2 locC 2 rank 1
2 8
3 9
++++++++

Local Matrix myrow 1 mycol 0 locR 2 locC 4 rank 2
12 18
13 19
16 22
17 23
++++++++
Local Matrix myrow 1 mycol 1 locR 2 locC 2 rank 3
14 20
15 21
++++++++
Local Matrix myrow 2 mycol 0 locR 2 locC 4 rank 4
24 30
25 31
28 34
29 35
++++++++
Local Matrix myrow 2 mycol 1 locR 2 locC 2 rank 5
26 32
27 33
++++++++

That is different that if I transpose it before the block distribution:
nprocs: 6 nprow 3 npcol 2
dgesv 1
dgesv 0
dgesv 0
dgesv 0
dgesv 0
dgesv 0
il 0 myrow 0 mycol 0 B 1
il 1 myrow 0 mycol 0 B 0
Local Matrix myrow 0 mycol 0 locR 2 locC 4 rank 0
0 6 24 30
1 7 25 31
++++++++
Local Matrix myrow 0 mycol 1 locR 2 locC 2 rank 1
12 18
13 19
++++++++
il 0 myrow 1 mycol 0 B 0
il 1 myrow 1 mycol 0 B 0
Local Matrix myrow 1 mycol 0 locR 2 locC 4 rank 2
2 8 26 32
3 9 27 33
++++++++
Local Matrix myrow 1 mycol 1 locR 2 locC 2 rank 3
14 20
15 21
++++++++
il 0 myrow 2 mycol 0 B 0
il 1 myrow 2 mycol 0 B 0
Local Matrix myrow 2 mycol 0 locR 2 locC 4 rank 4
4 10 28 34
5 11 29 35
++++++++
Local Matrix myrow 2 mycol 1 locR 2 locC 2 rank 5
16 22
17 23
++++++++

What is the right form of matrix distribution? What is the correct order of matrix distribution when PDGESV is called from c++?

Thank you!