MKL ScaLAPACK pdgetrf_

phaser75 — Thu, 03 Dec 2009 05:34:19 GMT

Hello!

I'm trying to run my simple code for testing ScaLAPACK, but there is an error when running the code.
The code is simple. I use 4 processors and have the prescribed Block-Cyclic distributed matrix as 4 files.
When the program is started, each processor reads a matrix from the files and do pdgetrf_().

If I don't call pdgetrf_(), the program is finished correctly with sign "Calculation END". But if I insert pdgetrf_(),
There is an error as follows:
0: Fatal error in PMPI_Reduce: Invalid MPI_Op, error stack:
0: PMPI_Reduce(1198)...........: MPI_Reduce(sbuf=0x7fbfffe810, rbuf=0x7fbfffe800, count=1, dtype=0x4c001013, MPI_MAXLOC, root=0, comm=0xc4000008) failed
0: MPIR_MAXLOC_check_dtype(151): MPI_Op MPI_MAXLOC operation not defined for this datatype
2: Fatal error in PMPI_Reduce: Invalid MPI_Op, error stack:
2: PMPI_Reduce(1198)...........: MPI_Reduce(sbuf=0x7fbfffe810, rbuf=0x7fbfffe800, count=1, dtype=0x4c001013, MPI_MAXLOC, root=0, comm=0xc4000003) failed
2: MPIR_MAXLOC_check_dtype(151): MPI_Op MPI_MAXLOC operation not defined for this datatype
rank 2 in job 56 node11_55838 caused collective abort of all ranks
exit status of rank 2: return code 1
rank 0 in job 56 node11_55838 caused collective abort of all ranks
exit status of rank 0: return code 1

Could you give some advices about this error (Where do I have to look into?)

The code is like this:

#include
#include
#include
#include
#include
#include

#include
#include

#define NRANSI
#include "NRUTIL.h"

#define MXLLDA 1320
#define MXLLDB 1320
#define MXLOCR 1320
#define MXLOCC 1320
#define DLEN_ 9

void Cblacs_pinfo(int *, int * );
void Cblacs_exit(int);
void Cblacs_get(int, int, int * );
void Cblacs_gridinit(int *, char *, int, int );
void Cblacs_gridinfo(int, int *, int *, int *, int * );
void Cblacs_gridexit(int);
int numroc_(int *, int *, int *, int *, int *);
void descinit_(int *, int *, int *, int *, int *, int *, int *, int *, int *, int *);

FILE *inp;
int ictxt, locr, locc, *ipvt;
int izero=0, ione=1, np;
int ProcNo, ProcID, nprow, npcol, MB_, NB_, desca[DLEN_], descb[DLEN_];

int main(int argc, char* argv[])
{
int i, j, maxa, maxb, info;
int myrow, mycol, iam, nnodes;
double **Aij, tmp;
int *indx;
char fname[100];

/* setup MPI stuff */
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &ProcNo);
MPI_Comm_rank(MPI_COMM_WORLD, &ProcID);

// processor info
nprow = 2;
npcol = 2;
MB_ = 4;
NB_ = 4;

maxa = MXLLDA;
maxb = MXLLDB;

MPI_Comm my_row_comm, my_col_comm;
MPI_Status stts;

Cblacs_pinfo(&iam, &nnodes);

np = 10;

Cblacs_get(0, 0, &ictxt);
Cblacs_gridinit(&ictxt, "R", nprow, npcol);
Cblacs_gridinfo(ictxt, &nprow, &npcol, &myrow, &mycol);

locr = numroc_(&np, &MB_, &myrow, &izero, &nprow);
locc = numroc_(&np, &NB_, &mycol, &izero, &npcol);

descinit_(desca, &np, &np, &MB_, &NB_, &izero, &izero,
&ictxt, &maxa, &info);
descinit_(descb, &np, &ione, &NB_, &ione, &izero, &izero,
&ictxt, &maxb, &info);

Aij = dmatrix(0,MXLOCC-1,0,MXLLDA-1);
ipvt = ivector(0,MXLOCR-1);

sprintf(fname,"Aij%d.dat",ProcID);
inp = fopen(fname,"r");

for (i=0; i = 0;
for (i=0; i{
for (j=0; j{
fscanf(inp,"%lf",&tmp);
Aij = tmp;
}
}

pdgetrf_(&np, &np, &Aij[0][0], &ione, &ione, desca, &ipvt[0], &info);

free_dmatrix(Aij,0,MXLOCC-1,0,MXLLDA-1);
free_ivector(ipvt,0,MXLOCR-1);

if (ProcID == 0) printf(" calculation END\n");
MPI_Barrier(MPI_COMM_WORLD);

Cblacs_gridexit(ictxt);
Cblacs_exit(0);

return 0;
}

Thank U in advance.

Re: MKL ScaLAPACK pdgetrf_

TimP — Thu, 03 Dec 2009 13:56:53 GMT

Quoting - phaser75

0: MPIR_MAXLOC_check_dtype(151): MPI_Op MPI_MAXLOC operation not defined for this datatype

#include

This looks like headers for incompatible MPI implementations are involved in your build. For example, you built some .o files with one version of MPI, and some with another, or you linked a library (including MKL) which is intended for a different version of MPI than your . You must use consistent MPI throughout the build and at run time.
For example, OSU MVAPICH1 uses different encoding of MPI data types than Intel MPI.

Re: MKL ScaLAPACK pdgetrf_

phaser75 — Sat, 05 Dec 2009 01:12:24 GMT

Quoting - tim18

Thanks a lot.

I'll check it out.

topic Re: MKL ScaLAPACK pdgetrf_ in Intel® oneAPI Math Kernel Library

MKL ScaLAPACK pdgetrf_

Re: MKL ScaLAPACK pdgetrf_

Re: MKL ScaLAPACK pdgetrf_