MKL Invalid Pointer in dsyevr (novice question)

[mspecian]

I am a young researcher without much CS experience trying to use the MKL routine "dsyevr" to diagonalize a real symmetric matrix. The matrix has approximately 6e9 8-byte elements (about 32.5GB). I have successfully executed this routine before for smaller matrices, but since moving to a new machine have experienced problems. If someone could point me in the right direction, I would be most appreciative.

Tech specs: I'm running MKL version 9 (I think) on a machine with16 64-bitXeon cores using em64t. I have 128GB of memory at my disposal. I call dsyevr from FORTRAN, but access the FORTRAN subroutine from C. Here's the C code:

////////////////////////////////////////////////////////////////////////////////////////////

// This routine calculates the eigenvalues and eigenvectors of the symmetric matrix

// of doubles fwritten to the file "fnameANM" using the dsyevr LAPACK procedure. Since

// memory constraints restrict the creation of a huge, complete eigenvector file, only

// the il-th through iu-th eigenvalues and eigenvectors are returned where 1<=il<=

// iu<=nrows. The eigenvalues are stored from most positive to most negative. This

// routine is somewhat faster than the packed version.

//

//Text files are written in matrix format with the eigenvectors in the columns.

// The "il"th eigenvector is in the first column while the "iu"th is in the last.

// Fwritten files store the eigenvectors sequentially. The eigenvector corresponding to

// the largest returned eigenvalue (i.e. the "il"th one takes up elements [0 through

// (nelements-1)]. The next largest eigenvector is stored to elements [nelements through

// (2*nelements-1)] and so forth.

////////////////////////////////////////////////////////////////////////////////////////////

void GetEigenvectors(char *fnameANM, int nrows, int il, int iu) {

double *w, *z, *work, rnrows, working, *EArray;

size_t size, ee, rr;

int ii, info, lz, lzp, lisuppz, lwork, m, *iwork, liwork, cc, iil, iiu;

info = NULL;

printf("\\nThis routine returns eigenmode #%d (larger eigenvalue) through %d (smaller eigenvalue).\\n\\n",il,iu);

//dsyevr puts the largest Evl's Evc last, so this fixes that

int swap = iu;

iu = nrows - il + 1;

il = nrows - swap + 1;

lz = iu-il+1; //Number of eigenvectors to be returned.

lzp = lz; //Same variable, but this doesn't get overwritten.

lisuppz = 4*lz; //Size of the iwork array

lwork = 50*nrows; //Size of work array

liwork= 40*nrows; //Size of lwork array

//////////////////// In case of seg fault, try upping the size of these arrays.

rnrows = nrows;

working = rnrows*rnrows;

size = floor(working+0.0001);

double *a = freadDMatrix(fnameANM,size); //Dynamically allocates memory and initializes matrix

printf("The final element of the matrix is %f.\\n",a[size-1]);

w = (double *)malloc(nrows*sizeof(double));

printf("Regarding memory allocation for w: "); ErrnoDescription(errno);

z = (double *)malloc(nrows*lz*sizeof(double));

printf("Regarding memory allocation for z: "); ErrnoDescription(errno);

work = (double *)malloc(lwork*sizeof(double));

printf("Regarding memory allocation for work: "); ErrnoDescription(errno);

iwork = (int *)malloc(liwork*sizeof(int));

printf("Regarding memory allocation for iwork: "); ErrnoDescription(errno);

printf("Executing dsyevrf.\\n");

dsyevrf_(&nrows, a, &il, &iu, &m, w, z, &lz, &lisuppz, work, &lwork, iwork, &liwork, &info);

}

void dsyevrf_(int nrows, double *matrix, int *il, int *iu, int *m, double *w, double *z, int *lz, int *lisuppz, double *work, int *lwork, int *iwork, int *liwork, int *info);

int main ()

{

//////////////////////////////////////////////////////////////////////////////////////////

///////////////////////////////// GetEigenvectors /////////////////////////////////////

//////////////////////////////////////////////////////////////////////////////////////////

size_t ncells6 = 66113;

char *fnameGCorr_sy = "/home/mspecian/PSnoise/noiseMat_Vec/GCorMat_R6sy_fwrite";

int il = 1;

int iu = 10000;

/*GetEigenvectors[1. Name of symmetric, fwritten ncells by ncells covariance function

to be diagonalized

2. Dimension of above matrix, also # of cells

3. Value >= 1 equaling # of first eigenvector returned (1 is largest)

4. Value >= il and <= ncells equaling # of final eigenvector returned]*/

GetEigenvectors(fnameGCorr_sy, ncells6, il, iu);

return 0;

}

The functiondsyevrf_ is written in FORTRAN. Here is how I define the subroutine:

c********************************************************************

subroutine dsyevrf(n,a,il,iu,m,w,z,lz,lisuppz,work,lwork,iwork,liwork,info)

c--------------------------------------------------------------------

c Calculates selected eigenvalues and eigenvectors of a

c symmetric matrix T.

c

c Based upon the dsyevr routine.

c

c This routine is called from the C function GetEigenvectors

c

c 'N' should be used if only eigenvalues are desired.

c 'V' should be used if both eigenvalues and eigenvalues are

c desired.

c For the second parameter, 'A' means all eigenvalues will be

c found. 'V' means all eigenvalues in the half-open interval

c (vl,vu] will be found. 'I' mean the eigenvalues with indices

c il through iu will be found.

c For the third parameter, 'U' means the symmetric matrix is

c stored in the upper triangle of a and 'L' means its stored in

c the lower triangle.

c "n" is the number of rows (columns) in the symmetric matrix

c "a" is the nXn array housing the symmetric matrix in its upper

c or lower triangle.

c "vl" and "vu" are the lower and upper bounds respectively of the

c interval to be searched for eigenvalues. "vl" <= "vu". These

c values are referenced only if the second parameter = 'V'

c "il" and "iu" are the indices of the smallest and largest eigenvalues

c to be returned such that 1<=il<=iu<=in. These values are referenced

c only if the second parameter = 'I'

c "abstol" is the absolute error tolerance for the eigenvalues.

c "z" is an output parameter. If 'N', this is not referenced. If

c 'V', then this will contain the "lz" eigenvectors specified by

c the range "il" through "iu".

c "m" is the total number of eigenvalues found.

c "w" is the array to which the first m eigenvalues are stored.

c "z" is the array to which the first m eigenvectors are stored.

c "isuppz" contains, upon output, the indices indicating the non-zero

c elements in "z". It has length "lisuppz".

c "work" is a workspace array of length "lwork"

c "iwork" is a workspace array of length "liwork" = 10n

c "info" is an integer. If all is ok, a 0 is output. If i, the

c algorithm did not converge and i indicates the number of

c elements of an intermediate tridiagonal form which did not

c converge to zero. If -i, the ith parameter had an illegal

c value.

c

c Using LAPACK.

c

implicit none

integer n,il,iu,m,lz,lisuppz,isuppz(lisuppz),lwork,liwork,iwork(liwork),info

real*8 a(n,n),vl,vu,abstol,dlamch,w(n),z(n,lz),work(lwork)

external dlamch

c

abstol = dlamch('S')

print *, 'a(final) = ', a(n,n)

print *, 'About to compute eigenvectors',il,'through',iu,'.'

print *, 'n = ',n

c

call dsyevr('V','I','L',n,a,n,vl,vu,il,iu,abstol,m,w,z,n,isuppz,work,lwork,iwork,liwork,info)

c

return

end

c

c

The code compiles. Upon execution, I receive the following error:

This routine returns eigenmode #1 (larger eigenvalue) through 10000 (smaller eigenvalue).

Regarding memory allocation for /home/mspecian/PSnoise/noiseMat_Vec/GCorMat_R6sy_fwrite: Memory allocated properly.

All values read from /home/mspecian/PSnoise/noiseMat_Vec/GCorMat_R6sy_fwrite in 49 seconds.

The final element of the average noise matrix is 48.310387.

Regarding memory allocation for w: Memory allocated properly.

Regarding memory allocation for z: Memory allocated properly.

Regarding memory allocation for work: Memory allocated properly.

Regarding memory allocation for iwork: Memory allocated properly.

Executing dsyevrf.

a(final) = 48.3103872000000

About to compute eigenvectors 56114 through 66113 .

n = 66113

MKL ERROR: Parameter 10 was incorrect on entry to DSYEVR

Parameter 10 had an illegal value.

*** glibc detected *** ./eigen.o: free(): invalid pointer: 0x00002b12ffffffff ***

======= Backtrace: =========

/lib64/libc.so.6[0x3b2566f444]

/lib64/libc.so.6(cfree+0x8c)[0x3b25672a6c]

./eigen.o[0x40723a]

./eigen.o[0x4082ea]

/lib64/libc.so.6(__libc_start_main+0xf4)[0x3b2561d8a4]

./eigen.o[0x4034a9]

======= Memory map: ========

00400000-004bc000 r-xp 00000000 00:23 2148785222 /home/mspecian/PSnoise/eigen.o

006bb000-006c2000 rw-p 000bb000 00:23 2148785222 /home/mspecian/PSnoise/eigen.o

006c2000-006c9000 rw-p 006c2000 00:00 0

1da45000-1da66000 rw-p 1da45000 00:00 0 [heap]

313fe00000-313fefc000 r-xp 00000000 08:12 196705 /usr/lib64/libfftw3.so.3.2.4

313fefc000-31400fc000 ---p 000fc000 08:12 196705 /usr/lib64/libfftw3.so.3.2.4

31400fc000-3140103000 rw-p 000fc000 08:12 196705 /usr/lib64/libfftw3.so.3.2.4

3acc400000-3acc40d000 r-xp 00000000 08:12 491255 /lib64/libgcc_s-4.1.2-20080825.so.1

3acc40d000-3acc60d000 ---p 0000d000 08:12 491255 /lib64/libgcc_s-4.1.2-20080825.so.1

3acc60d000-3acc60e000 rw-p 0000d000 08:12 491255 /lib64/libgcc_s-4.1.2-20080825.so.1

3b24400000-3b2441a000 r-xp 00000000 08:12 2714721 /lib64/ld-2.5.so

3b24619000-3b2461a000 r--p 00019000 08:12 2714721 /lib64/ld-2.5.so

3b2461a000-3b2461b000 rw-p 0001a000 08:12 2714721 /lib64/ld-2.5.so

3b25600000-3b25746000 r-xp 00000000 08:12 2714722 /lib64/libc-2.5.so

3b25746000-3b25946000 ---p 00146000 08:12 2714722 /lib64/libc-2.5.so

3b25946000-3b2594a000 r--p 00146000 08:12 2714722 /lib64/libc-2.5.so

3b2594a000-3b2594b000 rw-p 0014a000 08:12 2714722 /lib64/libc-2.5.so

3b2594b000-3b25950000 rw-p 3b2594b000 00:00 0

3b25a00000-3b25a82000 r-xp 00000000 08:12 2714723 /lib64/libm-2.5.so

3b25a82000-3b25c81000 ---p 00082000 08:12 2714723 /lib64/libm-2.5.so

3b25c81000-3b25c82000 r--p 00081000 08:12 2714723 /lib64/libm-2.5.so

3b25c82000-3b25c83000 rw-p 00082000 08:12 2714723 /lib64/libm-2.5.so

3b25e00000-3b25e02000 r-xp 00000000 08:12 2714724 /lib64/libdl-2.5.so

3b25e02000-3b26002000 ---p 00002000 08:12 2714724 /lib64/libdl-2.5.so

3b26002000-3b26003000 r--p 00002000 08:12 2714724 /lib64/libdl-2.5.so

3b26003000-3b26004000 rw-p 00003000 08:12 2714724 /lib64/libdl-2.5.so

3b26200000-3b26215000 r-xp 00000000 08:12 2714726 /lib64/libpthread-2.5.so

3b26215000-3b26414000 ---p 00015000 08:12 2714726 /lib64/libpthread-2.5.so

3b26414000-3b26415000 r--p 00014000 08:12 2714726 /lib64/libpthread-2.5.so

3b26415000-3b26416000 rw-p 00015000 08:12 2714726 /lib64/libpthread-2.5.so

3b26416000-3b2641a000 rw-p 3b26416000 00:00 0

2b09f1d32000-2b09f1d35000 rw-p 2b09f1d32000 00:00 0

2b09f1d67000-2b09f2004000 r-xp 00000000 00:20 5944241 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/libmkl_core.so

2b09f2004000-2b09f2104000 ---p 0029d000 00:20 5944241 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/libmkl_core.so

2b09f2104000-2b09f2109000 rw-p 0029d000 00:20 5944241 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/libmkl_core.so

2b09f2109000-2b09f2126000 rw-p 2b09f2109000 00:00 0

2b09f2126000-2b09f2787000 r-xp 00000000 00:20 5944270 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/libmkl_sequential.so

2b09f2787000-2b09f2887000 ---p 00661000 00:20 5944270 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/libmkl_sequential.so

2b09f2887000-2b09f2893000 rw-p 00661000 00:20 5944270 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/libmkl_sequential.so

2b09f2893000-2b09f289f000 rw-p 2b09f2893000 00:00 0

2b09f289f000-2b09f2b41000 r-xp 00000000 00:20 5944250 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/libmkl_intel_ilp64.so

2b09f2b41000-2b09f2c40000 ---p 002a2000 00:20 5944250 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/libmkl_intel_ilp64.so

2b09f2c40000-2b09f2c48000 rw-p 002a1000 00:20 5944250 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/libmkl_intel_ilp64.so

2b09f2c48000-2b09f2c52000 rw-p 2b09f2c48000 00:00 0

2b09f4000000-2b09f4021000 rw-p 2b09f4000000 00:00 0

2b09f4021000-2b09f8000000 ---p 2b09f4021000 00:00 0

2b1216fdd000-2b13547b5000 rw-p 2b1216fdd000 00:00 0

2b13547b5000-2b13552f9000 r-xp 00000000 00:20 5944257 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/libmkl_lapack.so

2b13552f9000-2b13553f9000 ---p 00b44000 00:20 5944257 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/libmkl_lapack.so

2b13553f9000-2b1355409000 rw-p 00b44000 00:20 5944257 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/libmkl_lapack.so

2b1355409000-2b135540a000 rw-p 2b1355409000 00:00 0

2b135540a000-2b135541f000 r--p 00000000 00:20 5944281 /data1/apps0/intel_fortran-11.1.069/x64-linux/mkl/lib/em64t/locale/en_US/mkl_msg.cat

7fffaa40f000-7fffaa43e000 rw-p 7ffffffd0000 00:00 0 [stack]

ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]

Abort

This seems like a memory issue, but I can't find where I've gone wrong.

[Gennady_F_Intel]

What exact MKL version you are using? look into \doc\ mklsupport.txt file. what is package_id?

--Gennady

[mspecian]

MKL 10.2 update 4

[barragan_villanueva_]

Hi,

Could you please show us your link-line using MKL libraries? Please check it with MKL Link Line Adviser on the top of MKL Forum.

[mspecian]

I figured it out. In C I was passing 4-byte integers, while in FORTRAN I was declaring 8-byte integers. This didn't always lead to obvious memory overwrite issues so it took me a while to identify. Thanks to anyone who read through.