error is pointing at CALL SCALAPACK_MAP

[skravoof]

Dear All,
I am using Intel fortran with imsl. I took the simple scalapack example for lslrg from the manual and did few modifications. However, if N=3 then the following code works. If I N=1000 the model falls a part saying stack overflow. The error is pointing at CALL SCALAPACK_MAP(A, DESCA, A0).
Any suggestions?
Thanks in advance.
Regards,
skravoof



Code:

program main
USE MPI_SETUP_INT
USE LSLRG_INT
USE WRRRN_INT
USE SCALAPACK_SUPPORT
IMPLICIT NONE
INCLUDE 'mpif.h'
include 'link_mpi_hpc.h'
! Declare variables
INTEGER N, DESCA(9), DESCX(9)
INTEGER INFO, MXCOL, MXLDA
integer i1,i2,i3
REAL, ALLOCATABLE :: A(:,:), B(:), X(:)
REAL, ALLOCATABLE :: A0(:,:), B0(:), X0(:)
PARAMETER (N=3)
! Set up for MPI
MP_NPROCS = MP_SETUP()
IF(MP_RANK .EQ. 0) THEN
ALLOCATE (A(N,N), B(N), X(N))
! Set values for A and B
A=0
do i1=1,N
A(i1,i1)=1.0 ! A is a diagonal matrix
B(i1)=10*i1
enddo
ENDIF

! Set up a 1D processor grid and define
! its context id, MP_ICTXT
CALL SCALAPACK_SETUP(N, N, .TRUE., .TRUE.)
! Get the array descriptor entities MXLDA,
! and MXCOL
CALL SCALAPACK_GETDIM(N, N, MP_MB, MP_NB, MXLDA, MXCOL)
! Set up the array descriptors
CALL DESCINIT(DESCA, N, N, MP_MB, MP_NB, 0, 0, MP_ICTXT, MXLDA, INFO)
CALL DESCINIT(DESCX, N, 1, MP_MB, 1, 0, 0, MP_ICTXT, MXLDA, INFO)
! Allocate space for the local arrays
ALLOCATE (A0(MXLDA,MXCOL), B0(MXLDA), X0(MXLDA))
! Map input arrays to the processor grid
CALL SCALAPACK_MAP(A, DESCA, A0)
CALL SCALAPACK_MAP(B, DESCX, B0)
! Solve the system of equations
CALL LSLRG (A0, B0, X0)
! Unmap the results from the distributed
! arrays back to a non-distributed array.
! After the unmap, only Rank=0 has the full
! array.
CALL SCALAPACK_UNMAP(X0, DESCX, X)
! Print results
! Only Rank=0 has the solution, X.
IF(MP_RANK .EQ. 0)CALL WRRRN ('X', X, 1, N, 1)
IF (MP_RANK .EQ. 0) DEALLOCATE(A, B, X)
DEALLOCATE(A0, B0, X0)
! Exit ScaLAPACK usage
CALL SCALAPACK_EXIT(MP_ICTXT)
! Shut down MPI
MP_NPROCS = MP_SETUP('FINAL')
END

[Steven_L_Intel1]

My guess is that this error is happening inside SCALAPACK_MAP - I don't see that the compiler would be creating a temporary array here. If you are building in Visual Studio, try setting the project property Linker > System > Stack reserve size to a larger value. The default is 1MB, so try 100000000.

[skravoof]

Dear Steve,
To my surprise initially stack reservoir size was set to 0. I changed it to 100000000 and it worked. However, why it was zero intially.Another question. I am using dual core x64 system. I launched the exe file using mpiexec. when I increase the N value I am expecting the job to be distributed between two cores. However, the maximum cpu usage never increased more than 55%. Am doing something wrong?

skravoof

[Steven_L_Intel1]

0 means default, which is 1MB.

I am not familiar with how IMSL uses MPI - but it could be that memory traffic is a limiting factor. If you are using a dual-core system, I suggest you look at Intel Math Kernel Library which has SCALAPACK routines and is optimized for multicore systems.

[skravoof]

The same program crashes again for n=2000. It works fine below 1000. I cant increase theStack reserve size anymore? any suggestions?

[Steven_L_Intel1]

I tried n=2000 with stack reserve of 100000000 and it worked. When the error occurs, the stack trace seems to be corrupted so I am having difficulty locating the culprit.

I will ask my colleagues at VNI if they can figure out where the temp is coming from.

[Steven_L_Intel1]

VNI showed me the code in SCALAPACK_MAP and fingered a use of RESHAPE that would definitely create a temporary array. There's nothing to be done for it for now other than to boost the stack size. I have suggested that in the future, IMSL be compiled with /heap-arrays to allow these array copies to be on the heap. Even on 64-bit Windows, the stack has a 1GB maximum size.

If you're happy with IMSL, then by all means keep using it. I note that MKL also offers an MPI SCALAPACK implementation, but don't know if it might suffer the same problem. It may be worth an experiment.

[skravoof]

Thank steve, I am kinda confused with the options that I have at the movement. is this means MPI SCALAPACK cant be applied to large matrics (like 50,000x50,000). Whats the point of having imsl with MPI support. I think I am missing something?
Thanks in advance.
skravoof

[Steven_L_Intel1]

I am not familiar enough (or really at all) with SCALAPACK to know whether this is a serious restriction of the IMSL implementation. I will admit that it creating an array expression for the argument is likely to be a problem. I have suggested to the IMSL developers that they be concerned about this.

All I can suggest at this time is to try the MKL version and see if it works better for you. Note that MKL is included in the product you have.