BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

[sfqian]

I use scalapack pdsyevx to calculate eigenvalues. It is available when the matrix is 87000*87000.However, when I increase dimension to 97200, the program is terminated.  I also increase a node to run this program and same fault occurred.

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 14440 RUNNING AT e19
= EXIT CODE: 9
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================

The source code:

*
*
      PROGRAM SAMPLE_PDSYEVX_CALL
*
*
*  -- ScaLAPACK routine (version 1.2) --
*     University of Tennessee, Knoxville, Oak Ridge National Laboratory,
*     and University of California, Berkeley.
*     May 10, 1996
*
*     This routine contains a sample call to PDSYEVX.
*     When compiled and run, it produces output which can be
*     pasted directly into matlab.
*
*
*     .. Parameters ..
      INTEGER            LWORK, MAXN, LIWORK
      DOUBLE PRECISION   ZERO
      PARAMETER          ( LWORK = 100000000, MAXN = 97210,
     $                   LIWORK = 5000,
     $                   ZERO = 0.0D+0 )
      INTEGER            LDA
      DOUBLE PRECISION   MONE
      INTEGER            MAXPROCS
      PARAMETER          ( LDA = MAXN, MONE = -1.0D+0, MAXPROCS = 1024 )
*     ..
*     .. Local Scalars ..
      INTEGER            CONTEXT, I, IAM, INFO, M, MYCOL, MYROW, N, NB,
     $                   NPCOL, NPROCS, NPROW, NZ
*     ..
*     .. Local Arrays ..
      INTEGER            DESCA( 200 ), DESCZ( 200 ),
     $                   ICLUSTR( MAXPROCS*2 ), IFAIL( MAXN ),
     $                   IWORK( LIWORK )
      DOUBLE PRECISION   A( LDA, LDA ), GAP( MAXPROCS ), W( MAXN ),
     $                   IWORK( LIWORK )
      DOUBLE PRECISION   A( LDA, LDA ), GAP( MAXPROCS ), W( MAXN ),
     $                   WORK( LWORK ), Z( LDA, LDA )
*     ..
*     .. External Subroutines ..
      EXTERNAL           BLACS_EXIT, BLACS_GET, BLACS_GRIDEXIT,
     $                   BLACS_GRIDINFO, BLACS_GRIDINIT, BLACS_PINFO,
     $                   BLACS_SETUP, DESCINIT, PDLAMODHILB, PDLAPRNT,
     $                   PDSYEVX
*     ..
*     .. Executable Statements ..
*
*
*     Set up the problem
*
      write (*, *), "aa"
      N = 97200
      NB = 32
      NPROW = 4
      NPCOL = 6
*
*
*     Initialize the BLACS
*
      write (*, *), "a"
      CALL BLACS_PINFO( IAM, NPROCS )
      IF( ( NPROCS.LT.1 ) ) THEN
         CALL BLACS_SETUP( IAM, NPROW*NPCOL )
      END IF
*
*
*     Initialize a single BLACS context
       write (*, *), "b"
      CALL BLACS_GET( -1, 0, CONTEXT )
      CALL BLACS_GRIDINIT( CONTEXT, 'R', NPROW, NPCOL )
      CALL BLACS_GRIDINFO( CONTEXT, NPROW, NPCOL, MYROW, MYCOL )
*
*     Bail out if this process is not a part of this context.
*
      IF( MYROW.EQ.-1 )
     $   GO TO 20
*
*
*     These are basic array descriptors
*
      write (*, *), "c"
      CALL DESCINIT( DESCA, N, N, NB, NB, 0, 0, CONTEXT, LDA, INFO )
      CALL DESCINIT( DESCZ, N, N, NB, NB, 0, 0, CONTEXT, LDA, INFO )
*
*     Build a matrix that you can create with
*     a one line matlab command:  hilb(n) + diag([1:-1/n:1/n])
*
      CALL PDLAMODHILB( N, A, 1, 1, DESCA, INFO )
*
*
*     Uncomment this line to see the matrix printed out.
*
*      CALL PDLAPRNT( N, N, A, 1, 1, DESCA, 0, 0, 'A', 6, WORK )
*
*
*     Ask PDSYEVX to compute the entire eigendecomposition
*
      CALL PDSYEVX( 'V', 'I', 'U', N, A, 1, 1, DESCA, ZERO, ZERO, 68590,
     $              68610, MONE, M, NZ, W, MONE, Z, 1, 1, DESCZ, WORK,
     $              LWORK, IWORK, LIWORK, IFAIL, ICLUSTR, GAP, INFO )
*
*
*     Print out the eigenvectors
*
*      CALL PDLAPRNT( N, 10, Z, 1, 1, DESCZ, 0, 0, 'Z', 6, WORK )
*
*
*     Print out matlab code which will check the residual
*
      IF( MYROW.EQ.0 .AND. MYCOL.EQ.0 ) THEN
         PRINT *, ' N =', N
         PRINT *, ' A =  hilb(N) + diag([1:-1/N:1/N])'
         DO 10 I = 1, N
            PRINT *, W( I )
   10    CONTINUE
         PRINT *, ' backerror = A - Z * diag(W) * Z'' '
         PRINT *, ' resid = A * Z - Z * diag(W)'
         PRINT *, ' ortho = Z'' * Z - eye(N)'
         PRINT *, ' norm(backerror)'
         PRINT *, ' norm(resid)'
         PRINT *, ' norm(ortho)'
      END IF
*
      CALL BLACS_GRIDEXIT( CONTEXT )
*
   20 CONTINUE
*
      CALL BLACS_EXIT( 0 )
*
*
*     Uncomment this line on SUN systems to avoid the useless print out
*
*      CALL IEEE_FLAGS( 'clear', 'exception', 'underflow', '')

[Gennady_F_Intel]

What version of mkl and mpi do you use?

How did you run this executable?

Could you show how did you link this case?

if we will build the code you showed, could we reproduce the issue? 

[sfqian]

I use intel2016.2.181 mkl and  intelmpi 5.1.3.181

compile: mpif90 -O2 -mcmodel=medium -o example1 sample_pdsyevx_call.f -L/opt/intel2016/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64 -mkl -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lmkl_lapack95_lp64 -lmpi -lmkl_sequential -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread -lm -ldl

run: mpirun -np 24 ./example1

I only increased N and MAXN in this code, and It did not work.

[Gennady_F_Intel]

Do you have an opportunity to check the current ( latest ) version 2020.1?

[Gennady_F_Intel · Accepted solution]

Could you please try to relink with ILP64 libraries and run once again. Here is the link to the MKL Linker Adviser to check how to do that: https://software.intel.com/content/www/us/en/develop/articles/intel-mkl-link-line-advisor.html

[sfqian]

The problem is solved! Thanks a lot.