problem with arrays with more than 2GB elements

[Udovydchenkov__Ilya]

Hi everyone.

I am trying to compile and run a code that calls a subroutine, which allocates an array with about 6 billion elements. I define the array in the main code as

integer(kind=8), dimension(:), allocatable  :: adjncy

and then in the subroutine as

integer(kind=8), dimension(0:sup_neighbour*nspec-1)  :: adjncy

Here my sup_neighbor=54, and nspec=109360000.

Then I assign array elements and return it to the main program. The issue is that once it is returned all elements in adjncy are wrong. No errors are produced during compilation or execution. I tried setting -mcmodel=medium or -mcmodel=large together with -shared-intel  compiler options for ifort, but it does not change anything. I am using composer_xe_2015.2.164 on Intel64 Linux cluster. For the mpiifort I use 5.0.3.048.

Thanks.

[TimP]

.

[TimP]

You must promote that size calculation to kind=8 or more portable kind=int64. Likewise any variable used to index the array. Mc_model shouldn't be needed in intel64 mode. Mpi messaging also needs specification of 64 bit data type.

[Udovydchenkov__Ilya]

Tim,

Thank you for the answer. I set kind=8 for all relevant integers. sup_neighbour and nspec are both kind=8. I also tried

integer(kind=8), dimension(0:int(sup_neighbour*nspec-1,8))  :: adjncy, but it does not help.

I am able to get around it if I set the static size in the main program, i.e. integer(kind=8), dimension(0:int(5905439999,8))  :: adjncy

and comment the allocate command allocate(adjncy(1:sup_neighbour*nspec),stat=ier)

In this case everything works correctly, but not with allocate. Also, how do I specify mpi messaging to be 64 bits?

Thanks.

[FortranFan]

Ilya U. wrote:

..also tried

integer(kind=8), dimension(0:int(sup_neighbour*nspec-1,8))  :: adjncy, but it does not help.

..  works correctly, but not with allocate. ..

Can you try out the following code in your system (note nspec is an order of magnitude smaller) and see what you get?

module m

   use, intrinsic :: iso_fortran_env, only : i8 => int64

   implicit none

   private

   public :: i8
   public :: foo

contains

   subroutine foo( adjncy )

      integer(kind=i8), intent(inout) :: adjncy(0:)

      integer(kind=i8) :: i

      print *, " lower bound of adjncy = ", lbound(adjncy, dim=1)
      print *, " upper bound of adjncy = ", ubound(adjncy, dim=1)
      
      adjncy = [( i, i = 0, ubound(adjncy,dim=1) )]

      return

   end subroutine foo

end module m

program p

   use m, only : i8, foo

   implicit none

   integer(i8), parameter :: sup_neighbor = 54_i8
   integer(i8), parameter :: nspec = 10936000_i8
   integer(i8), parameter :: L = 0_i8
   integer(i8), parameter :: U = sup_neighbor*nspec - 1_i8

   integer(i8), allocatable :: adjncy(:)
   integer(i8) :: istat
   character(len=80) :: erralloc

   allocate( adjncy(L:U), stat=istat, errmsg=erralloc )
   if (istat /= 0) then
      print *, " allocation of adjncy failed: stat=", istat, ", errmsg=", erralloc
      stop
   end if

   call foo( adjncy )
   
   print *, " sup_neighbor*nspec-1 = ", U 
   print *, " adjncy(0) = ", adjncy(0)
   print *, " adjncy(100) = ", adjncy(100)
   print *, " adjncy(sup_neighbor*nspec-1) = ", adjncy(U)

   stop

end program p

Upon execution with Intel Fortran 16.0 version and the smaller array size, I see no problems so the question would be whether an issue arises for you with your compiler version and the larger size array:

  lower bound of adjncy =  0
  upper bound of adjncy =  590543999
  sup_neighbor*nspec-1 =  590543999
  adjncy(0) =  0
  adjncy(100) =  100
  adjncy(sup_neighbor*nspec-1) =  590543999
Press any key to continue . . .

I'm curious because I'll working with a team soon doing some simulations where the number of nodes may range from 2**20 to 2**30 and 64-bit arithmetic may be involved.

Thanks,

[Udovydchenkov__Ilya]

FortranFan,

thank you for the code. Your example produces the output that you posted. With nspec=109360000 I get correct output except upper bound of adjncy = 1610472703, which I think is ok, because we probably need to specify kind=8 as a parameter for ubound.

I will try to adapt your code into the model.

Ilya.

[FortranFan]

Ilya U. wrote:

.. With nspec=109360000 I get correct output except upper bound of adjncy = 1610472703, which I think is ok, because we probably need to specify kind=8 as a parameter for ubound...

Ok, thanks for the feedback.  Yes, it may be better to write the subroutine as:

module m

   use, intrinsic :: iso_fortran_env, only : i8 => int64

   implicit none

   private

   public :: i8
   public :: foo

contains

   subroutine foo( adjncy )

      integer(kind=i8), intent(inout) :: adjncy(0:)

      integer(kind=i8) :: i

      print *, " lower bound of adjncy = ", lbound(adjncy, dim=1, kind=i8)
      print *, " upper bound of adjncy = ", ubound(adjncy, dim=1, kind=i8)

      adjncy = [( i, i = 0_i8, ubound(adjncy,dim=1,kind=i8) )]

      !.. I'd prefer to write the implied do loop this way, but Intel Fortran
      !   compiler gives an error
      !adjncy = [( i, i = int(0,kind=kind(i)), ubound(adjncy,dim=1,kind=kind(i)) )]

      return

   end subroutine foo

end module m

[Udovydchenkov__Ilya]

FortranFan,

thanks for help again. I have a follow-up question.

Does anyone have experience working with SCOTCH mesh practitioner?

The problem I have is that the adjacency array I need to pass to SCOTCH has more than 2GB elements (5,905,440,000 in my test case). With your help I verified that the array is created and indexed correctly in Fortran. Now I compiled SCOTCH with -DIDXSIZE64 -DINTSIZE64 flags. Calls to scotchfgraphbuild and scotchfgraphcheck produce no errors. When I call scotchfgraphpart it returns an array of correct size with all zeros in it.

Does anyone know if SCOTCH can handle adjacency arrays with more than 2GB elements? Is there a way to debug the issue?

Thanks.