Allocating large arrays in 64-bit

[Simon_Geard]

I'm having trouble understanding the rules associated with allocating large arrays. The code below shows what I'm trying to do and what assumptions I'm making. The compiler output is

ifort 64.f90  /warn:all /debug:all /heap-arrays:100

Intel(R) Visual Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 13.0.1.119 Build 20121008
Copyright (C) 1985-2012 Intel Corporation.  All rights reserved.

Microsoft (R) Incremental Linker Version 10.00.40219.01
Copyright (C) Microsoft Corporation.  All rights reserved.

-out:64.exe
-debug
-pdb:64.pdb
-subsystem:console
64.obj

[fortran]

! ifort 64.f90 /4I8 /warn:all /debug:all /heap-arrays:100
!      allocation succeeded (correct)
!      allocation failed (correct)
!      allocation failed (error)

! ifort 64.f90 /warn:all /debug:all /heap-arrays:100
!      allocation succeeded (correct)
!      allocation succeeded (error)
!      allocation succeeded (correct)

program eight_byte_array
    implicit none

    ! Variables
    integer(kind=4), allocatable :: iarray(:)
    integer(kind=4) :: isize,iostatus

    integer(kind=8), allocatable :: bigarray(:)
    integer(kind=8) :: bigsize

    isize = 2**31-1
    allocate(iarray(isize),stat=iostatus) ! Should succeed because iarray is integer(4)
    if (iostatus == 0 .and. isize == size(iarray)) then
        write(*,'(a)') 'allocation succeeded (correct)'
    else
        write(*,'(a)') 'allocation failed (error)'
    end if
        
    deallocate(iarray)
    bigsize = 2**32+1
    allocate(iarray(bigsize),stat=iostatus) ! Should fail because iarray is integer(4)
    if (iostatus == 0 .and. bigsize == size(iarray)) then
        write(*,'(a)') 'allocation succeeded (error)'
    else
        write(*,'(a)') 'allocation failed (correct)'
    end if
    
    allocate(bigarray(bigsize),stat=iostatus) ! Should succeed because bigarray is integer(8)
    if (iostatus == 0 .and. bigsize == size(bigarray)) then
        write(*,'(a)') 'allocation succeeded (correct)'
    else
        write(*,'(a)') 'allocation failed (error)'
    end if

end program eight_byte_array
[/fortran]

[Mark_Lewy]

1) Be aware that the Fortran standard specifies that integer expressions evaluate with the default integer kind, unless you explicitly specify the kind, so when you compile with the default (32-bit) integer kind you should write the expression to assign bigsize as bigsize = 2_8**32 + 1_8, otherwise bigsize will be set to 1.

2) The return value for size without a kind argument is of default integer kind; you want to use size(iarray, kind = 8) to return an integer(8) value.

[Simon_Geard]

Thank you. That explains the difference the /4I8 switch was making. It doesn't explain why the final allocation is failing even if I set bigsize to be the same as isize.

[fortran]

  ! ifort 64.f90 /heap-arrays:100
!      allocation succeeded (correct)
!      allocation failed (error): 41

program eight_byte_array
    implicit none

    ! Variables
    integer(kind=4), allocatable :: iarray(:)
    integer(kind=4) :: isize, a_status

    integer(kind=8), allocatable :: bigarray(:)
    integer(kind=8) :: bigsize

    isize = 2**31-1
    allocate(iarray(isize),stat=a_status) ! Should succeed because iarray is integer(4)
    if (a_status /= 0) then
        write(*,'(a,i0)') 'allocation failed (error): ',a_status
    else        
        if (isize == size(iarray)) then
            write(*,'(a)') 'allocation succeeded (correct)'
        else
            write(*,'(a,i0,2(a,z8.8))') 'allocation size mismatch (error): ',a_status,' ',isize,' ',size(iarray)
        end if
        deallocate(iarray)
    end if
        
    bigsize = 2_8**31-1_8
    allocate(bigarray(bigsize),stat=a_status) ! Should succeed because bigsize is the same as isize
    if (a_status /= 0) then
        write(*,'(a,i0)') 'allocation failed (error): ',a_status
    else        
        if (bigsize == size(bigarray, kind=8)) then
            write(*,'(a)') 'allocation succeeded (correct)'
        else
            write(*,'(a,i0,2(a,z16.16))') 'allocation size mismatch (error): ',a_status,' ',bigsize,' ',size(bigarray,kind=8)
        end if
        deallocate(iarray)
    end if

end program eight_byte_array

[/fortran]

outputs

allocation succeeded (correct)
allocation failed (error): 41

[Mark_Lewy]

Error 41 is insufficient virtual memory: remember that as iarray is already allocated, taking up 16 GB memory, your allocation of bigarray will take memory requirements up to 48 GB.  How much physical memory and page file do you have?

[Simon_Geard]

Yes I failed to consider the size of the array elements (8) in this case although iarray is explicitly deallocated (line 21) so it shouldn't still be hanging around. I have 8GB physical memory and a pagefile limit of 16GB.

I have now  been able to get a c++ and Fortran program to product the same results so I think I understand what is going on. Thanks again for your help.

[fortran]

program eight_byte_array
    implicit none

    integer(kind=4) :: a_status
    integer(kind=4), allocatable :: bigarray(:)
    integer(kind=8) :: bigsize
        
    bigsize = 2_8**31+1
    allocate(bigarray(bigsize),stat=a_status, source=0)
    if (a_status /= 0) then
        write(*,'(a)') 'allocation failed'
    else        
        write(*,'(z8.8)') bigsize
    end if

end program eight_byte_array
[/fortran]

[cpp]

#include <vector>
#include <iostream>
using namespace std;

int main(int, char**) {

    vector<int> iv;
    try {
        iv.assign((1LL<<31)+1, 0);
        cout << hex << iv.size() << endl;
    } catch (...) {
        cout << "allocation failed" << endl;
    }
    return 0;
}

[/cpp]