Maximum FORTRAN Array Sze or possible number of entries in FORTRAN ARRAY

[Qaiser_Bozdar]

Please asses following code. My PC fails to produce any output whenever dimension exceeds 4x10^8.

program ARRAY
!! Program to check the maximum array size of a FORTRAN Array
implicit none
!Variables

real A(500000000) !!! Array

INTEGER I
do 10 I=1,500000000
A(I)=I*1
! WRITE(*,*)A(I),I
10 CONTINUE
WRITE(*,*)A(I-1),I-1

END

I have program beside this one that needs an array of size more than 10^8.  But whenever 10^8  is exceeded, the build section prompts for the message of capacity overflow.

I have to work on three dimensional array such as A(1350,1350,1350,5). Every thing is fine up to A(135,135,135,5). My that code is oriented to Nuclear Science.

I am using Intel FORTRAN Compiler 2024.1 with MS Visual Studio Community Version 2022. 

Can paging system or mechanism help in this matter ? Or will increasing virtual memory effect the above code or not or will it increase the capacity of FORTRAN array?

How to access 64 space address on 64bit computer?

Thank you

Engr. Qaiser

[S-Poulsen]

Your example code Farray.f90 executes fine using ifx 2024.2.0 on linux. I think the single precision array A uses the storage 2GB approximately.

Have you selected x64 build in Visual Studio? See this thread for more information https://community.intel.com/t5/Intel-Fortran-Compiler/How-to-build-up-a-64-bit-application/m-p/1114853

[Qaiser_Bozdar]

Yes I have selected x64 build, and I also tried with x86. No effect, at all.

Thank you.

[jimdempseyatthecove]

real, allocatable :: A(:) !!! Array

INTEGER I
allocate(A(500000000))
do 10 I=1,500000000
A(I)=I*1
! WRITE(*,*)A(I),I
10 CONTINUE
WRITE(*,*)A(I-1),I-1

The upper limit then becomes the size of the heap.

If you build the application as x64 your limit on the heap is dependent on the O/S

(either physical RAM limit or remaining Page File size or some system policy setting)

Jim Dempsey

[Steve_Lionel]

On Windows, even 64-bit, you are limited to 2GB total size of code and static data. Windows will do strange things if you try to run an EXE that comes close to that limit. The linker may or may not complain, depending on just how the program is structured.

Linux allows larger static data if you use -mcmodel medium. If you make the array ALLOCATABLE, you can use more than 2GB on 64-bit Windows.

[Qaiser_Bozdar]

Sir How can I increase that 2GB limit? 

I have used ALLOCATABLE. Now, I get the insufficient virtual memory, and last accurate output is 2x10^9 (On a computer having 16GB RAM, core I7). My personal (Intel Core M3) computer went hang which has a RAM of 8GB when I set maximum array size of 2x10^9.

I have tested code by having virtual memory increased to 10GB. Still, same result.

I have attached two images. Those images show the result of code without the use of ALLOCATABLE. [PC CAN`T RUN, Access denied]

Check the following image and attached FORTRAN code:With ALLOCATABLE

Any way, can I get an array of size more than 8x10^9 and 10^10?

[jimdempseyatthecove]

>>I get the insufficient virtual memory

You will have to increase the page file size

Press the Windows key and search for "System".

Click on "Advanced System Settings".

Click on "Settings" under the Performance section.

Navigate to the "Advanced" tab and click on the "Change" button under Virtual Memory.

Uncheck the box next to "Automatically manage paging file size for all drives".

Select "Custom Size" and enter the desired values for "Initial size" and "Maximum size".

Click "Set", then "OK", and exit from the Virtual Memory, Performance Options, and Systems Properties Dialog.

Note, depending on how you access this(these) large arrays, performance can be very slow.

For better performance, you would want to tile the access to these large arrays.

Jim Dempsey

[Qaiser_Bozdar]

Highly excuses Sir @jimdempseyatthecove 

I have mentioned that I have increased virtual memory to 10GB. It did not worked out. In the reply to @Steve_Lionel 

>> I have tested code by having virtual memory increased to 10GB. Still, same result. (On a computer having 16GB RAM, core I7)

Please you should run this code and post the results here What kind of results you get on your computer? Your computer specifications (ex: RAM, Virtual memory allocated, Ghz)???

[jimdempseyatthecove]

I have mentioned that I have increased virtual memory to 10GB

Your last screenshot shows allocation attempt of 8,000,000,000 reals, 4 bytes each for a total of 32GB for that array alone.

Your test program could possibly run with a page file size of 33GB (though it may take a bit more if that fails try 36GB).

Jim Dempsey

[Qaiser_Bozdar]

Ohh, My goodness!! I can assure that you are right Sir.

I anticipated that but I was not sure. 

Anyway, how can I get an array of size more than 8x10^9 and 10^10? I have to work on three dimensional arrays of following type:

EXAMPLE:

REAL, ALLOCATABLE :: PHI(:,:,:,:)

ALLOCATE PHI((50000,50000,50000,5))

I have to use more than 10 variables same that I have mentioned in the above example.

[Steve_Lionel]

I tried this variant of your program:

real, allocatable:: A(:) !!! Array

INTEGER I
allocate (A(500000000))
do 10 I=1,500000000
A(I)=I*1
! WRITE(*,*)A(I),I
10 CONTINUE
WRITE(*,*)A(I-1),I-1

END

and here's the result:

D:\Projects>ifx t.f90
Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2024.2.0 Build 20240602
Copyright (C) 1985 - 2024 Intel Corporation. All rights reserved.

Microsoft (R) Incremental Linker Version 14.40.33811.0
Copyright (C) Microsoft Corporation.  All rights reserved.

-out:t.exe
-subsystem:console
t.obj

D:\Projects>t.exe
  5.0000000E+08   500000000

 This is a10th-gen Core i7 system (Intel NUC) with 64GB RAM and 4GB paging file.

[Qaiser_Bozdar]

Sir I appreciate your efforts. And, really you have a powerful computer, I liked that.

Sir will you test the code for the following inputs or array size exceeding 5x10^8.

A(50000000000)    ! 5x10^10

A(800000000)     !8x10^8

A(8000000000)  !!8x10^9

A(80000000000)!8x10^10

A(8000000000000)!8x10^12

I just want to know the maximum number of entries can I have in a FORTRAN array.

Q1. Does it depend on compiler?

Q2. Does compiler limits the size of array?

[Steve_Lionel]

The answer to both of these questions is is "No". You're limited by the operating system and amount of virtual memory available. At some point, though, the CPU time to churn through all of these elements will become too much. You should look at a different algorithm.

[andrew_4619]

Do you need all these arrays at the same time or can you allocate when needed and deallocate when no longer needed? That might save a lot of memory. Algorithm changes as Steve suggest if the way forward if that is possible.

[jimdempseyatthecove · Accepted solution]

Your array PHI will require 2,500,000,000,000,000 petabytes of memory, you have stated you have about 10 such arrays or 25 petabytes for these arrays alone. This is 25,000 terabytes.

Ask yourself:

Do you have that much disk space?

Do you have this much (or more) physical RAM?

If not, and you rely on a page file, you will need an additional 25 petabytes of disk for the page file.

Are these arrays, sparse arrays?

IOW the indices are for convenience. If so, and IIF your data size is reasonable, can you restructure your storage to collapse/compact it such that it fits your system.

You will also need to look at your CPU's virtual addressing capabilities as well as physical addressing capabilities. The 3rd generation Ice Lane Xeon Scalable processor has 57 bits (128PiB) of virtual addressing, and 52 bits (4PiB) physical addressing. So this CPU could handle your arrays, albeit with paging and may run extremely slow. Note, may CPU's do not have these capabilities for Virtual and physical addressing. Many have 48 bits of virtual address space or 0.25 petabytes.

To quote Clint Eastwood's character "Dirty Hary": A man's gotta know his limitations.

Maybe you should describe your problem requirements so we can better give you advice.

Jim Dempsey