Available memory command?

[David_Sall]

Is there a command I can run within a Fortran 90 code (using the Intel Visual Fortran version 11 compiler) that will output the amount of available memory I have (to allocate my arrays)?

Thank you very much.

Sincerely,

David

[David_Sall]

I just found a similar request on the forum from last year. Apparently (unfortuneately) there is no function call.

Thanks!


Sincerely,

David

[jimdempseyatthecove]

Generally you request allocation status when allocating

ALLOCATE(YourData, STAT=iStat)
IF(iStat .ne. 0) CALL YourAllocatonErrorRoutine()

Definition of "amount of available memory" is unclear.

Total unused memory in currently allocated heaps. Note this excludes additional space that can be acquired by expanding the heap. Note, when heap is fragmented this will not tell you if allocate will succeed.

Total unused Virtual Memory address space (exclusive of heaps allready allocated. Note this can exceed Page file limits. Allocation may succeed, but first touch may fail.

Note, event the ALLOCATE with STAT= will not assure you that the Page file will map the data once accessed.

Jim Dempsey

[Steven_L_Intel1]

The problem is that there is no way to know in advance whether an allocation will succeed or fail. Even if there is, in total, sufficient virtual memory available, it may be discontiguous or in a different memory pool.

[John_Campbell]

David,

Re largest array using ifort 64-bit.

I find the answers you have received from Jim and Steve to not be very helpful for what I expect you are trying to find out. I have been asking similar questions, but unfortunately there is not a simple answer.

I expect you are trying to find out how big an array you can obtain to enlarge the problem you are trying to solve. This size is limited mainly by the hardware you are using but also by the compiler and operating system you are using.

I presume you are asking about ifort 64-bit, as the answers for 32-bit are much clearer.

Lets look at the hardware constraints. If you are looking for a practical limit, which includes some assessment of the run time performance, then there are a number of issues to consider.

1) Physical Memory : How much physical memory is installed and available for your application? By limiting the amount of memory your program is using, your program should run in memory and not suffer from page faults. Depending on what other applications are running on your computer, my target is to allocate arrays to use about 80% of installed memory. You can calculate this as a simple spreadsheet calculation by combining the size of each array x the byte size of each element of each array. (Note that the intrinsic SIZE only returns an integer*4 answer so fails for arrays of size > 2^31 elements. An ifort bug?)

2) Virtual Memory : If you want to run a model whose size will exceed the physical memory available then you need to consider virtual storage. (or buy more memory!) The easiest way to program this is to use virtual memory, which is stored in pagefile.sys. This is shared by all processes that are running on your computer, so again you need to consider how much of this you should target. You can enlarge the size of this file to increase this limit. Using virtual memory comes at a performance cost. Programming techniques, such as localise addressing of memory can improve performance. At a simple level:

Doj = 1,n

Doi = 1,n

Array(i,j) = value

Will perform much faster than

Doi = 1,n

Doj = 1,n

Array(i,j) = value

Localising memory use when processing information in large arrays to minimise the number of page faults is what is required.

Recent introduction of Solid State Disks (SSD) has significantly improved the performance of virtual memory.

Alternatives of writing your own solutions for out of core approaches tend to provide superior performance, but with a significant coding effort. There is a rich history of these solutions for classical problems. (Combining these solutions with SSD has significantly extended the life of 32-bit compiler solutions.)

However, if you want a simple scaling up of the size of your problem, then virtual memory is the simple solution.

3) Fortran limit : How big an array can you code ? Any array larger than 2GB in size must be provided via ALLOCATE. Unfortunately ALLOCATE will not provide the answer you are looking for. The question of the array size limit is not how much memory is available to be addressed but also what is the largest contiguous memory that can be addresses. When you link your program, the linker puts different bits in different places, such as code, common, the stack, the heap and other operating system interfaces. These can be spread through memory, so the largest free space for ALLOCATE can be compromised by the way the linker spreads these elements of the memory usage. (This is a significant issue for 32-bit applications).

ALLOCATE can allocate arrays that are bigger than the combined physical and virtual space available. When allocated, the allocated array is allocated in the memory tables, referred to as allocated heaps, by Jim. The required memory is not delivered until it is addressed, so trying to use more memory than can be stored will fail when you try to use it, rather when you allocate it. There appear to be significant problems with the way these heaps are managed in the ifort implementation. I have questions outstanding about this, but no useful answers.

At this level, the question could become: If the practicality of sufficient physical memory and pagefile.sys size can be provided, what is the largest single array size that can be used with ifort 64-bit? While this is also limited by the operating system and the linkers operation, I would also like to know what this answer might be. (answers still welcome)

To answer this, I tried to use ALLOCATE to find the largest array I could allocate, by trying different sizes and testing the STAT= response, finding and allocating the largest available array. By repeating this process, say 20 times, I am able to map all large memory chunks that were not used before I started this scanning process. The bits in between identify the areas of memory that are already in use. This approach works well in 32-bit applications.

Unfortunately from my experience, ALLOCATE still has some significant deficiencies in its performance in 64-bit. The response appears to vary as the available memory tables are changed or updated, so the inconsistent results appear to fail my goal. (Id call this an ifort compiler bug!)

The answer I got for the largest array size when testing on my PC was about 36 GB, although there are other examples in this forum claiming to achieve 64GB. (My PC had 12GB of physical memory and an 18GB pagefile.sys file, which is less than the 36gb I allocated). Initially I thought there was a memory limit of 128 GB with ifort on Win-7, but my test did not go anywhere near that limit.

In short, the largest array I can ALLOCATE is bigger than the largest array size I can sensibly use. If you value your time, buy more physical memory and a SSD for pagefile.sys and see what you can achieve. Programming alternative solutions take a lot of time.

After this, the next question you might want to ask is how long will it take to run ?. Perhaps go for a cup of coffee, go home for the weekend, or take a vacation and check in every now and then. Once you resort to a dumb virtual memory solution, you are back to the good old days like in the 70s when programs took a long time to run! Smart out of core solutions reduce both the memory size and number of operations to be performed.

This is another area where theFortran Standard does not address these practical issues.

I hope my recent experience trying to answer these questions has helped.

John

[jimdempseyatthecove]

Execellent response John.

For additional information see:

http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx

Jim Dempsey

[David_Sall]

Thank you very much John for that very informative answer.

Thank you to all of you who responded to my post. I very much appreciate your timely responses!

Sincerely,

David