Memory Fragmentation

[ocean]

Steve,

We have a C++/Fortran application developed for VS2005/IVF 9.1 and running on Windows 2000 that suffers from memory fragmentation, which eventually results in the operating system running out of virtual memory. The main problem is thatour Fortran code has arrays that require allocating huge blocks of memory. I'm taking a look into the possibility of developing a piece of code that would do our own memory management in such a way thata large chunkof memory requested by the ALLOCATE function could be mapped into smaller pieces. I have seen codeover the Internet that would do just that for C++.Do you have any idea ifit is possible to implement such a capability inFortran, and if it is do you know of existing code that I could use?

Thanks,

Manny

[Steven_L_Intel1]

You can do that sort of thing in C/C++ because the memory allocation routines are just that, routines, which can be overridden by the user. In Fortran, the ALLOCATE statement is part of the language and there is no provision for user hooks into the memory allocation.

What you could do is create your own allocator and use POINTER arrays, with your allocator filling in the descriptor as documented in the Building Applications manual.

Easier would be to use the "Integer POINTER extension", not the standard POINTER syntax, and then you can call any memory allocator you want.

I am skeptical that this is really going to help you, though. It sounds to me as if you simply have insufficient virtual memory available for your application and that you shold consider a switch to the 64-bit platform (XP x64 or Vista x64). Do you really know that fragmentation is the problem? .

[jimdempseyatthecove]

What may be happening is

a) allocate large array
b) allocate anything else
c) deallocate large array
d) allocate anything else
e) allocate large array

Then depending on the size of the large array and interviening allocation the large array allocation may be walking up your virtual address space, creating less than large array sized holes (not suitable for subsequent large arrays).

The large array allocation/dealocation location in the application are likely centeralized in a few lines of code. What you could think about doing is to create a large arrayallocation pool. Then two support subroutines AllocateLargeArray and DeallocateLargeArray.

The AllocateLargeArray first checks the pool for suitable array for re-use. If not found it performs an ALLOCATE. if found it returns a descriptor to a section of the array that fulfills the allocation requirements.

The DeallocateLarge array determines if the array being returned was formerly a section of prior allocation, if so, returns prior allocated array to pool, if not returns array to pool. "return" here means place pointer to array (descriptor) into linked list of pointers to arrays in pool.

The AllocateLargeArray could return the best fit or first fit. Also, if allocation fails then you could have code to try to consolidate adjacent arrays, or simply deallocate all pooled arrays and begin again.

This would require using POINTER instead of ALLOCATABLE for the arrays in the application. Those are relatively easy changes. Also, if you want additional control you could create a derived type that contains the pointer to slice, allocatable array, and link pointer when in the pool.

You will also find that in addition to avoiding the memory allocation problem that the program may run faster.

Jim Dempsey

[ocean]

Steve,

Thanks for your input.I agree that the most practical solution is to switch to a 64-bit platform. I'm also taking a look into Windows XP Professional, whichmight give us temporary relieve by using the /3GB setting.

I have been running our application along with a virtual memory analyzer, which provides a real-time picture of the virtual memory address space, and it is clear thatwe are experiencing a certain degree of fragmentation.The tool also shows the locations and sizeof the system's DLLs within thevirtual memory space, and it is also clear that they are producing a great deal of memory fragmentationI'm not exactly sure what kind of scheme the operating system uses tolay down the system's DLLs in memory, but it would be nice if theywere close to each otherto avoid fragmentation.

Thanks,

Manny

[ocean]

Jim,

Thanks for the info. I like the idea and I'll take a look to see how hard it is to implement.

Manny