Problems with high memory usage

[grtoro]

Hi,

Ihave to increase thedimensions of the arrays and structures used by a program, in order to accommodate a larger problem. I have been gettinglinker warnings such as the following:

"LNK4084: total image size 1983963136 exceeds max (268435456); image may not run"

So far, theprogram was running on XP Pro despite this warning (albeit usingnearly 2GB of memory).

I had to increase the dimensions some more and now the program can't run at all. I am gettinga message that says

"programmyprogram.exe is not a valid Win32 application"

Are there any ways around this problem? Given the above message, I don't think this is a problem I can fix by increasing the paging file size.

I have tried to work around the problem bymaking some arrays allocatable. Unfortunately, some of the largest arrays are in named common blocks. From reading earlier postings, I understand I can use modules instead of common blocks to share allocatable arrays. I am new to allocatables and I have never understood modules. Can anyone provide a simple example of how to share an allocatable array between two subroutines? I would strongly prefernot to pass these arrays/structures as arguments.

Thanks,

Gabriel

[Steven_L_Intel1]

2GB is the upper limit on 32-bit Windows, except under unusual circumstances, and even them I don't think you can get more than 2GB of static code and data.

The linker warning refers to a 256MB limit on Windows 95 and can be ignored.

[grtoro]

Steve,

Thanks for your response. Is the 2GB limit for the combinedstatic + allocatable data? If not, what is the limit on allocatable data?

I am getting a little more space by making some arrays allocatable, butafter increasing sizes a bit moreI get runtime errors 41 or 179.

BTW: I amusing XP Pro and others may use the program in Win2000.

Gabriel

[Steven_L_Intel1]

2GB is the entire process virtiual address space limit, including static and dynamic data.

[John_Campbell]

Steve,

Does ifort support /3gb option ?
This could provide up to an extra 1gb of allocatable memory on XP and Win2000 OS, before requiring a x64 upgrade.

John

[TimP]

At best, the /3GB switch could make some non-contiguous space available for allocatable.  It's difficult to support Windows 2000; years ago Intel instituted a policy of disconnecting any machine detected running it. 

Perhaps you neglected to notice this thread has been inactive for many years and may expose a problem with the web site.

[SergeyKostrov]

>>...Does ifort support /3gb option ? It is a feature of some editions of Windows operating systems and Not All versions and editions support it (!). So, when it comes to user applications it is transparent but access to memory blocks outside of 2GB limit is done through some "window". I've completed some R&D project ( Windows 2003 .NET Server OS was used ) in December 2012 and my conclusion is it is much easier to switch to a regular 64-bit Windows when lots-lots memory is needed instead of implementing that /3GB support for an application ( costs are too high ).

[John_Campbell]

I didn't notice that all the posts listed on Monday were a blast from the past. A new feature of this IDE !

One advantage of support for /3gb during the linking stage is that in Windows 7_64, 32-bit applications get access to about 3.8gb of addressable memory. This can be a significant advance for a 32-bit program, without the need to convert to 64-bit. This allows for the continued use of 32-bit libraries where a 64-bit library might not be available.
Both 32-bit (using /3gb) and 64-bit programs require the use of ALLOCATE to access arrays above 2gb, so there is a similar development requirement to adapt from static to allocatable arrays. By replacing COMMON with a module and declaring arrays ALLOCATABLE can be a relatively easy approach that provides for transfer to 64-bit when the 64-bit library becomes available.

John

[SergeyKostrov]

John, None of Microsoft 64-bit Windows operating systems support that technology and as I've already mentioned I spent some real time on investigation how it works and what it does on Windows 2003 .NET Server. Check MSDN for more technical details.

[John_Campbell]

Sergey,

I am using Windows 7 Enterprise 64 bit OS and Salford 32-bit compiler and getting 3.8gb of addressable memory, so I am not sure what you are referring to. I was also able to access using XP_64 bit OS. I am finding that the 64-bit OS I have used default to a /3gb mode for 32-bit applications, as most of the OS is moved out of the 0-4gb address space.
The other issue in your testing is if the linker is providing access to above 2gb address by making executables "Large address aware".
I can not be certain of the ALLOCATE memory interface being used, but I thought it was based on MALLOC. It might be another

This is a good option where old 32-bit libraries are difficult to access for 64-bit, as old suppliers disappear.

John

[SergeyKostrov]

>>...I am using Windows 7 Enterprise 64 bit OS and Salford 32-bit compiler and getting 3.8gb of addressable memory... We're mixing 32-bit and 64-bit Windows operating systems, John. Actually, a little step is needed ( I mean port to 64-bit ) and application will be able to access tens-etc-of-GBs and 3.8GB is Not on my "table" any longer. I've spent lots of time on R&D related to that functionality using 32-bit (!) Windows Server .NET 2003 and my conclusion is it doesn't make sense to invest time and money on it. The point is, a 32-bit application could be "set" to a "64-bit-ready" state in a matter of weeks or months. As soon as that phase is completed ( let's call it preparations ) an actual port to 64-bit world won't take too long. It took ~2 weeks to complete the port to a 64-bit Windows platform for a middle-size project ( ~100K code lines ). >>The other issue in your testing is if the linker is providing access to above 2gb address by making executables "Large address aware"... That was not an issue and, of course, it was On.

[SergeyKostrov]

>>>>The other issue in your testing is if the linker is providing access to above 2gb address by making executables "Large address >>>>aware"... >> >>That was not an issue and, of course, it was On. Just in case I've verified and here is a makefile I used to compile an executable for tests on 3GB-aware 32-bit Windows Server .NET 2003: #-- Nmake macros for building Windows 32-Bit apps ----------------------------- !include "Win32.Mak" all: $(OUTDIR) $(OUTDIR)\AweTestApp3.exe #-- If OUTDIR does not exist, then create directory --------------------------- $(OUTDIR) : if not exist "$(OUTDIR)/$(NULL)" mkdir $(OUTDIR) $(OUTDIR)\AweTestApp3.obj: AweTestApp3.cpp $(cc) $(cflags) $(cvars) /D_WIN32_PLATFORM /WX /Fo"$(OUTDIR)\\" /Fd"$(OUTDIR)\\" AweTestApp3.cpp $(OUTDIR)\AweTestApp3.exe: $(OUTDIR)\AweTestApp3.obj $(link) $(conflags) /LARGEADDRESSAWARE -out:$(OUTDIR)\AweTestApp3.exe $(OUTDIR)\AweTestApp3.obj $(conlibs) #-- Clean Rule ---------------------------------------------------------------- #-- Rules for cleaning out those old files clean: $(CLEANUP)

[SergeyKostrov]

>>...I am using Windows 7 Enterprise 64 bit OS and Salford 32-bit compiler and getting 3.8gb of addressable memory... John, I'm simply interested if you tried to allocate more than 2GB of memory ( for example, 2.8GB ) for a 32-bit application when /3GB feature is on? Please post results and thanks in advance.

[John_Campbell]

Sergey,

I have successfully run programs where I have allocated 3.7 gb on a 64-bit OS using a 32-bit compiler and produced a 32-bit application.
I again wrote a simple test program yesterday and tested it. This one allocates a 1.8gb array and a 1.0gb array. The challenge is getting sufficiently large contiguous memory for the arrays to be allocated. I probably could have allocated 1.9 and 1.2 gb arrays, although the linker can sometimes mess up free memory between addresses 200mb and 1800mb, putting a few scraps there. Anyway it works and I am attaching the program and a log file. To achieve 3.7gb, you need to allocate 3 or 4 arrays. Using a combination of large common below 2gb and allocate above 2gb can give a more controlled result.

(You will note it uses a non-standard function core4 and dcore8, which writes the memory at that address as a I*4 or R*8. You could use TRANSFER, although I don't know how to do a direct memory address. There was a recent post doing this.)

The big advantage of this is you can still use the 32-bit libraries and access up to 3.8gb, although you need to be smart with the order of memory allocation. I have found that 64-bit OS ( both XP_64 and Win 7) defaults to the /3gb switch on, while with 32-bit OS (I tested XP), it can be a hastle to get both the startup and linker enabled for /3gb.

John

[SergeyKostrov]

Thanks for these technical details. >>...To achieve 3.7gb, you need to allocate 3 or 4 arrays... It is consistent with my results and I could not allocate more than 1GB of memory in a one block and optimal case was 3 by 0.8 GB to get 2.4 GB of memory in total for a 32-bit application. Then, management of these memory blocks during processing is a tricky task and that is why it is hard to integrate the solution based on /3GB with existing complex applications which use large data sets. >>...I have found that 64-bit OS ( both XP_64 and Win 7) defaults to the /3gb switch on... I think this is specific to Windows OS edition and I'll do a quick verification for Windows 7 Professional ( 64-bit ). The situation with Windows XP 32-bit editions is more complex since Home Edition does not support that feature at all.