Allocate failure unless /3gb, even though use < 2gb

[Matt_F_1]

I'm allocating a large array (pointer) of roughly 900 MB. The allocation fails (STATUS=41) unless I have the /3GB flag set in boot.ini (the machine has 9.5 GB RAM). However, with this flag, the total memory usage of the program is only 1.8 GB, below the 2 GB limit (as is the total computer usage). I am using the /align flag.

Is this a memory fragmentation issue? Our old CVF binary runs this problem fine, although it does pagefile on another machine with only 2GB physical memory.

We need this to run without requiring changes to users boot.ini.

Thanks,
Matt

[Steven_L_Intel1]

It could be fragmentation, though at 1.8GB you're running very close to the edge of what is usable. If anything is just a bit bigger, it could be an issue. 2GB is a theoretical limit and also includes the stack, data structures used by Windows, and any DLLs that get loaded - some of which may be "hooked" in at the OS level.

If you increased the stack size, try cutting it back. Link against the static rather than DLL libraries.

[Matt_F_1]

Ah, so the system DLLs would not show in in the memory usage in Task Manager, but would count towards the 2 GB? Is there any way to see the "full size" of an application including everything that counts towards the 2 GB limit?

I've rebuilt everything using static libraries where possible (except one third-party DLL) and reduced the stack to the default, but still no go.

Evidently the older version was really close to the limit, but just squeaked by.


Thanks,
Matt

[Steven_L_Intel1]

If you start the program under the VS debugger, and then do Debug > Windows > Modules, you'll see them all. They will not show in Task Manager, since they are not separate processes. This will also show you the memory ranges for each.

[Matt_F_1]

All DLLs only use about 30MB. However, if I understand Windows VM correctly, I think I've found the issue:

My DLL is loaded at an address of 256 MB (0x1000000), while one of the system DLLs (comctl32.dll) is loaded at 1488 MB (0x5d090000). Doesn't this mean that the largest contiguous block of memory would be at most 1232 MB? Most of the other Windows DLLs in this EXE load in the top 150 MB of memory (0x74320000 - 0x7e410000), but for some reason this errant DLL is loaded much lower.

Thus, once I allocate the one 500 MB array, there would be no contiguous space for the big 900MB one.

Is this a correct interpretation?

If so, I'll try to figure out why I'm loading two versions of comctl32.dll - the newer version loads at a much higher address.


Thanks,
Matt

[Steven_L_Intel1]

I can see that happening as well in some of my applications. One of the copies is from WindowsSystem32, the other is from a side-by-side assembly. It is not clear to me if this happens only when running under the debugger or alwsys. For example, I don't see the second copy if I run Dependency Walker.

However, I think you are on the right track. Isn't it amazing what gets loaded without your asking for it?

[Matt_F_1]

For me Dependency Walker only shows the multiple DLLs via a profile run.

Why does Windows set the preferred load address of so many system libraries to such low values? For example dot3api.dll loads at 1144 MB.

Does this really reduce the available contiguous memory? It seems the only plausible answer I've found so far.

- Matt

[Matt_F_1]

This is indeed what is happening.

Search for "rebase dll" and "virtual memory fragmentation" and you'll find quite a lot.

While "system" DLLs load above 0x70000000, related DLLs (such as OLE) are known to load much lower (0x50000000 or worse). In addition, Visual Studio (and Intel Fortran) default to loading user DLLs at 256MB (0x10000000). Depending on exactly which DLLs are loaded, this leads to roughly 1.2 GB of usable contiguous virtual memory (see, e.g., http://www.ittvis.com/services/techtip.asp?ttid=3346).

Unless one is running a server, I'd suggest using the /3gb boot.ini switch if you have more than 1 GB of RAM.

We'll see what I can do to avoid loading as many of these low-loading libraries as possible. Note that rebasing such system DLLs is not really an option.

I think this topic can be "closed."


- Matt

[Steven_L_Intel1]

Thanks for letting us know what you learned.