Continuing problem: error 41 for allocation

[john3]

In a 32 bit PC, IVF 9.1, Win XP, 3 GB memory, vRam 3 GB environment, I've continually had array allocation error 41 which is lack of memory space. The peak working set size at this failure is reported as: PeakWorkingSetSize....(MB): 1548, far less than 2 GB. And the array being allocated was 9 MB.

This occurs on my PC and on my customers machine (similar PC). Very frustrating after a 150 hour job for this to fail at this point.

Why is this? And what might be suggested work arrounds.

Thanks.

[Steven_L_Intel1]

"Why is this?" The OS is returning an error to the run-time library saying that more memory cannot be allocated. I'm not entirely sure that the normal allocation methods work up into the 3GB space, but they might.

By the way, peak working set size is not the same as virtual memory used. Working Set is the amount of physical memory alloted to your process. You should be looking at Peak PageFile Usage instead.

[john3]

As a simple physics guy, this is starring to get over my head. Here is the output at failure (as called from my fatalError routine after the allocation error):

/////////////////////////////////////
Solve_System_Using_ACA : Failed to allocate pSys%vDiagLUblk(iRowBlk)%vZ
iErr = 41

Stopping Program

PageFaultCount............: 143267754
PeakWorkingSetSize....(MB): 1548
WorkingSetSize........(MB): 1527
QuotaPeakPagedPoolUsage...: 25884
QuotaPeakNonPagedPoolUsage: 35160
QuotaNonPagedPoolUsage....: 21920
PagefileUsage.............: 1633566720
PeakPagefileUsage.........: 1654677504

What limits are being overrun? And what can be done about it?

Thanks

John

[Steven_L_Intel1]

I'm guessing that whatever you used to display these numbers reported the PeakPageFileUsage in bytes, as Windows typically reports this in KB. That's about 1.6GB and pretty close to what I consider the usable maximum in 32-bit Windows.

According to the MS documentation, you also have to build the executable with the /LARGEADDRESSAWARE linker option in order to use the expanded memory. You can call the Win32 API routine GlobalMemoryStatus (or GlobalMemoryStatusEx) to get the size of the virtual address space. The MS documentation also warns that there is a gap in the available address space right around 2GB, but I'd thimk this would not matter.

[ion_tiseanu]

Hi!

I found very insightful this discussion.

Indeed boot.ini with /3GB option and link option //LARGEADDRESSAWARE help increasing the size of allocated variables.

Ive got three variables each of about 1 GB. With boot.ini /GB but without /LARGEADDRESSAWARE only two variables of 1GB can be allocated.

Now, I have another problem.

I am building a Fortran DLL that it is called in a console C++ application.

Even if I do link my DLL with /LARGEADDRESSAWARE there is no way to increase the size of allocated variables.

Do I need somehow to link the calling C++ application with a similar /LARGEADDRESSAWARE option?

Thank you very much for any suggestion!

[Steven_L_Intel1]

Yes, the executable must be linked with that option.

[jimdempseyatthecove]

Steve is right, the PeakWorkingSetSize is the peak size of resident RAM occupied by the application. Whereas the PeakPagedMemorySize is the peak size of virtual memory (residentand not resident) allocated to the application.

If you know for a fact that the application accessible data is less than the maximum available to your application then the apparentmemory loss is likely due to data in address space hidden from the application. e.g. each memory allocation has 8, 16, or 32 bytes of overhead. So if your application has many small allocated objects then the overhead can be a significant problem. This can be fixed by allocating pools of these small objects, then maintaining free lists of the pools and objectsin linked lists. In this manner there is only one set of overhead bytes per size of individual pool.

In an application I am writing now, I allocate pools in multiples of 1024 objects. Additional pools are allocated as necessary. In my application I do not return the pools should one become full (i.e. all nodes not used). An additional advantage is that once the working set is attained then there is no (or relatively little) processing overhead to obtain or return a node to the pool (one of several pools).

My application is amulti-threaded application. ALLOCATE and DEALLOCATE are relatively expensive operations (entering and exiting critical sections). Properly implemented pools has virtually eliminated this overhead in the application. Yes, this is more work, but if it can save your application from failing at 150 hours and boost your performance as well, then would it be worth your effort to do the coding? I think yes.

Jim Dempsey