Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.
27086 Discussions

Runtime error 41, insufficient virtual memory

slaby__john
Beginner
684 Views

I am trying to allocate a large array with the following:

REAL*8, ALLOCATABLE :: work1(:)

allocate ( work1(900*202*400) )

Which leads to a virtual memory error when the ALLOCATE command is executed. This array is ~550MB, well within the 2GB limit of Win32. I've increased my page size to 16GB/32GB to no effect. But the problem seems to be related to having a single, large array because the following works fine:

REAL*8, ALLOCATABLE :: work1(:)

REAL*8, ALLOCATABLE :: work2(:)

allocate ( work1(900*202*300) )

allocate ( work1(900*202*300) )

This is Intel Fortran 19.0, Win32 in Visual Studio 2019. I'm executing this function through a VBA call from Excel, but I have seem similar behavior when running the DLL under other interfaces. I'm not seeing large memory usage in the Task Manager (at the time of the error memory usage by Excel is only 50MB). Is there some limit to a single array size?

0 Kudos
10 Replies
Steve_Lionel
Black Belt Retired Employee
672 Views

No, but there may not be that much contiguous memory available. That error is given only when Windows returns an error from the VirtualAlloc function. 

 
slaby__john
Beginner
603 Views

Steve, thanks for that information. What can be done about this? I've increased the page size to 16/32GB. The Task Manager indicates only ~30% of the 8GB is being utilized. I did a search on the VirtualAlloc function and tried using gpedit.msc to lock my user pages in memory, but that didn't help.

jimdempseyatthecove
Black Belt
658 Views

See what happens with a "Hello World" making those allocations.

If that fails, then there is likely a Windows limitation set somewhere,

Or perhaps you specified a humongous stack size, not leaving enough for your heap.

If the test succeeds, then something else has eaten the memory. Try placing these allocations early in the program. Note, if this is a result of fragmentation and ab-ends late in the program, place the offending arrays in a module (such that they are persistent), and allocate them once at program startup, Something else may fail, but not this allocation.

*** avoid using

work1 = (some array expression)

use

work1(:) = (some array expression) ! avoid realloc_lhs

or

do i=1,size(work1)
  work1(i) = ... ! avoid state temporary

or

   migrate to x64

Jim Dempsey

Jim Dempsey

slaby__john
Beginner
597 Views

Jim, thank you for these suggestions. I've stripped down my function to just the declaration and allocation:

SUBROUTINE DOG_XL ( IERR )
!DEC$ ATTRIBUTES DLLEXPORT, STDCALL, REFERENCE,  ALIAS:"DOG_XL" :: DOG_XL
IMPLICIT NONE
INTEGER*4 IERR
REAL*8, ALLOCATABLE :: WORK1(:)
ALLOCATE( WORK1(900*202*400) )
WORK1(:) = 1.0D0
RETURN
END

It is crashing on the ALLOCATE statement. Again, this is a DLL and I'm calling this function from VBA in Excel. Is there some setting in the DLL I'm missing? I've tried setting heap-arrays to zero and that didn't help.

jimdempseyatthecove
Black Belt
548 Views

Allocation failure indicates the heap does not have (or can obtain through expansion) a node of sufficient size to handle the request.

The other type of failure (which your described situation is not), is where the allocation succeeds (sufficient virtual address space), but where upon later first-touch, there was insufficient page file space (or process size restriction imposed by O/S) and then you observe a non-recoverable page fault.

In the code you provided, WORK1 is allocated at start of subroutine, then implicitly deallocated on return.

Should the first call to DOG_XL succeed (no indication from you if it fails on the first call or some call later), then after return, should VBA perform an allocation (VBA is very aggressive on allocations), which happens to split the returned node from prior call to DOG_XL, then the prior call's node is no longer available for the next call to DOG_XL. This is called heap fragmentation. And eventually you run out of heap.

Should the first call to DOG_XL succeed, then consider the following change:

SUBROUTINE DOG_XL ( IERR )
!DEC$ ATTRIBUTES DLLEXPORT, STDCALL, REFERENCE,  ALIAS:"DOG_XL" :: DOG_XL
IMPLICIT NONE
INTEGER*4 IERR
REAL*8, ALLOCATABLE, SAVE :: WORK1(:)
IF(.NOT. ALLOCATED(WORK1)) ALLOCATE( WORK1(900*202*400) )
WORK1(:) = 1.0D0
RETURN
END

 

However, that may not be sufficient.

Please read: Answers to a customer’s questions about memory and DLLs

What this seems to indicate is that because VBA is using COM is that you should be using the COM allocator. And, I suggest augmenting this by keeping the allocation persistent.

There are two ways to do this:

1) Make the allocation in VBA (once) and pass the base address into your Fortran DLL, or,
2) Have the Fortran call (once) a VBA function to allocate a blob of data which the Fortran program can then use C_F_POINTER to construct a SAVE pointer to the array.

If you intend to call DOG_XL more than once, keep the allocation around.

Jim Dempsey

 

slaby__john
Beginner
490 Views

Jim, thanks for this information.

The error is occurring on the first call, so it seems to not be a question of deallocation or fragmentation.

Further, and rather oddly, if instead of dynamically allocating memory I simply declare a large array:

REAL*8 WORK3(900*202*400)

Then I get run-time error 7 in the VBA that the DLL is not found. Though when I applied this test to our custom COM interface it worked. So this particular failure of a large, static array may be peculiar to VBA.

I'm not sure the article you provide is the case here as I am not trying to share the allocated memory across boundaries nor allocate in one and free in the other. It is local to the fortran dll only. Is there any way in which I can declare more heap space in the fortran dll? Using the /heap-arrays or /heap didn't seem to help.

 

Steve_Lionel
Black Belt Retired Employee
481 Views

Declaring it as a static array pushes you against the 2GB user static address space in Windows (even 64-bit). The symptoms of this can vary. VBA is brain-dead in reporting "DLL not found" if ANYTHING goes wrong while loading the DLL.

jimdempseyatthecove
Black Belt
444 Views

>>The error is occurring on the first call, ...

Then this would indicate that the COM (VBA) code has established a (COM) heap of a size that leaves too little room for the additional C Runtime Library heap to use.

Have you experimented with either allocating the array in VBA (and passing the array reference) .OR. having the Fortran DLL call VBA to allocate a persistant blob of data (and return the reference back to the Fortran code). In both cases, you keep the allocation around for the life of the program.

Jim Dempsey

Steve_Lionel
Black Belt Retired Employee
438 Views

A minor point, but on Windows, Intel Fortran uses the Windows API VirtualAlloc and not C malloc for ALLOCATE. That it works when not using VBA suggests to me, as it does to Jim, that there are other allocations not leaving enough contiguous memory.

 
IanH
Black Belt
460 Views

The VMMap process utility, part of the Microsoft sysinternals tool suite, may provide some insight (or feelings of terror) into the degree of fragmentation of the address space of the Excel (or whatever) process that is hosting your DLL.

Reply