Insufficient memory after one million allocations?

[arjen_markus1]

Hello,

I have a problem with the allocation of a large number of items:
The program stops with an error code 41 on the allocation statement
(apparently this means memory is exhausted), but the Task manager
does not indicate that it has reached the limits (still 450 MB of physical
memory available near the end of the program, 560 MB after the program
finished). Even more suspicious is that the runtime error occurs at the
same spot every time I run it (according to the counter:
after 1464915 allocations in the Fortran part - just before posting this
I tried three times with the same result, the third after starting some extra
programs).

Here is a short description of what it is doing:
- The program reads an XML file using a C library and puts
the various elements in a tree structure (implemented in Fortran)
- The file is about 10 MB and what is stored are short strings of
varying length (actually stored as character(len=1) arrays that
are allocated to fit the size of the string that is passed)
- There are a lot of other things going too before the XML file
is read, but that does not seem to cause any problems (at least
the debugger does not complain ;)).

I have written a small program to allocate a large number of
small arrays to see whether the problem is simply the number
of allocations, but that runs fine with 2 or even 4 million items.

Does anyone have suggestions for a solution?

(While I can use a different method to collect the data in this
case, I am really puzzled about the cause!)

Regards,

Arjen

[grg99]

Well, theoretically that shouldnt happen unless you're indexing off the end of one of those strings and bashing memory. Most heap implementations put pointers to the next and previous strings before and after each string, so if you accidentally bash string(0) or string(length+1), that breaks the heap allocator. Try turning on all range checks and look over your code for off-by-one errors.

This doesnt exactly answer your question, but you might find it better and faster to write your own allocator:: In your case, just allocate one big array, maybe 10% bigger than the input file size, and everytime you want to allocate a string, just plop the string into the next free spot in the array and bump the free pointer by the string length. If the strings are shorter than 256 bytes, precede each string body by a character representing the string length. If they're longer, you'll need to plop down two or more bytes for the length.

Or even simpler and faster: read the whole dang file into a big array, parse that, and use indexes into that array as the pointers to the data. Saves a whole lot of allocation and string copying.

[arjen_markus1]

Solutions in that direction are indeed possible (certainly in this case, due to the regular nature of the data), but it is really strange that this is happening. And it will make the library less general.

Regards,

Arjen

[jimdempseyatthecove]

When the insufficient memory for allocation occures does the error message indicate the size of the request? Of particular interest is:

Is the allocation for 0 bytes?

Is the allocation for -n bytes?

Also:

A string in C is typically NULL terminated with the NULL not being counted in the string size. Would your program possibly be allocating to number of bytes not counting the NULL then calling a C routine to perform a STRCPY into the FORTRAN array? If you are doing this then the terminating NULL _may_ clobber heap control structures that follow the allocation.

Jim Dempsey

[grg99]

What I'd do is start stripping out code and see when the problem goes away. For instance, strip out the xml parsing business and just make up strings to allocate. Eventually the problem should go away and that will point you to the culprit.

Also make sure you have ALL checking and debugging options on. it still sounds like an off-by one string overflow error.

Another thing to try: allocate a few more bytes in each array. If the problem goes away, somewhere you're storing past the supposed end of the array.

[arjen_markus1]

Well, the Fortran part uses transfer() to convert the single character string that is
passed via the C side to an array of single characters. A copy is stored in the
tree structure (allocated by Fortran). So, there is no particular reason to suspect a NUL character to come into play.

Anyway, I have solved it by using a completely different approach, using a Fortran library to read the XML file instead and a different data structure that requires much less allocations and deallocations.

Regards,

Arjen