Huge heap needed

[Bruce_Weaver]

Hi,

I need to keep very large arrays in memory.  The computer has 132 GB & I need to keep a 10 -20 GB array in memory as it is in constant use.  I think the array should be put on the heap as I know the stack is limited.  What is the heap limit & do I use heap-arrays for this?

thanks

[jimdempseyatthecove]

add to 3) , the previous OpenMP thread pool is not disbanded in the event that the prior thread makes a subsequent call.

Jim Dempsey

[Bruce_Weaver]

Hi Jim,

1) As far as I'm concerned, one of the major advantages of Fortran is that it keeps the pointers hidden from me.  I've seen way too many good programmers go temporarily nutso trying to sort pointers from the things they point at which are a pointer ... in languages like Aion of C.  I don't use'm.  I am big on the bug idea since, for example, I've had Fortran/linker complain about not having enough paging space when it didn't need any 'cause there was plenty of memory. 

2) I don't de-allocate as almost all my data structures were created by previous programs so the main program doesn't have to do any thing but read'em in.  There are one large one that is created in the main program but it is not de-allocated 'cause it is constantly used.

3) Don't mix C & Fortran.  I'm not fond of assembly languages & I don't think C is far enough removed.  When I (rarely) have to write C, I keep it to itself.  The code fails in heavily multi-threaded mode but my threading is quite simple so as to be as efficient as possible.  All the threads run continuously, doing pretty much the same thing except when the master blocks for writing results.  So once I start the threads, they don't stop until I control-C outa the program.  Hmmm...I wonder if that confuses it?

4) The first two are high on my list of suspects.  I was wondering if anyone else has had similar problems.

5) NO.  I use an m.2, which is fast enough for my disk needs, which is only to write out results as I go along.  I keep all the inputs (big tables) in memory....or, at least, I try to.  I can't possibly go to disk for the large tables; the program already runs for hours or days.  Ram monitors seem to indicate that all is well in terms of fraction of RAM being used.

6) I'm not sure what a Fixed Page Pool is.  Can I run out of memory in it (whatever it is) while there is still much memory unused in the machine?

Cheers,

[jimdempseyatthecove]

Bruce,

I agree with all points but 3): The code fails in heavily multi-threaded mode but my threading is quite simple so as to be as efficient as possible

Are you compiling with /recursive or /openmp or are all your subroutines and functions declared as recursive?

If not, then any of the procedure local routine arrays (and user defined types) may default to SAVE (and thus be not thread-safe).

An additional possibility are array temporaries that are created on heap. Array temporaries are often created with expressions that are composed of arrays as opposed to being explicitly iterated:

arrayR = sqrt(arrayX**2+arrayY**4+arrayZ**2)

While the compiler could generate code for this that does not include temporary array(s), it has been experienced at times that they are used. Try adding: /check:arg_temp_created and  see if any very large array temporaries are created. Note, this message only appears once per statement that creates the temporary (not on subsequent creations for that statement).

Jim Dempsey