Resize allocated array (F2003)

[davidspurr]

Could someone please clarify re resizing an allocated array.

For reading data from a file, I often need to allocate an array to store the data before I know how many records are in the file. To save reading through the file first to count the number of records (or reading to a large temp file first), it would be handy if the following was possible withou incurring a large overhead.

ALLOCATE (array (large_number)) ! large_number >> likely number of records

n = 0

DO

n = n + 1

READ(...) array(n)

if ( end of read) EXIT

END DO

array = array(:n) ! hopefully 'array' now has dimension (n) - NB: LHS 'array' smaller than RHS 'array'.

Questions:

1. Will this work also if 'array' is a derived type?

2. In the other thread, Steve stated 'To get the automatic reallocation that Tim mentions, you must compile with "-assume realloc_lhs" '. Does this apply for the above case also (/still)?

3. Does resizing as above incur a large overhead ?

Not that (3) above matters in most cases for me as it is generally done only once for a particular file - better to have tidy code.

[Steven_L_Intel1]

In theory, this should work. It will likely incur a large overhead and will almost certainly make a temporary copy of the array in the assignment. It does not matter if the array is derived type.

[davidspurr]

Thanks for the prompt reply.

Do I need to compile with "-assume realloc_lhs" ?

[davidspurr]

re "-assume realloc_lhs"

Is this option available in v 11.0.061 ?

If so, where is it applied (via UI)?

I tried adding it under "Command Line / Additional Options" but just got compiler messages as follows:

1>ifort: command line warning #10155: ignoring option '/assume'; argument required
1>ifort: command line warning #10161: unrecognized source type 'realloc_lhs'; object file assumed
1>ifort: warning #10146: no action performed for specified object file(s)

(have to confess my compiler is not up to date - started only updating when need arises)

[TimP]

-assume requires the colon ':' for Windows in place of the linux space ' '.

If you want to use many f2003 features, you should be updating your compiler frequently.

[jimdempseyatthecove]

Why not code it yourself, and eliminate the possibility of a temporary array

[bash]ALLOCATE (ReadInArray (large_number))      ! large_number >> likely number of records
n = 0
DO
  n = n + 1
  READ(...) ReadInArray(n)
  if ( end of read) EXIT
END DO
n=n-1
if(n == 0) goto NoData
ALLOCATE(array(n))
array = ReadInArray(:n)
DEALLOCATE(ReadInArray)
[/bash]

Jim Dempsey

[davidspurr]

Thanks Jim

That is how I have always done it.

Currently writing a new (relatively small) program & though it would look tidier code the "new" way.

Do not see that the temp array situation is much different either way - the old way there is a small overlap where two arrays are required.

I just tried it with Tim's correction (':' instead of ' ') & the resize works perfectly.

(knew there was a reason to stick the the Windows forum!)

[davidspurr]

Hmm - worked for a small test case (array size 5000 reduced to 41), but seems to crash when I try it with a 'production' size array; ie. 600,000 down to c.120,000 :(.

In real use I would do it 47 times, all 600,000 max, with reduced size varying from c. 100,000 to 500,000.

Is this a heap (stack?) issue or similar?

[davidspurr]

Answering my own question - yes.

I just set HEAP & STACK reserve sizes to 16,000,000.

Perhaps when I get time, I will test which one is needed - and how big is needed.

For now, it just works.

[davidspurr]

Continuing the monologue !

Answering my own question - yes.

I just set HEAP & STACK reserve sizes to 16,000,000.

Perhaps when I get time, I will test which one is needed - and how big is needed.

For now, it just works.

As noted, it works, but I now find, only in Debug mode. In Release mode, it crashes still, even after raising Heap & Stack reserve to 33,554,432.

Any suggestions?

[Steven_L_Intel1]

Set "Heap Arrays" to 0 under Fortran > Optimization. You are getting temporary arrays created and copied.

[jimdempseyatthecove]

>>Do not see that the temp array situation is much different either way - the old way there is a small overlap where two arrays are required.

In the sample code presented you overallocate temp array(s) read into the temporary array(s), then allocate target array, copy temp to target, deallocate temp (2 allocations and one copy)

In the (broken) auto-reallocate lhs, you over allocate target array, use auto-reallocate lhs which creates a temp, copies to temp, then deallocates lhs, then allocates lhs, then copies temp to lhs then deallocates temp. (potentially 3 allocations and 2 copies).

All this hoop jumping to "tidy" the code. In addition to unnecessary overhead it tends to fragment the heap.

Jim Dempsey

[davidspurr]

Quoting jimdempseyatthecove

>>Do not see that the temp array situation is much different either way - the old way there is a small overlap where two arrays are required.

In the sample code presented you overallocate temp array(s) read into the temporary array(s), then allocate target array, copy temp to target, deallocate temp (2 allocations and one copy)

In the (broken) auto-reallocate lhs, you over allocate target array, use auto-reallocate lhs which creates a temp, copies to temp, then deallocates lhs, then allocates lhs, then copies temp to lhs then deallocates temp. (potentially 3 allocations and 2 copies).

All this hoop jumping to "tidy" the code. In addition to unnecessary overhead it tends to fragment the heap.

Jim Dempsey

Steve, thanks, setting "Heap Arrays" to 0 did the trick. Works!

Jim, much appreciate your comments & assistance. Yes the 'old' way may involve less copying (esp). I will accept your comments re the heap fragmentation - that is beyond my expertise.

It is not critical for this particular application (I think ...!). It is a small program to consolidate & manipulate data for input to a simulation program. Though it involves a large amount of data (total ~8m records read in, combined with other data from serval sources, then output reformated), the computational effort is relatively minor. Total runtime is only ~3mins for the 8m+ records (~2GB output file).

Partly it was a matter of experimenting / gaining experience with f2003 constructs, and partly to make the code 'tidier', given that it was not a 'time critical' situation.

With the modification that Steve suggested it is now working fine (Debug & Release modes). Given your comments I guess I will avoid it in more critical applications.

My only 'concern' is your comment re heap fragmentation. Does that last only within the current application (not an issue, because the program works well), or is this a more permanent impact; ie. remains fragmented after the analysis has completed & hence will impact following programs until the PC has been re-booted

Sorry, showing my ignorance of what exactly heap fragmentation is. I assumed the heap is there only as long as the application is running & heap fragmentation hence has no 'permanent' effect.

[jimdempseyatthecove]

>>My only 'concern' is your comment re heap fragmentation. Does that last only within the current application (not an issue, because the program works well), or is this a more permanent impact; ie. remains fragmented after the analysis has completed & hence will impact following programs until the PC has been re-booted

Yes it is only within the current application. Fragmentation is a concern usually when you have a large number of allocations and deallocations going on within a program. But sometimes this will occure with a few very large allocations.

From the description of your program this should not be a problem.

Another side effect of auto-allocate lhs is you do not get an opportunity to programmatically work through a low memory condition. IOW you crash instead of dying gracefully or attempting to recover by returning other memory that you have held on to for performance reasons.

Jim Dempsey

[davidspurr]

Thanks for the clarification Jim.