Changing an array size without losing data

[alexismor]

Hello,

Is there any way to increase the size of an array without losing all the data that it currently holds? For example, I need to make a list of non-zero matrix elements in a huge matrix. The matrix is too big to fit into memory, and so I dynamically add the non-zero elements into the list as they are calculated. The problem is that I don't know how many of these elements there are going to be, and so I don't know how big to make my original list. Currently, the only way that I know how to increase the list size is to:

1)copy the list to a temporary one,
2)deallocate(list)
3)allocate(list(newsize))
4)list(1:newsize-1)=templist
5)list(newsize)=new_element

Is it possible to do this without copying data back and forth?

Thanks,

Alexis

[Steven_L_Intel1]

Not at present. Fortran 2003 defines a new intrinsic MOVE_ALLOC that "moves the allocation" of one allocatable variable to another. So with that your steps would be:

1. Allocate a new temporary array of the desired new size
2. Copy the old data
3. Call MOVE_ALLOC to transfer the allocation to the original array. (The temp array now becomes undefined and storage previously allocated to the original array is deallocated.)

We don't support MOVE_ALLOC - yet. (Checks watch...)

[nijhuis]

Hello Alexis,

It is still possible to save one copy, which can be interesting for large amounts of data. This can be achieved by allocating a new array of the new size, copy the old array to it, deallocate the old, assign the old array pointer to the new array and add the new element.

Because a lot of operations are to be performed for each addition, you can choose to allocate a larger part to the array and fill the new allocated part before allocating a new part.

I have added an example of the way I use it.

In the example a real array is used, but of course that can be replaced by other types.

Guus

[Jugoslav_Dujic]

We don't support MOVE_ALLOC - yet. (Checks watch...)

But I do :-).

Attached is sample workspace (tested on IVF8.1 and CVF6.6) which implements part of F2k MOVE_ALLOC. It supports only 1-d arrays with lower bound of 1. Note that it's quite a hack obtained by lots of reverse-engineering VF code -- that should be implemented in compiler, not in user-level code.

It took me half an hour to write it (and an hour to (incompletely) figure out arguments of VF RTL routines :-( ). So, can we expect it to appear in the next release? ;;-)

Jugoslav

Message Edited by JugoslavDujic on 08-18-2005 08:46 AM

program Test

use F2kFeatures

integer, allocatable::  ia(:)

type tfoo
   integer ia
   real x
   character s
end type tfoo
type(tfoo), allocatable:: foo(:)

allocate(ia(26))
ia = (/(i, i=1,26)/)
call move_alloc(ia, 40)
write(*,*) ia
write(*,*)
deallocate(ia)

allocate(foo(26))
foo(:)%ia = (/(i, i=1,26)/)
foo(:)%x = (/(i, i=1,26)/)
foo(:)%s = (/(char(i), i=64+1,64+26)/)
call move_alloc(foo, 40)
write(*,*) foo(:)%ia
write(*,*)
write(*,*) foo(:)%x
write(*,*)
write(*,*) foo(:)%s
deallocate(foo)

end program Test
!=====================================
module F2kFeatures

!Provide interface to external move_alloc

interface
   subroutine Move_Alloc(alloc, newsize)
   !DEC$ATTRIBUTES NO_ARG_CHECK:: alloc
   integer, allocatable, intent(inout):: alloc(:)
   integer, intent(in)::                 newsize
   end subroutine
end interface

end module F2kFeatures
!=======================================================================
subroutine Move_Alloc(oldDesc, newsize)

!Declarations of VF allocatable array descriptors
type VF_DIMENSION 
  integer(4)   iExtent
  integer(4)   iDistance
  integer(4)   iLowerBound
end type VF_DIMENSION 

type VF_1D_DESCRIPTIOR
   integer(int_ptr_kind()) lpAddress
   integer(4)   nSize
   integer(4)   iOffset
   integer(4)   iAllocated
   integer(4)   iRank
   !DEC$IF DEFINED(__INTEL_COMPILER)
   integer(4)   iWhatever
   !DEC$ENDIF
   type(VF_DIMENSION) Dim
end type VF_1D_DESCRIPTIOR

type VF_2D_DESCRIPTOR 
   integer(int_ptr_kind())  lpAddress
   integer(4)   nSize
   integer(4)   iOffset
   integer(4)   iAllocated
   integer(4)   iRank
   !DEC$IF DEFINED(__INTEL_COMPILER)
   integer(4)   iWhatever
   !DEC$ENDIF
   type(VF_DIMENSION) Dim1
   type(VF_DIMENSION) Dim2
end type VF_2D_DESCRIPTOR 

type(VF_1D_DESCRIPTIOR):: oldDesc
integer, intent(in)::      newsize

!Prototype of VF RTL routine which does the allocation
!(called by ALLOCATE). Obtained by reverse engineering :-)
interface
   subroutine for_alloc_allocatable(nBytes,pDescript,iwhatever)
   !DEC$ATTRIBUTES C, ALIAS: "_for_alloc_allocatable":: for_alloc_allocatable
   integer::                 nBytes
   integer(int_ptr_kind()):: pDescript
   integer::                 iwhatever
   end subroutine
end interface

!Standard C RTL routine
interface
   integer function memcpy(lpDest,lpSrc,count)
   !DEC$ATTRIBUTES C, ALIAS: "_memcpy":: memcpy
 
  integer(int_ptr_kind()):: lpDest
   integer(int_ptr_kind()):: lpSrc
   integer::                 count
   end function memcpy
end interface

!Prototype of VF RTL routine which does the deallocation
!(called by DEALLOCATE). Obtained by reverse engineering :-)
interface
   subroutine for_dealloc_allocatable(pAddress,iwhatever)
   !DEC$ATTRIBUTES C, ALIAS: "_for_dealloc_allocatable":: for_dealloc_allocatable
   integer(int_ptr_kind()):: pAddress
   integer::                 iwhatever
   end subroutine
end interface

!Local variable playing a new allocatable array
type(VF_1D_DESCRIPTIOR):: newDesc

newDesc = oldDesc
!Set the variable to deallocated
newDesc%iAllocated = 0
newDesc%lpAddress = 0
!New allocation size; assume that LowerBound = 1
newDesc%Dim%iExtent = newsize
nBytes = newsize * oldDesc%nSize

!Allocate new array
call for_alloc_allocatable(nBytes, loc(newDesc), 0)
!Copy the contents of old array
call memcpy(newDesc%lpAddress, oldDesc%lpAddress, newDesc%nSize * oldDesc%Dim%iExtent)
!Deallocate old array
call for_dealloc_allocatable(oldDesc%lpAddress, 0)

!Copy the new descriptor into old one:
oldDesc = newDesc

end subroutine Move_Alloc

Message Edited by JugoslavDujic on 08-18-2005 08:50 AM

[edmund_dunlop]

Alexis,

The attached sample code from Numerical Recipes (with a very simple main program) may be of interest. It works fine with IVF.

Best wishes,

Edmund.

[alexismor]

Hi Everyone,

Big thanks for all the suggestions and code samples! They have all helped me a lot.

Cheers,

Alexis

[Intel_C_Intel]

Hello,

Actually, I couldn't download attachments to see different ideas. What if use structure of list like this:

type List
integer value
pointer List => null()
end type List

We have some good perfomance (if we will keep not only adress of "head" but the "end" too). But memory usage increasing because of pointer in each node.

Stanislav