topic Memory consumed by pointers in Intel® Fortran Compiler

Memory consumed by pointers

gregory_p_herrick — Wed, 30 Aug 2006 22:21:40 GMT

I am seeking to employ (as targets) dynamically-allocated (one-dimensional) arrays of a derived data type which itself employs dynamically-allocated (multi-dimensional) component arrays. I would then like to create additional independent multi-dimensional pointers (of identical data type and dimension as the derived data type component arrays) to point to the component arrays of these targeted "primary variables". My motivation is to efficiently store large amounts of data which have one common dimension among them (the one dimension of the "primary variables" declared with the derived data type), while the sizes/extents of the other dimensions could vary greatly, making use of rectangular arrays sized to the largest extents highly inefficient with memory -- so those would be stored in dynamically allocated components of the derived data type "primary variables" (and presumably be successfully deferenced by the additional independent pointers). I seek to use pointers to dereference/extract the data contained within the derived data type for one primary reason: programming simplicity/ code legibiliity. (Because I realize that I am only complicating things further and consuming some additional -- how much? -- memory by using these pointers.)

I am presuming that using the derived data type for my "primary variables" will yield me the efficient memory management I seek; I am concerned about how much additional memory is consumed by the "additional" pointers. I understand that pointers consume no memory until associated; but -- once they are associated -- how much memory do they consume? Do pointers consume additional memory identical to their targets -- effectively doubling memory consumption for one array of actual data? Or -- as I hope -- do they only consume a little bit of additional memory to "map out" the memory of the associated target? I have come across information about "fortan array descriptors" which alludes to pointers and seems to imply that "fortran array descriptors" build a map of the associated target's memory -- consuming very little memory unto itself -- which is exactly the explanation I hope to confirm!

Questions, comments, assistance all appreciated.

EDIT/ADDITION: How much memory "overhead" is involved in array allocation/declaration? Do "Arrays = Pointers" in Fortran like C? Does a pointer declaration/allocation use more, less, or same amount of memory as "typical" array allocation/declaration?

Thanks!
Greg

Re: Memory consumed by pointers

Steven_L_Intel1 — Thu, 31 Aug 2006 03:43:45 GMT

Pointers in Fortran are not quite like pointers in C. For pointers to arrays, the pointer itself is that "descriptor" you read about - some additional storage that records the bounds of the array - but this is typically a few dozen bytes at most per pointer. The array data itself is no different in size than if you had declared it as a non-pointer array. For pointers to scalar objects, then it's just an address just like C.

Let me suggest that rather than use POINTER that you use ALLOCATABLE components in your derived types. These have subtle performance advantages in some cases and some nice semantics when you assign one structure to another.

Re: Memory consumed by pointers

gregory_p_herrick — Thu, 31 Aug 2006 04:14:30 GMT

MADsblionel:

Let me suggest that rather than use POINTER that you use ALLOCATABLE components in your derived types. These have subtle performance advantages in some cases and some nice semantics when you assign one structure to another.

My plan was to use "allocatable" components within the derived data type type-description, and then use "pointer" independent variables to point to those "allocated" components of the derived data type variable. [While "primary variables" would be "allocatable" and of "type (derived_type)" ]

Are we saying/thinking the same thing?

Thank you very much for your help,
Greg

Re: Memory consumed by pointers

Steven_L_Intel1 — Thu, 31 Aug 2006 05:02:54 GMT

Yes, that should work.

Re: Memory consumed by pointers

Steven_L_Intel1 — Thu, 31 Aug 2006 21:21:17 GMT

Note that if you use ALLOCATABLE components, the entire derived type variable must have the TARGET attribuite to allow you to pointer-assign to a component. If you make them POINTERs, then you don't. Perhaps for your purpose POINTER will be fine.

Re: Memory consumed by pointers

gregory_p_herrick — Fri, 01 Sep 2006 07:08:26 GMT

Thank you for your tips. I am implementing them now in my (large) code... then will come the fun of debugging!

Another question on the subject of "memory consumed by pointers

gregory_p_herrick — Wed, 06 Sep 2006 22:58:32 GMT

From Steve's comment above: "For pointers to scalar objects, then it's just an address just like C."

How may bytes does a scalar address consume?

For the simple fortran code below:

INTEGER :: I
INTEGER, POINTER :: I_PTR

I_PTR=>I

I understand that /I/ will consume however many bytes I set for "integers" in my compilation; how many "overhead" bytes will /I_PTR/ consume?

I ask this simple example specifically because it does *not* involve arrays -- where (I think I understand how) the "fortran array descriptors" facilitate the consumption of very little "overhead" memory in mapping large arrays.

Again, my concern is that /I/ and /I_PTR/ will consume "double" the required to memory to store what is actually a single integer datum.

Thanks in advance,
Greg

Re: Another question on the subject of "memory consumed by poin

Steven_L_Intel1 — Wed, 06 Sep 2006 23:14:18 GMT

An address is four bytes on a 32-bit system, 8 bytes on a 64-bit platform. The size of the variable itself does not matter.

Re: Memory consumed by pointers

jhliu — Thu, 07 Sep 2006 04:10:49 GMT

If the derived data structure has a pointer array with the deferred shape, the memory size of a pointer pointing to such derived data can be much more than you expected.

e.g.

Type :: test1
Integer var1
integer, dimension(:), pointer :: ptr
End type

Type (test1), pointer :: test_ptr1

You may use sizeof(test_ptr1) to see the size of this pointer, and compare the one without the pointer array. For my 64-bit machine, the size of pointer test_ptr1 is 80 bytes. comparing to 4 bytes if it is

Type :: test1
Integer var1
End type

Therefore, it is quite expensive to have an array of pointer in a derived data structure in FORTRAN 90. It seems that C does not have such problem.

Junhui

Re: Memory consumed by pointers

Steven_L_Intel1 — Thu, 07 Sep 2006 05:01:15 GMT

Actually, no, the pointer to the derived type object is the same size. I agree that the pointer to allocatable or pointer arrays inside the derived type is larger, but I had already said that.

C "does not have such problem" because C pointers are not Fortran pointers. C does not have the concept of arrays and array bounds and dimensions. All it has is simple address pointers and syntax for indexing into them. You cannot ask the shape of an array in C, for example. Fortran pointers for arrays keep much more information because arrays are a fundamental construct in Fortran whereas they are an afterthought in C.

Re: Memory consumed by pointers

gregory_p_herrick — Thu, 07 Sep 2006 21:33:26 GMT

I guess I should have written up a simple example to show my intentions:
======================
program junk
type :: type_variable
real*8,allocatable :: x(:,:,:)
end type type_variable

type(type_variable),allocatable,target :: var_data(:)
real*8,pointer :: x(:,:,:)

integer :: i

do i=...
x=>var_data(i)%x !using pointer 'x' for coding
!simplicity/legibility when accessing
!'var_data(i)%x'
::::
:::
enddo
end
=======================
I apologize for not writing up this simple example in my first post; I think this simple example should be 1000x clearer than my initial post's two paragraphs of prose.

Note that I am *not* using pointers *within* my derived data type.
I *am* using pointers to point to *allocatable* arrays within said target derived data type.

I am seeking coding simplicity/legibility/brevity by use of the pointer; I am using the derived data type (allocatable -- with allocatable components) for "efficient" allocation of memory.

Will I achieve my goal of "efficient" memory management using a scheme such as I typed above?

Or will pointer x consume nearly as much (additional) memory as the target var_data(i)%x?

Thank you for your responses and guidance; your help is greatly appreciated.

Greg

Re: Memory consumed by pointers

Steven_L_Intel1 — Thu, 07 Sep 2006 21:41:44 GMT

The appearance of an allocatable component (x) in the derived type type_variable means thaty for each element of array var_data, there will be a descriptr ofr component x. The size of that descriptor will be 24 + (12*rank) bytes, or in this case, 60 bytes. It would be the same if component x was a pointer rather than an allocatable.

Pointer x will also be 60 bytes since it is an array pointer of rank 3, not a scalar.

I can't comment on how efficient this is without knowing the typical size of the array components. You're not saving any memory by using the pointer variable x though the code will be more legible, I agree.

Re: Memory consumed by pointers

gregory_p_herrick — Thu, 07 Sep 2006 22:00:47 GMT

MADsblionel:
The appearance of an allocatable component (x) in the derived type type_variable means thaty for each element of array var_data, there will be a descriptr ofr component x. The size of that descriptor will be 24 + (12*rank) bytes, or in this case, 60 bytes. It would be the same if component x was a pointer rather than an allocatable.

Pointer x will also be 60 bytes since it is an array pointer of rank 3, not a scalar.

I can't comment on how efficient this is without knowing the typical size of the array components. You're not saving any memory by using the pointer variable x though the code will be more legible, I agree.

I understand that the memory consumed by the array descriptors will be identical (and counted twice: once for component x, once for pointer x). But -- aside from the two identical array descriptors -- no other memory will be "double bookkept", correct? The 3-dimensions of x are usually sized ~50:50:50. So I am hoping I can have brief, legible code accessing the x data with only 60 bytes (array descriptor) "excess" memory consumed. The 50:50:50 x data should consume 1MB unto itself - - and not twice that (one for component, one for pointer), correct?

Thanks.

Re: Memory consumed by pointers

Steven_L_Intel1 — Thu, 07 Sep 2006 22:30:59 GMT

Correct. The data itself is not duplicated.