Does (simple) array slicing produce a copy or a pointer to the original array?

[Karanta__Antti]

I am trying to turn on interface checking in an old codebase. This requires quite a few changes, as many things that work and have (unfortunately) been used quite extensively are not according to the Fortran standard and cause compilation errors when using /gen-interfaces:nosource

Most are fairly straight forward to fix.

The case I am wondering about is passing a "subarray" as a parameter. As an example

! assume this subroutine is not in a module and the caller does not have an explicit interface
subroutine assign_zeros(a, n)
  integer a(*), n
  integer i

  do i=1,n
    a(i)=0
  end do
end subroutine

somewhere else:

integer x(10)

...

! are these equivalent?
! assign zeros to array x from index 5 to 10
call assign_zeros( x(5), 6 ) ! this way of passing a portion of an array as an array is common in the code base. Works fine as x(5) passes pointer to the fifth element in the array
call assign_zeros( x(5:), 6 )

To make the compilation pass when using interface checks, I thought I'd just change from the first form to the latter. But getting an access violation in a given spot in the code when simply adding the : seems to indicate that x(5:) actually creates a copy, where I thought it would just pass a pointer to the fifth element in the array. Is this what happens? Can I affect it somehow so no copy is made?
I do understand that e.g. in some more complex uses of slices the compiler must make a copy (e.g. 5:10:2), but for a simple case like x(5:) for a simple contiguous array x I thought this would be equivalent to doing something like this is C:

// C code
int x[10];
assign_zeros(x + 4, 6); // this would be equivalent to the above fortran calls

If Fortran semantics for slices actually do always make a copy, is there some other way I could obtain a "subarray" in the way that I need?
Changing all the routines so that instead of an array they take an array and a starting subscript is not possible, as it would require huge amount of work.

The change that caused an access violation is from this:

CALL DM_ARRAYCOPY32( II( IR + PROCESSED_WORDS ), TDBB( ISTT ), 1 )

To this

CALL DM_ARRAYCOPY32( II( IR + PROCESSED_WORDS: ), TDBB( ISTT ), 1 )

II having been defined as

INTEGER, PARAMETER :: DMSIZE = 2
INTEGER II
COMMON II(DMSIZE)

There is out of bounds access here, as II is used as a sort of base from where to point to memory of a given address. But that has been used for a long time and should not be the issue here, unless slicing somehow reacts badly to taking a slice that is out of the array bounds?

The called routine is simply

SUBROUTINE DM_ARRAYCOPY32( I1, I2, NW )
  IMPLICIT NONE 
  INTEGER, INTENT(IN) :: NW
  INTEGER*4, INTENT(IN) :: I1(*)
  INTEGER*4, INTENT(OUT) :: I2(*)

  INTEGER I
  DO I = 1, NW
    I2( I ) = I1( I )
  END DO
END

NB: there is also a datatype mismatch here, but that is intentional in the calling code and from the point of interface checking a separate solvable issue.

Environment: windows 10 x64, ifort 19.0.5.281 Build 20190815

[Karanta__Antti]

A simple experiment shows that the slice actually creates a copy, at least in this case

INTEGERADDR1, ADDR2
ADDR1 = LOC( II(IR + PROCESSED_WORDS) ) 
ADDR2 = LOC( II(IR + PROCESSED_WORDS:) )

Visual Studio debugger:

	Name	Value	Type
	ADDR1	930586288	INTEGER(8)
	ADDR2	140738418941616	INTEGER(8)

[Arjen_Markus]

I do not know all the ins and outs of argument passing, but I suspect that the form "call assign( x(5:), 6)" causes the compiler to pass an array descriptor rather than a plain address. What you should do instead is:

subroutine assign( x, n ) 
    integer, dimension(:) :: x
    integer               :: n
    ...

to indicate that you want to work with arrays in the Fortran 90 sense.

That, of course, has to be acompanied by several other changes: the interface must now be made explicit - I recommend putting the routine in a module and using that module.

[Karanta__Antti]

Thanks Markus for your idea.

Unfortunately we are talking of thousands of source files and some millions of lines of code. The kind of modification you are suggesting is quite a bit bigger than just modifying the problematic calls. But if that is the only option I have to consider that or just give up on turning on interface checking. Which would be a pity, as having compiler check the calls versus the interface definitions would help hugely in quality assurance.

As for the compiler passing a descriptor: my understanding is that if the explicit interface of the called function is not known, no descriptors are passed, as those would not work correctly with e.g. old style fortran routines (like the samples I posted). I am pretty sure a pointer is passed.

[Arjen_Markus]

You are probably correct about no array descriptor being passed and indeed it is a pity that the task is gargantuan. Still, I am a bit puzzled about the crash you reported when using "5:". Well, I guess others will be able to provide more and better suggestions.

[Steve_Lionel]

An array descriptor is passed only when there is a visible explicit interface specifying that the dummy argument is assumed-shape. The usage in the example does not change how the array is passed and does not make a copy (at least when I try it.)

If the array section is not contiguous or uses a "vector subscript" (an array of elements), then the compiler will pass a copy. That's not the case here.

Generated interface checking, which would cause a complaint about mismatches, isn't supposed to change the generated code. I know that some times in the past it did, but those were bugs. I have not heard of recent versions doing this.

I would like to see a complete example demonstrating the problem.

[Karanta__Antti]

Hi Steve! Thanks for the reply!

The interface checking likely has no effect on how the array is handled. I just brought it up to explain the background on what I am trying to do and why.

There is a clear difference in what the loc intrinsic gives as I showed from my debugging session. 

Here's a simplified standalone sample that produces the same effect:

! ifort /4I8 /nologo arraytest.f90
program Arraytest

    implicit none

      INTEGER II

      COMMON II(2)
      
      integer indx

    indx = 2305825417532254121

    call tell_address( ii(indx) )
    call tell_address( ii(indx:) )

  write (*,*) loc(ii(indx))
  write (*,*) loc(ii(indx:))


end program

subroutine tell_address( p )
  implicit none
  integer p(*)
  
  write (*,*) loc(p)
end subroutine
  

Here's what happens when its run

C:\work>ifort /4I8 /nologo arraytest.f90  && arraytest.exe
          -32875742496
       140704612612832
          -32875742496
       140704612612832

For a smaller value of indx (e.g. 10000) the printed values are the same.

The value I used for indx in the sample is from an actual run. As I told, II is used as a sort of reference point to point to certain memory locations allocated from the heap by using malloc & such in C (i.e. they are from the heap).

NB: I'll be on xmas vacation till early January. So unfortunately I likely will not be responding again before that. Have a great Christmas everyone!

[jimdempseyatthecove]

There are problems with your test code:

Subroutine tell_address specifies the dummy argument p as an integer array of assumed size.
call tell_address(ii(indx)) is passing a scalar as opposed to an array

call tell_address(ii(indx:)) is the same as using ii(2305825417532254121:2)
Two issues with this:
2305825417532254121 is larger than huge(integer)
The truncated value is likely larger than 2. As an example, assume it is 42.
ii(42:2) is an array of decreasing indexes, and thus constructs a temporary with the cell orders reversed and indexed as temp(1:41).

Jim Dempsey

[Steve_Lionel]

/4I8 tells the compiler to use 64-bit integers. The preferred spelling is /integer-size:64. With that, and a 64-bit build, I get:

-35342500160
-35342500160
-35342500160
-35342500160

Why is that? It's because your value of indx is SO large that ii(indx) is outside of 64-bit address space!

But this also shows that no copy is being made.

[Karanta__Antti]

Thanks for your answers Jim and Steve!

"Subroutine tell_address specifies the dummy argument p as an integer array of assumed size.
call tell_address(ii(indx)) is passing a scalar as opposed to an array"

When not using explicit interfaces, using ii(indx) as a parameter passes a pointer to the indx:th element of ii, which is effectively what passing an assumed size array does. I do understand that this does not adhere to the standard, but as this is how several compilers that have been used w/ the code base I am working on have worked, it has ended up being used quite extensively. Like I stated previously, I am now cleaning up the code base in order to be able to turn the interface checks on.

"2305825417532254121 is larger than huge(integer)"

As Steve pointed out, the value I used is not too large for a 64 bit integer. However, it may end up being outside the address space in this particular tiny example, as I used a value I obtained from debugger investigating our actual software.

"The truncated value is likely larger than 2. As an example, assume it is 42.
ii(42:2) is an array of decreasing indexes, and thus constructs a temporary with the cell orders reversed and indexed as temp(1:41)."

For a moderate value of indx the values printed out are the same. I tried w/ 10000 and now also w/ 42. Which ofc is an answer to much larger topics than discussed here.

So, if it were the case thata temporary array was constructed w/ the cell order reversed, I would expect the behavior to be the same for all values greater than the array size. Which it is not (see example in the code sample in this post).

Could it be that then using the slice syntax, the indexes given are treated as 32 bit integers even if given as 64 bit integers? That might explain the difference in behavior.

Steve:

"Why is that? It's because your value of indx is SO large that ii(indx) is outside of 64-bit address space!"

My problem stems from that ii(indx) and ii(indx:) evaluate differently although I would have thought when used as a parameter in a routine call where no explicit interface is known, they would end up passing the pointer to the indx:th element of ii. Which they do not, as shown by printing out loc(ii(indx)) and loc(ii(indx:)).

Here is a code sample closer to what is actually happening in the code base, i.e. a common array used as a base to reference arbitrary memory:

! ifort /integer-size:64 /nologo arraytest2.f90
program Arraytest

    implicit none

      INTEGER II

      COMMON II(2)
      
      integer :: offset
      integer, dimension(:), allocatable :: arr

    allocate(arr(10))

    arr(:) = 12345
    arr(5) = 42

    offset = (loc(arr) - loc(ii)) / 8 ! the integers being eight bytes
    ! elements in arr can now be referenced by using ii as a base
    ! arr( N ) == ii( offset + N )
    write (*,*) "the offset from beginning of ii to beginning of arr"
    write (*,*) offset
    write (*,*) "the location of the first element in arr using both arr and ii"
    write (*,*) loc(arr(1))
    write (*,*) loc(ii(offset + 1))

  write (*,*) "the value of the fifth element in arr referenced via arr and ii (should be 42)"
  write (*,*) arr(5)
  write (*,*) ii(offset + 5)


  write (*,*) "the location of the fifth element in arr referenced using ii as the base. Would expect these to be the same, but the one using slice syntax is somewhere off"
  write (*,*) loc(ii(offset + 5))
  write (*,*) loc(ii(offset + 5:))
  
  write (*,*) "however, this is not so when referencing via arr"
  write (*,*) loc(arr(5))
  write (*,*) loc(arr(5:))
  write (*,*) "even when going over the actual bounds (by a moderate amount) using plain index and slice syntax result to the same memory location"
  write (*,*) loc(arr(size(arr) + 100))
  write (*,*) loc(arr(size(arr) + 100:))
  

  write (*,*) "This is a problem because in the code base I am working on, which is rather large and has a history spanning around 40 years, "
  write (*,*) "the fact that when given as parameter to a routine w/ no explicit interface, arr(n) and arr(n:) both pass the pointer to the n:th element in arr is extensively used"
  write (*,*) "and taking explicit interface checks to use requires change to the slice syntax as naturally passing a scalar is incorrect where an array is expected" 
  write (*,*) "(even though it works ok due to both evaluating to the exact same pointer)."

  write (*,*) "using the slice syntax as in the fourth call below results in something else being passed"
  call tell_address( arr(5) )
  call tell_address( arr(5:) )
  call tell_address( ii(offset + 5) )
  call tell_address( ii(offset + 5:) )
end program

subroutine tell_address( p )
  implicit none
  integer p(*)
  
  write (*,*) "     location of the given array in memory"
  write (*,*) loc(p)
  write (*,*) "     value in the first slot in the given array"
  write (*,*) p(1)
end subroutine

The result

C:\work>ifort /integer-size:64 /nologo arraytest2.f90  && arraytest2.exe
 the offset from beginning of ii to beginning of arr
       -17392831058724
 the location of the first element in arr using both arr and ii
         1555340080352
         1555340080352
 the value of the fifth element in arr referenced via arr and ii (should be 42)
                    42
                    42
 the location of the fifth element in arr referenced using ii as the base. Would
  expect these to be the same, but the one using slice syntax is somewhere off
         1555340080384
       140712280470784
 however, this is not so when referencing via arr
         1555340080384
         1555340080384
 even when going over the actual bounds (by a moderate amount) using plain index
  and slice syntax result to the same memory location
         1555340081224
         1555340081224
 This is a problem because in the code base I am working on, which is rather lar
 ge and has a history spanning around 40 years,
 the fact that when given as parameter to a routine w/ no explicit interface, ar
 r(n) and arr(n:) both pass the pointer to the n:th element in arr is extensivel
 y used
 and taking explicit interface checks to use requires change to the slice syntax
  as naturally passing a scalar is incorrect where an array is expected
 (even though it works ok due to both evaluating to the exact same pointer).
 using the slice syntax as in the fourth call below results in something else be
 ing passed
      location of the given array in memory
         1555340080384
      value in the first slot in the given array
                    42
      location of the given array in memory
         1555340080384
      value in the first slot in the given array
                    42
      location of the given array in memory
         1555340080384
      value in the first slot in the given array
                    42
      location of the given array in memory
       140712280470784
      value in the first slot in the given array
forrtl: severe (157): Program Exception - access violation
Image              PC                Routine            Line        Source
arraytest2.exe     00007FF6CD971B9B  Unknown               Unknown  Unknown
arraytest2.exe     00007FF6CD9C1332  Unknown               Unknown  Unknown
arraytest2.exe     00007FF6CD9C16C0  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFFDC417BD4  Unknown               Unknown  Unknown
ntdll.dll          00007FFFDDB2CE51  Unknown               Unknown  Unknown

[Steve_Lionel]

It's perfectly fine to pass ii(indx) to an array dummy - this is called "sequence association" and the effective argument is the elements from indx to the end of ii. The same would be true of passing ii(indx:) as an argument. (Some exceptions apply - see Doctor Fortran in "I've Come Here For An Argument" - Doctor Fortran (stevelionel.com))

But when you use ii(indx:) in an expression, THAT'S a copy.

[Karanta__Antti]

My original assumption that a copy was being made is apparently wrong. But something that I do not understand is happening here. 

In the code sample of my previous reply, I do not understand why these two lines produce the same result

write (*,*) loc(arr(5)) 
write (*,*) loc(arr(5:))

But these produce a different one:

write (*,*) loc(ii(offset + 5))
write (*,*) loc(ii(offset + 5:))

Also, as shown previously in the code all these should refer to the same memory, but the last one fails (as shown above) - why?

call tell_address( arr(5) )
call tell_address( arr(5:) )
call tell_address( ii(offset + 5) )
call tell_address( ii(offset + 5:) )

[Karanta__Antti]

Experimenting some more, it seems that arrays handle 64 bit indexers ok (as shown in the previous example), but something goes wrong when using the slice syntax if the index value is greater than what fits a 32 bit integer. 

Here's the experiment:

! ifort /integer-size:64 /nologo arraytest3.f90
program Arraytest3

    implicit none
    
    integer, dimension(10) :: arr
    integer :: i
    
    do i = 1, huge(i)
       if ( loc(arr(i)) /= loc(arr(i:)) ) then
          write (*,*) "the first index for which the addresses do not match is"
          write (*,*) i
          exit
       end if
    end do
    write (*,*) "done"
    
end program

The output

C:\work>ifort /integer-size:64 /nologo arraytest3.f90

C:\work>arraytest3.exe
 the first index for which the addresses do not match is
            2147483648
 done

The last index for which the addresses matched was 2147483647 == 0x7FFFFFFF, i.e. the maximum value for a signed 32 bit integer.

Now the question is: is the array indexer being supported as 64 bit value but the slice syntax only supporting 32 bit value (I assume there being a cast behind the scenes) an intentional difference in the compiler or a bug? Is there some compiler switch that might affect this?

[jimdempseyatthecove]

First question to ask

Are you using the 32-bit compiler? (generating a 32-bit program)

If so, LOC returns a 32-bit (signed)address. .AND. indexing beyond 214748364/sizeof(arr(1)) is pointless.

If you are generating a 64-bit program...

IMHO this is a compiler/runtime bug.

What happens when you make arr allocatable, allocate it with:

    allocate(arr(10_8)) ! force literal 10 to integer(8)

If this fails at same index, then LOC is at fault.
If this fails at different index (or doesn't fail), then the compiler  for the local defined array, was generating (or interpreting) the array descriptor of arr as being 32-bit indexed.

Jim Dempsey

[Karanta__Antti]

Hi Jim!

I am generating a 64 bit program. 

C:\work>ifort
Intel(R) Visual Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 19.0.5.281 Build 20190815
...
C:\work>dumpbin /headers arraytest3.exe
...
FILE HEADER VALUES
            8664 machine (x64)

You can also see this by looking at the output of my sample program where I output the loc of several memory locations. They are clearly larger than max 32 bit ints. 

Making the array allocatable in the previous sample, i.e. 

! ifort /integer-size:64 /nologo arraytest3.f90
program Arraytest3

    implicit none
    
    integer, dimension(:), allocatable :: arr
    integer :: i
    
    allocate( arr(10_8) )
    
    do i = 1, huge(i)
       if ( loc(arr(i)) /= loc(arr(i:)) ) then
          write (*,*) "the first index for which the addresses do not match is"
          write (*,*) i
          exit
       end if
    end do
    write (*,*) "done"
    
end program

The result is the same, i.e. the last index where loc results agree is max 32 bit signed int.

Though I slightly disagree w/ your conclusion: I doubt there is anything wrong w/ loc, I think the problem is in 64 bit values being supported as indexers in arrays but not when using the slice syntax.

Should I file a bug report? 

[andrew_4619]

program Arraytest3
    implicit none
    integer, dimension(:), allocatable :: arr
    integer(8) :: i
    integer(8) :: isz = 2147483648_8 + 100_8
    allocate( arr(isz) )
    do i = 1, isz
       if ( loc(arr(i)) /= loc(arr(i:)) ) then
          write (*,*) "the first index for which the addresses do not match is"
          write (*,*) i
          exit
       end if
    end do
    write (*,*) "done"
end program

This program runs to completion which is beyond the 2147483648 limit of your test. The fact that your test program is invalid because you address beyond the allocated size has a bearing I feel.

[Karanta__Antti]

That's a good added observation Andrew!

The program you modified was to investigate the issue further. The original issue is better illustrated in my previous post in this thread (https://community.intel.com/t5/Intel-Fortran-Compiler/Does-simple-array-slicing-produce-a-copy-or-a-pointer-to-the/m-p/1243130/highlight/true#M153704), i.e. that arbitrary memory is referenced using a common area array as a base. 

I do understand accessing memory beyond the original extent of the array as it has been declared is hacky and does not adhere to the standard.

So, using your observation we can say that the compiler does not have a bug in this case as such if that is defined by supporting standard adhering usage. 

I don't know how common the kind of use I demonstrated in the reply I linked to above is in other code bases, so I can't say whether this issue is of general importance.

I can likely work around the issue by using some C interop.

[andrew_4619]

I guess another way of looking at it is that if you obscure the size of the array to the compiler then the compiler has to make some assumptions. One assumption is that  array pointers (indexes) are within the bounds. Another assumption has to be the maximum size of the bounds because it has to use some "kind" for the integer index. The Fortran standard is undoubtedly silent on that matter, but it would seem that Intel have used 32bits (default integer) unless they know better. In that respect it is a limitation/constraint I feel rather than a bug. Standard compliant coding is always the ultimate goal IMO as it removes many such problems that we get when we wander into the Delta zone....

[andrew_4619]

Additionally legacy style code that make much use of unknown array bounds like in your earlier example is also not likely to see such problems as the old hardware that the old code was written for would not be hitting problems of addressing such a large amount of physical memory! 

[Karanta__Antti]

Hi Andrew!

I fully agree that one should aim to write standards compliant code. However, when you "inherit" a few million lines of old code, it is not really up to choice and one just has to make what one has to work.

This problem is not theoretical, as the code I am speaking of is being maintained and developed and has been ported to x64 and the program is addressing the 64 bit address space. If that was not the case, I would never have run into the problem in the first place, 

[andrew_4619]

I understand your problem is real but the real problem is bugs in the old code that are exposed by the march of time. I guess whatever out opinions are the net result is that some work needs to be done so I guess in  the first instance you need some workaround sticky plasters as you eluded to earlier. Interesting thread and good luck!