!DIR$ OFFLOAD TARGET: Disable implicit copying of variables?

[PKM]

Hi 

I have a code that uses derived types that includes allocatables inside a region that I want to offload. In order to get the derives type onto the device I try to manually transfer the content of the allocatables and then re-assemble the derived type on the phi. However, I have run into the problem that some kind of implicit copying must be going on ...

The issue is illustrated in the small example below. In the offload region transfer of MyType is not allowed, so I state that only TransferInt,TransferFloats should be copied to the device. However, when you get to the very last line of the program it appears that MyType has been included implicitly as in INOUT variable. An access violation is now produced runtime, since MyType could not be properly copied back from the device.

Is there any way I can turn of implicit copying in order to get around this problem?

I use the latest version of parallel studio XE.

    module mTestType
      type TTestType
        integer :: Int
        real*8,allocatable :: Floats(:)
      end type TTestType
      !dir$ attributes offload : mic :: TTestType
      type(TTestType) :: MyType
      !dir$ attributes offload : mic :: MyType
    end module mTestType
 
    
    program Console1
    use mTestType
    implicit none
    integer            :: TransferInt
    real*8,allocatable :: TransferFloats(:)
    !dir$ attributes offload : mic :: TransferInt
    MyType%Int=1d0
    allocate(MyType%Floats(4))
    MyType%Floats(:)=2d0
    allocate(TransferFloats(4))
    TransferFloats(:)=MyType%Floats(:)
    TransferInt=MyType%Int
    !DIR$ OFFLOAD BEGIN TARGET(mic:0) IN(TransferInt,TransferFloats)
    allocate(MyType%Floats(4))
      MyType%Floats(:)=TransferFloats(:)
      MyType%Int=TransferInt
      print *,'On phi ...'
      print *,MyType%Int
      print *,MyType%Floats(:)
    !DIR$ END OFFLOAD  
    print *,'On host ...'
    print *,MyType%Int
    print *,MyType%Floats(:)
    end program Console1

[Kevin_D_Intel]

Unfortunately because the compiler disallows the appearance of a derived-type with an allocatable component in the IN/INOUT/NOCOPY clauses one cannot use NOCOPY here to avoid the transfer of MyType that leads to the run-time failure. By default, the compiler transfers variables with the lexical scope of the offload construct with INOUT. As you found, this was applied to MyType which appears to corrupt the allocatable component leading to the fault in the CPU code following the offload construct. I will check with Development whether there's anyway to avoid this; however, in the meantime, with the current 14.0 compiler, the only work around I found was to declare another variable (e.g. MyType2) and use that within the offload code section instead of MyType.

In mTestType.f90:

      ...
      type(TTestType) :: MyType, MyType2

      !dir$ attributes offload : mic :: MyType, MyType2
      ...

In Console.f90:

    ...
    !DIR$ OFFLOAD BEGIN TARGET(mic:0) IN(TransferInt,TransferFloats) 
      allocate(MyType2%Floats(4))

      MyType2%Floats(:)=TransferFloats(:)
      MyType2%Int=TransferInt

      print *,'On phi ...'
      print *,MyType2%Int
      print *,MyType2%Floats(:)	 
    !DIR$ END OFFLOAD
    ...

I also wanted to note that with the upcoming CXE 2015 (15.0 compiler) release (that is still in Beta testing) you can offload the derived-type with the allocatable component. I confirmed this trivial case will offload using the support in the coming release for offloading the derived-type with the simple allocatable component.

  program Console1

    use mTestType
    implicit none

    integer            :: TransferInt
    real*8,allocatable :: TransferFloats(:)

    !dir$ attributes offload : mic :: TransferInt

    MyType%Int=1d0

    allocate(MyType%Floats(4))
    MyType%Floats(:)=2d0

    allocate(TransferFloats(4))

    TransferFloats(:)=MyType%Floats(:)
    TransferInt=MyType%Int

    !DIR$ OFFLOAD BEGIN TARGET(mic:0) IN(MyType) IN(MyType%Floats)
      print *,'On phi ...'
      print *,MyType%Int
      print *,MyType%Floats(:)
    !DIR$ END OFFLOAD  

    print *,'On host ...'
    print *,MyType%Int
    print *,MyType%Floats(:)

    end program Console1

Compiled and Run and follows:

C:\> ifort mTestType.f90 Console1.f90 /FeConsole1.exe
Intel(R) Visual Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.0.070 Beta Build 20140530
Copyright (C) 1985-2014 Intel Corporation.  All rights reserved.

Microsoft (R) Incremental Linker Version 12.00.21005.1
Copyright (C) Microsoft Corporation.  All rights reserved.

-out:Console1.exe
-subsystem:console
mTestType.obj
Console1.obj
C:\Users\kddavis\AppData\Local\Temp\2\2604333.obj
-defaultlib:liboffload
-defaultlib:libiomp5md
ofldbegin.obj
ofldend.obj

C:\> .\Console1.exe
 On phi ...
           1
   2.00000000000000        2.00000000000000        2.00000000000000
   2.00000000000000
 On host ...
           1
   2.00000000000000        2.00000000000000        2.00000000000000
   2.00000000000000

[PKM]

Thanks Kevin ...

It would be very nice if the compiler allowed putting these forbidden derived types in NOCOPY clauses - or even better - did it by default. I use some nested derived types of allocatables and these will not be supported by the new functionality in 15.0 right?

[Kevin_D_Intel]

Yes, I agree. I was already working on that and a couple of other issues I found from investigating the issue you reported.

You are correct, nested allocations are not supported in the new functionality in 15.0.

[Kevin_D_Intel]

I submitted defect reports with Development about the original problem you reported and others that I found during my investigation.

I filed one report against 14.0 for the original issue you reported. The others I filed against 15.0 since that release contains explicit support for offload of the derive-type w/allocatable component. With 15.0, the implicit data transfer of the derived-type w/allocatable component fails with a Seg-fault at run-time. Explicit transfer of the derived-type and the derived-type allocatable component (as I showed in my earlier example) runs successfully. The other report filed relates to not permitting use of NOCOPY and OUT clauses with the derived type w/allocatable component.

I will keep you posted on these as I learn more.

(Internal tracking id: DPD200359169 - Run-time SegV on CPU with implicit transferred derived-type allocatable component (14.0))
(Internal tracking id: DPD200359184 - Run-time SegV in offloaded code with implicit transferred derived-type allocatable component)
(Internal tracking id: DPD200359188 - Derived-type w/allocatable component not permitted in NOCOPY and OUT clauses)

[pbkenned1]

I'm helping Kevin evaluate the various problem reports associated with this thread. 

For DPD200359188 - Derived-type w/allocatable component not permitted in NOCOPY and OUT clauses), ifort-15.0.3 now accepts both NOCOPY and OUT clauses for such a type, so that resolves DPD200359188.

#ifdef SHOW_error1
! Compiler does not allow derived-type w/allocatable in NOCOPY
! so we lose ability to use data persistence from previous offload
    !DIR$ OFFLOAD BEGIN TARGET(mic:0) NOCOPY (MyType)
      print *,'On phi block 2...'
      print *,MyType%Int
      print *,MyType%Floats(:)
    !DIR$ END OFFLOAD

    !DIR$ OFFLOAD BEGIN TARGET(mic:0) NOCOPY (MyType%Floats)
      print *,'On phi block 2B...'
      print *,MyType%Int
      print *,MyType%Floats(:)
    !DIR$ END OFFLOAD
#endif
#ifdef SHOW_error2
! Compiler does not allow derived-type w/allocatable in OUT
! so we lose ability to use data persistence from previous offload
! and return updated values
    !DIR$ OFFLOAD BEGIN TARGET(mic:0) OUT (MyType)
      print *,'On phi block 3...'
      print *,MyType%Int
      print *,MyType%Floats(:)
    !DIR$ END OFFLOAD

    !DIR$ OFFLOAD BEGIN TARGET(mic:0) OUT (MyType%Floats)
      print *,'On phi block 3B...'
      print *,MyType%Int
      print *,MyType%Floats(:)
    !DIR$ END OFFLOAD
#endif

[U518387]$ ifort -V
Intel(R) Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.3.187 Build 20150407
Copyright (C) 1985-2015 Intel Corporation.  All rights reserved.

[U518387]$ ifort DPD200359188.f90 -fpp -DSHOW_error1 -c
[U518387]$ ifort DPD200359188.f90 -fpp -DSHOW_error2 -c
[U518387]$

I'll check remaining problem reports DPD200359169 and DPD200359184 and update this thread accordingly.

Patrick

[pbkenned1]

DPD200359169 - Run-time SegV on CPU with implicit transferred derived-type allocatable component (14.0))

This was specific to the 14.0 compiler, and was considered too risky to fix in that version.  The 15.0 compiler behaves differently WRT to the DPD200359169 test case since it supports transfer of the derived-type w/allocatable.

Patrick

[pbkenned1]

DPD200359184 - Run-time SegV in offloaded code with implicit transferred derived-type allocatable component.
If you have a derived type with an allocatable component, it must be explicitly transferred.  Implicit transfer will not work.

While allocatable components of objects can be copied back and forth between CPU and MIC by copying the containing object, their values can only be used where allocated. An allocate statement on CPU or MIC is one way of allocating an allocatable object. Listing an allocatable object in an offload directive is a second way of allocating on object on MIC.

If I have:

    module mTestType
      type TTestType
        integer :: Int
        real*8,allocatable :: Floats(:)
      end type TTestType

      !dir$ attributes offload : mic :: MyType
      type(TTestType) :: MyType
    end module mTestType
...
    allocate(MyType%Floats(4)) !allocate on CPU
    MyType%Floats(:)=2d0

    !DIR$ OFFLOAD BEGIN TARGET(mic:0) IN(MyType%Floats) !explicit transfer works fine
      print *,'On phi block 1...'
      print *,MyType%Int
      print *,MyType%Floats(:)                          ! this is usable on MIC
    !DIR$ END OFFLOAD

However, implicit transfer fails:

    !DIR$ OFFLOAD BEGIN TARGET(mic:0) ! implicit transfer will fail
      print *,'On phi block 2...'
      print *,MyType%Int
      print *,MyType%Floats(:)        !this is not usable on MIC
    !DIR$ END OFFLOAD

The above implicit transfer of MyType only does a bitwise copy.  Thhis works fine for MyType%Int,
but won't work for MyType%Floats(:); any access to the latter will have undefined results.

The offload directives deal with one object at a time; they do not process an object by iterating over its individual components.
An allocatable object of intrinsic type that is not a component of a derived type can be implicitly transferred.
However, an allocatable object that is part of a derived type has to be explicitly transferred.

Patrick