topic problem using array arg on MIC in subroutine in Software Archive

problem using array arg on MIC in subroutine

Gene_Wagenbreth — Thu, 14 Nov 2013 03:09:08 GMT

This example is a very shortened version of what I am trying to do in real code. I am trying to put an array on MIC, then pass the array to a routine on the host, then do a loop in the routine on the MIC. If I put the MIC loop inline, it works. But if I put it in a subroutine, it fails. Thank you Gene W genew@cy015:~/testdir> cat subargprob1.F program t implicit none real matrix(10000) integer i print *,'start' print *,'b4 offload_transfer' !dir$ offload_transfer target(mic:0) nocopy(matrix: & alloc_if(.true.) free_if(.false.)) print *,'af offload_transfer' !THE FOLLOWING CODE WORKS INLINE BUT FAILS IN SUBROUTINE !dir$ offload target(mic:0) inout(matrix(1:10000): & alloc_if(.false.) free_if(.false.)) !$omp parallel do do i=1,10000 matrix(i)=i end do print '(a,z)','GW b4 call t3s2 loc(matrix)=',loc(matrix) !CODE FAILS INSIDE THIS SUBROUTINE call t3s2(matrix) end subroutine t3s2(matrix) implicit none real matrix(10000) integer i print *,'GW start t3s2' print '(a,z)','GW inside t3s2 loc(matrix)=',loc(matrix) !THE FOLLOWING CODE WORKS INLINE BUT FAILS IN SUBROUTINE !dir$ offload target(mic:0) inout(matrix(1:10000): & alloc_if(.false.) free_if(.false.)) !$omp parallel do do i=1,10000 matrix(i)=i end do end mpiifort -openmp subargprob1.F -o subargprob1 -list=subargprob1.list ./subargprob1 start b4 offload_transfer [Offload] [MIC 0] [File] subargprob1.F [Offload] [MIC 0] [Line] 8 [Offload] [MIC 0] [Tag] Tag 0 [Offload] [HOST] [Tag 0] [CPU Time] 1.900814 (seconds) [Offload] [MIC 0] [Tag 0] [MIC Time] 0.000000 (seconds) af offload_transfer [Offload] [MIC 0] [File] subargprob1.F [Offload] [MIC 0] [Line] 13 [Offload] [MIC 0] [Tag] Tag 1 [Offload] [HOST] [Tag 1] [CPU Time] 0.659075 (seconds) [Offload] [MIC 0] [Tag 1] [MIC Time] 0.323732 (seconds) GW b4 call t3s2 loc(matrix)= 7FFFFFFF3890 GW start t3s2 GW inside t3s2 loc(matrix)= 7FFFFFFF3890 [Offload] [MIC 0] [File] subargprob1.F [Offload] [MIC 0] [Line] 32 [Offload] [MIC 0] [Tag] Tag 2 offload error: cannot find data associated with pointer variable 0x7fffffff3890

I modified your example a bit

Kevin_D_Intel — Thu, 14 Nov 2013 19:33:00 GMT

The modified version below demonstrates using an allocatable array passed as an argument with data persistence.

[cpp] program example2
implicit none

integer, parameter :: SZ = 10000
real, allocatable,dimension(:) :: matrix
integer i

allocate(matrix(SZ))

! Allocate memory on coprocessor only
!dir$ offload_transfer target(mic:0) in(matrix : length(SZ) &
alloc_if(.true.) free_if(.false.))

! Initialize values on the host
!$omp parallel do private(i)
do i=1,SZ
    matrix(i)=99.0
end do

print *,'start'
print *,'b4 offload_transfer'
print '(a,f6.2)','matrix(1)= ',matrix(1)
print '(a,f6.2)','matrix(2)= ',matrix(2)
print '(a,I0,a,f6.2)','matrix(',SZ,')= ',matrix(SZ)

! Send current values to coprocessor only and reuse previous allocation
!dir$ offload_transfer target(mic:0) in(matrix : length(SZ) &
alloc_if(.false.) free_if(.false.))

print *,'af offload_transfer'

! change this element on the host only to demonstate values on
! coprocessor are used
matrix(1) = 0.0

print '(a,z)','GW b4 call t3s2 loc(matrix)=',loc(matrix)

call t3s2(matrix,SZ)

! Transfer current values on coprocessor to host and free allocation
!dir$ offload_transfer target(mic:0) out(matrix : length(SZ) &
alloc_if(.false.) free_if(.true.))

print '(a)','Expected values are 99.0, 100.0, 98.0'
print '(a,f6.2)','matrix(1)= ',matrix(1)
print '(a,f6.2)','matrix(2)= ',matrix(2)
print '(a,I0,a,f6.2)','matrix(',SZ,')= ',matrix(SZ)
end program example2

subroutine t3s2(matA,siz)
implicit none

integer, intent(in) :: siz
real, dimension(*), intent(inout) :: matA

integer i

print *,'GW start t3s2'
print '(a,z)','GW inside t3s2 loc(matA)=',loc(matA)

! With length(0), no data is transferred to the coprocessor
! With IN and no allocation set, the pointer value for this
! offload is only initialized to the value created in the calling
! scope for the associated actual argument in the caller;
! therefore, this re-uses the previous allocation on the coprocessor
! for the associated actual argument

! Modify one existing value (i.e. 99.0) on the coprocessor and
! leave value on coprocessor

!dir$ offload begin target(mic:0) in(siz) in(matA : length(0) &
    alloc_if(.false.) free_if(.false.))
matA(siz)= matA(siz) - 1
!dir$ end offload

! Re-use values on coprocessor, update some elements of array,
! leave values on coprocessor
!dir$ offload target(mic:0) in(siz) nocopy (matA : length(0) &
    alloc_if(.false.) free_if(.false.))
!$omp parallel do private(i)
do i=2,(siz-1)
matA(i)=100.0
end do

end subroutine t3s2 [/cpp]

[plain]$ ./a.out
[Offload] [MIC 0] [File]            example2.F90
[Offload] [MIC 0] [Line]            11
[Offload] [MIC 0] [Tag]             Tag 0
[Offload] [HOST] [Tag 0] [CPU Time]        1.005746(seconds)
[Offload] [MIC 0] [Tag 0] [CPU->MIC Data]   40064 (bytes)
[Offload] [MIC 0] [Tag 0] [MIC Time]        0.000188(seconds)
[Offload] [MIC 0] [Tag 0] [MIC->CPU Data]   0 (bytes)

start
b4 offload_transfer
matrix(1)= 99.00
matrix(2)= 99.00
matrix(10000)= 99.00
[Offload] [MIC 0] [File]            example2.F90
[Offload] [MIC 0] [Line]            27
[Offload] [MIC 0] [Tag]             Tag 1
[Offload] [HOST] [Tag 1] [CPU Time]        0.000265(seconds)
[Offload] [MIC 0] [Tag 1] [CPU->MIC Data]   40072 (bytes)
[Offload] [MIC 0] [Tag 1] [MIC Time]        0.000014(seconds)
[Offload] [MIC 0] [Tag 1] [MIC->CPU Data]   0 (bytes)

af offload_transfer
GW b4 call t3s2 loc(matrix)=                1CAA690
GW start t3s2
GW inside t3s2 loc(matA)=                1CAA690
[Offload] [MIC 0] [File]            example2.F90
[Offload] [MIC 0] [Line]            71
[Offload] [MIC 0] [Tag]             Tag 2
[Offload] [HOST] [Tag 2] [CPU Time]        0.004952(seconds)
[Offload] [MIC 0] [Tag 2] [CPU->MIC Data]   12 (bytes)
[Offload] [MIC 0] [Tag 2] [MIC Time]        0.000058(seconds)
[Offload] [MIC 0] [Tag 2] [MIC->CPU Data]   8 (bytes)

[Offload] [MIC 0] [File]            example2.F90
[Offload] [MIC 0] [Line]            78
[Offload] [MIC 0] [Tag]             Tag 3
[Offload] [HOST] [Tag 3] [CPU Time]        0.236283(seconds)
[Offload] [MIC 0] [Tag 3] [CPU->MIC Data]   16 (bytes)
[Offload] [MIC 0] [Tag 3] [MIC Time]        0.227425(seconds)
[Offload] [MIC 0] [Tag 3] [MIC->CPU Data]   4 (bytes)

[Offload] [MIC 0] [File]            example2.F90
[Offload] [MIC 0] [Line]            41
[Offload] [MIC 0] [Tag]             Tag 4
[Offload] [HOST] [Tag 4] [CPU Time]        0.028915(seconds)
[Offload] [MIC 0] [Tag 4] [CPU->MIC Data]   72 (bytes)
[Offload] [MIC 0] [Tag 4] [MIC Time]        0.000257(seconds)
[Offload] [MIC 0] [Tag 4] [MIC->CPU Data]   40000 (bytes)

Expected values are 99.0, 100.0, 98.0
matrix(1)= 99.00
matrix(2)= 100.00
matrix(10000)= 98.00[/plain]

Kevin Davis

Gene_Wagenbreth — Thu, 14 Nov 2013 20:45:21 GMT

Kevin Davis Thank you for taking the time to look at my problem. Your code runs. If I change "OUT" at line 38 to "NOCOPY", I get a seg fault on the MIC. Why is this ? NOCOPY is what I really want to do. In my actual code, I want to allocate an array on the MIC. Then I call subroutines, passing the array as an argument, which have loops that read and set the array. I want to leave a lot of the code on the host, but execute the loops on the MIC. I want to initialize the array on the host, alllocate it on the MIC and transfer it to the MIC, then leave the array on the MIC executing lots of loops in subroutines on the MIC, but executing scalar code on the host. I was told by someone else that transfers of dummy arguments is not allowed. Your example shows that it works sometimes. But experiments I have run indicate that it does not always work. The seg fault when line 38 of your example is changed from OUT ot NOCOPY is one example. I am experimenting with passing the array in COMMON rather that as a subroutine argument, but that is not a very general solution. Thanks again Gene W