OpenMP Target use_device_ptr/use_device_addr with Fortran derived types

[caplanr]

Hi,

I am testing a miniapp that tests multi-gpu Fortran that is used in our production code.

The code is located at:  multigpu-test-code 

The specific version being tested here is:   psi_multigpu_test_code_stdpar_ompdata.f90

I am testing on the Stampede3 system at TACC on the MAX 1550 GPU.

When I try to compile the code with:

mpif90 -f90=ifx -fiopenmp -fopenmp-target-do-concurrent -fopenmp-targets=spir64 -fopenmp-do-concurrent-maptype-modifier=present psi_multigpu_test_code_stdpar_ompdata.f90 -o psi_multigpu_test_code_stdpar_ompdata

I get the error:

psi_multigpu_test_code_stdpar_ompdata.f90(669): error #9036: A list item that appears in a use_device_ptr or use_device_addr clause must not be a component of a derived type. [R]
!$omp target data use_device_ptr(v%r,v%t,v%p)
-----------------------------------^

Is this error due to the OpenMP Target specification, or is it a temporary limitation of the compiler?   

The code works correctly on NVIDIA GPUs with the nvfortran compiler, implying the latter (although nvfortran is known to support the unsupported sometimes...)

In either case, is there a current workaround to allow passing derived type arrays into GPU-aware MPI/library calls?

Thanks!

 - Ron

[TobiasK · Accepted solution]

@caplanr

The restriction is removed in OpenMP 6.0, for 5.2 you may workaround the restriction with something like this:

program test_use_device_addr
  implicit none
  type d
     real :: a
  end type d
  type(d) :: a
  real,pointer ::a_ptr
  call assign_ptr(a%a,a_ptr)  
!$omp target data use_device_addr(a_ptr)

!$omp end target data
call wrapper(a%a)
contains
  subroutine wrapper(a)
    real :: a
    !$omp target data use_device_addr(a)                                                                                                                                                     
    !$omp end target data                                                                                                                                                                    
  end subroutine wrapper
  subroutine assign_ptr(source,ptr)
    real,target :: source
    real,pointer :: ptr
    ptr=>source
  end subroutine assign_ptr
end program test_use_device_ptr

[caplanr]

Thanks!

Will OpenMP 6.0 (at least this component of it) be implemented in the next version of IFX?

Any ETA on the release?

 - Ron

[TobiasK]

@caplanr

sorry for the long delay here. It took me several discussions with the internal OMP team and I have to tell you that it will be out of scope for the near future. My interpretation of the standard was wrong - the note about structure components was removed but it was only removed because it was a duplicate. There will be no change regarding use_device_addr and structure components.

So the only way to handle this is going through a wrapper function the remove the struct or assign temporary pointers to the structure components and use them in the use_device_addr() clause.

[caplanr]

Hi,

That is not great news.  Is there a reason for the restriction in the spec?

Is there a way to request the feature to the the OpenMP spec committee?

For our code, it would be very cumbersome to wrap every routine where a derived type is used as a device pointer.

I think for our main code that uses derived types we will stick with OpenACC for now as its "host_data use_device" works with derived type arrays  (since Intel is skipping its next data-center GPU and no high-end Battlemage GPUs have been announced, we can wait for the spec to catch up).

For our other codes, we will stick with OpenMP target so we can run on Intel GPUs (check out our latest paper where we have some great Intel GPU results: https://arxiv.org/abs/2501.06377 Figures 9 and 10)

Thank you for the info!

 -- Ron