topic Re: OpenMP Target use_device_ptr/use_device_addr with Fortran derived types in Intel® MPI Library

OpenMP Target use_device_ptr/use_device_addr with Fortran derived types

caplanr — Tue, 17 Dec 2024 01:56:22 GMT

Hi,

I am testing a miniapp that tests multi-gpu Fortran that is used in our production code.

The code is located at: multigpu-test-code

The specific version being tested here is: psi_multigpu_test_code_stdpar_ompdata.f90

I am testing on the Stampede3 system at TACC on the MAX 1550 GPU.

When I try to compile the code with:

mpif90 -f90=ifx -fiopenmp -fopenmp-target-do-concurrent -fopenmp-targets=spir64 -fopenmp-do-concurrent-maptype-modifier=present psi_multigpu_test_code_stdpar_ompdata.f90 -o psi_multigpu_test_code_stdpar_ompdata

I get the error:

psi_multigpu_test_code_stdpar_ompdata.f90(669): error #9036: A list item that appears in a use_device_ptr or use_device_addr clause must not be a component of a derived type. [R]
!$omp target data use_device_ptr(v%r,v%t,v%p)
-----------------------------------^

Is this error due to the OpenMP Target specification, or is it a temporary limitation of the compiler?

The code works correctly on NVIDIA GPUs with the nvfortran compiler, implying the latter (although nvfortran is known to support the unsupported sometimes...)

In either case, is there a current workaround to allow passing derived type arrays into GPU-aware MPI/library calls?

Thanks!

- Ron

Re: OpenMP Target use_device_ptr/use_device_addr with Fortran derived types

TobiasK — Fri, 27 Dec 2024 11:55:44 GMT

@caplanr

The restriction is removed in OpenMP 6.0, for 5.2 you may workaround the restriction with something like this:

program test_use_device_addr implicit none type d real :: a end type d type(d) :: a real,pointer ::a_ptr call assign_ptr(a%a,a_ptr) !$omp target data use_device_addr(a_ptr) !$omp end target data call wrapper(a%a) contains subroutine wrapper(a) real :: a !$omp target data use_device_addr(a) !$omp end target data end subroutine wrapper subroutine assign_ptr(source,ptr) real,target :: source real,pointer :: ptr ptr=>source end subroutine assign_ptr end program test_use_device_ptr

Re: OpenMP Target use_device_ptr/use_device_addr with Fortran derived types

caplanr — Fri, 27 Dec 2024 20:42:17 GMT

Thanks!

Will OpenMP 6.0 (at least this component of it) be implemented in the next version of IFX?

Any ETA on the release?

- Ron

Re:OpenMP Target use_device_ptr/use_device_addr with Fortran derived types

TobiasK — Fri, 14 Mar 2025 18:06:29 GMT

@caplanr

sorry for the long delay here. It took me several discussions with the internal OMP team and I have to tell you that it will be out of scope for the near future. My interpretation of the standard was wrong - the note about structure components was removed but it was only removed because it was a duplicate. There will be no change regarding use_device_addr and structure components.

So the only way to handle this is going through a wrapper function the remove the struct or assign temporary pointers to the structure components and use them in the use_device_addr() clause.

Re: OpenMP Target use_device_ptr/use_device_addr with Fortran derived types

caplanr — Mon, 17 Mar 2025 16:42:20 GMT

Hi,

That is not great news. Is there a reason for the restriction in the spec?

Is there a way to request the feature to the the OpenMP spec committee?

For our code, it would be very cumbersome to wrap every routine where a derived type is used as a device pointer.

I think for our main code that uses derived types we will stick with OpenACC for now as its "host_data use_device" works with derived type arrays (since Intel is skipping its next data-center GPU and no high-end Battlemage GPUs have been announced, we can wait for the spec to catch up).

For our other codes, we will stick with OpenMP target so we can run on Intel GPUs (check out our latest paper where we have some great Intel GPU results: https://arxiv.org/abs/2501.06377 Figures 9 and 10)

Thank you for the info!

-- Ron