topic Re: Error of OpenMP data transfer in GPU offloading in Intel® Fortran Compiler

Error of OpenMP data transfer in GPU offloading

Longxiang_Li — Sun, 18 Feb 2024 07:44:18 GMT

Dear all,

Currently, we are using the IFX compiler to migrate our code to run on the GPU. We found that ifx supports OpenMP 5.1 spec and could declare a mapper to control the data transfer processes for the user-defined type variables. However, we found the declared mapper could not work correctly as we expected.

There are mainly two problems:

1. The map directive with the mapper will cash at the second usage.

For example, we use a user-defined type 'struct_test', and declare the corresponding mapper as follows:

type struct_test integer :: len integer, pointer :: int_ptr(:) real, pointer :: float_ptr(:) end type struct_test !$omp declare mapper ( struct_mapper: struct_test :: var ) & !$omp& map( var%len, var%int_ptr, var%float_ptr )

We also define a test function and check whether the array components could remain the same with the map directives.

subroutine test_data_map(len, init_val) integer :: init_val, len ! local var type(struct_test) :: test call init_struct(test, len, init_val, float(init_val)) write(*, *) !============================================================ ! test omp offloading with int vector from structure !============================================================ !$omp target enter data map(mapper(struct_mapper), to: test) test%int_ptr = 1 !$omp target exit data map(mapper(struct_mapper), from: test) write(*, *) "test with data structure" write(*, *) "test%int_ptr = ", test%int_ptr !============================================================ ! test omp offloading with float vector from structure !============================================================ !$omp target enter data map(mapper(struct_mapper), to: test) test%float_ptr = 2 !$omp target exit data map(mapper(struct_mapper), from: test) write(*, *) "test with data structure" write(*, *) "test%float_ptr = ", test%float_ptr call delete_struct(test) end subroutine test_data_map

When we called the test function the second time, the program crashed with the following error message.

$ ./test.exe test with data structure test%int_ptr = 2 3 4 5 6 test with data structure test%float_ptr = 2.000000 3.000000 4.000000 5.000000 6.000000 Libomptarget message: explicit extension not allowed: host address specified is 0x00007ffc2bfda598 (152 bytes), but device allocation maps to host at 0x00007ffc2bfda588 (152 bytes) Libomptarget error: Call to getTargetPointer returned null pointer (device failure or illegal mapping). Libomptarget error: Call to targetDataBegin via targetDataMapper for custom mapper failed. Libomptarget error: Run with Libomptarget error: LIBOMPTARGET_DEBUG=1 to display basic debug information. Libomptarget error: LIBOMPTARGET_DEBUG=2 to display calls to the compute runtime. Libomptarget error: LIBOMPTARGET_INFO=4 to dump host-target pointer mappings. Libomptarget error: Source location information not present. Compile with -g or -gline-tables-only. Libomptarget fatal error 1: failure of target construct while offloading is mandatory forrtl: error (76): Abort trap signal Image PC Routine Line Source libpthread-2.31.s 00007F9EEAEB3420 Unknown Unknown Unknown libc-2.31.so 00007F9EEACF000B gsignal Unknown Unknown libc-2.31.so 00007F9EEACCF859 abort Unknown Unknown libomptarget.so 00007F9EEB1A6CB8 Unknown Unknown Unknown libomptarget.so 00007F9EEB19CBFC __tgt_target_data Unknown Unknown test.exe 0000000000405A15 Unknown Unknown Unknown test.exe 000000000040527D Unknown Unknown Unknown libc-2.31.so 00007F9EEACD1083 __libc_start_main Unknown Unknown test.exe 000000000040519E Unknown Unknown Unknown [1] 5290 abort ./test.exe

2. The map directive with mapper could not work correctly for the user-defined type arrays.

We also checked if the mapper could work functionally with the user-defined type array. For example, we extend the test function as follows:

subroutine test_data_array(len_array, len, init_val) integer :: len_array, len, init_val ! local var type(struct_test), allocatable :: test(:) integer :: i allocate(test(len_array)) do i = 1, len_array call init_struct(test(i), len, init_val + len_array, float(init_val + len_array)) !$omp target enter data map(mapper(struct_mapper), to: test(i)) end do do i = 1, len_array test(i)%int_ptr = 0 test(i)%float_ptr = 1 end do do i = 1, len_array !$omp target exit data map(mapper(struct_mapper), from: test(i)) end do write(*, *) write(*, *) "test with data structure array, len_array = ", len_array write(*, *) "test(", len_array, ")%int_ptr = ", test(len_array)%int_ptr do i = 1, len_array call delete_struct(test(i)) end do deallocate(test) end subroutine test_data_array

We could not get the initial values in the array components of 'int_ptr' and 'float_ptr'. It means the map directives do not work correctly.

$ ./test.exe test with data structure array, len_array = 2 test( 2 )%int_ptr = 0 0 0 0 0 test with data structure array, len_array = 2 test( 2 )%int_ptr = 0 0 0 0 0

Re: Error of OpenMP data transfer in GPU offloading

Barbara_P_Intel — Tue, 20 Feb 2024 21:57:25 GMT

Thank you for the reproducer. Please tell me the output you expect from each of the two test cases.

What version of ifx are you using?

I am using the version of the compiler that is planned to be released in 6-8 weeks. Your first test case does not abort. Good news!

Re: Error of OpenMP data transfer in GPU offloading

Ron_Green — Tue, 20 Feb 2024 23:27:17 GMT

@Longxiang_Li we are still doing some work with mappers in the compiler. Make sure to upgrade to 2024.1, Update 1, as soon as it is available. There are many important fixes in 2024.1 for OMP mapper

Re: Error of OpenMP data transfer in GPU offloading

Longxiang_Li — Wed, 21 Feb 2024 01:34:29 GMT

Hi Barbara,

I am using the latest version of the ifx 2024.0.2.

Could you help check if the new compiler could work functionally for the second test?

Thank you

Re: Error of OpenMP data transfer in GPU offloading

Longxiang_Li — Wed, 21 Feb 2024 01:37:27 GMT

Thanks for your suggestion.

I found the latest version of the ifx compiler is still 2024.0.2 (https://www.intel.com/content/www/us/en/developer/articles/tool/oneapi-standalone-components.html#fortran). Where could I get the compiler of 2024.1 version?

Re: Error of OpenMP data transfer in GPU offloading

Barbara_P_Intel — Wed, 21 Feb 2024 16:24:32 GMT

ifx 2024.1.0 is not yet released. Look for it in 6-8 weeks.

I tried both of your tests with a preview version of ifx 2024.1.0. Please tell me the expected output for each test. I know what I think the output should be, but it may not match your expectations.

If there are still issues, I will let you know and get bug reports filed.

Re: Error of OpenMP data transfer in GPU offloading

Longxiang_Li — Thu, 22 Feb 2024 02:16:07 GMT

Hi Barbara,

In the main function, the test functions are called with the following parameters:

call test_data_map(5, 2) call test_data_array(2, 5, 2)

The expected result of the first test function should be

test with data structure test%int_ptr = 2 3 4 5 6 test with data structure test%float_ptr = 2.000000 3.000000 4.000000 5.000000 6.000000

The result of the second test function should be

test with data structure array, len_array = 2 test( 1 )%int_ptr = 4 5 6 7 8 test( 1 )%float_ptr = 4.000000 5.000000 6.000000 7.000000 8.000000 test( 2 )%int_ptr = 4 5 6 7 8 test( 2 )%float_ptr = 4.000000 5.000000 6.000000 7.000000 8.000000

To find the correct result, I have added the test function `test_data_array_map`, which utilized the map directive instead of the mapper. However, there is another error with mapping the structure arrays. The test function is shown as follows.

subroutine test_data_array_map(len_array, len, init_val) integer :: len_array, len, init_val ! local var type(struct_test), allocatable :: test(:) integer :: i allocate (test(len_array)) do i = 1, len_array call init_struct(test(i), len, init_val + len_array, float(init_val + len_array)) !$omp target enter data map(to: test(i)%int_ptr, test(i)%float_ptr) end do do i = 1, len_array test(i)%int_ptr = 0 test(i)%float_ptr = 1 end do do i = 1, len_array !$omp target exit data map(from: test(i)%int_ptr, test(i)%float_ptr) end do write (*, *) write (*, *) "test with data structure array, len_array = ", len_array do i = 1, len_array write (*, *) "test(", i, ")%int_ptr = ", test(i)%int_ptr write (*, *) "test(", i, ")%float_ptr = ", test(i)%float_ptr end do do i = 1, len_array call delete_struct(test(i)) end do deallocate (test) end subroutine test_data_array_map

After mapping the test array from device to host, only the first structure element gets the correct initial values, whereas the other elements' pointer array %int_ptr and %float_ptr is empty.

test with data structure array, len_array = 2 test( 1 )%int_ptr = 4 5 6 7 8 test( 1 )%float_ptr = 4.000000 5.000000 6.000000 7.000000 8.000000 test( 2 )%int_ptr = test( 2 )%float_ptr =

Is this also a bug for the map directive? Could you help to check if this error could be solved in 2024.1 update?

Best wishes

Longxiang Li

Re: Error of OpenMP data transfer in GPU offloading

Barbara_P_Intel — Thu, 22 Feb 2024 18:06:27 GMT

Thank you for the additional information.

This is getting confusing with 3 different test cases. I created a separate file for each case. I'll post info on each in a separate post.

For the first case, attached is test.data.map.f90. According to your test.f90, test_data_map is called twice.

call test_data_map(5, 2) call test_data_map(3, 4)

Using the preview ifx 2024.1.0 I get this output. The first call to test_data_map matches the output you sent. Is the output from the second call correct?

test with data structure test%int_ptr = 2 3 4 5 6 test with data structure test%float_ptr = 2.000000 3.000000 4.000000 5.000000 6.000000 test with data structure test%int_ptr = 0 0 4 test with data structure test%float_ptr = 2.000000 3.000000 4.000000

Re: Error of OpenMP data transfer in GPU offloading

Barbara_P_Intel — Thu, 22 Feb 2024 19:10:13 GMT

For the second test, I created test.data.array.f90 (attached) from your test.f90. I think there's a mismatch between what you sent for this subroutine and the subroutine that printed the results you sent. Please confirm.

Three things:

(1) The version of subroutine test.data.array that you sent does not have write statements for test%float_ptr. Since your output prints that I added the appropriate write statements.

(2) The main routine has these calls for test_data_array. Those subroutine calls don't match the output you sent. Please confirm.

call test_data_array(2, 5, 2) call test_data_array(2, 5, 2)

(3) The version of subroutine test.data.array that you sent sets the array test%int_ptr to 0 and the array test%float_ptr to 1.

Re: Error of OpenMP data transfer in GPU offloading

Barbara_P_Intel — Thu, 22 Feb 2024 19:23:53 GMT

For the 3rd test case, please provide the statements that call test_data_array_map. The arguments are key.

Re: Error of OpenMP data transfer in GPU offloading

Longxiang_Li — Fri, 23 Feb 2024 02:03:58 GMT

The second call gets some errors in the result.

In the parameters of `test_data_map`, the first represents the length of the vector and the second represents the initial value of the vector '%int_ptr' and `%float_ptr`. According to the input parameters, there are two errors in the output.

1. In the second call of `test_data_map`, the start value of the vector should be 4, but the two vectors still start from 2.

2. The vectors `%int_ptr` and `%float_ptr` should have the same values in each element, but there are many zeros in the second output of `%int_ptr`.

Re: Error of OpenMP data transfer in GPU offloading

Longxiang_Li — Fri, 23 Feb 2024 01:58:15 GMT

Hi,

I have added some output commands to show the correct initial value and all the array `test` elements. Please use this source code for checking.

The correct value of the structure elements `test(1)` and `test(2)` should start from 2 and 7 respectively. I will explain the reason.

1. The structure element is initialized inside the `init_struct` subroutine with the given initial value, and these initial values are copied to the device. (I have edited the source code to assign different initial values for the structure elements)

2. The assignment of the vectors with `test(i)%int_ptr = 0` and `test(i)%float_ptr = 1` is conducted by the host, so the device still gets the initial values.

3. After exiting the data region, the device values are copied back to the host. Therefore, the vectors of the structure elements get their initial values.

I used the compiler ifx 2024.0.2 to run the test function, the results are far from the correct values.

$ ./test.data.array.exe Initial value of test( 1 ) is 2 Initial value of test( 2 ) is 7 test with data structure array, len_array = 2 test( 1 ) %int_ptr = 0 0 0 0 0 %float_ptr = 1.000000 1.000000 1.000000 1.000000 1.000000 test( 2 ) %int_ptr = 0 0 0 0 0 %float_ptr = 1.000000 1.000000 1.000000 1.000000 1.000000 Initial value of test( 1 ) is 2 Initial value of test( 2 ) is 7 test with data structure array, len_array = 2 test( 1 ) %int_ptr = 0 0 0 0 0 %float_ptr = 1.000000 1.000000 1.000000 1.000000 1.000000 test( 2 ) %int_ptr = 0 0 0 0 0 %float_ptr = 1.000000 1.000000 1.000000 1.000000 1.000000

Re: Error of OpenMP data transfer in GPU offloading

Longxiang_Li — Fri, 23 Feb 2024 02:12:28 GMT

I have added a new file to check with the test_data_array_map function.

In this function, I discard the mapper directive and utilize the map directive to copy the vectors of each structure element between the host and the device. The result shows the map directive also could not work functionally with the structure array.

My check results are

$ ./test.data.map.array.exe Initial value of test( 1 ) is 2 Initial value of test( 2 ) is 7 test with data structure array, len_array = 2 test( 1 ) %int_ptr = %float_ptr = test( 2 ) %int_ptr = %float_ptr = Initial value of test( 1 ) is 2 Initial value of test( 2 ) is 7 test with data structure array, len_array = 2 test( 1 ) %int_ptr = %float_ptr = test( 2 ) %int_ptr = %float_ptr =

Re: Error of OpenMP data transfer in GPU offloading

Barbara_P_Intel — Mon, 26 Feb 2024 20:03:05 GMT

Thank you. I filed a bug, CMPLRLLVM-56396.

Curiously, if I comment out the first call, the second call prints the right answers. I don't see anything obvious in subroutine test_data_map that would cause that.

Re: Error of OpenMP data transfer in GPU offloading

Barbara_P_Intel — Mon, 26 Feb 2024 20:52:17 GMT

Thank you. I filed a bug report, CMPLRLLVM-56399, for this case, too.

Re: Error of OpenMP data transfer in GPU offloading

Barbara_P_Intel — Mon, 26 Feb 2024 21:31:41 GMT

Thanks for this complete reproducer. I filed CMPLRLLVM-56400 on your behalf.

Re:Error of OpenMP data transfer in GPU offloading

Barbara_P_Intel — Mon, 15 Apr 2024 15:53:26 GMT

Look for a fix for the 3rd test case in ifx 2024.2.0. It is planned to be released in mid-2024.