Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.
28963 Discussions

Error of OpenMP data transfer in GPU offloading

Longxiang_Li
Novice
3,382 Views

Dear all,

Currently, we are using the IFX compiler to migrate our code to run on the GPU. We found that ifx supports OpenMP 5.1 spec and could declare a mapper to control the data transfer processes for the user-defined type variables. However, we found the declared mapper could not work correctly as we expected. 

There are mainly two problems:

1. The map directive with the mapper will cash at the second usage.

For example, we use a user-defined type 'struct_test', and declare the corresponding mapper as follows:

   type struct_test
      integer           :: len
      integer, pointer  :: int_ptr(:)
      real,    pointer  :: float_ptr(:)
   end type struct_test

   !$omp declare mapper ( struct_mapper: struct_test :: var ) &
   !$omp&         map( var%len, var%int_ptr, var%float_ptr )

We also define a test function and check whether the array components could remain the same with the map directives.

   subroutine test_data_map(len, init_val)
      integer :: init_val, len

      ! local var
      type(struct_test) :: test

      call init_struct(test, len, init_val, float(init_val))

      write(*, *)
      !============================================================
      ! test omp offloading with int vector from structure
      !============================================================
      !$omp target enter data map(mapper(struct_mapper), to: test)

      test%int_ptr = 1

      !$omp target exit data map(mapper(struct_mapper), from: test)

      write(*, *) "test with data structure"
      write(*, *) "test%int_ptr = ", test%int_ptr

      !============================================================
      ! test omp offloading with float vector from structure
      !============================================================
      !$omp target enter data map(mapper(struct_mapper), to: test)

      test%float_ptr = 2

      !$omp target exit data map(mapper(struct_mapper), from: test)

      write(*, *) "test with data structure"
      write(*, *) "test%float_ptr = ", test%float_ptr


      call delete_struct(test)
   end subroutine test_data_map

When we called the test function the second time, the program crashed with the following error message.

$ ./test.exe
 
 test with data structure
 test%int_ptr =            2           3           4           5           6
 test with data structure
 test%float_ptr =    2.000000       3.000000       4.000000       5.000000    
   6.000000    
 
Libomptarget message: explicit extension not allowed: host address specified is 0x00007ffc2bfda598 (152 bytes), but device allocation maps to host at 0x00007ffc2bfda588 (152 bytes)
Libomptarget error: Call to getTargetPointer returned null pointer (device failure or illegal mapping).
Libomptarget error: Call to targetDataBegin via targetDataMapper for custom mapper failed.
Libomptarget error: Run with
Libomptarget error: LIBOMPTARGET_DEBUG=1 to display basic debug information.
Libomptarget error: LIBOMPTARGET_DEBUG=2 to display calls to the compute runtime.
Libomptarget error: LIBOMPTARGET_INFO=4 to dump host-target pointer mappings.
Libomptarget error: Source location information not present. Compile with -g or -gline-tables-only.
Libomptarget fatal error 1: failure of target construct while offloading is mandatory
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
libpthread-2.31.s  00007F9EEAEB3420  Unknown               Unknown  Unknown
libc-2.31.so       00007F9EEACF000B  gsignal               Unknown  Unknown
libc-2.31.so       00007F9EEACCF859  abort                 Unknown  Unknown
libomptarget.so    00007F9EEB1A6CB8  Unknown               Unknown  Unknown
libomptarget.so    00007F9EEB19CBFC  __tgt_target_data     Unknown  Unknown
test.exe           0000000000405A15  Unknown               Unknown  Unknown
test.exe           000000000040527D  Unknown               Unknown  Unknown
libc-2.31.so       00007F9EEACD1083  __libc_start_main     Unknown  Unknown
test.exe           000000000040519E  Unknown               Unknown  Unknown
[1]    5290 abort      ./test.exe

2. The map directive with mapper could not work correctly for the user-defined type arrays.

We also checked if the mapper could work functionally with the user-defined type array. For example, we extend the test function as follows:

   subroutine test_data_array(len_array, len, init_val)
      integer :: len_array, len, init_val

      ! local var
      type(struct_test), allocatable :: test(:)
      integer :: i

      allocate(test(len_array))
      do i = 1, len_array
         call init_struct(test(i), len, init_val + len_array, float(init_val + len_array))
         !$omp target enter data map(mapper(struct_mapper), to: test(i))
      end do

      do i = 1, len_array
         test(i)%int_ptr = 0
         test(i)%float_ptr = 1
      end do

      do i = 1, len_array
         !$omp target exit data map(mapper(struct_mapper), from: test(i))
      end do

      write(*, *)
      write(*, *) "test with data structure array, len_array = ", len_array
      write(*, *) "test(", len_array, ")%int_ptr = ", test(len_array)%int_ptr

      do i = 1, len_array
         call delete_struct(test(i))
      end do
      deallocate(test)
   end subroutine test_data_array

 We could not get the initial values in the array components of 'int_ptr' and 'float_ptr'. It means the map directives do not work correctly.

$ ./test.exe
 
 test with data structure array, len_array =            2
 test(           2 )%int_ptr =            0           0           0           0
           0
 
 test with data structure array, len_array =            2
 test(           2 )%int_ptr =            0           0           0           0
           0

 

 

Labels (2)
0 Kudos
16 Replies
Barbara_P_Intel
Employee
3,299 Views

Thank you for the reproducer. Please tell me the output you expect from each of the two test cases.

What version of ifx are you using?

I am using the version of the compiler that is planned to be released in 6-8 weeks. Your first test case does not abort. Good news!

 

 

0 Kudos
Longxiang_Li
Novice
3,271 Views

Hi Barbara,

I am using the latest version of the ifx 2024.0.2.

Could you help check if the new compiler could work functionally for the second test? 

Thank you

0 Kudos
Ron_Green
Moderator
3,285 Views

@Longxiang_Li we are still doing some work with mappers in the compiler.  Make sure to upgrade to 2024.1, Update 1, as soon as it is available.  There are many important fixes in 2024.1 for OMP mapper

0 Kudos
Longxiang_Li
Novice
3,271 Views

Thanks for your suggestion.

I found the latest version of the ifx compiler is still 2024.0.2  (https://www.intel.com/content/www/us/en/developer/articles/tool/oneapi-standalone-components.html#fortran). Where could I get the compiler of 2024.1 version?

0 Kudos
Barbara_P_Intel
Employee
3,238 Views

ifx 2024.1.0 is not yet released. Look for it in 6-8 weeks.

I tried both of your tests with a preview version of ifx 2024.1.0. Please tell me the expected output for each test. I know what I think the output should be, but it may not match your expectations.

If there are still issues, I will let you know and get bug reports filed.

 

0 Kudos
Longxiang_Li
Novice
3,216 Views

Hi Barbara,

In the main function, the test functions are called with the following parameters:

    call test_data_map(5, 2) 
    call test_data_array(2, 5, 2)

 The expected result of the first test function should be

 test with data structure
 test%int_ptr =            2           3           4           5           6
 test with data structure
 test%float_ptr =    2.000000       3.000000       4.000000       5.000000    
   6.000000 

The result of the second test function should be 

 test with data structure array, len_array =            2
 test(           1 )%int_ptr =            4           5           6           7
           8
 test(           1 )%float_ptr =    4.000000       5.000000       6.000000    
   7.000000       8.000000
 test(           2 )%int_ptr =            4           5           6           7
           8
 test(           2 )%float_ptr =    4.000000       5.000000       6.000000    
   7.000000       8.000000

 

To find the correct result, I have added the test function `test_data_array_map`, which utilized the map directive instead of the mapper. However, there is another error with mapping the structure arrays. The test function is shown as follows.

   subroutine test_data_array_map(len_array, len, init_val)
      integer :: len_array, len, init_val

      ! local var
      type(struct_test), allocatable :: test(:)
      integer :: i

      allocate (test(len_array))
      do i = 1, len_array
         call init_struct(test(i), len, init_val + len_array, float(init_val + len_array))
         !$omp target enter data map(to: test(i)%int_ptr, test(i)%float_ptr)
      end do

      do i = 1, len_array
         test(i)%int_ptr = 0
         test(i)%float_ptr = 1
      end do

      do i = 1, len_array
         !$omp target exit data map(from: test(i)%int_ptr, test(i)%float_ptr)
      end do

      write (*, *)
      write (*, *) "test with data structure array, len_array = ", len_array
      do i = 1, len_array
         write (*, *) "test(", i, ")%int_ptr = ", test(i)%int_ptr
         write (*, *) "test(", i, ")%float_ptr = ", test(i)%float_ptr
      end do

      do i = 1, len_array
         call delete_struct(test(i))
      end do
      deallocate (test)
   end subroutine test_data_array_map

After mapping the test array from device to host, only the first structure element gets the correct initial values, whereas the other elements' pointer array %int_ptr and %float_ptr is empty. 

 test with data structure array, len_array =            2
 test(           1 )%int_ptr =            4           5           6           7
           8
 test(           1 )%float_ptr =    4.000000       5.000000       6.000000    
   7.000000       8.000000    
 test(           2 )%int_ptr = 
 test(           2 )%float_ptr = 

Is this also a bug for the map directive? Could you help to check if this error could be solved in 2024.1 update? 

 

Best wishes

Longxiang Li

0 Kudos
Barbara_P_Intel
Employee
3,171 Views

Thank you for the additional information.

This is getting confusing with 3 different test cases. I created a separate file for each case. I'll post info on each in a separate post.

 

For the first case, attached is test.data.map.f90. According to your test.f90, test_data_map is called twice.

    call test_data_map(5, 2)
    call test_data_map(3, 4)

Using the preview ifx 2024.1.0 I get this output. The first call to test_data_map matches the output you sent.  Is the output from the second call correct?

 test with data structure
 test%int_ptr =            2           3           4           5           6
 test with data structure
 test%float_ptr =    2.000000       3.000000       4.000000       5.000000
   6.000000

 test with data structure
 test%int_ptr =            0           0           4
 test with data structure
 test%float_ptr =    2.000000       3.000000       4.000000

 

0 Kudos
Longxiang_Li
Novice
3,128 Views

The second call gets some errors in the result. 

In the parameters of `test_data_map`, the first represents the length of the vector and the second represents the initial value of the vector '%int_ptr' and `%float_ptr`. According to the input parameters, there are two errors in the output.

1. In the second call of `test_data_map`, the start value of the vector should be 4, but the two vectors still start from 2.

2. The vectors `%int_ptr` and `%float_ptr` should have the same values in each element, but there are many zeros in the second output of `%int_ptr`.

0 Kudos
Barbara_P_Intel
Employee
3,016 Views

Thank you. I filed a bug, CMPLRLLVM-56396.

Curiously, if I comment out the first call, the second call prints the right answers. I don't see anything obvious in subroutine test_data_map that would cause that.

 

0 Kudos
Barbara_P_Intel
Employee
3,165 Views

For the second test, I created test.data.array.f90 (attached) from your test.f90. I think there's a mismatch between what you sent for this subroutine and the subroutine that printed the results you sent. Please confirm.

Three things:

(1) The version of subroutine test.data.array that you sent does not have write statements for test%float_ptr. Since your output prints that I added the appropriate write statements. 

(2) The main routine has these calls for test_data_array. Those subroutine calls don't match the output you sent. Please confirm.

   call test_data_array(2, 5, 2)
   call test_data_array(2, 5, 2)

(3) The version of subroutine test.data.array that you sent sets the array test%int_ptr to 0 and the array test%float_ptr to 1.

0 Kudos
Longxiang_Li
Novice
3,124 Views

Hi,

I have added some output commands to show the correct initial value and all the array `test` elements. Please use this source code for checking.

 

The correct value of the structure elements `test(1)` and `test(2)` should start from 2 and 7 respectively. I will explain the reason.

1. The structure element is initialized inside the `init_struct` subroutine with the given initial value, and these initial values are copied to the device. (I have edited the source code to assign different initial values for the structure elements)

2. The assignment of the vectors with `test(i)%int_ptr = 0` and `test(i)%float_ptr = 1` is conducted by the host, so the device still gets the initial values.

3. After exiting the data region, the device values are copied back to the host. Therefore, the vectors of the structure elements get their initial values.

 

I used the compiler ifx 2024.0.2 to run the test function, the results are far from the correct values.

$ ./test.data.array.exe

 Initial value of test(           1 ) is            2
 Initial value of test(           2 ) is            7

 test with data structure array, len_array =            2
 test(           1 )
   %int_ptr =            0           0           0           0           0
   %float_ptr =    1.000000       1.000000       1.000000       1.000000
   1.000000
 test(           2 )
   %int_ptr =            0           0           0           0           0
   %float_ptr =    1.000000       1.000000       1.000000       1.000000
   1.000000

 Initial value of test(           1 ) is            2
 Initial value of test(           2 ) is            7

 test with data structure array, len_array =            2
 test(           1 )
   %int_ptr =            0           0           0           0           0
   %float_ptr =    1.000000       1.000000       1.000000       1.000000
   1.000000
 test(           2 )
   %int_ptr =            0           0           0           0           0
   %float_ptr =    1.000000       1.000000       1.000000       1.000000
   1.000000

 

0 Kudos
Barbara_P_Intel
Employee
3,011 Views

Thank you. I filed a bug report, CMPLRLLVM-56399, for this case, too.

 

0 Kudos
Barbara_P_Intel
Employee
3,159 Views

For the 3rd test case, please provide the statements that call test_data_array_map. The arguments are key.

 

0 Kudos
Longxiang_Li
Novice
3,123 Views

I have added a new file to check with the test_data_array_map function.

In this function, I discard the mapper directive and utilize the map directive to copy the vectors of each structure element between the host and the device. The result shows the map directive also could not work functionally with the structure array.

My check results are

$ ./test.data.map.array.exe

 Initial value of test(           1 ) is            2
 Initial value of test(           2 ) is            7

 test with data structure array, len_array =            2
 test(           1 )
   %int_ptr =
   %float_ptr =
 test(           2 )
   %int_ptr =
   %float_ptr =

 Initial value of test(           1 ) is            2
 Initial value of test(           2 ) is            7

 test with data structure array, len_array =            2
 test(           1 )
   %int_ptr =
   %float_ptr =
 test(           2 )
   %int_ptr =
   %float_ptr =
0 Kudos
Barbara_P_Intel
Employee
3,005 Views

Thanks for this complete reproducer. I filed CMPLRLLVM-56400 on your behalf.

 

0 Kudos
Barbara_P_Intel
Employee
2,262 Views

Look for a fix for the 3rd test case in ifx 2024.2.0. It is planned to be released in mid-2024.



0 Kudos
Reply