topic How to use IFX and offload openMP to GPU? in Intel® oneAPI Math Kernel Library

How to use IFX and offload openMP to GPU?

CRquantum — Mon, 04 Apr 2022 18:39:31 GMT

While looking at MKL examples, I notice that intel new Fortran Compiler IFX seems can offload openMP at least to Intel's GPU.
I mean say I have a xeon 2186M and with intel P630 GPU inside the chip, it seems IFX should be able to offload openMP to intel's GPU,

https://www.intel.com/content/www/us/en/develop/documentation/get-started-with-cpp-fortran-compiler-openmp/top.html

I use Intel OneAPI 2022.0.3 + visual studio 2019 on windows, and I tried to compile and run the offload examples in Intel OneAPI MKL examples in the examples folder,
```
C:\Program Files (x86)\Intel\oneAPI\mkl\2022.0.3\examples
```

The example I am trying the run openMP offload to intel GPU is ```vsinh.f90``` located at,
```
C:\Program Files (x86)\Intel\oneAPI\mkl\2022.0.3\examples\examples_offload_f\f_offload\vml\source
```

However, it seems if I enable openMP offload as below,

CRquantum_0-1649097472065.png

it just give error at linking stage. However I am not sure if I what I set at linking stage is correct or not, below is what I set,

CRquantum_1-1649097503748.png

The error at linking is below,
```
Build started...
1>------ Build started: Project: MKL_test (IFX), Configuration: Release x64 ------
1>Linking...
1>Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2022.0.0 Build 20211123
1>Copyright (C) 1985-2021 Intel Corporation. All rights reserved.
1>2828013-vsinh.obj : warning LNK4078: multiple '__CLANG_OFFLOAD_BUNDLE__openmp-s' sections found with different attributes (40000800)
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VMLSETMODE_OMP_OFFLOAD_ILP64 referenced in function TEST_FLOAT
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VSSINH_OMP_OFFLOAD_ILP64 referenced in function TEST_FLOAT
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VMSSINH_OMP_OFFLOAD_ILP64 referenced in function TEST_FLOAT
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VSSINHI_OMP_OFFLOAD_ILP64 referenced in function TEST_FLOAT
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VMSSINHI_OMP_OFFLOAD_ILP64 referenced in function TEST_FLOAT
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VDSINH_OMP_OFFLOAD_ILP64 referenced in function TEST_DOUBLE
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VMDSINH_OMP_OFFLOAD_ILP64 referenced in function TEST_DOUBLE
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VDSINHI_OMP_OFFLOAD_ILP64 referenced in function TEST_DOUBLE
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VMDSINHI_OMP_OFFLOAD_ILP64 referenced in function TEST_DOUBLE
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VCSINH_OMP_OFFLOAD_ILP64 referenced in function TEST_FLOAT_COMPLEX
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VMCSINH_OMP_OFFLOAD_ILP64 referenced in function TEST_FLOAT_COMPLEX
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VCSINHI_OMP_OFFLOAD_ILP64 referenced in function TEST_FLOAT_COMPLEX
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VMCSINHI_OMP_OFFLOAD_ILP64 referenced in function TEST_FLOAT_COMPLEX
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VZSINH_OMP_OFFLOAD_ILP64 referenced in function TEST_DOUBLE_COMPLEX
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VMZSINH_OMP_OFFLOAD_ILP64 referenced in function TEST_DOUBLE_COMPLEX
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VZSINHI_OMP_OFFLOAD_ILP64 referenced in function TEST_DOUBLE_COMPLEX
1>2828013-vsinh.obj : error LNK2019: unresolved external symbol MKL_VM_VMZSINHI_OMP_OFFLOAD_ILP64 referenced in function TEST_DOUBLE_COMPLEX
1>x64\Release\MKL_test.exe : fatal error LNK1120: 17 unresolved externals
1>
```

However if I disable offload like below then the code can compile and link (with some minor warnings) and runs fine.

CRquantum_2-1649097523345.png

But I just do not know how to corrected set in Visual Studio or in command line how to let IFX offload to GPU and run the code correctly.

Just curious, does anyone knows how to use IFX and offload openMP to GPU?

Thanks much in advance!

PS.

A similar post is here too,

https://fortran-lang.discourse.group/t/how-to-use-ifx-and-offload-openmp-to-gpu/3104

Re: How to use IFX and offload openMP to GPU?

Ron_Green — Tue, 05 Apr 2022 00:52:59 GMT

I have not tried the MKL offload feature. So I will move this question to the MKL User Forum for them to answer.

Just a quick question:

did you install the correct driver for your system from the DRIVERS DOWNLOADS
Matching the right driver to your "generation" of processor can be a problem, but there is a link below the list for "let us identify your product" that will detect your system and find the right driver.

ANd did you install the oneAPI Base Toolkit and the HPC Toolkit or just the HPC Toolkit?

OR did you use the component installers for Fortran and MKL?

I'll move this issue to MKL Forum.

ron

Re: How to use IFX and offload openMP to GPU?

CRquantum — Tue, 05 Apr 2022 04:22:58 GMT

Thank you Ron!

1. My laptop is ThinkPad P72 with Xeon-2186M, it is with Intel UHD P630. Yes, I have installed the Intel Graphics - WIndows DCH Driver as below,

https://www.intel.com/content/www/us/en/support/products/126790/graphics/graphics-for-8th-generation-intel-processors/intel-uhd-graphics-630.html#drivers-software

But I can try reinstall it.

2. I have installed both Intel OneAPI base toolkit and HPC toolkit. I have installed all the components in base toolkit and HPC toolkit. They are integrated with Visual Studo 2019 Enterprise.

Thank you Ron for moving it to MKL. Yeah, I think at least in principle, IFX seems should be able to offload openMP to Intel's GPU. Especially usually Intel's CPU and Intel GPU packed inside in chip. Also all the offload examples are included in the OneAPI installation folder. So intel perhaps have tested them already.

If there are more examples that perhaps could show some instructions step-by-step it will be great!

Best regards,

Rong

Re: How to use IFX and offload openMP to GPU?

VidyalathaB_Intel — Wed, 06 Apr 2022 06:23:39 GMT

Hi Rong,

Thanks for reaching out to us.

We suggest you run your code from Intel oneAPI command prompt and take help from the link line advisor (https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl-link-line-advisor.html#gs.vvmn6g) which is a great tool for recommending the necessary options such as compiling and linking options according to your environment (please check the box of the option Enable OpenMP offload feature to GPU and select Intel Fortran compiler beta).

I have also tried to run the sample code vsinh.f90 from the Intel command prompt with the following commands and could see that it got executed successfully.

Compilation command:

ifx -c /Qiopenmp /Qopenmp-targets:spir64 /DMKL_ILP64 /4I8 -I"%MKLROOT%\include" /MD /fpp -I"C:\Program Files (x86)\Intel\oneAPI\mkl\2022.0.2\examples\examples_offload_f\f_offload\vml\source" Vsinh.f90

Linking Command:

ifx Vsinh.obj /Qiopenmp /Qopenmp-targets:spir64 -fsycl mkl_sycl.lib mkl_intel_ilp64.lib mkl_intel_thread.lib mkl_core.lib libiomp5md.lib OpenCL.lib sycl.lib

Now you can see that the executable got generated successfully named *.exe.

VidyalathaB_Intel_2-1649225932108.png

Hope the provided information helps in resolving the issue.

Regards,

Vidya.

Re: How to use IFX and offload openMP to GPU?

CRquantum — Thu, 07 Apr 2022 07:29:26 GMT

Thank you very much Vidya! The link line advisor is very useful! Cool.

I was able to complie and link and run. Thanks.

Just one more comment, in windows with OneAPI, need to make sure

C:\Program Files (x86)\Intel\oneAPI

is in the system environment path.

Then in any cmd window, need to type

setvars

first, then it will load Intel's some variables in the cmd, I mean cmd will show,

```

:: initializing oneAPI environment...
Initializing Visual Studio command-line environment...
Visual Studio version 17.1.3 environment configured.
"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\"
Visual Studio command-line environment initialized for: 'x64'
: advisor -- latest
: compiler -- latest
: dal -- latest
: debugger -- latest
: dev-utilities -- latest
: dnnl -- latest
: dpcpp-ct -- latest
: dpl -- latest
: inspector -- latest
: intelpython -- latest
: ipp -- latest
: ippcp -- latest
: itac -- latest
: mkl -- latest
: mpi -- latest
: tbb -- latest
: vpl -- latest
: vtune -- latest
:: oneAPI environment initialized ::

```

Then can type the compile and link command you mentioned.

However there some warning messages after the compile command,

ifx -c /Qiopenmp /Qopenmp-targets:spir64 /DMKL_ILP64 /4I8 -I"%MKLROOT%\include" /MD /fpp vsinh.f90

Message is below

```

vsinh.f90(99): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------^
vsinh.f90(99): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRES1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------------^
vsinh.f90(105): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------^
vsinh.f90(105): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRES1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------------^
vsinh.f90(111): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------^
vsinh.f90(111): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------------^
vsinh.f90(117): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------^
vsinh.f90(117): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------------^
vsinh.f90(214): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------^
vsinh.f90(214): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRES1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------------^
vsinh.f90(220): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------^
vsinh.f90(220): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRES1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------------^
vsinh.f90(226): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------^
vsinh.f90(226): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------------^
vsinh.f90(232): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------^
vsinh.f90(232): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------------^
vsinh.f90(329): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------^
vsinh.f90(329): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRES1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------------^
vsinh.f90(335): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------^
vsinh.f90(335): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRES1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------------^
vsinh.f90(341): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------^
vsinh.f90(341): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------------^
vsinh.f90(347): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------^
vsinh.f90(347): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------------^
vsinh.f90(444): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------^
vsinh.f90(444): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRES1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------------^
vsinh.f90(450): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------^
vsinh.f90(450): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRES1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------------^
vsinh.f90(456): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------^
vsinh.f90(456): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------------^
vsinh.f90(462): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------^
vsinh.f90(462): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------------^
vsinh.f90(99): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------^
vsinh.f90(99): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRES1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------------^
vsinh.f90(105): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------^
vsinh.f90(105): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRES1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------------^
vsinh.f90(111): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------^
vsinh.f90(111): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------------^
vsinh.f90(117): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------^
vsinh.f90(117): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------------^
vsinh.f90(214): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------^
vsinh.f90(214): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRES1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------------^
vsinh.f90(220): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------^
vsinh.f90(220): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRES1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------------^
vsinh.f90(226): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------^
vsinh.f90(226): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------------^
vsinh.f90(232): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------^
vsinh.f90(232): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------------^
vsinh.f90(329): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------^
vsinh.f90(329): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRES1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------------^
vsinh.f90(335): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------^
vsinh.f90(335): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRES1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------------^
vsinh.f90(341): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------^
vsinh.f90(341): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------------^
vsinh.f90(347): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------^
vsinh.f90(347): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------------^
vsinh.f90(444): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------^
vsinh.f90(444): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRES1]
!$omp target variant dispatch use_device_ptr(varg1,vres1)
---------------------------------------------------^
vsinh.f90(450): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------^
vsinh.f90(450): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRES1]
!$omp target variant dispatch use_device_ptr(varg1,vmres1)
---------------------------------------------------^
vsinh.f90(456): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------^
vsinh.f90(456): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vresi1)
---------------------------------------------------^
vsinh.f90(462): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VARG1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------^
vsinh.f90(462): remark #8980: A list item in an is_device_ptr or a use_device_ptr clause must be of type C_PTR. The support is deprecated otherwise. [VMRESI1]
!$omp target variant dispatch use_device_ptr(varg1,vmresi1)
---------------------------------------------------^

```

After typing the linking command,

ifx Vsinh.obj /Qiopenmp /Qopenmp-targets:spir64 -fsycl mkl_sycl.lib mkl_intel_ilp64.lib mkl_intel_thread.lib mkl_core.lib libiomp5md.lib OpenCL.lib sycl.lib

At the end of linking there some warnings,

```

...

-defaultlib:omptarget.lib
1255624-Vsinh.obj : warning LNK4078: multiple '__CLANG_OFFLOAD_BUNDLE__openmp-s' sections found with different attributes (40000800)
1255657-libsycl-complex.obj : warning LNK4078: multiple '__CLANG_OFFLOAD_BUNDLE__sycl-spi' sections found with different attributes (40000800)
Creating library Vsinh.lib and object Vsinh.exp

```

But nonetheless the code seems working, output windows is,

```

Running sinh functions:
Running sinh with single precision real data type:
vsinh<HA, simple>[1]( 6.8037548 ) = 450.61188 ; expected = 450.61188 ; ulp = 0.00 :PASS
vmsinh<HA, simple>[1]( 6.8037548 ) = 450.61188 ; expected = 450.61188 ; ulp = 0.00 :PASS
vsinhi<HA, strided>[1]( 6.8037548 ) = 450.61188 ; expected = 450.61188 ; ulp = 0.00 :PASS
vmsinhi<HA, strided>[1]( 6.8037548 ) = 450.61188 ; expected = 450.61188 ; ulp = 0.00 :PASS
vsinh<HA, simple>[2]( -2.1123414 ) = -4.0733109 ; expected = -4.0733109 ; ulp = 0.00 :PASS
vmsinh<HA, simple>[2]( -2.1123414 ) = -4.0733109 ; expected = -4.0733109 ; ulp = 0.00 :PASS
vsinhi<HA, strided>[2]( -2.1123414 ) = 777.00000 ; expected = 777.00000 ; ulp = 0.00 :PASS
vmsinhi<HA, strided>[2]( -2.1123414 ) = 777.00000 ; expected = 777.00000 ; ulp = 0.00 :PASS
vsinh<HA, simple>[3]( 5.6619844 ) = 143.85779 ; expected = 143.85779 ; ulp = 0.00 :PASS
vmsinh<HA, simple>[3]( 5.6619844 ) = 143.85779 ; expected = 143.85779 ; ulp = 0.00 :PASS
vsinhi<HA, strided>[3]( 5.6619844 ) = 143.85779 ; expected = 143.85779 ; ulp = 0.00 :PASS
vmsinhi<HA, strided>[3]( 5.6619844 ) = 143.85779 ; expected = 143.85779 ; ulp = 0.00 :PASS
vsinh<HA, simple>[4]( 5.9688005 ) = 195.51691 ; expected = 195.51691 ; ulp = 0.00 :PASS
vmsinh<HA, simple>[4]( 5.9688005 ) = 195.51691 ; expected = 195.51691 ; ulp = 0.00 :PASS
vsinhi<HA, strided>[4]( 5.9688005 ) = 777.00000 ; expected = 777.00000 ; ulp = 0.00 :PASS
vmsinhi<HA, strided>[4]( 5.9688005 ) = 777.00000 ; expected = 777.00000 ; ulp = 0.00 :PASS
vsinh<LA, simple>[1]( 6.8037548 ) = 450.61188 ; expected = 450.61188 ; ulp = 0.00 :PASS
vmsinh<LA, simple>[1]( 6.8037548 ) = 450.61188 ; expected = 450.61188 ; ulp = 0.00 :PASS
vsinhi<LA, strided>[1]( 6.8037548 ) = 450.61188 ; expected = 450.61188 ; ulp = 0.00 :PASS
vmsinhi<LA, strided>[1]( 6.8037548 ) = 450.61188 ; expected = 450.61188 ; ulp = 0.00 :PASS
vsinh<LA, simple>[2]( -2.1123414 ) = -4.0733109 ; expected = -4.0733109 ; ulp = 0.00 :PASS
vmsinh<LA, simple>[2]( -2.1123414 ) = -4.0733109 ; expected = -4.0733109 ; ulp = 0.00 :PASS
vsinhi<LA, strided>[2]( -2.1123414 ) = 777.00000 ; expected = 777.00000 ; ulp = 0.00 :PASS
vmsinhi<LA, strided>[2]( -2.1123414 ) = 777.00000 ; expected = 777.00000 ; ulp = 0.00 :PASS
vsinh<LA, simple>[3]( 5.6619844 ) = 143.85779 ; expected = 143.85779 ; ulp = 0.00 :PASS
vmsinh<LA, simple>[3]( 5.6619844 ) = 143.85779 ; expected = 143.85779 ; ulp = 0.00 :PASS
vsinhi<LA, strided>[3]( 5.6619844 ) = 143.85779 ; expected = 143.85779 ; ulp = 0.00 :PASS
vmsinhi<LA, strided>[3]( 5.6619844 ) = 143.85779 ; expected = 143.85779 ; ulp = 0.00 :PASS
vsinh<LA, simple>[4]( 5.9688005 ) = 195.51691 ; expected = 195.51691 ; ulp = 0.00 :PASS
vmsinh<LA, simple>[4]( 5.9688005 ) = 195.51691 ; expected = 195.51691 ; ulp = 0.00 :PASS
vsinhi<LA, strided>[4]( 5.9688005 ) = 777.00000 ; expected = 777.00000 ; ulp = 0.00 :PASS
vmsinhi<LA, strided>[4]( 5.9688005 ) = 777.00000 ; expected = 777.00000 ; ulp = 0.00 :PASS
vsinh<EP, simple>[1]( 6.8037548 ) = 450.61377 ; expected = 450.61188 ; ulp = 62.0 :PASS
vmsinh<EP, simple>[1]( 6.8037548 ) = 450.61377 ; expected = 450.61188 ; ulp = 62.0 :PASS
vsinhi<EP, strided>[1]( 6.8037548 ) = 450.61377 ; expected = 450.61188 ; ulp = 62.0 :PASS
vmsinhi<EP, strided>[1]( 6.8037548 ) = 450.61377 ; expected = 450.61188 ; ulp = 62.0 :PASS
vsinh<EP, simple>[2]( -2.1123414 ) = -4.0733027 ; expected = -4.0733109 ; ulp = 17.0 :PASS
vmsinh<EP, simple>[2]( -2.1123414 ) = -4.0733027 ; expected = -4.0733109 ; ulp = 17.0 :PASS
vsinhi<EP, strided>[2]( -2.1123414 ) = 777.00000 ; expected = 777.00000 ; ulp = 0.00 :PASS
vmsinhi<EP, strided>[2]( -2.1123414 ) = 777.00000 ; expected = 777.00000 ; ulp = 0.00 :PASS
vsinh<EP, simple>[3]( 5.6619844 ) = 143.85716 ; expected = 143.85779 ; ulp = 41.0 :PASS
vmsinh<EP, simple>[3]( 5.6619844 ) = 143.85716 ; expected = 143.85779 ; ulp = 41.0 :PASS
vsinhi<EP, strided>[3]( 5.6619844 ) = 143.85716 ; expected = 143.85779 ; ulp = 41.0 :PASS
vmsinhi<EP, strided>[3]( 5.6619844 ) = 143.85716 ; expected = 143.85779 ; ulp = 41.0 :PASS
vsinh<EP, simple>[4]( 5.9688005 ) = 195.51376 ; expected = 195.51691 ; ulp = 206. :PASS
vmsinh<EP, simple>[4]( 5.9688005 ) = 195.51376 ; expected = 195.51691 ; ulp = 206. :PASS
vsinhi<EP, strided>[4]( 5.9688005 ) = 777.00000 ; expected = 777.00000 ; ulp = 0.00 :PASS
vmsinhi<EP, strided>[4]( 5.9688005 ) = 777.00000 ; expected = 777.00000 ; ulp = 0.00 :PASS
sinh single precision real result: PASS
Running sinh with double precision real data type:
vsinh<HA, simple>[1]( 6.803754343094191 ) = 450.6116721871068 ; expected = 450.6116721871068 ; ulp = 0.00 :PASS
vmsinh<HA, simple>[1]( 6.803754343094191 ) = 450.6116721871068 ; expected = 450.6116721871068 ; ulp = 0.00 :PASS
vsinhi<HA, strided>[1]( 6.803754343094191 ) = 450.6116721871068 ; expected = 450.6116721871068 ; ulp = 0.00 :PASS
vmsinhi<HA, strided>[1]( 6.803754343094191 ) = 450.6116721871068 ; expected = 450.6116721871068 ; ulp = 0.00 :PASS
vsinh<HA, simple>[2]( -2.112341463618139 ) = -4.073311222615664 ; expected = -4.073311222615664 ; ulp = 0.00 :PASS
vmsinh<HA, simple>[2]( -2.112341463618139 ) = -4.073311222615664 ; expected = -4.073311222615664 ; ulp = 0.00 :PASS
vsinhi<HA, strided>[2]( -2.112341463618139 ) = 777.0000000000000 ; expected = 777.0000000000000 ; ulp = 0.00 :PASS
vmsinhi<HA, strided>[2]( -2.112341463618139 ) = 777.0000000000000 ; expected = 777.0000000000000 ; ulp = 0.00 :PASS
vsinh<HA, simple>[3]( 5.661984475172117 ) = 143.8577860426789 ; expected = 143.8577860426789 ; ulp = 0.00 :PASS
vmsinh<HA, simple>[3]( 5.661984475172117 ) = 143.8577860426789 ; expected = 143.8577860426789 ; ulp = 0.00 :PASS
vsinhi<HA, strided>[3]( 5.661984475172117 ) = 143.8577860426789 ; expected = 143.8577860426789 ; ulp = 0.00 :PASS
vmsinhi<HA, strided>[3]( 5.661984475172117 ) = 143.8577860426789 ; expected = 143.8577860426789 ; ulp = 0.00 :PASS
vsinh<HA, simple>[4]( 5.968800669521466 ) = 195.5169251084481 ; expected = 195.5169251084481 ; ulp = 0.00 :PASS
vmsinh<HA, simple>[4]( 5.968800669521466 ) = 195.5169251084481 ; expected = 195.5169251084481 ; ulp = 0.00 :PASS
vsinhi<HA, strided>[4]( 5.968800669521466 ) = 777.0000000000000 ; expected = 777.0000000000000 ; ulp = 0.00 :PASS
vmsinhi<HA, strided>[4]( 5.968800669521466 ) = 777.0000000000000 ; expected = 777.0000000000000 ; ulp = 0.00 :PASS
vsinh<LA, simple>[1]( 6.803754343094191 ) = 450.6116721871068 ; expected = 450.6116721871068 ; ulp = 0.00 :PASS
vmsinh<LA, simple>[1]( 6.803754343094191 ) = 450.6116721871068 ; expected = 450.6116721871068 ; ulp = 0.00 :PASS
vsinhi<LA, strided>[1]( 6.803754343094191 ) = 450.6116721871068 ; expected = 450.6116721871068 ; ulp = 0.00 :PASS
vmsinhi<LA, strided>[1]( 6.803754343094191 ) = 450.6116721871068 ; expected = 450.6116721871068 ; ulp = 0.00 :PASS
vsinh<LA, simple>[2]( -2.112341463618139 ) = -4.073311222615663 ; expected = -4.073311222615664 ; ulp = 1.00 :PASS
vmsinh<LA, simple>[2]( -2.112341463618139 ) = -4.073311222615663 ; expected = -4.073311222615664 ; ulp = 1.00 :PASS
vsinhi<LA, strided>[2]( -2.112341463618139 ) = 777.0000000000000 ; expected = 777.0000000000000 ; ulp = 0.00 :PASS
vmsinhi<LA, strided>[2]( -2.112341463618139 ) = 777.0000000000000 ; expected = 777.0000000000000 ; ulp = 0.00 :PASS
vsinh<LA, simple>[3]( 5.661984475172117 ) = 143.8577860426789 ; expected = 143.8577860426789 ; ulp = 1.00 :PASS
vmsinh<LA, simple>[3]( 5.661984475172117 ) = 143.8577860426789 ; expected = 143.8577860426789 ; ulp = 1.00 :PASS
vsinhi<LA, strided>[3]( 5.661984475172117 ) = 143.8577860426789 ; expected = 143.8577860426789 ; ulp = 1.00 :PASS
vmsinhi<LA, strided>[3]( 5.661984475172117 ) = 143.8577860426789 ; expected = 143.8577860426789 ; ulp = 1.00 :PASS
vsinh<LA, simple>[4]( 5.968800669521466 ) = 195.5169251084481 ; expected = 195.5169251084481 ; ulp = 0.00 :PASS
vmsinh<LA, simple>[4]( 5.968800669521466 ) = 195.5169251084481 ; expected = 195.5169251084481 ; ulp = 0.00 :PASS
vsinhi<LA, strided>[4]( 5.968800669521466 ) = 777.0000000000000 ; expected = 777.0000000000000 ; ulp = 0.00 :PASS
vmsinhi<LA, strided>[4]( 5.968800669521466 ) = 777.0000000000000 ; expected = 777.0000000000000 ; ulp = 0.00 :PASS
vsinh<EP, simple>[1]( 6.803754343094191 ) = 450.6116721871068 ; expected = 450.6116721871068 ; ulp = 0.00 :PASS
vmsinh<EP, simple>[1]( 6.803754343094191 ) = 450.6116721871068 ; expected = 450.6116721871068 ; ulp = 0.00 :PASS
vsinhi<EP, strided>[1]( 6.803754343094191 ) = 450.6116721871068 ; expected = 450.6116721871068 ; ulp = 0.00 :PASS
vmsinhi<EP, strided>[1]( 6.803754343094191 ) = 450.6116721871068 ; expected = 450.6116721871068 ; ulp = 0.00 :PASS
vsinh<EP, simple>[2]( -2.112341463618139 ) = -4.073311222615663 ; expected = -4.073311222615664 ; ulp = 1.00 :PASS
vmsinh<EP, simple>[2]( -2.112341463618139 ) = -4.073311222615663 ; expected = -4.073311222615664 ; ulp = 1.00 :PASS
vsinhi<EP, strided>[2]( -2.112341463618139 ) = 777.0000000000000 ; expected = 777.0000000000000 ; ulp = 0.00 :PASS
vmsinhi<EP, strided>[2]( -2.112341463618139 ) = 777.0000000000000 ; expected = 777.0000000000000 ; ulp = 0.00 :PASS
vsinh<EP, simple>[3]( 5.661984475172117 ) = 143.8577860426789 ; expected = 143.8577860426789 ; ulp = 1.00 :PASS
vmsinh<EP, simple>[3]( 5.661984475172117 ) = 143.8577860426789 ; expected = 143.8577860426789 ; ulp = 1.00 :PASS
vsinhi<EP, strided>[3]( 5.661984475172117 ) = 143.8577860426789 ; expected = 143.8577860426789 ; ulp = 1.00 :PASS
vmsinhi<EP, strided>[3]( 5.661984475172117 ) = 143.8577860426789 ; expected = 143.8577860426789 ; ulp = 1.00 :PASS
vsinh<EP, simple>[4]( 5.968800669521466 ) = 195.5169251084481 ; expected = 195.5169251084481 ; ulp = 0.00 :PASS
vmsinh<EP, simple>[4]( 5.968800669521466 ) = 195.5169251084481 ; expected = 195.5169251084481 ; ulp = 0.00 :PASS
vsinhi<EP, strided>[4]( 5.968800669521466 ) = 777.0000000000000 ; expected = 777.0000000000000 ; ulp = 0.00 :PASS
vmsinhi<EP, strided>[4]( 5.968800669521466 ) = 777.0000000000000 ; expected = 777.0000000000000 ; ulp = 0.00 :PASS
sinh double precision real result: PASS
Running sinh with single precision complex data type:
vsinh<HA, simple>[1]( -2.1123414 +6.8037548 *i)
= -3.5337441 +2.0861197 *i
expected = -3.5337441 +2.0861197 *i;
ulp = 0.00 :PASS

vmsinh<HA, simple>[1]( -2.1123414 +6.8037548 *i)
= -3.5337441 +2.0861197 *i
expected = -3.5337441 +2.0861197 *i;
ulp = 0.00 :PASS

vsinhi<HA, strided>[1]( -2.1123414 +6.8037548 *i)
= -3.5337441 +2.0861197 *i
expected = -3.5337441 +2.0861197 *i;
ulp = 0.00 :PASS

vmsinhi<HA, strided>[1]( -2.1123414 +6.8037548 *i)
= -3.5337441 +2.0861197 *i
expected = -3.5337441 +2.0861197 *i;
ulp = 0.00 :PASS

vsinh<HA, simple>[2]( 5.9688005 +5.6619844 *i)
= 158.99046 -113.79469 *i
expected = 158.99046 -113.79469 *i;
ulp = 0.00 :PASS

vmsinh<HA, simple>[2]( 5.9688005 +5.6619844 *i)
= 158.99046 -113.79469 *i
expected = 158.99046 -113.79469 *i;
ulp = 0.00 :PASS

vsinhi<HA, strided>[2]( 5.9688005 +5.6619844 *i)
= 777.00000 +777.00000 *i
expected = 777.00000 +777.00000 *i;
ulp = 0.00 :PASS

vmsinhi<HA, strided>[2]( 5.9688005 +5.6619844 *i)
= 777.00000 +777.00000 *i
expected = 777.00000 +777.00000 *i;
ulp = 0.00 :PASS

vsinh<HA, simple>[3]( -6.0489726 +8.2329483 *i)
= 78.371567 +196.80936 *i
expected = 78.371567 +196.80936 *i;
ulp = 0.00 :PASS

vmsinh<HA, simple>[3]( -6.0489726 +8.2329483 *i)
= 78.371567 +196.80936 *i
expected = 78.371567 +196.80936 *i;
ulp = 0.00 :PASS

vsinhi<HA, strided>[3]( -6.0489726 +8.2329483 *i)
= 78.371567 +196.80936 *i
expected = 78.371567 +196.80936 *i;
ulp = 0.00 :PASS

vmsinhi<HA, strided>[3]( -6.0489726 +8.2329483 *i)
= 78.371567 +196.80936 *i
expected = 78.371567 +196.80936 *i;
ulp = 0.00 :PASS

vsinh<HA, simple>[4]( 5.3645916 -3.2955451 *i)
= -105.58589 +16.385576 *i
expected = -105.58589 +16.385576 *i;
ulp = 0.00 :PASS

vmsinh<HA, simple>[4]( 5.3645916 -3.2955451 *i)
= -105.58589 +16.385576 *i
expected = -105.58589 +16.385576 *i;
ulp = 0.00 :PASS

vsinhi<HA, strided>[4]( 5.3645916 -3.2955451 *i)
= 777.00000 +777.00000 *i
expected = 777.00000 +777.00000 *i;
ulp = 0.00 :PASS

vmsinhi<HA, strided>[4]( 5.3645916 -3.2955451 *i)
= 777.00000 +777.00000 *i
expected = 777.00000 +777.00000 *i;
ulp = 0.00 :PASS

vsinh<LA, simple>[1]( -2.1123414 +6.8037548 *i)
= -3.5337441 +2.0861197 *i
expected = -3.5337441 +2.0861197 *i;
ulp = 0.00 :PASS

vmsinh<LA, simple>[1]( -2.1123414 +6.8037548 *i)
= -3.5337441 +2.0861197 *i
expected = -3.5337441 +2.0861197 *i;
ulp = 0.00 :PASS

vsinhi<LA, strided>[1]( -2.1123414 +6.8037548 *i)
= -3.5337441 +2.0861197 *i
expected = -3.5337441 +2.0861197 *i;
ulp = 0.00 :PASS

vmsinhi<LA, strided>[1]( -2.1123414 +6.8037548 *i)
= -3.5337441 +2.0861197 *i
expected = -3.5337441 +2.0861197 *i;
ulp = 0.00 :PASS

vsinh<LA, simple>[2]( 5.9688005 +5.6619844 *i)
= 158.99046 -113.79469 *i
expected = 158.99046 -113.79469 *i;
ulp = 0.00 :PASS

vmsinh<LA, simple>[2]( 5.9688005 +5.6619844 *i)
= 158.99046 -113.79469 *i
expected = 158.99046 -113.79469 *i;
ulp = 0.00 :PASS

vsinhi<LA, strided>[2]( 5.9688005 +5.6619844 *i)
= 777.00000 +777.00000 *i
expected = 777.00000 +777.00000 *i;
ulp = 0.00 :PASS

vmsinhi<LA, strided>[2]( 5.9688005 +5.6619844 *i)
= 777.00000 +777.00000 *i
expected = 777.00000 +777.00000 *i;
ulp = 0.00 :PASS

vsinh<LA, simple>[3]( -6.0489726 +8.2329483 *i)
= 78.371567 +196.80936 *i
expected = 78.371567 +196.80936 *i;
ulp = 0.00 :PASS

vmsinh<LA, simple>[3]( -6.0489726 +8.2329483 *i)
= 78.371567 +196.80936 *i
expected = 78.371567 +196.80936 *i;
ulp = 0.00 :PASS

vsinhi<LA, strided>[3]( -6.0489726 +8.2329483 *i)
= 78.371567 +196.80936 *i
expected = 78.371567 +196.80936 *i;
ulp = 0.00 :PASS

vmsinhi<LA, strided>[3]( -6.0489726 +8.2329483 *i)
= 78.371567 +196.80936 *i
expected = 78.371567 +196.80936 *i;
ulp = 0.00 :PASS

vsinh<LA, simple>[4]( 5.3645916 -3.2955451 *i)
= -105.58589 +16.385576 *i
expected = -105.58589 +16.385576 *i;
ulp = 0.00 :PASS

vmsinh<LA, simple>[4]( 5.3645916 -3.2955451 *i)
= -105.58589 +16.385576 *i
expected = -105.58589 +16.385576 *i;
ulp = 0.00 :PASS

vsinhi<LA, strided>[4]( 5.3645916 -3.2955451 *i)
= 777.00000 +777.00000 *i
expected = 777.00000 +777.00000 *i;
ulp = 0.00 :PASS

vmsinhi<LA, strided>[4]( 5.3645916 -3.2955451 *i)
= 777.00000 +777.00000 *i
expected = 777.00000 +777.00000 *i;
ulp = 0.00 :PASS

vsinh<EP, simple>[1]( -2.1123414 +6.8037548 *i)
= -3.5337441 +2.0861197 *i
expected = -3.5337441 +2.0861197 *i;
ulp = 0.00 :PASS

vmsinh<EP, simple>[1]( -2.1123414 +6.8037548 *i)
= -3.5337441 +2.0861197 *i
expected = -3.5337441 +2.0861197 *i;
ulp = 0.00 :PASS

vsinhi<EP, strided>[1]( -2.1123414 +6.8037548 *i)
= -3.5337441 +2.0861197 *i
expected = -3.5337441 +2.0861197 *i;
ulp = 0.00 :PASS

vmsinhi<EP, strided>[1]( -2.1123414 +6.8037548 *i)
= -3.5337441 +2.0861197 *i
expected = -3.5337441 +2.0861197 *i;
ulp = 0.00 :PASS

vsinh<EP, simple>[2]( 5.9688005 +5.6619844 *i)
= 158.99046 -113.79469 *i
expected = 158.99046 -113.79469 *i;
ulp = 0.00 :PASS

vmsinh<EP, simple>[2]( 5.9688005 +5.6619844 *i)
= 158.99046 -113.79469 *i
expected = 158.99046 -113.79469 *i;
ulp = 0.00 :PASS

vsinhi<EP, strided>[2]( 5.9688005 +5.6619844 *i)
= 777.00000 +777.00000 *i
expected = 777.00000 +777.00000 *i;
ulp = 0.00 :PASS

vmsinhi<EP, strided>[2]( 5.9688005 +5.6619844 *i)
= 777.00000 +777.00000 *i
expected = 777.00000 +777.00000 *i;
ulp = 0.00 :PASS

vsinh<EP, simple>[3]( -6.0489726 +8.2329483 *i)
= 78.371567 +196.80936 *i
expected = 78.371567 +196.80936 *i;
ulp = 0.00 :PASS

vmsinh<EP, simple>[3]( -6.0489726 +8.2329483 *i)
= 78.371567 +196.80936 *i
expected = 78.371567 +196.80936 *i;
ulp = 0.00 :PASS

vsinhi<EP, strided>[3]( -6.0489726 +8.2329483 *i)
= 78.371567 +196.80936 *i
expected = 78.371567 +196.80936 *i;
ulp = 0.00 :PASS

vmsinhi<EP, strided>[3]( -6.0489726 +8.2329483 *i)
= 78.371567 +196.80936 *i
expected = 78.371567 +196.80936 *i;
ulp = 0.00 :PASS

vsinh<EP, simple>[4]( 5.3645916 -3.2955451 *i)
= -105.58589 +16.385576 *i
expected = -105.58589 +16.385576 *i;
ulp = 0.00 :PASS

vmsinh<EP, simple>[4]( 5.3645916 -3.2955451 *i)
= -105.58589 +16.385576 *i
expected = -105.58589 +16.385576 *i;
ulp = 0.00 :PASS

vsinhi<EP, strided>[4]( 5.3645916 -3.2955451 *i)
= 777.00000 +777.00000 *i
expected = 777.00000 +777.00000 *i;
ulp = 0.00 :PASS

vmsinhi<EP, strided>[4]( 5.3645916 -3.2955451 *i)
= 777.00000 +777.00000 *i
expected = 777.00000 +777.00000 *i;
ulp = 0.00 :PASS

sinh single precision complex result: PASS
Running sinh with double precision complex data type:
vsinh<HA, simple>[1]( -2.112341463618139 +6.803754343094191 *i)
= -3.533745332757388 +2.086118168674303 *i
expected = -3.533745332757388 +2.086118168674303 *i;
ulp = 0.00 (PASS)

vmsinh<HA, simple>[1]( -2.112341463618139 +6.803754343094191 *i)
= -3.533745332757388 +2.086118168674303 *i
expected = -3.533745332757388 +2.086118168674303 *i;
ulp = 0.00 (PASS)

vsinhi<HA, strided>[1]( -2.112341463618139 +6.803754343094191 *i)
= -3.533745332757388 +2.086118168674303 *i
expected = -3.533745332757388 +2.086118168674303 *i;
ulp = 0.00 (PASS)

vmsinhi<HA, strided>[1]( -2.112341463618139 +6.803754343094191 *i)
= -3.533745332757388 +2.086118168674303 *i
expected = -3.533745332757388 +2.086118168674303 *i;
ulp = 0.00 (PASS)

vsinh<HA, simple>[2]( 5.968800669521466 +5.661984475172117 *i)
= 158.9904813936415 -113.7946992092927 *i
expected = 158.9904813936415 -113.7946992092927 *i;
ulp = 0.00 (PASS)

vmsinh<HA, simple>[2]( 5.968800669521466 +5.661984475172117 *i)
= 158.9904813936415 -113.7946992092927 *i
expected = 158.9904813936415 -113.7946992092927 *i;
ulp = 0.00 (PASS)

vsinhi<HA, strided>[2]( 5.968800669521466 +5.661984475172117 *i)
= 777.0000000000000 +777.0000000000000 *i
expected = 777.0000000000000 +777.0000000000000 *i;
ulp = 0.00 (PASS)

vmsinhi<HA, strided>[2]( 5.968800669521466 +5.661984475172117 *i)
= 777.0000000000000 +777.0000000000000 *i
expected = 777.0000000000000 +777.0000000000000 *i;
ulp = 0.00 (PASS)

vsinh<HA, simple>[3]( -6.048972614132321 +8.232947158735687 *i)
= 78.37134542800175 +196.8094430413420 *i
expected = 78.37134542800175 +196.8094430413420 *i;
ulp = 0.00 (PASS)

vmsinh<HA, simple>[3]( -6.048972614132321 +8.232947158735687 *i)
= 78.37134542800175 +196.8094430413420 *i
expected = 78.37134542800175 +196.8094430413420 *i;
ulp = 0.00 (PASS)

vsinhi<HA, strided>[3]( -6.048972614132321 +8.232947158735687 *i)
= 78.37134542800175 +196.8094430413420 *i
expected = 78.37134542800175 +196.8094430413420 *i;
ulp = 0.00 (PASS)

vmsinhi<HA, strided>[3]( -6.048972614132321 +8.232947158735687 *i)
= 78.37134542800175 +196.8094430413420 *i
expected = 78.37134542800175 +196.8094430413420 *i;
ulp = 0.00 (PASS)

vsinh<HA, simple>[4]( 5.364591896238082 -3.295544885702220 *i)
= -105.5859236313347 +16.38555846778798 *i
expected = -105.5859236313347 +16.38555846778798 *i;
ulp = 0.00 (PASS)

vmsinh<HA, simple>[4]( 5.364591896238082 -3.295544885702220 *i)
= -105.5859236313347 +16.38555846778798 *i
expected = -105.5859236313347 +16.38555846778798 *i;
ulp = 0.00 (PASS)

vsinhi<HA, strided>[4]( 5.364591896238082 -3.295544885702220 *i)
= 777.0000000000000 +777.0000000000000 *i
expected = 777.0000000000000 +777.0000000000000 *i;
ulp = 0.00 (PASS)

vmsinhi<HA, strided>[4]( 5.364591896238082 -3.295544885702220 *i)
= 777.0000000000000 +777.0000000000000 *i
expected = 777.0000000000000 +777.0000000000000 *i;
ulp = 0.00 (PASS)

vsinh<LA, simple>[1]( -2.112341463618139 +6.803754343094191 *i)
= -3.533745332757388 +2.086118168674303 *i
expected = -3.533745332757388 +2.086118168674303 *i;
ulp = 0.00 (PASS)

vmsinh<LA, simple>[1]( -2.112341463618139 +6.803754343094191 *i)
= -3.533745332757388 +2.086118168674303 *i
expected = -3.533745332757388 +2.086118168674303 *i;
ulp = 0.00 (PASS)

vsinhi<LA, strided>[1]( -2.112341463618139 +6.803754343094191 *i)
= -3.533745332757388 +2.086118168674303 *i
expected = -3.533745332757388 +2.086118168674303 *i;
ulp = 0.00 (PASS)

vmsinhi<LA, strided>[1]( -2.112341463618139 +6.803754343094191 *i)
= -3.533745332757388 +2.086118168674303 *i
expected = -3.533745332757388 +2.086118168674303 *i;
ulp = 0.00 (PASS)

vsinh<LA, simple>[2]( 5.968800669521466 +5.661984475172117 *i)
= 158.9904813936415 -113.7946992092927 *i
expected = 158.9904813936415 -113.7946992092927 *i;
ulp = 0.00 (PASS)

vmsinh<LA, simple>[2]( 5.968800669521466 +5.661984475172117 *i)
= 158.9904813936415 -113.7946992092927 *i
expected = 158.9904813936415 -113.7946992092927 *i;
ulp = 0.00 (PASS)

vsinhi<LA, strided>[2]( 5.968800669521466 +5.661984475172117 *i)
= 777.0000000000000 +777.0000000000000 *i
expected = 777.0000000000000 +777.0000000000000 *i;
ulp = 0.00 (PASS)

vmsinhi<LA, strided>[2]( 5.968800669521466 +5.661984475172117 *i)
= 777.0000000000000 +777.0000000000000 *i
expected = 777.0000000000000 +777.0000000000000 *i;
ulp = 0.00 (PASS)

vsinh<LA, simple>[3]( -6.048972614132321 +8.232947158735687 *i)
= 78.37134542800175 +196.8094430413420 *i
expected = 78.37134542800175 +196.8094430413420 *i;
ulp = 0.00 (PASS)

vmsinh<LA, simple>[3]( -6.048972614132321 +8.232947158735687 *i)
= 78.37134542800175 +196.8094430413420 *i
expected = 78.37134542800175 +196.8094430413420 *i;
ulp = 0.00 (PASS)

vsinhi<LA, strided>[3]( -6.048972614132321 +8.232947158735687 *i)
= 78.37134542800175 +196.8094430413420 *i
expected = 78.37134542800175 +196.8094430413420 *i;
ulp = 0.00 (PASS)

vmsinhi<LA, strided>[3]( -6.048972614132321 +8.232947158735687 *i)
= 78.37134542800175 +196.8094430413420 *i
expected = 78.37134542800175 +196.8094430413420 *i;
ulp = 0.00 (PASS)

vsinh<LA, simple>[4]( 5.364591896238082 -3.295544885702220 *i)
= -105.5859236313347 +16.38555846778798 *i
expected = -105.5859236313347 +16.38555846778798 *i;
ulp = 0.00 (PASS)

vmsinh<LA, simple>[4]( 5.364591896238082 -3.295544885702220 *i)
= -105.5859236313347 +16.38555846778798 *i
expected = -105.5859236313347 +16.38555846778798 *i;
ulp = 0.00 (PASS)

vsinhi<LA, strided>[4]( 5.364591896238082 -3.295544885702220 *i)
= 777.0000000000000 +777.0000000000000 *i
expected = 777.0000000000000 +777.0000000000000 *i;
ulp = 0.00 (PASS)

vmsinhi<LA, strided>[4]( 5.364591896238082 -3.295544885702220 *i)
= 777.0000000000000 +777.0000000000000 *i
expected = 777.0000000000000 +777.0000000000000 *i;
ulp = 0.00 (PASS)

vsinh<EP, simple>[1]( -2.112341463618139 +6.803754343094191 *i)
= -3.533745332757388 +2.086118168674303 *i
expected = -3.533745332757388 +2.086118168674303 *i;
ulp = 0.00 (PASS)

vmsinh<EP, simple>[1]( -2.112341463618139 +6.803754343094191 *i)
= -3.533745332757388 +2.086118168674303 *i
expected = -3.533745332757388 +2.086118168674303 *i;
ulp = 0.00 (PASS)

vsinhi<EP, strided>[1]( -2.112341463618139 +6.803754343094191 *i)
= -3.533745332757388 +2.086118168674303 *i
expected = -3.533745332757388 +2.086118168674303 *i;
ulp = 0.00 (PASS)

vmsinhi<EP, strided>[1]( -2.112341463618139 +6.803754343094191 *i)
= -3.533745332757388 +2.086118168674303 *i
expected = -3.533745332757388 +2.086118168674303 *i;
ulp = 0.00 (PASS)

vsinh<EP, simple>[2]( 5.968800669521466 +5.661984475172117 *i)
= 158.9904813936415 -113.7946992092927 *i
expected = 158.9904813936415 -113.7946992092927 *i;
ulp = 0.00 (PASS)

vmsinh<EP, simple>[2]( 5.968800669521466 +5.661984475172117 *i)
= 158.9904813936415 -113.7946992092927 *i
expected = 158.9904813936415 -113.7946992092927 *i;
ulp = 0.00 (PASS)

vsinhi<EP, strided>[2]( 5.968800669521466 +5.661984475172117 *i)
= 777.0000000000000 +777.0000000000000 *i
expected = 777.0000000000000 +777.0000000000000 *i;
ulp = 0.00 (PASS)

vmsinhi<EP, strided>[2]( 5.968800669521466 +5.661984475172117 *i)
= 777.0000000000000 +777.0000000000000 *i
expected = 777.0000000000000 +777.0000000000000 *i;
ulp = 0.00 (PASS)

vsinh<EP, simple>[3]( -6.048972614132321 +8.232947158735687 *i)
= 78.37134542800175 +196.8094430413420 *i
expected = 78.37134542800175 +196.8094430413420 *i;
ulp = 0.00 (PASS)

vmsinh<EP, simple>[3]( -6.048972614132321 +8.232947158735687 *i)
= 78.37134542800175 +196.8094430413420 *i
expected = 78.37134542800175 +196.8094430413420 *i;
ulp = 0.00 (PASS)

vsinhi<EP, strided>[3]( -6.048972614132321 +8.232947158735687 *i)
= 78.37134542800175 +196.8094430413420 *i
expected = 78.37134542800175 +196.8094430413420 *i;
ulp = 0.00 (PASS)

vmsinhi<EP, strided>[3]( -6.048972614132321 +8.232947158735687 *i)
= 78.37134542800175 +196.8094430413420 *i
expected = 78.37134542800175 +196.8094430413420 *i;
ulp = 0.00 (PASS)

vsinh<EP, simple>[4]( 5.364591896238082 -3.295544885702220 *i)
= -105.5859236313347 +16.38555846778798 *i
expected = -105.5859236313347 +16.38555846778798 *i;
ulp = 0.00 (PASS)

vmsinh<EP, simple>[4]( 5.364591896238082 -3.295544885702220 *i)
= -105.5859236313347 +16.38555846778798 *i
expected = -105.5859236313347 +16.38555846778798 *i;
ulp = 0.00 (PASS)

vsinhi<EP, strided>[4]( 5.364591896238082 -3.295544885702220 *i)
= 777.0000000000000 +777.0000000000000 *i
expected = 777.0000000000000 +777.0000000000000 *i;
ulp = 0.00 (PASS)

vmsinhi<EP, strided>[4]( 5.364591896238082 -3.295544885702220 *i)
= 777.0000000000000 +777.0000000000000 *i
expected = 777.0000000000000 +777.0000000000000 *i;
ulp = 0.00 (PASS)

sinh double precision complex result: PASS
sinh function result: PASS

```

Re: How to use IFX and offload openMP to GPU?

CRquantum — Thu, 07 Apr 2022 07:33:45 GMT

By the way Vidya, I have two questions,

1. Is there some example that I check and make sure the computation is really offload to Intel's GPU?

2. Is there some example that I can compile and run, so that can compare the speed of the same code on CPU vs on GPU?

Just to wanted to see under what circumstances and how GPU can accelerate the computation compared with CPU.

Thanks much.

Best regards,

Rong

Re:How to use IFX and offload openMP to GPU?

VidyalathaB_Intel — Fri, 08 Apr 2022 16:44:07 GMT

Hi Rong,

Yes, there are examples with both the cases i.e. running the code on the host and running code on the device. You can find them in the same example folder which is named "example_offload_f"(please see examples in blas folder).

Additionally, you can refer to the below as well for more details

https://www.intel.com/content/www/us/en/develop/documentation/onemkl-developer-reference-fortran/top/openmp-offload/openmp-offload-for-onemkl.html

To see if the code is really offloaded to Intel GPU you can try setting the environment variable "LIBOMPTARGET_PLUGIN_PROFILE=T" before running the code as referred in the below documentation link https://www.intel.com/content/www/us/en/develop/documentation/get-started-with-cpp-fortran-compiler-openmp/top.html

For more details regarding the offloading environment variables refer

https://www.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/software-development-process/debugging-the-dpc-and-openmp-offload-process/oneapi-debug-tools.html

Hope the provided information helps.

Regards,

Vidya.

Re:How to use IFX and offload openMP to GPU?

VidyalathaB_Intel — Fri, 15 Apr 2022 17:07:22 GMT

Hi Rong,

A gentle reminder:

Could you please provide us with an update regarding the issue? Please do let us know if the provided information helps.

Regards,

Vidya.

Re:How to use IFX and offload openMP to GPU?

VidyalathaB_Intel — Fri, 22 Apr 2022 08:25:57 GMT

Hi Rong,

We are closing this thread assuming that your issue is resolved. Please post a new question if you need any additional information from Intel as this thread will no longer be monitored.

Regards,

Vidya.