GCC OpenMP 4.0 Offloading to a Real Knights Corner Xeon Phi Card

[Xuhao_C_]

https://gcc.gnu.org/wiki/Offloading

"Patches enabling OpenMP 4.0 offloading to Intel MIC are merged to trunk. They include general infrastructure changes, mkoffload tool, libgomp plugin, Intel MIC runtime offload library liboffloadmic and an emulator. This emulator lies under liboffloadmic and reproduces MIC's HW and SW stack behavior allowing to run offloaded code in a separate address space using the host machine. The emulator consists of 4 shared libraries which replace COI and MYO libraries from Intel Manycore Platform Software Stack (MPSS). In case of real offloading, user is supposed to specify path to MPSS libraries in LD_LIBRARY_PATH, this will overload emulator libraries on runtime."

I am trying to use the new GCC 6.2 release to offload OpenMP blocks to the Intel MIC (i.e. the Xeon Phi) KNC.

The GCC wiki says it supports emulation, but I want to run the OpenMP program on a real MIC (KNC).

As the wiki says, in case of real offloading, user is supposed to specify path to MPSS libraries in LD_LIBRARY_PATH, this will overload emulator libraries on runtime. I tried it, however, the execution failed at CheckProcessBinary() in line 879 of mpss-coi-3.6/src/mechanism/process/process_source.cpp (MPSS-3.6) when creating a process on MIC (_COIProcess::_COIProcess()).

So what's wrong with it? Is it possible to offload OpenMP program to a real Knights Corner Xeon Phi card using GCC? If yes, how? Thank you.

[Ilya_Verbin]

Please look at the top of that page:

GCC 5 and later support two offloading configurations:

 * OpenMP to Intel MIC targets (upcoming Intel Xeon Phi products codenamed KNL) as well as MIC emulation on host.

[...]

So, offloading to KNC is not supported.

[Xuhao_C_]

Thank you so much for your reply! I understand KNC is not supported, but what is exactly missing to support KNC except modifying the cross (offload) compiler? i.e. If I want to port GCC to support KNC, what else changes should I apply to liboffloadmic, libgomp and so on? Thank you!

Ilya V. wrote:

Please look at the top of that page:

GCC 5 and later support two offloading configurations:

 * OpenMP to Intel MIC targets (upcoming Intel Xeon Phi products codenamed KNL) as well as MIC emulation on host.

[...]

So, offloading to KNC is not supported.

[Ilya_Verbin]

Xuhao C. wrote:
Thank you so much for your reply! I understand KNC is not supported, but what is exactly missing to support KNC except modifying the cross (offload) compiler? i.e. If I want to port GCC to support KNC, what else changes should I apply to liboffloadmic, libgomp and so on? Thank you!

GCC lacks support of KNC code-generation. Of course it can emit all basic x86 insns, however it doesn't support the main thing - KNC vector insns.

[Xuhao_C_]

Ilya V. wrote:

Quote:

Xuhao C. wrote:

Thank you so much for your reply! I understand KNC is not supported, but what is exactly missing to support KNC except modifying the cross (offload) compiler? i.e. If I want to port GCC to support KNC, what else changes should I apply to liboffloadmic, libgomp and so on? Thank you!

GCC lacks support of KNC code-generation. Of course it can emit all basic x86 insns, however it doesn't support the main thing - KNC vector insns.

Thank you. So if my OpenMP program doesn't contain any vector instructions, theoretically, it can run successfully on KNC card with current GCC offload environment?

[Ilya_Verbin]

Xuhao C. wrote:
Thank you. So if my OpenMP program doesn't contain any vector instructions, theoretically, it can run successfully on KNC card with current GCC offload environment?

Theoretically yes. If you really want to try it, you should get this patch from MPSS GCC and apply it to GCC 6.2: https://github.com/apc-llc/gcc-5.1.1-knc/commit/cee3ea59cd33411db673dc639ee179bcf2d3fcef

But please note that offloading to KNC was never tested, and do not expect any performance gains, because of absent vector instructions.

[Xuhao_C_]

Ilya V. wrote:

Quote:

Xuhao C. wrote:

Thank you. So if my OpenMP program doesn't contain any vector instructions, theoretically, it can run successfully on KNC card with current GCC offload environment?

Theoretically yes. If you really want to try it, you should get this patch from MPSS GCC and apply it to GCC 6.2: https://github.com/apc-llc/gcc-5.1.1-knc/commit/cee3ea59cd33411db673dc63...

But please note that offloading to KNC was never tested, and do not expect any performance gains, because of absent vector instructions.

Thank you so much. I tried to patch it to GCC-6.2, but failed with errors. However, I successfully patched it to GCC-5.1.0. Then when I build this patched GCC-5.1.0, how should I configure it?

Originally, for building accel compiler:

../configure --build=x86_64-intelmicemul-linux-gnu --host=x86_64-intelmicemul-linux-gnu --target=x86_64-intelmicemul-linux-gnu --enable-as-accelerator-for=x86_64-pc-linux-gnu

For building host compiler:

../configure --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --target=x86_64-pc-linux-gnu --enable-offload-targets=x86_64-intelmicemul-linux-gnu=/install/prefix

I tried to replace "x86_64-intelmicemul-linux-gnu" with "k1om-mpss-linux", and it works for building accel compiler, but compiling host compiler failed because it didn't recognize "k1om-mpss-linux".

Should I replace "intelmicemul-linux" with "intelmic-linux"?

Thank you.

[Ilya_Verbin]

Xuhao C. wrote:
Should I replace "intelmicemul-linux" with "intelmic-linux"?

No, you should make GCC understand "k1om-mpss-linux" as offload target by adding it to configure.ac (search for enable_offload_targets) and running autoconf.

[Xuhao_C_]

Ilya V. wrote:

Quote:

Xuhao C. wrote:

Should I replace "intelmicemul-linux" with "intelmic-linux"?

No, you should make GCC understand "k1om-mpss-linux" as offload target by adding it to configure.ac (search for enable_offload_targets) and running autoconf.

Thank you. I added k1om-* in following files:
configure.ac
gcc/config.gcc
libgomp/plugin/configfrag.ac
liboffloadmic/plugin/configure.ac
liboffloadmic/configure.tgt

Now I can build this patched GCC-5.1.0 successfully:

$ ../gcc-5.1.0/configure --prefix=/home/cxh/work/gcc-install/gcc-5.1.0 --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=k1om-mpss-linux --enable-as-accelerator-for=x86_64-pc-linux-gnu --enable-languages=c,c++ --enable-threads=posix --disable-multilib --with-gmp=/usr/local/gmp-4.3.2 --with-mpfr=/usr/local/mpfr-2.4.2 --with-mpc=/usr/local/mpc-0.8.1 --disable-libitm --with-sysroot=/opt/mpss/3.6/sysroots/k1om-mpss-linux

$ make -j16

$ make -j16 install

$ ../gcc-5.1.0/configure --prefix=/home/cxh/work/gcc-install/gcc-5.1.0 --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --target=x86_64-pc-linux-gnu --enable-offload-targets=k1om-mpss-linux=/home/cxh/work/gcc-install/gcc-5.1.0 --enable-languages=c,c++ --enable-threads=posix --disable-multilib --with-gmp=/usr/local/gmp-4.3.2 --with-mpfr=/usr/local/mpfr-2.4.2 --with-mpc=/usr/local/mpc-0.8.1

$ make -j16

$ make -j16 install

However, when I tried to compile a simple OpenMP program using this GCC, I got this error:

$ export PATH=/home/cxh/work/gcc-install/gcc-5.1.0/bin:/usr/local/cuda/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

$ export LD_LIBRARY_PATH=/home/cxh/work/gcc-install/gcc-5.1.0/lib64:/usr/local/mpc-0.8.1/lib:/usr/local/mpfr-2.4.2/lib:/usr/local/gmp-4.3.2/lib:/usr/local/cuda/lib64:/usr/local/lib

$ gcc vec_mult.c -o vec_mult -fopenmp
x86_64-pc-linux-gnu-accel-k1om-mpss-linux-gcc: error: libgomp.spec: No such file or directory
mkoffload-intelmic: fatal error: x86_64-pc-linux-gnu-accel-k1om-mpss-linux-gcc returned 1 exit status
compilation terminated.
lto-wrapper: fatal error: /home/cxh/work/gcc-install/gcc-5.1.0/libexec/gcc/x86_64-pc-linux-gnu/5.1.0//accel/k1om-mpss-linux/mkoffload returned 1 exit status
compilation terminated.
/usr/bin/ld: lto-wrapper failed
collect2: error: ld returned 1 exit status

My OS is RedHat 7 with Linux kernel 3.10.0-123.el7.x86_64.

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/home/cxh/work/gcc-install/gcc-5.1.0/libexec/gcc/x86_64-pc-linux-gnu/5.1.0/lto-wrapper
OFFLOAD_TARGET_NAMES=k1om-mpss-linux
Target: x86_64-pc-linux-gnu
Configured with: ../gcc-5.1.0/configure --prefix=/home/cxh/work/gcc-install/gcc-5.1.0 --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --target=x86_64-pc-linux-gnu --enable-offload-targets=k1om-mpss-linux=/home/cxh/work/gcc-install/gcc-5.1.0 --enable-languages=c,c++ --enable-threads=posix --disable-multilib --with-gmp=/usr/local/gmp-4.3.2 --with-mpfr=/usr/local/mpfr-2.4.2 --with-mpc=/usr/local/mpc-0.8.1
Thread model: posix
gcc version 5.1.0 (GCC)

Why libgomp.spec exists, but can not be found?

$ find ../ -name libgomp.spec
../gcc-5.1.0/k1om-mpss-linux/lib64/libgomp.spec
../gcc-5.1.0/lib64/libgomp.spec

Thank you!

[TimP]

If libgomp wasn't built, on account of being "unsupported," you might try adding to configure --enable-libgomp.  I would suspect then some feature of libgomp will be missing.

[Xuhao_C_]

Tim P. wrote:

If libgomp wasn't built, on account of being "unsupported," you might try adding to configure --enable-libgomp.  I would suspect then some feature of libgomp will be missing.

libgomp is already built, and the subdirectory lib64/ contains libgomp.so and other libraries.

[Ilya_Verbin]

Xuhao C. wrote:

$ gcc vec_mult.c -o vec_mult -fopenmp

x86_64-pc-linux-gnu-accel-k1om-mpss-linux-gcc: error: libgomp.spec: No such file or directory
mkoffload-intelmic: fatal error: x86_64-pc-linux-gnu-accel-k1om-mpss-linux-gcc returned 1 exit status
compilation terminated.
lto-wrapper: fatal error: /home/cxh/work/gcc-install/gcc-5.1.0/libexec/gcc/x86_64-pc-linux-gnu/5.1.0//accel/k1om-mpss-linux/mkoffload returned 1 exit status
compilation terminated.
/usr/bin/ld: lto-wrapper failed
collect2: error: ld returned 1 exit status

Pretty good progress! This message means that host gcc is able to find and run k1om accel gcc. But accel gcc can't find its own libgomp (I don't know why). Maybe something like this will help:

$ gcc vec_mult.c -o vec_mult -fopenmp -foffload="-L../gcc-5.1.0/k1om-mpss-linux/lib64/"

or

$ gcc vec_mult.c -o vec_mult -fopenmp -foffload="-B../gcc-5.1.0/k1om-mpss-linux/lib64/"

[Xuhao_C_]

Ilya V. wrote:

Quote:

Xuhao C. wrote:

$ gcc vec_mult.c -o vec_mult -fopenmp

x86_64-pc-linux-gnu-accel-k1om-mpss-linux-gcc: error: libgomp.spec: No such file or directory
mkoffload-intelmic: fatal error: x86_64-pc-linux-gnu-accel-k1om-mpss-linux-gcc returned 1 exit status
compilation terminated.
lto-wrapper: fatal error: /home/cxh/work/gcc-install/gcc-5.1.0/libexec/gcc/x86_64-pc-linux-gnu/5.1.0//accel/k1om-mpss-linux/mkoffload returned 1 exit status
compilation terminated.
/usr/bin/ld: lto-wrapper failed
collect2: error: ld returned 1 exit status

Pretty good progress! This message means that host gcc is able to find and run k1om accel gcc. But accel gcc can't find its own libgomp (I don't know why). Maybe something like this will help:

$ gcc vec_mult.c -o vec_mult -fopenmp -foffload="-L../gcc-5.1.0/k1om-mpss-linux/lib64/"

or

$ gcc vec_mult.c -o vec_mult -fopenmp -foffload="-B../gcc-5.1.0/k1om-mpss-linux/lib64/"

Finally, it works. Putting it all together:

Building GCC for offloading KNC card

Gcc source code（patched）: /home/cxh/work/gcc-5.1.0/

Gcc install path: /home/cxh/work/gcc-install/gcc-5.1.0/

OpenMP test program: /home/cxh/work/vec_mult.c (vector multiplication)

Environment setup: see Q5

MPSS version installed: 3.6

Accel compiler

../gcc-5.1.0/configure --prefix=/home/cxh/work/gcc-install/gcc-5.1.0 --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=k1om-mpss-linux --enable-as-accelerator-for=x86_64-pc-linux-gnu --enable-languages=c,c++ --enable-threads=posix --disable-multilib --with-gmp=/usr/local/gmp-4.3.2 --with-mpfr=/usr/local/mpfr-2.4.2 --with-mpc=/usr/local/mpc-0.8.1 --disable-libitm --with-sysroot=/opt/mpss/3.6/sysroots/k1om-mpss-linux --with-ld=/opt/mpss/3.6/sysroots/x86_64-mpsssdk-linux/usr/bin/k1om-mpss-linux/k1om-mpss-linux-ld --with-as=/opt/mpss/3.6/sysroots/x86_64-mpsssdk-linux/usr/bin/k1om-mpss-linux/k1om-mpss-linux-as

Host compiler

../gcc-5.1.0/configure --prefix=/home/cxh/work/gcc-install/gcc-5.1.0 --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --target=x86_64-pc-linux-gnu --enable-offload-targets=k1om-mpss-linux=/home/cxh/work/gcc-install/gcc-5.1.0 --enable-languages=c,c++ --enable-threads=posix --disable-multilib --with-gmp=/usr/local/gmp-4.3.2 --with-mpfr=/usr/local/mpfr-2.4.2 --with-mpc=/usr/local/mpc-0.8.1

Questions and Answers

Q1: k1om-mpss-linux unsupported

A1: Get the patch from https://github.com/apc-llc/gcc-5.1.1-knc/commit/cee3ea59cd33411db673dc639ee179bcf2d3fcef

Add k1om-* in following files and run autoconf

configure.ac

gcc/config.gcc

libgomp/plugin/configfrag.ac

liboffloadmic/plugin/configure.ac

liboffloadmic/configure.tgt

Q2:

$ gcc vec_mult.c -o vec_mult -fopenmp

x86_64-pc-linux-gnu-accel-k1om-mpss-linux-gcc: error: libgomp.spec: No such file or directory

mkoffload-intelmic: fatal error: x86_64-pc-linux-gnu-accel-k1om-mpss-linux-gcc returned 1 exit status

compilation terminated.

lto-wrapper: fatal error: /home/cxh/work/gcc-install/gcc-5.1.0/libexec/gcc/x86_64-pc-linux-gnu/5.1.0//accel/k1om-mpss-linux/mkoffload returned 1 exit status

compilation terminated.

/usr/bin/ld: lto-wrapper failed

collect2: error: ld returned 1 exit status

export PATH=/home/cxh/work/gcc-install/gcc-5.1.0/bin:/usr/local/cuda/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

export LD_LIBRARY_PATH=/home/cxh/work/gcc-install/gcc-5.1.0/lib64:/usr/local/mpc-0.8.1/lib:/usr/local/mpfr-2.4.2/lib:/usr/local/gmp-4.3.2/lib:/usr/local/cuda/lib64:/usr/local/lib

A2:

Add -foffload="-B/home/cxh/work/gcc-install/gcc-5.1.0/k1om-mpss-linux/lib64/" when compiling the OpenMP program

Q3:

$ gcc vec_mult.c -o vec_mult -fopenmp -foffload="-B/home/cxh/work/gcc-install/gcc-5.1.0/k1om-mpss-linux/lib64/"

collect2: fatal error: cannot find 'ld'

compilation terminated.

mkoffload-intelmic: fatal error: x86_64-pc-linux-gnu-accel-k1om-mpss-linux-gcc returned 1 exit status

compilation terminated.

lto-wrapper: fatal error: /home/cxh/work/gcc-install/gcc-5.1.0/libexec/gcc/x86_64-pc-linux-gnu/5.1.0//accel/k1om-mpss-linux/mkoffload returned 1 exit status

compilation terminated.

/usr/bin/ld: lto-wrapper failed

collect2: error: ld returned 1 exit status

A3: add --with-ld=/opt/mpss/3.6/sysroots/x86_64-mpsssdk-linux/usr/bin/k1om-mpss-linux/k1om-mpss-linux-ld when building GCC

Q4:

$ gcc vec_mult.c -o vec_mult -fopenmp -foffload="-B/home/cxh/work/gcc-install/gcc-5.1.0/k1om-mpss-linux/lib64/"

/opt/mpss/3.6/sysroots/x86_64-mpsssdk-linux/usr/bin/k1om-mpss-linux/k1om-mpss-linux-ld: cannot find crtbeginS.o: No such file or directory

/opt/mpss/3.6/sysroots/x86_64-mpsssdk-linux/usr/bin/k1om-mpss-linux/k1om-mpss-linux-ld: cannot find -lgcc

collect2: error: ld returned 1 exit status

mkoffload-intelmic: fatal error: x86_64-pc-linux-gnu-accel-k1om-mpss-linux-gcc returned 1 exit status

compilation terminated.

lto-wrapper: fatal error: /home/cxh/work/gcc-install/gcc-5.1.0/libexec/gcc/x86_64-pc-linux-gnu/5.1.0//accel/k1om-mpss-linux/mkoffload returned 1 exit status

compilation terminated.

/usr/bin/ld: lto-wrapper failed

collect2: error: ld returned 1 exit status

A4:

$ find ./ -name "crtbeginS.o"

./gcc-install/gcc-5.1.0/lib/gcc/k1om-mpss-linux/5.1.0/accel/k1om-mpss-linux/crtbeginS.o

./gcc-install/gcc-5.1.0/lib/gcc/x86_64-pc-linux-gnu/5.1.0/crtbeginS.o

$cp gcc-install/gcc-5.1.0/lib/gcc/k1om-mpss-linux/5.1.0/accel/k1om-mpss-linux/* gcc-install/gcc-5.1.0/lib/gcc/x86_64-pc-linux-gnu/5.1.0/accel/k1om-mpss-linux/

Q5:

$ ./vec_mult

offload error: cannot start process on the device 0 (error code 9)

A5:

This is because device libraries are not found. Note that some libraries (e.g. libgomp) have the same name for both host and device. Make sure the correct libraries are found for the device (The order in LD_LIBRARY_PATH matters).  The correct environment setup:

export LD_LIBRARY_PATH=/opt/mpss/3.6/sysroots/k1om-mpss-linux/usr/lib64:/opt/mpss/3.6/sysroots/k1om-mpss-linux/lib64:/home/cxh/mpss-3.6/src/mpss-coi-3.6/install-host/usr/local/lib64:/home/cxh/work/gcc-install/gcc-5.1.0/k1om-mpss-linux/lib64:/home/cxh/work/gcc-install/gcc-5.1.0/lib64:/usr/local/mpc-0.8.1/lib:/usr/local/mpfr-2.4.2/lib:/usr/local/gmp-4.3.2/lib:/usr/local/cuda/lib64:/usr/local/lib

Maybe we can set SINK_LD_LIBRARY_PATH to make it more clear? but for now, the above setup works at least.