as if native compilation on Xeon PHI using host compilers

[supersmith]

This file describes how to use an "as if native" intel compilation mode
to reliably compile open source packages for the MIC / Xeon PHI.

A few basic points to make this work.

1) Use wrapper scripts on the mic that invoke the intel compilers and
tools ON THE HOST from the mic, using passwordless ssh.

2) Use special "sanitize" code with these scripts, to correctly escape
the various special characters such as ' " ` \ that otherwise would be
lost via ssh. THIS METHOD WILL NOT WORK WITHOUT THE SANITIZE CODE.

3) Have the same home directories mounted on the MIC as well as the HOST,
such the directory names are the same. i.e. /home/userx on both systems.

4) Have these home directories mounted TWICE on the MIC. A second
mount of /home at /home.replicant in our example code, mounted with
actimeo=0 This is absolutely necessary in order that files
created on the HOST from passwordless-ssh-enabled compiler scipts are
instantly and synchronously visible on the MIC. THIS METHOD WILL NOT
WORK OTHERWISE. Happy to hear a concrete real alternative to this,
please keep idle speculation to yourself.

There are a number of files:

a) README.txt This file. Read it.

b) background.pdf Read it. Explains the motivation.

c) rel.tar.gz Untar this directory "rel" to your HOME DIRECTORY.

c) site.tar.gz Untar this directory "site" to your HOME IDRECTORY.

The scripts located in:

$HOME/site/mic/intel/compiler_mic.1.1

ar cpp icc icpc ifort perl ranlib

are ready to use to do "as if" compiling. Let's look at that icc:

#!/bin/sh HERE=`pwd | sed "s/.replicant//"`
set -- $($HOME/site/mic/intel/compiler_mic.1.1/utility/sanitize.mic -n -- "$@")
ssh -o loglevel=error host "cd $HERE; $HOME/site/mic/intel/compiler_mic.1.1/utility/sanitize.host -d -- icc -mmic $@"

(To rebuild the sanitize code, cd to
$HOME/site/mic/intel/compiler_mic.1.1/utility, and run ./go.build)

To run this script on the mic, the idea is that you have your home
directories mounted at /home and /home.replicant as described above. So,
you'll compile like:

cd /home.replicant/userx
icc hello.c

after putting $HOME/site/mic/intel/compiler_mic.1.1 in your PATH on the
MIC. Of course, you'll have to have icc in your PATH on HOST as well.

You can see that the sanitize code runs on the mic and host side, with
different flags, encoding and decoding the arguments in such a way as to
preserve special escape characters through the ssh invocation. Don't ask
me how it works, it's MAGIC provided by my colleague Dale Talcott.

To show you how we use this in practice, I've gone to some length to create a
script for you, "rel/go.build.basic_module" that builds all the opensource
packages needed to create the "modules" package. These binaries and
libraries are built up into the "site" directory. You will no doubt
have to try several times to get this to work. Before each attempt, you
should run the "go.reset" script to recreate the "site" directory from
the original tar file I've provided. Since the contents of stuff built
into "site" are used as the build progresses, garbage left in "site"
will possibly mess up a repeat of the build process, SO BE SURE TO RUN
go.reset BETWEEN EACH ATTEMPT.

What we do, is put the following into /etc/csh.cshrc on the MIC:

alias module 'eval `/xxxx/mic/basic/basic.2.0/bin/modulecmd tcsh \!*`'
setenv MODULEPATH /xxxx/modulefiles/mic

In a user's .cshrc on the mic, we put:

if ( `basename $MODULEPATH` == host) then
module load intel/composer_xe_2013.1.117 intel/intel_mpi.4.1.0.024 netcdf/4.1.1 interocitor.1.1
endif
if ( `basename $MODULEPATH` == mic) then
module load intel/composer_xe_2013.1.117 intel/intel_mpi.4.1.0.024 intel_compiler_mic.1.1 coreutils-8.17 netcdf/4.1.1 basic.2.0 binutils.2.1.3653-8
endif

The compiler and mpi modules have the same names on host and MIC, doing
different things. On the MIC, they just setup the runtime.
Of course, we have built full-up versions of bash and tcsh (not shown
here) and installed them in /bin on the MICs.

After untarring, cd to the "rel" directory. Have a look at
"go.unpack". You need to locate and download all the indicated compressed
open source packages. Don't try using any other versions of the packages
other than the precise ones listed. They may not work. Put all those
compressed files in the "rel" directory, and run "go.build.basic_module"
from the "rel" directory, perhaps after doing "go.reset".

So, these docs and sample scripts should give you a flavour of how we
are going about creating a reliable modules-based environment and correct
build procedures for the MIC.

I'm willing to offer help if you email me, BUT NOT IF YOU DON"T READ
AND FOLLOW THESE DIRECTIONS.

Full sentences and some attempt at proper grammar will help to get a
response. As I mentioned earlier, I'm not interested in hearing random
speculation.

email ---> dbarker@supersmith.com

[BelindaLiviero]

Thank you for sharing this process with the community!

[Eric_B_]

Here's a script that (as long as you have bash available on both sides and as login shell on host) does the argument protection for a similar setup in a shell script. Just create symlinks to the script with the names of executables you want run via ssh on the host and place them in your path. The desired executable must be available in PATH on the host. 

[bash]

#!/bin/bash

THERE=`pwd | sed -e 's/\.replicant//'`

unset a i

for arg; do
  printf -v 'a[i++]' %q "$arg"
done
exec ssh host "cd $THERE; `basename $0` ${a[@]}"

[/bash]

Adapted from: http://mywiki.wooledge.org/BashFAQ/096

[kiran_s_]

can you please help me about how to run host+offload programming model on host and mic

i have one only one mic card on host.

please specify the steps for the same.with sample example code. and different offload options.in out and inout. 

[TaylorIoTKidd]

Kiran,

You can find example code in the compiler Samples directory (/opt/intel/composer_xe/Samples/). Look under the directory, mic_samples.

There is a lot of very good documentation and tutorials on the developer website, software.intel.com/mic-developer.

Regards
--
Taylor