topic Hi all, in Software Archive

Failed to explicit offload on Xeon Phi KNC

Rolly_N_ — Fri, 10 Mar 2017 17:53:30 GMT

Dear all,

I am referring to the guide on Explicit offload QE to Xeon Phi KNC (7120P) here,

https://software.intel.com/en-us/articles/explicit-offload-for-quantum-espresso

I tried to follow the above steps but I failed to run pw.x (QE v5.3.0) on 2 Xeon Phi 7120P using the mpirun.sh script. The error reads:

allocating buffers 2048 2048 1024
on device 0
threshold 20000000000.0000
allocating buffers 2048 2048 1024
on device 0
threshold 20000000000.0000
offload error: cannot create buffer on device 0 (error code 14)
offload error: cannot create buffer on device 0 (error code 14)

This is how I run the script,

[qeuser@node09 ~]$ ~/mpirun/mpirun.sh -p 1 -w ~/libxphi/xphilibwrapper.sh -x ~/QE530-KNC-OL/espresso-5.3.0/bin/pw.x -i ~/rolly/AUSURF112/ausurf.in

I have already scp all the lib and bin files to each Xeon Phi 7120P and I have also compiled the libxphi lib. This is how the libxphi directory reads,

[qeuser@node09 libxphi]$ ls
build-library.sh libmkl_proxy.so LICENSE README.md xphilibmod.mod xphilib.o xphilib_proxy.o
clean.sh libxphi.so mkl_proxy.c xphilib.f90 xphilibmod.modmic xphilib_proxy.f90 xphilibwrapper.sh

I suppose this is okay.

However, I found it interesting that I can run a single instance on mic0 but it is very slow. This is how I did it,

[qeuser@node09 ~]$ export LD_LIBRARY_PATH=/home/qeuser/libxphi/:$LD_LIBRARY_PATH

[qeuser@node09 ~]$ LD_PRELOAD="/home/qeuser/libxphi/libxphi.so" /home/qeuser/QE530-KNC-OL/espresso-5.3.0/bin/pw.x < /home/qeuser/rolly/AUSURF112/ausurf.in

Error messages were also produced,

allocating buffers 2048 2048 1024
on device 0
threshold 20000000000.0000
ERROR: ld.so: object '/home/qeuser/libxphi/libxphi.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/home/qeuser/libxphi/libxphi.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
buffer allocation 4.02019500732422 s

The AUSURF112 benchmark completed, it took very long time, and it reads

PWSCF : 9h56m CPU 0h36m WALL

This run was terminated on: 2:25: 1 11Mar2017

=------------------------------------------------------------------------------=
JOB DONE.
=------------------------------------------------------------------------------=
2
offload error: cannot unload library from the device 0 (error code 14)

On the host I can see one copy of pw.x is running, and on mic0 I can see that offload_main and coi_daemon are running by the micuser. But it is very slow.

So, is this offload error: cannot create buffer on device 0 (error code 14) related to the mpirun.sh script and the libxphi.so were not preloaded even it is present and the libxphi cannot be unload after completion???

With two Xeon E5-2683v3 CPU of 56 threads, it tooks only 9 mins and 15 seconds, but with this single instance offloaded to Xeon Phi, it tooks 10 hours?

I am running CentOS 7.1 + Intel MPSS 3.8.1 + Intel psxe 2017 update 1, and I have already made a symbolic link of the psxevars.sh to /etc/profile.d and I can use mpirun to pw.x on the host, but not offload to mic0 and mic1

Are these compatibility issues because the libxphi and mpirun.sh were written 2 years ago? How can these be fixed?

Thank you,

Rolly

Hi all,

Rolly_N_ — Mon, 27 Mar 2017 15:11:07 GMT

Hi all,

Thanks to Dr. Dahnken, developer of the libxphi library. He found the solution to my question.

https://github.com/cdahnken/libxphi

https://github.com/hfp/mpirun/blob/master/mpirun.py

In configuration with CentOS 7.1+ MPSS-3.8.1 + Intel PSXE 2017 update 1, a line have to be removed from the mpirun.py

runstring = "mpirun -bootstrap ssh"
if (None == args.mri): runstring = runstring \
                    + " -genv I_MPI_PIN_DOMAIN=auto" \
#                   + " -genv OFFLOAD_INIT=on_start" \
                    + " -genv MIC_USE_2MB_BUFFERS=2m" \
                    + " -genv MIC_ENV_PREFIX=" + micenv \
                    + " -genv " + micenv + "_KMP_AFFINITY=" + args.micaffinity + ",granularity=fine" \
                    + " -genv " + micenv + "_OMP_SCHEDULE=" + args.schedule \
                    + " -genv " + micenv + "_OMP_NUM_THREADS=" + str(max((mcores + min(0, args.reserved)) * 4, args.mthreads))

After removal of the line "+ " -genv OFFLOAD_INIT=on_start" \", it runs offloading to on 2x 7120P.

With these parameters, 2x 7120P complete the QE AUSURF112 benchmark in 17min56sec,

$ QE_MIC_BLOCKSIZE_M=2048
$ QE_MIC_BLOCKSIZE_N=2048
$ QE_MIC_BLOCKSIZE_K=512

By increasing K to 1024, 2x 7120P complete the QE AUSURF112 benchmark in 17min17sec,

I found 3 warning are given by the code:

(1) OMP: Info #256: KMP_PLACE_THREADS variable deprecated, please use KMP_HW_SUBSET instead.
(2) OMP: Warning #250: KMP_HW_SUBSET "o" offset designator deprecated, please use @ prefix for offset value.
(3) offload error: cannot unload library from the device 0 (error code 14)

and I am looking into these and I will report my findings.

Thanks for your attention.

Rolly

Re 1: KMP_PLACE_THREADS has

James_C_Intel2 — Mon, 27 Mar 2017 15:23:53 GMT

Re 1: KMP_PLACE_THREADS has been replaced with KMP_HW_SUBSET which better captures what the environmnet variable actually does.(The link below also points this out!)

Re 2: KMP_HW_SUBSET is documented in the compiler manual at https://software.intel.com/en-us/node/684224#DA65065D-8C30-4E3A-BA61-4AAE42DD505C (and elsewhere).

Hi James,

Rolly_N_ — Tue, 28 Mar 2017 09:11:00 GMT

Hi James,

Thanks for the info.

I took a look at the link, I believe KMP_HW_SUBSET reads something like this, KMP_HW_SUBSET=socketsS[@offset],coresC[@offset],threadsT

The mpirun.py script generates something like this,

mpirun 
-bootstrap ssh 
-genv I_MPI_PIN_DOMAIN=auto 
-genv MIC_USE_2MB_BUFFERS=2m 
-genv MIC_ENV_PREFIX=MIC 
-genv MIC_KMP_AFFINITY=balanced,granularity=fine 
-genv MIC_OMP_SCHEDULE=static 
-genv MIC_OMP_NUM_THREADS=32 
-host node09 
-np 1 
-env KMP_AFFINITY=compact,1,granularity=fine 
-env OMP_SCHEDULE=static 
-env OMP_NUM_THREADS=4 
-env OFFLOAD_DEVICES=0 
-env MIC_KMP_PLACE_THREADS=8c,4t,0o 
-env MIC_OMP_SCHEDULE=static 
/home/qeuser/libxphi/xphilibwrapper.sh  
/home/qeuser/QE530-KNC-OL/espresso-5.3.0/bin/pw.x  < /home/qeuser/rolly/AUSURF112/ausurf.in : 
-host node09 
-np 1 
-env KMP_AFFINITY=compact,1,granularity=fine 
-env OMP_SCHEDULE=static 
-env OMP_NUM_THREADS=4 
-env OFFLOAD_DEVICES=0 
-env MIC_KMP_PLACE_THREADS=8c,4t,8o 
-env MIC_OMP_SCHEDULE=static 
/home/qeuser/libxphi/xphilibwrapper.sh  
/home/qeuser/QE530-KNC-OL/espresso-5.3.0/bin/pw.x  < /home/qeuser/rolly/AUSURF112/ausurf.in :
...
-host node09 
-np 1 
-env KMP_AFFINITY=compact,1,granularity=fine 
-env OMP_SCHEDULE=static 
-env OMP_NUM_THREADS=4 
-env OFFLOAD_DEVICES=1 
-env MIC_KMP_PLACE_THREADS=8c,4t,48o 
-env MIC_OMP_SCHEDULE=static 
/home/qeuser/libxphi/xphilibwrapper.sh  
/home/qeuser/QE530-KNC-OL/espresso-5.3.0/bin/pw.x  < /home/qeuser/rolly/AUSURF112/ausurf.in

So, is it correct to change environmental variable MIC_KMP_PLACE_THREADS to MIC_KMP_HW_SUBSET? Shall it looks like this?

-host node09 
-np 1 
-env KMP_AFFINITY=compact,1,granularity=fine 
-env OMP_SCHEDULE=static 
-env OMP_NUM_THREADS=4 
-env OFFLOAD_DEVICES=0 
-env MIC_KMP_HW_SUBSET=0s,8c@0,4t 
-env MIC_OMP_SCHEDULE=static 
/home/qeuser/libxphi/xphilibwrapper.sh  
/home/qeuser/QE530-KNC-OL/espresso-5.3.0/bin/pw.x  < /home/qeuser/rolly/AUSURF112/ausurf.in :
-host node09 
-np 1 
-env KMP_AFFINITY=compact,1,granularity=fine 
-env OMP_SCHEDULE=static 
-env OMP_NUM_THREADS=4 
-env OFFLOAD_DEVICES=0 
-env MIC_KMP_HW_SUBSET=0s,8c@8,4t 
-env MIC_OMP_SCHEDULE=static 
/home/qeuser/libxphi/xphilibwrapper.sh  
/home/qeuser/QE530-KNC-OL/espresso-5.3.0/bin/pw.x  < /home/qeuser/rolly/AUSURF112/ausurf.in :
...
-host node09 
-np 1 
-env KMP_AFFINITY=compact,1,granularity=fine 
-env OMP_SCHEDULE=static 
-env OMP_NUM_THREADS=4 
-env OFFLOAD_DEVICES=1 
-env MIC_KMP_HW_SUBSET=0s@1,8c@48,4t 
-env MIC_OMP_SCHEDULE=static 
/home/qeuser/libxphi/xphilibwrapper.sh  
/home/qeuser/QE530-KNC-OL/espresso-5.3.0/bin/pw.x  < /home/qeuser/rolly/AUSURF112/ausurf.in :

The MIC_KMP_PLACE_THREADS also assign different thread numbers to different core via the "o" offset designator, but how can I do the same with MIC_KMP_HW_SUBSET? By the way, is MIC_KMP_HW_SUBSET the correct environmental variable?

Thanks,

Rolly

Hi all,

Rolly_N_ — Thu, 30 Mar 2017 02:37:14 GMT

Hi all,

I am able to remove the first two warnings by modifying the mpirun.py. Can anyone check if these are correct? Thank you!

#! /usr/bin/env python
###############################################################################
## Copyright (c) 2014-2015, Intel Corporation                                ##
## All rights reserved.                                                      ##
##                                                                           ##
## Redistribution and use in source and binary forms, with or without        ##
## modification, are permitted provided that the following conditions        ##
## are met:                                                                  ##
## 1. Redistributions of source code must retain the above copyright         ##
##    notice, this list of conditions and the following disclaimer.          ##
## 2. Redistributions in binary form must reproduce the above copyright      ##
##    notice, this list of conditions and the following disclaimer in the    ##
##    documentation and/or other materials provided with the distribution.   ##
## 3. Neither the name of the copyright holder nor the names of its          ##
##    contributors may be used to endorse or promote products derived        ##
##    from this software without specific prior written permission.          ##
##                                                                           ##
## THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS       ##
## "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT         ##
## LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR     ##
## A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT      ##
## HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,    ##
## SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED  ##
## TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR    ##
## PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF    ##
## LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING      ##
## NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS        ##
## SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.              ##
###############################################################################
## Hans Pabst, Christopher Dahnken, Intel Corporation                        ##
###############################################################################
import platform
import sys
import os

try:
    import argparse
except ImportError, e:
    print 'Failed to import the argparse module: ', e
    print 'Please follow the troubleshooting instructions at:'
    print 'https://github.com/hfp/mpirun#troubleshooting'


def micshift(i, rcores, ncores, nthreads):
    return str(max(ncores + min(0, rcores), nthreads)) + "c@" + str(i * ncores) + "," + str(nthreads) + "t"  


nodename = platform.node().split(".")[0]
micenv = "MIC"

parser = argparse.ArgumentParser()
parser.add_argument("-n", "--nodelist", help="list of comma separated node names", default=["localhost", nodename]["" != nodename])
parser.add_argument("-p", "--cpuprocs", help="number of processes per socket (host)", type=int, default=1)
parser.add_argument("-q", "--micprocs", help="number of processes per mic (native)", type=int, default=1)
parser.add_argument("-s", "--nsockets", help="number of sockets per node", type=int, default=1)
parser.add_argument("-d", "--ndevices", help="number of devices per node", type=int, default=0)
parser.add_argument("-e", "--cpucores", help="number of CPU cores per socket", type=int, default=1)
parser.add_argument("-t", "--nthreads", help="number of CPU threads per core", type=int, default=2)
parser.add_argument("-m", "--miccores", help="number of MIC cores per device", type=int, default=57)
parser.add_argument("-r", "--reserved", help="number of MIC cores reserved", type=int, default=sys.maxint)
parser.add_argument("-u", "--mthreads", help="number of MIC threads per core", type=int, default=4)
parser.add_argument("-a", "--cpuaffinity", help="affinity (CPU) e.g., compact", default="compact")
parser.add_argument("-b", "--micaffinity", help="affinity (MIC) e.g., balanced", default="balanced")
parser.add_argument("-c", "--schedule", help="schedule, e.g. dynamic", default="static")
parser.add_argument("-w", "--wrapper", help="wrapper")
parser.add_argument("-g", "--debugger", help="debugger")
parser.add_argument("-i", "--inputfile", help="inputfile (<)")
parser.add_argument("-0", "--hr0", help="executable (rank-0)")
parser.add_argument("-x", "--hri", help="executable (host)")
parser.add_argument("-y", "--mri", help="executable (mic)")
parser.add_argument("-z", "--micpre", help="prefixed mic name", action="store_true")
parser.add_argument("-v", "--dryrun", help="dryrun", action="store_true")
args, unknown = parser.parse_known_args()

if (None != args.inputfile):
    arguments = " ".join(unknown) + " < " + args.inputfile
else:
    arguments = " ".join(unknown)
if ("" != arguments): arguments = " " + arguments

if (None != args.wrapper):
    wrapper = args.wrapper + [" ", " " + str(args.debugger) + " -ex run --args"][None != args.debugger]
else:
    wrapper = ["", str(args.debugger) + " -ex run --args"][None != args.debugger]
if ("" != wrapper): wrapper = wrapper + " "

if (1 < args.nsockets): args.cpuaffinity = args.cpuaffinity + ",1"
if (1 < args.nthreads): args.cpuaffinity = args.cpuaffinity + ",granularity=fine"

if (None == args.mri):
    if (sys.maxint == args.reserved): args.reserved = 1
    if (0 < args.ndevices):
        cpusockets = min(args.nsockets, args.ndevices)
    else:
        cpusockets = args.nsockets
    nparts = args.cpuprocs
else:
    if (sys.maxint == args.reserved): args.reserved = 0
    cpusockets = args.nsockets
    nparts = args.micprocs

cputhreads = cpusockets * args.cpucores * args.nthreads
mcores = int(0 < nparts) * max((args.miccores - max(args.reserved, 0)), nparts) / max(nparts, 1)
pthreads = int(0 < (args.cpuprocs * cpusockets)) * cputhreads / max(args.cpuprocs * cpusockets, 1)
remainder = cputhreads - pthreads * args.cpuprocs * cpusockets


runstring = "mpirun -bootstrap ssh"
if (None == args.mri): runstring = runstring \
                    + " -genv I_MPI_PIN_DOMAIN=auto" \
		    + " -genv MIC_USE_2MB_BUFFERS=2m" \
                    + " -genv MIC_ENV_PREFIX=" + micenv \
                    + " -genv " + micenv + "_KMP_AFFINITY=" + args.micaffinity + ",granularity=fine" \
                    + " -genv " + micenv + "_OMP_SCHEDULE=" + args.schedule \
                    + " -genv " + micenv + "_OMP_NUM_THREADS=" + str(max((mcores + min(0, args.reserved)) * 4, args.mthreads))
else: runstring = runstring \
                    + " -genv I_MPI_MIC=1"
if (None != args.hr0): runstring = runstring \
                    + " -host " + args.nodelist.split(",")[0] + " -np 1" \
                    + " -env I_MPI_PIN_DOMAIN=auto" \
                    + " -env KMP_AFFINITY=" + args.cpuaffinity \
                    + " -env OMP_NUM_THREADS=" + str(args.nthreads) \
                    + " " + wrapper + args.hr0 + arguments \
                    + " :"
for n in args.nodelist.split(","):
    for s in range(0, cpusockets):
        for p in range(0, args.cpuprocs):
            runstring = runstring \
                    + " -host " + n + " -np 1" \
                    + " -env KMP_AFFINITY=" + args.cpuaffinity \
                    + " -env OMP_SCHEDULE=" + args.schedule \
                    + " -env OMP_NUM_THREADS=" + str(pthreads - [0, args.nthreads][None != args.hr0 and 0 == s and 0 == p])
            if (None == args.mri): runstring = runstring \
                    + " -env OFFLOAD_DEVICES=" + str(s) \
                    + " -env " + micenv + "_KMP_HW_SUBSET=" + str(s) + "s," + micshift(p, args.reserved, mcores, args.mthreads) \
                    + " -env " + micenv + "_OMP_SCHEDULE=" + args.schedule
            else: runstring = runstring \
                    + " -env I_MPI_PIN_DOMAIN=auto"
            if (None != args.hri): runstring = runstring \
                    + " " + wrapper + args.hri + arguments \
                    + " :"
    if (None != args.mri):
        for d in range(0, args.ndevices):
            for m in range(0, args.micprocs):
                runstring = runstring \
                    + " -host " + ["", n + "-"][int(args.micpre)] + "mic" + str(d) + " -np 1" \
                    + " -env LD_LIBRARY_PATH=$MIC_LD_LIBRARY_PATH" \
                    + " -env I_MPI_PIN=off" \
                    + " -env KMP_HW_SUBSET=" + str(d) + "s," + micshift(m, args.reserved, mcores, args.mthreads) \
                    + " -env KMP_AFFINITY=" + args.micaffinity + ",granularity=fine" \
                    + " -env OMP_SCHEDULE=" + args.schedule \
                    + " -env OMP_NUM_THREADS=" + str(max(mcores + min(0, args.reserved), args.mthreads) * 4) \
                    + " " + args.mri + arguments \
                    + " :"
        if (None != args.hri and 0 < remainder and 0 < cputhreads and 0 < args.cpuprocs): runstring = runstring \
                    + " -host " + n + " -np 1" \
                    + " -env I_MPI_PIN_DOMAIN=auto" \
                    + " -env KMP_AFFINITY=" + args.cpuaffinity \
                    + " -env OMP_SCHEDULE=" + args.schedule \
                    + " -env OMP_NUM_THREADS=" + str(remainder) \
                    + " " + wrapper + args.hri + arguments \
                    + " :"

runstring = runstring[0:len(runstring)-2]
print runstring
print

result = 0
if (False == args.dryrun):
    result = os.system(runstring)

sys.exit([0, 1][0 != result])

But for the 3rd warning, (3) offload error: cannot unload library from the device 0 (error code 14), can anyone help?

Thanks,

Rolly