/tmp/coi_procs libiomp5.so fail

[holmz]

<Edit>
I decided to put it into reverse and back up to a known starting point.
 

> cp intel composer_xe_2013_sp1.1.106/Samples/en_US/Fortran/mic_samples/LEO_Fortran_into/* .
> make mic
...
 Build complete

> ./leoF_intro.out

And what I get is:

System configuration pre-check
  Checking for Intel(R) Xeon Phi (TM) (Target CPU) devices...

  Number of Target devices installed:  1

]7848: renam failed: No such file or directory on /tmp/coi_procs/1/.tmp_40_44/libimf.so -> /tmp/coi_procs/1/7848/libimf.so
offload error: cannot start process on the deviice 0 (error code 1)

============================ end of edit ==========================

I believe that I have have some code that runs on the mic natively, as I was using the micload a couple of weeks ago. I assume this because it took a while to get the SOs moved around and the environment variables set to point to where the .so files were at before it started working.

<later...> I have a small program that uses OpenMP and parallel Do SIMD on the host, a single routine was tested to give the proper results and some benchmarking as done on the host with various threads=# given. It is a bit of a different beast as my vector code is in a library, which I compile with a makefile and have the xiar and xild used to create the libmic.a file. The calling (host) program is compiled seperately and I am not using xiar nor xild for that... Nor do I see an easy way to get there with the framework I am using which wraps the calling Fortran in a C wrapper.

I am to the point, after many hours of xiar and other issues, where it is now appearing to want run on the mic. Namely at the call to the mic offload I am seeing:

7305: rename failed: No such file or directory on /tmp/coi_procs/1/.tmp_04_22/libiomp5.so -> /tmp/coi_procs/1/7305/libiomp5.so

offload error: cannot start process on the device 0 (error code 1)

As there was no /tmp/coi_procs, nor /tmp/coi_procs/1 , I created both and chmod 777 to both. Either the subdirectories in /tmp/coi_procs get destroyed or they never get created... (I am not sure which). When I ssh'ed into mic:0 I saw that there is a coi demon running.

--- Took a crack at removing the framework...

Apologies in advance as the machine is not connected to the net so I have to fat-finger it in.
ifort is: composer_xe-2013
 

MODULE VECTORF
USE OMP_LIB
USE MIC_LIB
USE ISO_C_BINDING
IMPLICIT NONE

INTEGER(KIND=4) :: MICTHREADS = 10
INTEGER(KIND=4) :: HOSTTHREADS = 40

TYPE(OFFLOAD_STATUS) :: MIC_STATUS

PUBLIC :: RH$INIT, RH$CVMUL  ! And a lot more
PRIVATE

INTERFACE RH$CVMUL
  MODULE PROCEDURE RH$VSMUL_C8, RH$CVMUL_R4 ! more
END INTERFACE
...
CONTAINS

!PURE
SUBROUTINE  RH$INIT(NTHREADS, Use_Mic)
INTEGER(KIND=4)               , INTENT(IN   ) :: NTHREADS
LOGICAL(KIND=4)               , INTENT(IN   ) :: Use_Mic

MIC_Threads = 120
Host_Threads = OMP_GET_NUM_PROCS()

! More use_Mic and other stuff goes in here !

WRITE(*,*)'VF:1123  MIC_Threads =',MIC_Threads ,' Host_Threads  =',Host_Threads 

RETURN
END SUBROUTINE  RH$INIT


!DIR$ ATTRIBUTES OFFLOAD:mic :: RH$VSMUL_C8R8C8
!PURE
SUBROUTINE  RH$VSMUL_C8R8C8(X, Scale, Z, N)
INTEGER(KIND=4)                   , INTENT(IN   ) :: N
COMPLEX(KIND=4), DIMENSION(N), INTENT(IN   ) :: X
COMPLEX(KIND=4), DIMENSION(N), INTENT(  OUT) :: Z
REAL(KIND=8)                      , INTENT(IN   ) :: Scale

INTEGER(KIND=4)                       :: I
INTEGER(KIND=4)                       :: Parallel

!DIR$ ASSUME_ALIGNED X:64
!DIR$ ASSUME_ALIGNED Z:64

Parallel = OMP_IN_PARALLEL

IF(Parallel) THEN
  !DIR$ PREFETCH X:1:16
  !$OMP DO SIMD NUM_THREADS=(MICTHREADS) ALIGNED(X, Z) SHARED(X,Z,Scale)
  DO I = 1 , N
    Z(I)= X(I) * Scale
  ENDDO
  !$OMP END DO SIMD
ELSE
  !DIR$ PREFETCH X:1:16
  !$OMP PARALLEL DO SIMD NUM_THREADS=(MICTHREADS)  SHARED(X,Z,Scale)
  DO I = 1 , N
    Z(I)= X(I) * Scale
  ENDDO
  !$OMP END PARALLEL DO SIMD 
ENDIF

RETURN
END SUBROUTINE  RH$VSMUL_C8R8C8

END MODULE VECTORF

The make file is:

DO_DBG = -nod-lines
F90_WARN = ifort -O0 -132 -fixed ${DO_DBG } -check all - warn all -recursive
F90_NOWARN = ifort -O3 -132 -fixed ${DO_DBG } -recursive

RHFLAGS = -shared -Wl

RHINKS = -I$(<mydir here>)/inc
RHINKS2 = -I$(<mydir here>)/lib/mic
RHINKS3 = -I$(<mydir here>)/inc
RHINKS4 = -I$(<anothedir here>)/inc
RHINKS5 = -I$(<ifort area>)/mkl/include
LIB = ../../lib/libmic.a

rh objects vectorf.0 #others

#--- Default rules ----
all: $(lLIB)

#--- Implicite rule ----
#%.0: %.f90
#        $(F90) -c $< -o $@

#--- Depenedancies ----
vectorf.o: vectorf.f90
     $(F90_NOWARN) $(RHINKS) $(RHINKS3) $(RHINKS4) -offload -offload-option,mic.compiler,"-z defs" -offload-attribute-target=mic -align -cpp -openmp -opemmp-report2 -axAVX -diag-file=vectorf.compile_txt -opt-report 2 -opt-report-file=vectorf.opt-rpt -Winline -vec-report=3 -no-vec-guard-write -c $, $@

clean: rm -f *.0 *.micmiod *.miod *.core.* *.compile_txt *.fpp *.opt_rpt ../libmic.a

$(LIB): $(rhobjects)
#was  ar sr $(LIB) $(rhobjects)
           xild -lib -qoffload-build rcs $(LIB) $(rhobjects)
           xiar -goffload-build rcs $(LIB) $(rhobjects)

And it builds.

Host code:

PROGRAM TEST_LIB
USE VECTORF
IMPLICIT NONE
!DIR$ ATTRIBUTES ALIGN: 64:: cinbuf
!DIR$ ATTRIBUTES ALIGN: 64:: ctemp
COMPLEX(KIND=4), DIMENSION(:), ALLOCATABLE :: cinbuf
COMPLEX(KIND=4), DIMENSION(:), ALLOCATABLE :: ctemp

REAL(KIND=8) :: Scale = 0.5

INTEGER(KIND=4) :: Iter                  = 1e6
INTEGER(KIND=4) :: I
INTEGER(KIND=4) :: nFFT              = 1048576
INTEGER(KIND=4) :: NTHREADS = 8
LOGICAL(KIND=4) :: Use_MIC      = .TRUE.

ALLOCATE(CTEMP(NFFT))
ALLOCATE(Cinbuf(NFFT))

DO I = 1, NFFT
  cinbuf(I) = CMPLX(I,(I-1))
ENDDO

DO I = 1, Iter
  !DIR$ OFFLOAD VBEGIN TEARGET(mic:0) MANDATOR IN(cinbuf:LENGTH(nFFT) IN(Scale) OUT(cTemp:LENGTH(nFFT))
  CALL RH$VSMUL(cInbuf, Scale, cTemp, nFFT)
  !DIR$ END OFFLOAD
ENDDO


IF(ALLOCATED(CTEMP)  DEALLOCATE(CTEMP)
IF(ALLOCATED(Cinbuf)  DEALLOCATE(Cinbuf)

END PROGRAM TEST_LIB

command to build the host/main:

ifort -O3 -cpp -fixed -132 -axAVX -openmp -L<mydir> -lmic test_lib.for -o test_lib.0

I either get the coi_proc issue with /tmp/coi_procs/1/libiomp5.so not copying, or I get mp_vectorf_rh$init_ not found

[holmz · Accepted solution]

A combination of the Samples code and drdobbs Xeon Phi: A Jumpstart Introduction , and a coworker resulted in success.

Required is that the environment variable SINK_LD_LIBRARY_PATH needs to contain the following (maybe there is a .csh that sets some of this?)

#tsch
setenv SINK_LD_LIBRARY_PATH <intel>/compiler/lib/mic: <intel>/mkl/lib/mic:<my library path>

When that is NOT correct then the /tmp/coi_procs/1/.?? does NOT get the library to transfer up to the MIC, and the result is the error message "7305: error rename failed".