HDF5 error

[joshbowden]

Hi,

I have some compiler assisted offload code that when I use a wrapper for the HDF library (v5 1.8.8) breaks with the following error:

On the remote process, dlopen() failed. The error message sent back from the sink is /var/volatile/tmp/coi_procs/1/199062/load_lib/icpcoutRTBEtd: undefined symbol: H5P_CLS_FILE_ACCESS_g
On the sink, dlopen() returned NULL. The result of dlerror() is "/var/volatile/tmp/coi_procs/1/199062/load_lib/icpcoutRTBEtd: undefined symbol: H5P_CLS_FILE_ACCESS_g"
offload error: cannot load library to the device 0 (error code 20)

I do not use the HDF library on the MIC card, so I wonder why it requires the library.

Thanks,

Josh

[Kevin_D_Intel]

Somehow when building your app with offload code a reference to HDF was included in the target compilation leading to the reference in the target image. I’m guessing you have the Composer XE 2013 SP1 release where the target image is a Linux shared object which is permitted to have unresolved references at link time, but, when those are not resolved at run-time you receive an error in the form you cited.

It is most often possible to flush out where the reference originates at compile/link time using the -zdefs option to the final link for the part of the application that includes the offload code. You need to use the compiler driver (icc, icpc, ifort) for the link and add that option using the -offload-option and use the form that’s appropriate for the build step.

For example:

When compiling and linking in one step, use this form:

icc -offload-option,mic,compiler,”-Wl,-zdefs” <along with source files and other options>

When linking only in one step, use this form:

icc -offload-option,mic,ld,”-zdefs” <along with object files and other options>

Knowing the origin of the reference should help know how to adjust the build to avoid the reference.

[joshbowden]

Hi Kevin,

Thanks for the link option tip. The output I get from it indicates that the HDF 5 library functions I am using are not defined.

The file the hdf functions are being called from (test_jacobi.C) does not have any offload functionality so I do not know why it is needs to be compiled into a MIC.o object anyway? Does the HDF library have to be compiled for offload mic use?

Example compiler output with the -zdefs flag:

icpc -o test_jacobi CBasicFunctions.o jacobi_rcpp_micomp.o  test_jacobi.o  CTimer.o -L/apps/intel/mkl/11.1/mkl/lib/intel64  -openmp -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread  -lhdf5  -lz  -lm -offload-attribute-target=mic -offload-option,mic,ld,"  -L/apps/intel/mkl/11.1/mkl/lib/mic -zdefs -lmkl_intel_lp64 -lmkl_core -lmkl_sequential" 
test_jacobiMIC.o: In function `CHDF5File<double>::CreateFullHDFGroupPath(std::string, bool)':
test_jacobi.C:(.text._ZN9CHDF5FileIdE22CreateFullHDFGroupPathESsb[_ZN9CHDF5FileIdE22CreateFullHDFGroupPathESsb]+0x329): undefined reference to `H5Gopen2'
test_jacobi.C:(.text._ZN9CHDF5FileIdE22CreateFullHDFGroupPathESsb[_ZN9CHDF5FileIdE22CreateFullHDFGroupPathESsb]+0x338): undefined reference to `H5Gclose'
test_jacobi.C:(.text._ZN9CHDF5FileIdE22CreateFullHDFGroupPathESsb[_ZN9CHDF5FileIdE22CreateFullHDFGroupPathESsb]+0x52f): undefined reference to `H5Gopen2'
test_jacobi.C:(.text._ZN9CHDF5FileIdE22CreateFullHDFGroupPathESsb[_ZN9CHDF5FileIdE22CreateFullHDFGroupPathESsb]+0x540): undefined reference to `H5Gclose'
test_jacobi.C:(.text._ZN9CHDF5FileIdE22CreateFullHDFGroupPathESsb[_ZN9CHDF5FileIdE22CreateFullHDFGroupPathESsb]+0x624): undefined reference to `H5Gcreate2'
test_jacobi.C:(.text._ZN9CHDF5FileIdE22CreateFullHDFGroupPathESsb[_ZN9CHDF5FileIdE22CreateFullHDFGroupPathESsb]+0x951): undefined reference to `H5Gcreate2'

Thanks for you help.

Josh

[joshbowden]

Hi Kevin,

I have got the issue sorted. I compiled the file containing offload regions separately using: -offload-attribute-target=mic -offload-option,mic,compiler,"  -L/apps/intel/mkl/11.1/mkl/lib/mic -zdefs -lmkl_intel_lp64 -lmkl_core -lmkl_sequential" , along with its dependencies and then linked the *.MIC.o and *.o in a separate line.

I'm not sure how to get my Makefile to do this for me yet though.

Also, when I want to use the hdf5 library functions in a file that contains offload regions, do I have to do a conditional compile using #ifndef __MIC__ around the code that uses hdf?

Thanks again,

Josh

[Kevin_D_Intel]

Hi Josh,

Yes, the linking option exposes those undefined references that caused the earlier run-time failure but with the added insight about the location of the references. An aside, the nice feature in the Intel Parallel Studio XE 2015 (15.0 compiler) release is the underlying changes to the target binary that automatically expose those undefined references at link-time without the need for manual intervention with -zdefs.

You should not manipulate the MIC.o file yourself. Whatever the final resolution is for you here it will not include that. The manipulation of the MIC.o must be done by the compiler itself, and in future releases this file disappears from the user view with greater use of FAT binary files containing both the host and target binary images.

I pretty sure the issue relates to unnecessary use of -offload-attribute-target=mic for source files containing routines that do not participate in the execution of the offload code; therefore, should not be compiled with that option. The option has a special use which you may already understand. Adding offload code sections in some files does not require compiling that file (or others) with that option. The compiler auto-detects the presence of offload language extensions and triggers the target compilation automatically. For offload code with external dependencies on other routines, selectively use the __attribute__((target(mic))) to declare those, and there is no need to use the -offload-attribute-target option when compiling files containing this attribute either. One should paint only with the broader brush of the -offload-attribute-target=mic where all or a significant number of routines in the source file participate in the offload execution. A typical application of the option is building a library in support of offload code execution where many of the routines may be called during execution of offload constructs.   
 
So, if you have a single file containing self-contained offload code sections then it’s likely your build does not require any manual adjustments in terms of the compilation of that file and thus no need for the -offload-attribute-target option. If you are not calling MKL routines within the offload code, then you also do not need special compilation/link options to link MKL for the target image as you showed in your command-line.

As for conditional compilation, that is applicable in some cases to avoid inclusion of code from participating in the target compilation, but it does not sound like the necessary direction for your case just yet.

[joshbowden]

Hi Kevin,

Your description of the situation makes sense to me. I have tested the conditional compilation within the file that was causing issues and this did fix the problem and I'll write a better makefile to deal with my code having offload sections and i'll test out removing the -offload-attribute-target=mic altogether. I used the MKL link line advisor to get the full comand line originally, as I need to link in MKL functions withing the offload section and I have possibly added/removed commands since then. I also have been tripped up by trying to link *MIC.o files explicitly in the Makefile previously, so now I just let the compiler do its thing with minimal intervention. The compiler has suggested using __attribute__(target(mic)) for some functions that were called within the offload section, so that was helpful.

Thanks for your help and sugestions.

Kind regards,

Josh

[John_R_2]

I saw this thread and am just curious - has anyone tried using parallel HDF5 on Xeon Phi?   How did it go?

John Readey

HDF Group