Compile warnings and offload errors

[Cheng_C_]

(I could not publish my previous post, which is in the draft status after I submitted it, so I'm duplicating another one)

Dear all, I got some compile warnings and offload errors when I offloaded part of my C++ code to Intel Phi for acceleration. I hope I could get some help here.

I first read some tiff images using libtiff and make some first-step calculations to the images on the host. Then I want to make some further calculations to the first-step results on Phi using MKL. The code offloaded to Phi only uses the standard C++ and MKL libraries. When I compile the code, there are some warnings associated with the stdlib.h file. When I run the code, I got some undefined symbol errors about the libtiff library. The offloaded code has nothing to do with libtiff. I attached my makefile, the compile warnings and runtime errors. Thanks for the help.

The makefile:

CC = icpc 
LDIR = -L$(MKLROOT)/lib/intel64
IDIR = -I$(MKLROOT)/include 
LLIB = -lmkl_intel_ilp64 -lmkl_core -lmkl_intel_thread -lpthread -lm -liomp5 -ltiff
FLAG = -no-multibyte-chars -openmp
DEF = -DDEBUG_MAIN -DWRITE_TO_FILE -DMKL_ILP64
MIC = -offload-option,mic,compiler,"-I/usr/include -L$(MKLROOT)/lib/mic -lmkl_intel_ilp64 -lmkl_core -lmkl_intel_thread" -offload-attribute-target=mic -watch=mic-cmd

main: main.cpp dpc.cpp dpc.h
        $(CC) main.cpp dpc.cpp $(LLIB) -o main $(MIC) $(IDIR) $(LDIR) $(DEF) $(FLAG)

clean:
        rm -rf *o main

The compile warnings:

rm -rf *o main
icpc  main.cpp dpc.cpp -lmkl_intel_ilp64 -lmkl_core -lmkl_intel_thread -lpthread -lm -liomp5 -ltiff -o main -offload-option,mic,compiler,"-I/usr/include -L/software/intel/2013/composer_xe_2013_sp1/mkl/lib/mic  -lmkl_intel_ilp64 -lmkl_core -lmkl_intel_thread" -offload-attribute-target=mic -watch=mic-cmd -I/software/intel/2013/composer_xe_2013_sp1/mkl/include  -L/software/intel/2013/composer_xe_2013_sp1/mkl/lib/intel64 -DDEBUG_MAIN -DWRITE_TO_FILE -DMKL_ILP64 -no-multibyte-chars -openmp
MIC command line:
icpc -I/usr/include -L/software/intel/2013/composer_xe_2013_sp1/mkl/lib/mic -I/software/intel/2013/composer_xe_2013_sp1/mkl/include main.cpp dpc.cpp -o main -offload-attribute-target=mic -L/software/intel/2013/composer_xe_2013_sp1/mkl/lib/intel64 -DDEBUG_MAIN -DWRITE_TO_FILE -DMKL_ILP64 -no-multibyte-chars -openmp -lmkl_intel_ilp64 -lmkl_core -lmkl_intel_thread
In file included from /usr/include/stdlib.h(43),
                 from /software/intel/2013/composer_xe_2013_sp1.1.106/compiler/include/omp.h(142),
                 from main.cpp(17):
/usr/include/bits/waitstatus.h(75): warning #3218: *MIC* class/struct may fail when offloaded because this field is misaligned or contains data that is misaligned
  	unsigned int __w_retcode:8; /* Return code if exited normally.  */
  	             ^

In file included from /usr/include/stdlib.h(43),
                 from /software/intel/2013/composer_xe_2013_sp1.1.106/compiler/include/omp.h(142),
                 from main.cpp(17):
/usr/include/bits/waitstatus.h(89): warning #3218: *MIC* class/struct may fail when offloaded because this field is misaligned or contains data that is misaligned
  	unsigned int __w_stopsig:8; /* Stopping signal.  */
  	             ^

In file included from /usr/include/stdlib.h(43),
                 from /software/intel/2013/composer_xe_2013_sp1.1.106/compiler/include/omp.h(142),
                 from main.cpp(17):
/usr/include/bits/waitstatus.h(84): warning #3218: *MIC* class/struct may fail when offloaded because this field is misaligned or contains data that is misaligned
        } __wait_terminated;
          ^

In file included from /usr/include/stdlib.h(43),
                 from dpc.h(6),
                 from dpc.cpp(2):
/usr/include/bits/waitstatus.h(75): warning #3218: *MIC* class/struct may fail when offloaded because this field is misaligned or contains data that is misaligned
  	unsigned int __w_retcode:8; /* Return code if exited normally.  */
  	             ^

In file included from /usr/include/stdlib.h(43),
                 from dpc.h(6),
                 from dpc.cpp(2):
/usr/include/bits/waitstatus.h(89): warning #3218: *MIC* class/struct may fail when offloaded because this field is misaligned or contains data that is misaligned
  	unsigned int __w_stopsig:8; /* Stopping signal.  */
  	             ^

In file included from /usr/include/stdlib.h(43),
                 from dpc.h(6),
                 from dpc.cpp(2):
/usr/include/bits/waitstatus.h(84): warning #3218: *MIC* class/struct may fail when offloaded because this field is misaligned or contains data that is misaligned
        } __wait_terminated;
          ^

The runtime error:

The time for loading images and calculating xsum and ysum: 3.87822
On the sink, dlopen() returned NULL. The result of dlerror() is "/tmp/coi_procs/1/160843/load_lib/icpcoutVJ7iyF: undefined symbol: TIFFGetField"
On the remote process, dlopen() failed. The error message sent back from the sink is /tmp/coi_procs/1/160843/load_lib/icpcoutVJ7iyF: undefined symbol: TIFFGetField
offload error: cannot load library to the device 0 (error code 20)

[Frances_R_Intel]

It has taken us a while to answer you, so you have probably already found the answer to this but -

When using the offload directive, only data which is bitwise copyable can be passed by address to the coprocessor. For example, arrays can be passed back and forth. However, class objects are, in general, not bitwise copyable. If you want to pass those back and forth, you will need to use _Cilk_offload. You can find information on this in the C/C++ reference manual https://software.intel.com/en-us/node/512559 as well as under the Programming tab on https://software.intel.com/mic-developer. Check out especially "Parallelization with Intel® Cilk™ Plus" under https://software.intel.com/en-us/articles/programming-and-compiling-for-intel-many-integrated-core-architecture. Also, the coprocessor may be looking to load libtiff if some function in the class uses libtiff, even if that function is never called on the coprocessor.

[Cheng_C_]

Frances Roth (Intel) wrote:

It has taken us a while to answer you, so you have probably already found the answer to this but -

When using the offload directive, only data which is bitwise copyable can be passed by address to the coprocessor. For example, arrays can be passed back and forth. However, class objects are, in general, not bitwise copyable. If you want to pass those back and forth, you will need to use _Cilk_offload. You can find information on this in the C/C++ reference manual https://software.intel.com/en-us/node/512559 as well as under the Programming tab on https://software.intel.com/mic-developer. Check out especially "Parallelization with Intel® Cilk™ Plus" under https://software.intel.com/en-us/articles/programming-and-compiling-for-.... Also, the coprocessor may be looking to load libtiff if some function in the class uses libtiff, even if that function is never called on the coprocessor.

Frances, thanks very much for the reply. The problem has not been solved yet. I got stuck in it.

I only passed 2 3D arrays to the coprocessor. The compiler didn't complain that the data is not bitwise copyable. I still get the same error, even though I simplify the offload code as the following:

#pragma offload target(mic)
{
    #ifdef __MIC__
        printf("The offload part is running on Phi\n");
    #else
        printf("The offload part is running on the host\n");
    #endif
}

I have another two relevant questions:

1. Does 3D arrays in the form of array[100][100][100] bitwise copiable? How about 2D arrays like array[100][100]?

2. In the non-shared memory offload mode, suppose I only pass some simple 1D arrays to or from the coprocessor. The offload code only needs MKL. The code outside the offload region uses libtiff. I want to double check with you guys that, the coprocessor doesn't need the libtiff library to be installed on it for the host-side compilation and execution. Am I right?

Thanks for the help.

[Frances_R_Intel · Accepted solution]

Let's start off by simplifying things.- you have executed:

 source /opt/intel/composerxe/bin/compilervars.sh intel64

right? This script will set things up for use with the Intel® compilers; what I say here applies only to the Intel® compilers.

With this script, you do not need to specify many of the things you are currently specifying on your command line. Among other things, it means you do not need to specify the location of the libraries for either the host or the coprocessor. Those are set for you, using the LIBRARY_PATH, MIC_LIBRARY_PATH, LD_LIBRARY_PATH and MIC_LD_LIBRARY_PATH environment variables, which are automatically recognized by the compiler.

You already specify '-openmp' in your list of flags. It is not necessary to also specify the library using a -l option. Also, if you specify '-mkl' in your flags, you do not need to separately list the libraries or include files.

Are you actually using both pthreads and openmp in the same program? If you did not include any calls to pthread routines, you can omit the pthread library from your command line. If you did need it, you could include it by using '-pthread' in your list of flags. (You may have been copying an example using the g++ compiler here - in that case you do need to specify both openmp and pthread libraries in order to use the Intel® OpenMP library, just as you would need to explicitly specify library names and locations for mkl.)

Now for 'offload-attribute-target=mic'. This will cause any file-scope functions and data objects to be compiled for both the host and the coprocessor. I think this is where you are getting your error messages from. Any variables you have that are instances of a class will be marked for offloading. Any functions defined in the file will be marked for offloading (including functions that call the tiff routines.) So, in effect, you are trying to offload class objects and calls to the tiff routines.

You need a finer level of control here. In your source code, find those include files and function and variable declarations that need to be offloaded to the coprocessor - and only those include files and function and variable declarations - and enclose them between these pragmas:

[bash]

#pragma offload_attribute (push, target(mic)

#pragma offload_attribute (pop)

[/bash]

So, now we are down to:

[bash]

CC = icpc

LLIB = -lm -ltiff

FLAG = -no-multibyte-chars -openmp -mkl

DEF = -DDEBUG_MAIN -DWRITE_TO_FILE -DMKL_ILP64

MIC = -watch=mic-cmd

main: main.cpp dpc.cpp dpc.h

        $(CC) main.cpp dpc.cpp $(LLIB) -o main $(MIC) $(DEF) $(FLAG)

clean:

        rm -rf *o main

[/bash]

To get more information, I highly recommend Programming and Compiling for Intel® Many Integrated Core Architecture. I still learn things when I skim through that. Oh, and yes, multi-dimensional arrays are bit-wise copyable. 

If, when you go through what I have written, you find mistakes (and sadly, I do sometimes make mistakes) let me know.

[Cheng_C_]

Frances, thanks very much for the detailed and enlightening explanations. It solves all my problems!!! Your recommended materials seem useful as well.

But I do need to add the following option to tell Phi the include files of libtiff. I think it is a little strange as Phi doesn't need the libtiff library in my case, why do I have to add it?

-offload-option,mic,compiler,"-I/usr/include"

In my offload code, I used some standard libraries, like complex. I don't need to add it between the following pragmas. Is it because the compiler automatically offload the standard libraries or the standard libraries are installed on Phi?

#pragma offload_attribute (push, target(mic)
#pragma offload_attribute (pop)

Thanks and have a good day.

Frances Roth (Intel) wrote:

Let's start off by simplifying things.- you have executed:

 source /opt/intel/composerxe/bin/compilervars.sh intel64

right? This script will set things up for use with the Intel® compilers; what I say here applies only to the Intel® compilers.

With this script, you do not need to specify many of the things you are currently specifying on your command line. Among other things, it means you do not need to specify the location of the libraries for either the host or the coprocessor. Those are set for you, using the LIBRARY_PATH, MIC_LIBRARY_PATH, LD_LIBRARY_PATH and MIC_LD_LIBRARY_PATH environment variables, which are automatically recognized by the compiler.

You already specify '-openmp' in your list of flags. It is not necessary to also specify the library using a -l option. Also, if you specify '-mkl' in your flags, you do not need to separately list the libraries or include files.

Are you actually using both pthreads and openmp in the same program? If you did not include any calls to pthread routines, you can omit the pthread library from your command line. If you did need it, you could include it by using '-pthread' in your list of flags. (You may have been copying an example using the g++ compiler here - in that case you do need to specify both openmp and pthread libraries in order to use the Intel® OpenMP library, just as you would need to explicitly specify library names and locations for mkl.)

Now for 'offload-attribute-target=mic'. This will cause any file-scope functions and data objects to be compiled for both the host and the coprocessor. I think this is where you are getting your error messages from. Any variables you have that are instances of a class will be marked for offloading. Any functions defined in the file will be marked for offloading (including functions that call the tiff routines.) So, in effect, you are trying to offload class objects and calls to the tiff routines.

You need a finer level of control here. In your source code, find those include files and function and variable declarations that need to be offloaded to the coprocessor - and only those include files and function and variable declarations - and enclose them between these pragmas:

#pragma offload_attribute (push, target(mic)
#pragma offload_attribute (pop)

So, now we are down to:

CC = icpc
LLIB = -lm -ltiff
FLAG = -no-multibyte-chars -openmp -mkl
DEF = -DDEBUG_MAIN -DWRITE_TO_FILE -DMKL_ILP64
MIC = -watch=mic-cmd
main: main.cpp dpc.cpp dpc.h
        $(CC) main.cpp dpc.cpp $(LLIB) -o main $(MIC) $(DEF) $(FLAG)
clean:
        rm -rf *o main

To get more information, I highly recommend Programming and Compiling for Intel® Many Integrated Core Architecture. I still learn things when I skim through that. Oh, and yes, multi-dimensional arrays are bit-wise copyable. 

If, when you go through what I have written, you find mistakes (and sadly, I do sometimes make mistakes) let me know.

[Frances_R_Intel]

I'm not quite sure what is going on. Would you be willing to let me see the file that complains when you do not add that offload option? You can send it as an attachment to a private message (Send Author A Message) if you don't want to post the code to the forum.