Community
cancel
Showing results for 
Search instead for 
Did you mean: 
psing51
Beginner
447 Views

intel v19 compilation failure (type name not allowed) , gcc 4.8 works fine : tensorflow-1.12.0

Hi,
I was able to compile tensorflowv1.12.0 (CPU version) with gcc 4.8.5 (system default) from source, and i am currently having issues while compiling tensorflow with intel. Here are some details wrt. build environment - 

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS Linux release 7.6.1810 (Core)
  • Python version: 3.6.8
  • Bazel version : 0.19.1
  • GCC/Compiler version : gcc 4.8.5 +  intel/19.0.0.117
  • CUDA/cuDNN version: NA (CPU Build)
  • GPU model and memory: NA (CPU build)

Primary reason for rebuilding with intel/19.0.0.117 is to compare relative performance gain on some sample TF benchmarks.

tensorflow-1.12.0]$ icc --version
icc (ICC) 19.0.0.117 20180804
Copyright (C) 1985-2018 Intel Corporation.  All rights reserved.

The command used for compilation is as follows -

CC=icc bazel build --config=mkl --define=grpc_no_ares=true --copt=-xHOST 
//tensorflow/tools/pip_package:build_pip_package

Error messages with aforementioned command line can be found in build_logs.txt & build_logs2.txt

 

Since bazel seems to be using parallel make (make -j N) type biild, I tried "make"ing tensorflow in "serial mode" as -

CC=icc bazel build -s --config=mkl --jobs=1 --define=grpc_no_ares=true --copt=-xHOST 
//tensorflow/tools/pip_package:build_pip_package 

compilation logs (for --jobs=1) are in file build_logs3_serial1. Please let me know if any further information is required from my end. 
I have also posted the issue at tensorflow forum , but as issue is related to intel compilers, i am not expecting much help from tensorflow forum.
Eagerly awaiting your replies.

 

0 Kudos
10 Replies
psing51
Beginner
447 Views

I retried using CC=icpc as - 

CC=icpc bazel build -s --config=mkl --jobs=1 --define=grpc_no_ares=true --copt=-xHOST //tensorflow/tools/pip_package:build_pip_package 

Breif error message - 
 

/usr/include/string.h(402): error: linkage specification is incompatible with previous "strnlen" (declared at line 160 of "external/nasm/include/compiler.h")
  extern size_t strnlen (const char *__string, size_t __maxlen)
                ^


Error log file attached.
 

psing51
Beginner
447 Views

I tried manual compilation of a .cc file from bazel cache directory  /home/puneet/.cache/bazel/_bazel_puneet/000cc71254e2a8499cd324ebddae445d/execroot/org_tensorflow using the command line which fails during bazel build .

/opt/intel/compilers_and_libraries_2019.0.117/linux/bin/intel64/icpc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/opt/intel/compilers_and_libraries_2019.0.117/linux/bin/intel64 -B/usr/bin -Wunused-but-set-parameter -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/external/boringssl/_objs/ssl/s3_pkt.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/boringssl/_objs/ssl/s3_pkt.pic.o' -fPIC -iquote external/boringssl -iquote bazel-out/k8-opt/genfiles/external/boringssl -iquote bazel-out/k8-opt/bin/external/boringssl -iquote external/bazel_tools -iquote bazel-out/k8-opt/genfiles/external/bazel_tools -iquote bazel-out/k8-opt/bin/external/bazel_tools -isystem external/boringssl/src/include -isystem bazel-out/k8-opt/genfiles/external/boringssl/src/include -isystem bazel-out/k8-opt/bin/external/boringssl/src/include -xHOST -Wa,--noexecstack '-D_XOPEN_SOURCE=700' -Wall -Werror '-Wformat=2' -Wsign-compare -Wmissing-field-initializers -Wwrite-strings -Wshadow -fno-common '-std=c++11' -Wmissing-declarations -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/boringssl/src/ssl/s3_pkt.cc -o bazel-out/k8-opt/bin/external/boringssl/_objs/ssl/s3_pkt.pic.o

 

here are the terminal logs -

 

icpc: command line warning #10006: ignoring unknown option '-frandom-seed=bazel-out/k8-opt/bin/external/boringssl/_objs/ssl/s3_pkt.pic.o'
In file included from external/boringssl/src/ssl/s3_pkt.cc(122):
external/boringssl/src/ssl/internal.h(191): error #1418: external function definition with no prior declaration
  T *New(Args &&... args) {
     ^

In file included from external/boringssl/src/ssl/s3_pkt.cc(122):
external/boringssl/src/ssl/internal.h(204): error #1418: external function definition with no prior declaration
  void Delete(T *t) {
       ^

In file included from external/boringssl/src/ssl/s3_pkt.cc(122):
external/boringssl/src/ssl/internal.h(223): error #1418: external function definition with no prior declaration
  UniquePtr<T> MakeUnique(Args &&... args) {
               ^

.c and .h files are attached.

please let me know if any further information is required.

UPDATE: -Werror flag seems to be creating the issue, manual compilation now succeeds. Will try to remove the same from build process config.

psing51
Beginner
447 Views

Hi,
-Werror helped and compilation seems to be moving forward. However stuck with another compilation issue.

cd /home/puneet/.cache/bazel/_bazel_puneet/000cc71254e2a8499cd324ebddae445d/execroot/org_tensorflow && \
  exec env - \
    LD_LIBRARY_PATH=/home/puneet/MySoftwares/COMPILER/JDK/1.8.0.201_precompiled/lib:/home/puneet/MySoftwares/COMPILER/JDK/1.8.0.201_precompiled/lib64:/home/puneet/MySoftwares/UTILS/BAZEL/0.19.1_gcc4.8.5/lib:/home/puneet/MySoftwares/UTILS/BAZEL/0.19.1_gcc4.8.5/lib64:/home/puneet/MySoftwares/COMPILER/PYTHON/3.6.8/lib:/home/puneet/MySoftwares/COMPILER/PYTHON/3.6.8/lib64:/opt/intel/compilers_and_libraries_2019.0.117/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.0.117/linux/mpi/intel64/libfabric/lib:/opt/intel/compilers_and_libraries_2019.0.117/linux/mpi/intel64/lib/release:/opt/intel/compilers_and_libraries_2019.0.117/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2019.0.117/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.0.117/linux/tbb/lib/intel64/gcc4.7:/opt/intel/debugger_2019/libipt/intel64/lib:/opt/intel/compilers_and_libraries_2019.0.117/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.0.117/linux/daal/../tbb/lib/intel64_lin/gcc4.4 \
    PATH=/home/puneet/MySoftwares/COMPILER/JDK/1.8.0.201_precompiled/bin:/home/puneet/MySoftwares/UTILS/BAZEL/0.19.1_gcc4.8.5/bin:/home/puneet/MySoftwares/PYTHONPACKAGES/3.6.8_gnu4.8.5_intel19.0.0.117/PACKAGESUITE/1/bin:/home/puneet/MySoftwares/COMPILER/PYTHON/3.6.8/bin:/opt/intel/compilers_and_libraries_2019.0.117/linux/bin/intel64:/opt/intel/compilers_and_libraries_2019.0.117/linux/mpi/intel64/libfabric/bin:/opt/intel/compilers_and_libraries_2019.0.117/linux/mpi/intel64/bin:/opt/intel/debugger_2019/gdb/intel64/bin:/opt/xcat/bin:/opt/xcat/sbin:/opt/xcat/share/xcat/tools:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/puneet/.local/bin:/home/puneet/bin \
    PWD=/proc/self/cwd \
    /opt/intel/compilers_and_libraries_2019.0.117/linux/bin/intel64/icc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/opt/intel/compilers_and_libraries_2019.0.117/linux/bin/intel64 -B/usr/bin -Wunused-but-set-parameter -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/host/bin/tensorflow/core/kernels/_objs/non_max_suppression_op/non_max_suppression_op.pic.d '-frandom-seed=bazel-out/host/bin/tensorflow/core/kernels/_objs/non_max_suppression_op/non_max_suppression_op.pic.o' -fPIC -DEIGEN_MPL2_ONLY '-DEIGEN_MAX_ALIGN_BYTES=64' -D__CLANG_SUPPORT_DYN_ANNOTATION__ -DTF_USE_SNAPPY -iquote . -iquote bazel-out/host/genfiles -iquote bazel-out/host/bin -iquote external/nsync -iquote bazel-out/host/genfiles/external/nsync -iquote bazel-out/host/bin/external/nsync -iquote external/bazel_tools -iquote bazel-out/host/genfiles/external/bazel_tools -iquote bazel-out/host/bin/external/bazel_tools -iquote external/eigen_archive -iquote bazel-out/host/genfiles/external/eigen_archive -iquote bazel-out/host/bin/external/eigen_archive -iquote external/local_config_sycl -iquote bazel-out/host/genfiles/external/local_config_sycl -iquote bazel-out/host/bin/external/local_config_sycl -iquote external/com_google_absl -iquote bazel-out/host/genfiles/external/com_google_absl -iquote bazel-out/host/bin/external/com_google_absl -iquote external/gif_archive -iquote bazel-out/host/genfiles/external/gif_archive -iquote bazel-out/host/bin/external/gif_archive -iquote external/jpeg -iquote bazel-out/host/genfiles/external/jpeg -iquote bazel-out/host/bin/external/jpeg -iquote external/protobuf_archive -iquote bazel-out/host/genfiles/external/protobuf_archive -iquote bazel-out/host/bin/external/protobuf_archive -iquote external/com_googlesource_code_re2 -iquote bazel-out/host/genfiles/external/com_googlesource_code_re2 -iquote bazel-out/host/bin/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/host/genfiles/external/farmhash_archive -iquote bazel-out/host/bin/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/host/genfiles/external/fft2d -iquote bazel-out/host/bin/external/fft2d -iquote external/highwayhash -iquote bazel-out/host/genfiles/external/highwayhash -iquote bazel-out/host/bin/external/highwayhash -iquote external/zlib_archive -iquote bazel-out/host/genfiles/external/zlib_archive -iquote bazel-out/host/bin/external/zlib_archive -iquote external/png_archive -iquote bazel-out/host/genfiles/external/png_archive -iquote bazel-out/host/bin/external/png_archive -isystem external/nsync/public -isystem bazel-out/host/genfiles/external/nsync/public -isystem bazel-out/host/bin/external/nsync/public -isystem third_party/eigen3/mkl_include -isystem bazel-out/host/genfiles/third_party/eigen3/mkl_include -isystem bazel-out/host/bin/third_party/eigen3/mkl_include -isystem external/eigen_archive -isystem bazel-out/host/genfiles/external/eigen_archive -isystem bazel-out/host/bin/external/eigen_archive -isystem external/gif_archive/lib -isystem bazel-out/host/genfiles/external/gif_archive/lib -isystem bazel-out/host/bin/external/gif_archive/lib -isystem external/protobuf_archive/src -isystem bazel-out/host/genfiles/external/protobuf_archive/src -isystem bazel-out/host/bin/external/protobuf_archive/src -isystem external/farmhash_archive/src -isystem bazel-out/host/genfiles/external/farmhash_archive/src -isystem bazel-out/host/bin/external/farmhash_archive/src -isystem external/zlib_archive -isystem bazel-out/host/genfiles/external/zlib_archive -isystem bazel-out/host/bin/external/zlib_archive -isystem external/png_archive -isystem bazel-out/host/genfiles/external/png_archive -isystem bazel-out/host/bin/external/png_archive -g0 -g0 -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare -fno-exceptions '-ftemplate-depth=900' '-DINTEL_MKL=1' -DEIGEN_USE_VML -DENABLE_MKL -fopenmp -msse3 -pthread '-DINTEL_MKL=1' -DENABLE_MKL -Wno-builtin-macro-redefined -Wno-error '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c tensorflow/core/kernels/non_max_suppression_op.cc -o bazel-out/host/bin/tensorflow/core/kernels/_objs/non_max_suppression_op/non_max_suppression_op.pic.o)

compilation error messages -
 

ERROR: /home/puneet/MySoftwares/INSTALLATION_ROOT/python3/intel/tensorflow-1.12.0/tensorflow/core/kernels/BUILD:2220:1: C++ compilation of rule '//tensorflow/core/kernels:non_max_suppression_op' failed (Exit 2)

tensorflow/core/kernels/non_max_suppression_op.cc(120): error: type name is not allowed
    typename TTypes<float, 2>::ConstTensor overlaps_data =
    ^

tensorflow/core/kernels/non_max_suppression_op.cc(120): error: expected a ";"
    typename TTypes<float, 2>::ConstTensor overlaps_data =
                                           ^

tensorflow/core/kernels/non_max_suppression_op.cc(122): error: identifier "overlaps_data" is undefined
    return std::bind(&OverlapsGreaterThanThreshold, overlaps_data,


tensorflow/core/kernels/non_max_suppression_op.cc(112): error: expected a ";"
    typename TTypes<T, 2>::ConstTensor boxes_data = boxes.tensor<T, 2>();
                                       ^

tensorflow/core/kernels/non_max_suppression_op.cc(113): error: identifier "boxes_data" is undefined
    return std::bind(&IOUGreaterThanThreshold<T>, boxes_data,
               

Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 135.802s, Critical Path: 119.45s, Remote (0.00% of the time): [queue: 0.00%, setup: 0.00%, process: 0.00%]
INFO: 2777 processes: 2777 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

I am attaching files involved in this issue.

Note that the in intel compilation environment,  gcc compilers are able to compile the file in question (non_max_suppression_op.cc), but intel compilers are messing up!. Please have a look at - intelv19_issue_non_max_suppression_op.txt .
similar post - https://github.com/tensorflow/tensorflow/issues/25896
 please advice.

 

 

Viet_H_Intel
Moderator
447 Views

Can you create  a preprocess file (non_max_suppression_op.i) for us?

psing51
Beginner
447 Views

Hi,
I have created 2 files using following flags - 

  • -P (non_max_suppression_op.i)   
  • -E (non_max_suppression_op_1.i)


Files are attached, Please let me know if i can provide any further information.


 

Viet_H_Intel
Moderator
447 Views

Can you see if you can compile the test.cpp with gcc in your ENV? then provide us with your command line options?

psing51
Beginner
447 Views

Seems i am having issues while compiling the file you shared with both gcc & intel,
Could you please help me with correct command line to compile a preprocessed file ?

 I tried g++ -stc=c++11 ...... -x cpp-output -c test.cpp  -o test.o ?
Terminal log file is attached - for my attempt.

Also, i did not add any additional compilation flag for building tensorflow/non_max_suppression_op.cc. You may use the flags from the log file - shared with my previous comment.  bazel  is picking up compilation flags from the config files defined deep down the TF source code (seems CROSSTOOL file has setting for same). I need to confirm the config file for compilation flags with tensorflow forum.

Viet_H_Intel
Moderator
447 Views

test.cpp is your non_max_suppression_op.i after removing all the empty lines.

to get a pre-processed file: replace -c with -E and redirect the output to a file. 

after creating a preprocessed file, can you replace non_max_suppression_op.cc with non_max_suppression_op.i and see if it compiles with g++? 

psing51
Beginner
447 Views


g++ & gcc are able to compile the preprocessed file (and icpc/icc failed).
Terminal log (having compilation commands) is attached.

Viet_H_Intel
Moderator
447 Views

I encountered error when compiling  your non_max_suppression_op.i with g++. Since you compile the pre-processed file, can you minimize your command lines options and then provide it to us? 

Is non_max_suppression_op.i you attached earlier the same with the one you compiled successfully with g++?

 

Reply