intel v19 compilation failure (type name not allowed) , gcc 4.8 works fine : tensorflow-1.12.0

[psing51]

Hi,
I was able to compile tensorflowv1.12.0 (CPU version) with gcc 4.8.5 (system default) from source, and i am currently having issues while compiling tensorflow with intel. Here are some details wrt. build environment - 

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS Linux release 7.6.1810 (Core)
• Python version: 3.6.8
• Bazel version : 0.19.1
• GCC/Compiler version : gcc 4.8.5 +  intel/19.0.0.117
• CUDA/cuDNN version: NA (CPU Build)
• GPU model and memory: NA (CPU build)

Primary reason for rebuilding with intel/19.0.0.117 is to compare relative performance gain on some sample TF benchmarks.

tensorflow-1.12.0]$ icc --version
icc (ICC) 19.0.0.117 20180804
Copyright (C) 1985-2018 Intel Corporation.  All rights reserved.

The command used for compilation is as follows -

CC=icc bazel build --config=mkl --define=grpc_no_ares=true --copt=-xHOST 
//tensorflow/tools/pip_package:build_pip_package

Error messages with aforementioned command line can be found in build_logs.txt & build_logs2.txt

Since bazel seems to be using parallel make (make -j N) type biild, I tried "make"ing tensorflow in "serial mode" as -

CC=icc bazel build -s --config=mkl --jobs=1 --define=grpc_no_ares=true --copt=-xHOST 
//tensorflow/tools/pip_package:build_pip_package 

compilation logs (for --jobs=1) are in file build_logs3_serial1. Please let me know if any further information is required from my end. 
I have also posted the issue at tensorflow forum , but as issue is related to intel compilers, i am not expecting much help from tensorflow forum.
Eagerly awaiting your replies.

 

[psing51]

I retried using CC=icpc as - 

CC=icpc bazel build -s --config=mkl --jobs=1 --define=grpc_no_ares=true --copt=-xHOST //tensorflow/tools/pip_package:build_pip_package 

Breif error message - 
 

/usr/include/string.h(402): error: linkage specification is incompatible with previous "strnlen" (declared at line 160 of "external/nasm/include/compiler.h")
  extern size_t strnlen (const char *__string, size_t __maxlen)
                ^

Error log file attached.
 

[psing51]

I tried manual compilation of a .cc file from bazel cache directory  /home/puneet/.cache/bazel/_bazel_puneet/000cc71254e2a8499cd324ebddae445d/execroot/org_tensorflow using the command line which fails during bazel build .

/opt/intel/compilers_and_libraries_2019.0.117/linux/bin/intel64/icpc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/opt/intel/compilers_and_libraries_2019.0.117/linux/bin/intel64 -B/usr/bin -Wunused-but-set-parameter -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/external/boringssl/_objs/ssl/s3_pkt.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/boringssl/_objs/ssl/s3_pkt.pic.o' -fPIC -iquote external/boringssl -iquote bazel-out/k8-opt/genfiles/external/boringssl -iquote bazel-out/k8-opt/bin/external/boringssl -iquote external/bazel_tools -iquote bazel-out/k8-opt/genfiles/external/bazel_tools -iquote bazel-out/k8-opt/bin/external/bazel_tools -isystem external/boringssl/src/include -isystem bazel-out/k8-opt/genfiles/external/boringssl/src/include -isystem bazel-out/k8-opt/bin/external/boringssl/src/include -xHOST -Wa,--noexecstack '-D_XOPEN_SOURCE=700' -Wall -Werror '-Wformat=2' -Wsign-compare -Wmissing-field-initializers -Wwrite-strings -Wshadow -fno-common '-std=c++11' -Wmissing-declarations -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/boringssl/src/ssl/s3_pkt.cc -o bazel-out/k8-opt/bin/external/boringssl/_objs/ssl/s3_pkt.pic.o

here are the terminal logs -

 

icpc: command line warning #10006: ignoring unknown option '-frandom-seed=bazel-out/k8-opt/bin/external/boringssl/_objs/ssl/s3_pkt.pic.o'
In file included from external/boringssl/src/ssl/s3_pkt.cc(122):
external/boringssl/src/ssl/internal.h(191): error #1418: external function definition with no prior declaration
  T *New(Args &&... args) {
     ^

In file included from external/boringssl/src/ssl/s3_pkt.cc(122):
external/boringssl/src/ssl/internal.h(204): error #1418: external function definition with no prior declaration
  void Delete(T *t) {
       ^

In file included from external/boringssl/src/ssl/s3_pkt.cc(122):
external/boringssl/src/ssl/internal.h(223): error #1418: external function definition with no prior declaration
  UniquePtr<T> MakeUnique(Args &&... args) {
               ^

.c and .h files are attached.

please let me know if any further information is required.

UPDATE: -Werror flag seems to be creating the issue, manual compilation now succeeds. Will try to remove the same from build process config.

[psing51]

Hi,
-Werror helped and compilation seems to be moving forward. However stuck with another compilation issue.

cd /home/puneet/.cache/bazel/_bazel_puneet/000cc71254e2a8499cd324ebddae445d/execroot/org_tensorflow && \
  exec env - \
    LD_LIBRARY_PATH=/home/puneet/MySoftwares/COMPILER/JDK/1.8.0.201_precompiled/lib:/home/puneet/MySoftwares/COMPILER/JDK/1.8.0.201_precompiled/lib64:/home/puneet/MySoftwares/UTILS/BAZEL/0.19.1_gcc4.8.5/lib:/home/puneet/MySoftwares/UTILS/BAZEL/0.19.1_gcc4.8.5/lib64:/home/puneet/MySoftwares/COMPILER/PYTHON/3.6.8/lib:/home/puneet/MySoftwares/COMPILER/PYTHON/3.6.8/lib64:/opt/intel/compilers_and_libraries_2019.0.117/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.0.117/linux/mpi/intel64/libfabric/lib:/opt/intel/compilers_and_libraries_2019.0.117/linux/mpi/intel64/lib/release:/opt/intel/compilers_and_libraries_2019.0.117/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2019.0.117/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.0.117/linux/tbb/lib/intel64/gcc4.7:/opt/intel/debugger_2019/libipt/intel64/lib:/opt/intel/compilers_and_libraries_2019.0.117/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.0.117/linux/daal/../tbb/lib/intel64_lin/gcc4.4 \
    PATH=/home/puneet/MySoftwares/COMPILER/JDK/1.8.0.201_precompiled/bin:/home/puneet/MySoftwares/UTILS/BAZEL/0.19.1_gcc4.8.5/bin:/home/puneet/MySoftwares/PYTHONPACKAGES/3.6.8_gnu4.8.5_intel19.0.0.117/PACKAGESUITE/1/bin:/home/puneet/MySoftwares/COMPILER/PYTHON/3.6.8/bin:/opt/intel/compilers_and_libraries_2019.0.117/linux/bin/intel64:/opt/intel/compilers_and_libraries_2019.0.117/linux/mpi/intel64/libfabric/bin:/opt/intel/compilers_and_libraries_2019.0.117/linux/mpi/intel64/bin:/opt/intel/debugger_2019/gdb/intel64/bin:/opt/xcat/bin:/opt/xcat/sbin:/opt/xcat/share/xcat/tools:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/puneet/.local/bin:/home/puneet/bin \
    PWD=/proc/self/cwd \
    /opt/intel/compilers_and_libraries_2019.0.117/linux/bin/intel64/icc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/opt/intel/compilers_and_libraries_2019.0.117/linux/bin/intel64 -B/usr/bin -Wunused-but-set-parameter -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/host/bin/tensorflow/core/kernels/_objs/non_max_suppression_op/non_max_suppression_op.pic.d '-frandom-seed=bazel-out/host/bin/tensorflow/core/kernels/_objs/non_max_suppression_op/non_max_suppression_op.pic.o' -fPIC -DEIGEN_MPL2_ONLY '-DEIGEN_MAX_ALIGN_BYTES=64' -D__CLANG_SUPPORT_DYN_ANNOTATION__ -DTF_USE_SNAPPY -iquote . -iquote bazel-out/host/genfiles -iquote bazel-out/host/bin -iquote external/nsync -iquote bazel-out/host/genfiles/external/nsync -iquote bazel-out/host/bin/external/nsync -iquote external/bazel_tools -iquote bazel-out/host/genfiles/external/bazel_tools -iquote bazel-out/host/bin/external/bazel_tools -iquote external/eigen_archive -iquote bazel-out/host/genfiles/external/eigen_archive -iquote bazel-out/host/bin/external/eigen_archive -iquote external/local_config_sycl -iquote bazel-out/host/genfiles/external/local_config_sycl -iquote bazel-out/host/bin/external/local_config_sycl -iquote external/com_google_absl -iquote bazel-out/host/genfiles/external/com_google_absl -iquote bazel-out/host/bin/external/com_google_absl -iquote external/gif_archive -iquote bazel-out/host/genfiles/external/gif_archive -iquote bazel-out/host/bin/external/gif_archive -iquote external/jpeg -iquote bazel-out/host/genfiles/external/jpeg -iquote bazel-out/host/bin/external/jpeg -iquote external/protobuf_archive -iquote bazel-out/host/genfiles/external/protobuf_archive -iquote bazel-out/host/bin/external/protobuf_archive -iquote external/com_googlesource_code_re2 -iquote bazel-out/host/genfiles/external/com_googlesource_code_re2 -iquote bazel-out/host/bin/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/host/genfiles/external/farmhash_archive -iquote bazel-out/host/bin/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/host/genfiles/external/fft2d -iquote bazel-out/host/bin/external/fft2d -iquote external/highwayhash -iquote bazel-out/host/genfiles/external/highwayhash -iquote bazel-out/host/bin/external/highwayhash -iquote external/zlib_archive -iquote bazel-out/host/genfiles/external/zlib_archive -iquote bazel-out/host/bin/external/zlib_archive -iquote external/png_archive -iquote bazel-out/host/genfiles/external/png_archive -iquote bazel-out/host/bin/external/png_archive -isystem external/nsync/public -isystem bazel-out/host/genfiles/external/nsync/public -isystem bazel-out/host/bin/external/nsync/public -isystem third_party/eigen3/mkl_include -isystem bazel-out/host/genfiles/third_party/eigen3/mkl_include -isystem bazel-out/host/bin/third_party/eigen3/mkl_include -isystem external/eigen_archive -isystem bazel-out/host/genfiles/external/eigen_archive -isystem bazel-out/host/bin/external/eigen_archive -isystem external/gif_archive/lib -isystem bazel-out/host/genfiles/external/gif_archive/lib -isystem bazel-out/host/bin/external/gif_archive/lib -isystem external/protobuf_archive/src -isystem bazel-out/host/genfiles/external/protobuf_archive/src -isystem bazel-out/host/bin/external/protobuf_archive/src -isystem external/farmhash_archive/src -isystem bazel-out/host/genfiles/external/farmhash_archive/src -isystem bazel-out/host/bin/external/farmhash_archive/src -isystem external/zlib_archive -isystem bazel-out/host/genfiles/external/zlib_archive -isystem bazel-out/host/bin/external/zlib_archive -isystem external/png_archive -isystem bazel-out/host/genfiles/external/png_archive -isystem bazel-out/host/bin/external/png_archive -g0 -g0 -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare -fno-exceptions '-ftemplate-depth=900' '-DINTEL_MKL=1' -DEIGEN_USE_VML -DENABLE_MKL -fopenmp -msse3 -pthread '-DINTEL_MKL=1' -DENABLE_MKL -Wno-builtin-macro-redefined -Wno-error '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c tensorflow/core/kernels/non_max_suppression_op.cc -o bazel-out/host/bin/tensorflow/core/kernels/_objs/non_max_suppression_op/non_max_suppression_op.pic.o)

compilation error messages -
 

ERROR: /home/puneet/MySoftwares/INSTALLATION_ROOT/python3/intel/tensorflow-1.12.0/tensorflow/core/kernels/BUILD:2220:1: C++ compilation of rule '//tensorflow/core/kernels:non_max_suppression_op' failed (Exit 2)

tensorflow/core/kernels/non_max_suppression_op.cc(120): error: type name is not allowed
    typename TTypes<float, 2>::ConstTensor overlaps_data =
    ^

tensorflow/core/kernels/non_max_suppression_op.cc(120): error: expected a ";"
    typename TTypes<float, 2>::ConstTensor overlaps_data =
                                           ^

tensorflow/core/kernels/non_max_suppression_op.cc(122): error: identifier "overlaps_data" is undefined
    return std::bind(&OverlapsGreaterThanThreshold, overlaps_data,


tensorflow/core/kernels/non_max_suppression_op.cc(112): error: expected a ";"
    typename TTypes<T, 2>::ConstTensor boxes_data = boxes.tensor<T, 2>();
                                       ^

tensorflow/core/kernels/non_max_suppression_op.cc(113): error: identifier "boxes_data" is undefined
    return std::bind(&IOUGreaterThanThreshold<T>, boxes_data,
               

Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 135.802s, Critical Path: 119.45s, Remote (0.00% of the time): [queue: 0.00%, setup: 0.00%, process: 0.00%]
INFO: 2777 processes: 2777 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

I am attaching files involved in this issue.

Note that the in intel compilation environment,  gcc compilers are able to compile the file in question (non_max_suppression_op.cc), but intel compilers are messing up!. Please have a look at - intelv19_issue_non_max_suppression_op.txt .
similar post - https://github.com/tensorflow/tensorflow/issues/25896
 please advice.

[Viet_H_Intel]

Can you create  a preprocess file (non_max_suppression_op.i) for us?

[psing51]

Hi,
I have created 2 files using following flags - 

• -P (non_max_suppression_op.i)   
• -E (non_max_suppression_op_1.i)

Files are attached, Please let me know if i can provide any further information.

[Viet_H_Intel]

Can you see if you can compile the test.cpp with gcc in your ENV? then provide us with your command line options?

[psing51]

Seems i am having issues while compiling the file you shared with both gcc & intel,
Could you please help me with correct command line to compile a preprocessed file ?

 I tried g++ -stc=c++11 ...... -x cpp-output -c test.cpp  -o test.o ?
Terminal log file is attached - for my attempt.

Also, i did not add any additional compilation flag for building tensorflow/non_max_suppression_op.cc. You may use the flags from the log file - shared with my previous comment.  bazel  is picking up compilation flags from the config files defined deep down the TF source code (seems CROSSTOOL file has setting for same). I need to confirm the config file for compilation flags with tensorflow forum.

[Viet_H_Intel]

test.cpp is your non_max_suppression_op.i after removing all the empty lines.

to get a pre-processed file: replace -c with -E and redirect the output to a file. 

after creating a preprocessed file, can you replace non_max_suppression_op.cc with non_max_suppression_op.i and see if it compiles with g++? 

[psing51]

g++ & gcc are able to compile the preprocessed file (and icpc/icc failed).
Terminal log (having compilation commands) is attached.

[Viet_H_Intel]

I encountered error when compiling  your non_max_suppression_op.i with g++. Since you compile the pre-processed file, can you minimize your command lines options and then provide it to us? 

Is non_max_suppression_op.i you attached earlier the same with the one you compiled successfully with g++?