Error in Linking Sycl with Libtorch for project on heterogeneous acceleration

[YuanM]

Hi! I am working on an RL acceleration project in DPC++ involving different classes and primitives, which relies heavily on the Libtorch (C++ distribution of pytorch). However, I was not able to build a simple SYCL+libtorch project on DevCloud due to a strange linker error. I am using the vector-add example provided by vector-add-buffers.cpp to show a step-by-step minimal reproducible example of the error:

1. I installed libtorch using the following command in the home directory:

wget https://download.pytorch.org/libtorch/nightly/cpu/libtorch-shared-with-deps-latest.zip
unzip libtorch-shared-with-deps-latest.zip

2.  I used the following command (without cmake/make) to compile and link the project:

mkdir build
cd build
[compile] icpx -fsycl -I/home/u186670/libtorch/include -I/home/u186670/libtorch/include/torch/csrc/api/include -D_GLIBCXX_USE_CXX11_ABI=0 -std=gnu++17 -L/home/u186670/libtorch/lib -Wl,-R/home/u186670/libtorch/lib -ltorch -ltorch_cpu -lc10 -o vector-add-buffers.cpp.o -c ../src/vector-add-buffers.cpp
[link] icpx -fsycl -I/home/u186670/libtorch/include -I/home/u186670/libtorch/include/torch/csrc/api/include -D_GLIBCXX_USE_CXX11_ABI=0 -std=gnu++17 -L/home/u186670/libtorch/lib -Wl,-R/home/u186670/libtorch/lib -ltorch -ltorch_cpu -lc10 vector-add-buffers.cpp.o -o vector-add-buffers

Note: the only libtorch code added to the vector-add-buffers.cpp are simply:

#include <torch/torch.h>
// dpc++ code for vector add ...
torch::Tensor tensor = torch::rand({2, 3});
std::cout << tensor << std::endl;
return 0; //retun for main()
} //ending main()

Result & error:

The compilation was successful. The link gives the following error (seems to be a problem with sycl exception handler):

/home/u186670/tmp/icpx-515249/vector-add-buffers-7e91ab.o: In function `std::_Function_handler<void (sycl::_V1::handler&), VectorAdd(sycl::_V1::queue&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> >&)::{lambda(sycl::_V1::handler&)#1}>::_M_invoke(std::_Any_data const&, sycl::_V1::handler&)':
vector-add-buffers-32895f.cpp:(.text+0x28d1): undefined reference to `sycl::_V1::handler::verifyUsedKernelBundle(std::string const&)'
/home/u186670/tmp/icpx-515249/vector-add-buffers-7e91ab.o: In function `sycl::_V1::exception::exception(std::error_code, std::string const&, int, std::shared_ptr<sycl::_V1::context>)':
vector-add-buffers-32895f.cpp:(.text._ZN4sycl3_V19exceptionC2ESt10error_codeRKSsiSt10shared_ptrINS0_7contextEE[_ZN4sycl3_V19exceptionC2ESt10error_codeRKSsiSt10shared_ptrINS0_7contextEE]+0x23d): undefined reference to `sycl::_V1::exception::exception(std::error_code, std::shared_ptr<sycl::_V1::context>, std::string const&)'
icpx: error: linker command failed with exit code 1 (use -v to see invocation)

Could anyone please give some pointers on what could possibly cause such linker error, and/or whether linking DPC++ code with libtorch is possible in general (if so, any minimal working examples)?

[VaishnaviV_Intel · Accepted solution]

Hi,

Thanks for posting in Intel Communities.

We have tried with Intel Pytorch and we didn't face any issue.

Please follow the below steps: 

1) To locate shared libraries at runtime use the following command,

 export LD_LIBRARY_PATH=/glob/development-tools/versions/oneapi/2023.0.1/oneapi/intelpython/latest/envs/pytorch/lib/python3.9/site-  packages/torch/lib/:$LD_LIBRARY_PATH

2) Please use the following command to compile the code:

icpx -fsycl vector-add-buffer.cpp -I /glob/development-tools/versions/oneapi/2023.0.1/oneapi/intelpython/latest/envs/pytorch/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I /glob/development-tools/versions/oneapi/2023.0.1/oneapi/intelpython/latest/envs/pytorch/lib/python3.9/site-packages/torch/include -L /glob/development-tools/versions/oneapi/2023.0.1/oneapi/intelpython/latest/envs/pytorch/lib/python3.9/site-packages/torch/lib -ltorch -ltorch_cpu -lc10

3) ./a.out

Please refer to the below screenshot for further details.

If your issue still persists, please let us know.

Thanks and Regards,

Vankudothu Vaishnavi.

[YuanM]

Thank you a lot for the prompt assistance. I followed exactly the provided steps, but am still facing (another) linker error as shown in the screenshot.

FYI, I inlcude my PATH and LD_LIBRARY_PATH below for you reference. Could you please let me know what are the environment variables you used to produce the working example, so I can cross-reference and check if there is anything different that caused the linker error?

u186670@login-2:~/build$ echo $PATH
/home/u186670/.vscode-server/bin/441438abd1ac652551dbe4d408dfcec8a499b8bf/bin/remote-cli:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/vtune/2023.0.0/bin64:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/vpl/2023.0.0/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/ospray_studio/0.11.1/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/ospray/2.10.0/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/openvkl/1.3.1/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/oidn/1.4.3/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/mpi/2021.8.0//libfabric/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/mpi/2021.8.0//bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/mkl/2023.0.0/bin/intel64:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/itac/2021.8.0/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/ispc/1.18.1/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/intelpython/latest/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/intelpython/latest/condabin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/inspector/2023.0.0/bin64:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/embree/3.13.5/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/dpcpp-ct/2023.0.0/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/dev-utilities/2021.8.0/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/debugger/2023.0.0/gdb/intel64/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/bin/intel64:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/bin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/clck/2021.7.2/bin/intel64:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/advisor/2023.0.0/bin64:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/u186670/.local/bin:/home/u186670/bin:/bin:/glob/intel-python/python2/bin:/home/u186670/.local/bin:/home/u186670/bin:/bin

u186670@login-2:~/build$ echo $LD_LIBRARY_PATH
/glob/development-tools/versions/oneapi/2022.3/oneapi/intelpython/latest/envs/pytorch/lib/python3.9/site-packages/torch/lib/:/glob/development-tools/versions/oneapi/2022.3.1/oneapi/intelpython/latest/envs/pytorch/lib/python3.9/site-packages/torch/lib/:/glob/development-tools/versions/oneapi/2023.0/oneapi/intelpython/latest/envs/pytorch/lib/python3.9/site-packages/torch/lib/:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/intelpython/latest/envs/pytorch/lib/python3.9/site-packages/torch/lib/:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/vpl/2023.0.0/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/tbb/2021.8.0/env/../lib/intel64/gcc4.8:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/rkcommon/1.10.0/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/ospray_studio/0.11.1/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/ospray/2.10.0/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/openvkl/1.3.1/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/oidn/1.4.3/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/mpi/2021.8.0//libfabric/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/mpi/2021.8.0//lib/release:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/mpi/2021.8.0//lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/mkl/2023.0.0/lib/intel64:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/itac/2021.8.0/slib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/ispc/1.18.1/lib/lib64:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/ipp/2021.7.0/lib/intel64:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/ippcp/2021.6.3/lib/intel64:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/ipp/2021.7.0/lib/intel64:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/embree/3.13.5/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/dnnl/2023.0.0/cpu_dpcpp_gpu_dpcpp/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/debugger/2023.0.0/gdb/intel64/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/debugger/2023.0.0/libipt/intel64/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/debugger/2023.0.0/dep/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/dal/2023.0.0/lib/intel64:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/x64:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/lib/oclfpga/host/linux64/lib:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/compiler/2023.0.0/linux/compiler/lib/intel64_lin:/glob/development-tools/versions/oneapi/2023.0.1/oneapi/ccl/2021.8.0/lib/cpu_gpu_dpcpp

[VaishnaviV_Intel]

Hi,

Could you please try running in the compute node, using the following command:

qsub -I -l nodes=1:gpu:ppn=2 -d .

>>Could you please let me know what are the environment variables you used to produce the working example?

We have exported LD_LIBRARY_PATH before execution.

LD_LIBRARY_PATH=/glob/development-tools/versions/oneapi/2023.0.1/oneapi/intelpython/latest/envs/pytorch/lib/python3.9/site-packages/torch/lib/:$LD_LIBRARY_PATH

Please refer to the below screenshot for more details.

If your issue still persists, please let us know.

Thanks & Regards,

Vankudothu Vaishnavi.

[YuanM]

Thank you a lot, switching to a compute node made it compile and run successfully.

[VaishnaviV_Intel]

Hi,

Thanks for accepting our solution. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel. 

Thanks & Regards,

Vankudothu Vaishnavi.