[Moved from old forum] GLIBC issues on the devcloud for FPGA

[Vile_Lasagna]

TL;DR: fpga_compile nodes on the devCloud have a newer glibc installed than the nodes with actual FPGAs so code build on the former cannot run on the latter



I've been trying for a few days to test some code on the FPGAs in the dev cloud but I keep running into blocking issues.

It took me a long time to even get things building (thanks to Susannah from the Discord for helping me troubleshoot environment setup issues on the devcloud) but now that I've finally got my code building, it turns out I can't run anything.

After looking at logs yesterday and double checking things today it seems that... yeah... Unless I'm missing something there's a fundamental issue with the cluster setup.

All of the nodes in the cluster, from what I can tell run Ubuntu. They're all at Ubuntu 20 EXCEPT the nodes with the stratix10 or the arria10 FPGAs. These are Ubuntu 18, but NOT the fpga_compile nodes. 

This means that code built on the compile nodes cannot run on the FPGA nodes due to GLIBC version mismatch which I'd THINK someone would've run into immediately.

I'm not sure if this is the ideal feedback channel for this. I'd still like to report the issue that prevented me from compiling in the first place (the wrong version of python 2 gets used by default and the scripts in quartus start looking into nonexistent locations for stuff).

For context:
The code I'm building can be found in Gitlab . One can find the code in question under the `raymarched/SYCL/` folder if you want to verify. It's C++ with SYCL that I know itself works as I've validated it both with hipSYCL as well with oneAPI on the GPUs in the devCloud so not worried there. Everything is built with CMake so I'm not calling the compiler on my own or writing makefiles by hand.

And here's just some validation of what i'm going on about. After building, I get this when trying to run:


/home/u121770/repos/toyBrot/redist/bin/rmSYCL-fpga-arria10: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /home/u121770/repos/toyBrot/redist/bin/rmSYCL-fpga-arria10)
/home/u121770/repos/toyBrot/redist/bin/rmSYCL-fpga-stratix10: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /home/u121770/repos/toyBrot/redist/bin/rmSYCL-fpga-stratix10)


And double checking with cat /etc/os-release and nm on glib confirms that, yes, this error completely makes sense

# Resources: neednodes=1:fpga_compile:ppn=2,nodes=1:fpga_compile:ppn=2,walltime=06:00:00
########################################################################

NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
<....>
0000000000000000 A GLIBCXX_3.4.24@@GLIBCXX_3.4.24
0000000000000000 A GLIBCXX_3.4.25@@GLIBCXX_3.4.25
0000000000000000 A GLIBCXX_3.4.26@@GLIBCXX_3.4.26
0000000000000000 A GLIBCXX_3.4.27@@GLIBCXX_3.4.27
0000000000000000 A GLIBCXX_3.4.28@@GLIBCXX_3.4.28
0000000000000000 A GLIBCXX_3.4.3@@GLIBCXX_3.4.3

# Resources: neednodes=1:stratix10:ppn=2,nodes=1:stratix10:ppn=2,walltime=06:00:00
########################################################################

NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
<....>
0000000000000000 A GLIBCXX_3.4.24@@GLIBCXX_3.4.24
0000000000000000 A GLIBCXX_3.4.25@@GLIBCXX_3.4.25
0000000000000000 A GLIBCXX_3.4.3@@GLIBCXX_3.4.3

EDIT:

It's been a few days and I came back to this issue hoping I'd be able to try and build the code correctly using a container in my own dev machine but that'd apparently cost me 4000USD Dollars because Quartus so... as far as I can tell, I'm SoL here and the dream is very much dead when it comes to validating the workflow and seeing how the FPGAs perform, etc....

[nielskm]

Hi @BoonBengT_Intel,

As mentioned before in this thread, the dependency on GLIBC comes from the Intel oneMKL library. So in order to close this issue, you should at least provide a minimum working example of a program that uses oneMKL (to e.g. multiply two matrices) that compiles on the fpga_compile nodes and runs on the fpga nodes.

Best wishes,

Niels

[PSath2]

This is still an issue with a non-MKL bare-bones SYCL build, even *emulated* builds on the fpga_compile nodes link to GLIBCXX_3.4.26.

It has nothing to do with the FPGA design, it's failing to run the *host code* because of a *basic C++ dynamic linking issue*.

EDIT:
The janky workaround of cping the libstdc++.so.6 to your home directory does work. Then just LD_PRELOAD it when running your application on the stratix10 nodes, so it provides the libstdc++ symbols before the system's outdated version. This allows you use the fpga_compile nodes as intended to keep synthesis off the run nodes, but suffers from ease-of-use.