quartus_map crash, possibly due to shared library shenanigans

[SimonRichter]

Hi,

I keep getting reproducible crashes from quartus_map when called inside a Docker container, and right now I'm at the stage in debugging where I'm wondering how this could ever have worked.

Basically, quartus_map is, for some reason, linked against libtbbmalloc_proxy, which provides malloc/realloc/free and is meant to override these for the entire process. Later, libsys_cpt.so uses

dlopen("libudev.so.1", RTLD_LAZY|RTLD_DEEPBIND);

to get the system libudev, which is then used to scan for connected devices in the hope of finding something uniquely identifiable to attach licensing to. libudev, at several points, calls strdup (which is in libc, but uses malloc from libtbbmalloc_proxy) and realloc (which is in libc, and not overridden because the RTLD_DEEPBIND puts the libc definition first).

Now, since strdup returns a pointer from malloc, it should be safe to call realloc on it, which is what libudev does -- however it tries to use the libc realloc on a pointer that was allocated through libtbbmalloc_proxy, and this falls over in a spectacular fashion:

Info: Command: quartus_map raid
realloc(): invalid pointer
Aborted

Now I have no clue how this could have ever worked, or why that only happens inside Docker -- it should crash everywhere.

I can use LD_PRELOAD to add another gross hack on top:

#define _GNU_SOURCE 1

#include <dlfcn.h>

static void *(*orig_dlopen)(char const *, int) = 0;

void *dlopen(char const *name, int flags)
{
    if(!orig_dlopen)
        orig_dlopen = dlsym(RTLD_NEXT, "dlopen");
    flags &= ~RTLD_DEEPBIND;
    return orig_dlopen(name, flags);
}

When I compile this

$ gcc -shared -o /tmp/hack.so hack.c -ldl

and use LD_PRELOAD to inject it into the quartus_map process

LD_PRELOAD=/tmp/hack.so quartus_map raid

then my design (almost) compiles, the remaining problems seem unrelated.

Since this is a wobbly stack of hacks upon hacks, it would be nice if there was a proper solution that doesn't require me to inject code into the Quartus process. Is there a good way to get this bug to the attention of the software team?

[RichardTanSY_Intel]

Hi @SimonRichter 

The Quartus Prime software is not validated in Docker.  

You may checkout the previous forum case below for a solution. 

https://community.intel.com/t5/Intel-Quartus-Prime-Software/Quartus-failed-to-run-inside-Docker-Linux/td-p/241058

You may try to set the env export RTLD_DEEPBIND=0 and see if it works. 

Could you help to clarify which Quartus version and OS that you are using? 

Best Regards,
Richard Tan

p/s: If any answer from the community or Intel support are helpful, please feel free to give Kudos. 

[SimonRichter]

Hi Richard,

alas, there is no environment variable to override what applications are doing through libdl, the environment affects only normal library loading.

This is Quartus Lite 20.1.1.720 on Debian Linux, once installed normally on a laptop, and once installed into Docker as follows:

1. First, import the installer tar as a Docker image:

   docker import Quartus-lite-20.1.1.720-linux.tar quartus-installer:20.1.1.720​

2. Then, build this Dockerfile:

   FROM debian
   RUN [ "apt-get", "update" ]
   RUN [ "apt-get", "--yes", "install", "gcc" ]
   ADD hack.c hack.c
   RUN [ "gcc", "-shared", "-o", "hack.so", "hack.c", "-ldl" ]
   
   FROM debian
   COPY --from=quartus-installer:20.1.1.720 / /tmp/setup
   RUN [ "/tmp/setup/setup.sh", \
           "--accept_eula", "1", \
           "--mode", "unattended", \
           "--unattendedmodeui", "none", \
           "--installdir", "/opt/altera/20.1" ]
   
   FROM debian
   COPY --from=1 /opt/altera /opt/altera
   RUN [ "apt-get", "update" ]
   RUN [ "apt-get", "--yes", "install", "--no-install-recommends", \
    "locales", \
    "libfontconfig1", \
    "libglib2.0-0", \
    "libxrender1", \
    "libxtst6", \
    "make", \
    "xauth", \
    "xvfb" ]
   RUN [ "sed", "-i", "-e", "/en_US\\.UTF-8/s/^# *//", "/etc/locale.gen" ]
   RUN [ "locale-gen" ]
   COPY --from=0 hack.so /opt/hack.so
   ADD ld.so.preload /etc/ld.so.preload
   RUN [ "useradd", "build" ]
   USER build​

Remarks:

1. The "ld.so.preload" file just points to /opt/hack.so, so it is added to every process, this allows the compiler to work.
2. The "hack.c" file is from the other post.
3. I install "locales" and generate the "en_US" locale to avoid some warnings.
4. The four libraries are required as system libraries because they aren't shipped with QuartusII.
5. I have to add GNU Make or the sequencer of the UniPHY based memory controller fails to build. That is a bit weird because GNU Make is included in the Quartus distribution package.
6. I pack a virtual X server and xauth in there so I can regenerate IP components. For some reason, mw-regenerate falls over without a display present, even if it's just generating, so Jenkins is instructed to set up a fake display.

This is where I'm at currently, this seems to be generally working for everything but Qsys, so most of my projects are covered by that, but the amount of trickery I needed to make this reproducible is still a bit high. After all, the goal is to be able to generate programming files on a central server and become independent from developers' workstations.

This post already describes the Qsys problem: I cannot reference the Verilog modules generated from Qsys from a VHDL toplevel file.

The Qsys generator looks at ~/.altera.quartus/quartus2.ini to see whether it should generate VHDL or Verilog, so a project that compiles fine on my laptop (where the preference is set to VHDL) will fail when I run it over night on the server where the INI file doesn't exist and it falls back to Verilog.

This problem can be reproduced with a simple test project, which I've attached: whether this compiles depends on the setting of Tools->Options->IP Settings->IP Generation HDL Preference: in VHDL mode it succeeds and in Verilog mode, it fails.

   Simon

[RichardTanSY_Intel]

Hi @SimonRichter 

Unfortunately, Quartus is not official support in Docker. Hopefully the community here that come across this similar issue will be able to help you on this. 

With that, I will now transition this thread to community support. If you have a new question, feel free to open a new thread to get the support from Intel experts. Otherwise, the community users will continue to help you on this thread. Thank you.

Best Regards,
Richard Tan

p/s: If any answer from the community or Intel support are helpful, please feel free to give Kudos.