advixe crashes immediately

[DH123]

Hi,

I've been trying to run a sample intel example that does matrix multiplication using advixe-cl or advixe. I issue the following command:

advixe-cl -collect survey -project-dir my_result -- ./1_mmult_serial

and get the following output (on multiple platforms running fedora 33):

Intel(R) Advisor Command Line Tool
Copyright (C) 2009-2020 Intel Corporation. All rights reserved.
advisor: Collection started. To stop the collection, either press CTRL-C or enter from another console window: advisor -r /home/dhalpern/Downloads/mmult/my_result/e000/hs002 -command stop.
advisor: Warning: [2020.12.27 18:00:23] /lib64/libc.so.6 _init() instrumentation failed. Profiling data may be missing.
vcs/tpss2/tpss/src/tpss/runtime/linux/exe/tpss_deepbind.c:235 tpss_deepbind_notify_on_pthread_loaded: Assertion '((tpss_pthread_key_create_call_t)(((((tpss_probe_t*)g_tpss_probes_table) + g_tpss_pt_id[(tpss_pi_pthread_key_create)]))->trampoline)) != ((void *)0) && ((tpss_pthread_setspecific_call_t)(((((tpss_probe_t*)g_tpss_probes_table) + g_tpss_pt_id[(tpss_pi_pthread_setspecific)]))->trampoline)) != ((void *)0) && ((tpss_pthread_getspecific_call_t)(((((tpss_probe_t*)g_tpss_probes_table) + g_tpss_pt_id[(tpss_pi_pthread_getspecific)]))->trampoline)) != ((void *)0) && ((tpss_pthread_getattr_np_call_t)(((((tpss_probe_t*)g_tpss_probes_table) + g_tpss_pt_id[(tpss_pi_pthread_getattr_np)]))->trampoline)) != ((void *)0) && ((tpss_pthread_attr_getstack_call_t)(((((tpss_probe_t*)g_tpss_probes_table) + g_tpss_pt_id[(tpss_pi_pthread_attr_getstack)]))->trampoline)) != ((void *)0) && ((tpss_pthread_attr_getstacksize_call_t)(((((tpss_probe_t*)g_tpss_probes_table) + g_tpss_pt_id[(tpss_pi_pthread_attr_getstacksize)]))->trampoline)) != ((void *)0) && ((tpss_pthread_attr_setstack_call_t)(((((tpss_probe_t*)g_tpss_probes_table) + g_tpss_pt_id[(tpss_pi_pthread_attr_setstack)]))->trampoline)) != ((void *)0) && ((tpss_pthread_attr_setstacksize_call_t)(((((tpss_probe_t*)g_tpss_probes_table) + g_tpss_pt_id[(tpss_pi_pthread_attr_setstacksize)]))->trampoline)) != ((void *)0) && ((tpss__pthread_cleanup_push_call_t)(((((tpss_probe_t*)g_tpss_probes_table) + g_tpss_pt_id[(tpss_pi__pthread_cleanup_push)]))->trampoline)) != ((void *)0) && ((tpss__pthread_cleanup_pop_call_t)(((((tpss_probe_t*)g_tpss_probes_table) + g_tpss_pt_id[(tpss_pi__pthread_cleanup_pop)]))->trampoline)) != ((void *)0)' failed.

I've tried different executable programs (fortran and c) and get the same error message about /lib64/libc.so.6.

Any suggestions would be appreciated.

DH

[GouthamK_Intel]

Hi David,

Thanks for reaching out to us!

As your issue is related to Advisor tool, we are redirecting this thread to Intel Advisor Forum for faster response.

Have a Good day!

Thanks & Regards

Goutham

[ChithraJ_Intel]

Hi David,

We have tried to run Survey analysis for a sample matrix application in Ubuntu machine(Ubuntu 18.04.4 LTS). But, we couldn't observe any issues. So, could you please try the below work arounds as of now.

1. As checking your error log we found that you are using an older version of advisor. Could you please upgrade it to the latest version and let us know the updates.

2. Please check other analysis types as well.

3. If it's a GUI enabled machine, try to launch advisor through GUI.

Please try any of the above work arounds and let us know the updates.

Regards,

Chithra

[DH123]

Hi,

Thanks for your reply.

1. I am using the latest version 2021.1 that comes with oneapi. So, I am not sure what you mean by "OLD".

2. I've tried and get the same problems.

3. I used the gui versions of advixe and advisor and get exactly the same errors. Also the gui and non-gui versions of advixe and advisor crash ungracefully. I have to end up doing a kill -9 PID on a separate window,

[DRW]

I think I am seeing the same problem, /lib64/libc.6.0 _init() instrumentation failed, also with Fedora 33 which I just updated to about a week ago, and Advisor GUI at the 2021.1 version that's part of the Oneapi install archive.

I am using the Advisor GUI to analyze a simple program using AVX512 that runs about 30 seconds.

I created a new project and just specified the name of the executable in the dialog asking for run parameters, and running the  Vectorization and Code insights analysis

Once I get the error, the GUI is stalled, but the UI is still active. I end up manually killing my program and the advisor processes manually to recover.

Maybe there's something unique about the system software, libc and others, in Fedora 33 that is causing this.

[DH123]

Hi,

Thanks for the response. In the end, do you get any meaningful output from advisor that you can use?

DH

[DRW]

If you're asking me, no I do not. The GUI is stalled and I have to kill it and my application process. I did try starting the GUI again and saw there is a project folder. I tried opening that and got a popup assert and hung GUI that I cannot copy test from. The assert is at line 722 db_interface_smpl_sqlite.cpp: Attribute table already exists. There's what' apparently a preceding error message 'No data is collected'.

[ChithraJ_Intel]

Hi David,

Did you try out any of the work arounds provided? Please let us know the updates.

Regards,

Chithra

[ChithraJ_Intel]

Hi David,

We are forwarding this case to Subject Matter Experts(SME) who can help you out and they will get back to you soon.

Regards,

Chithra

[Johny_P_Intel]

Is this issue occurring only on a particular fedora system? Can you also reproduce this issue on a similar but clean installation? We suspect an issue with your libc and that's the reason behind these additional questions.

Regards,

Johny Paul.

[DRW]

I created a virtual machine, VirtualBox 6.1.16, installed a new copy of Fedora 33 and then installed the OneApi code. I get the same error when I run advisor-gui

[DRW]

I decided to try one more thing to narrow this down.

I downloaded a small OpenMP example program https://computing.llnl.gov/tutorials/openMP/samples/C/omp_mm.c and compiled it on my base Fedora 33 system where advisor does not work

icc -qopenmp omp_mm.c omp_mm.c

I have the base docker containers for Fedora 31 and Fedora 33 (presumably from dockerhub)

I have OneApi final December 2020 version, including Advisor installed on my base Fedora system

I started the Fedora 33 container 'docker container run -t -i -v /opt/intel:/opt/intel <imagename> sh' to get a container with a shell prompt and have OneApi available to the container

I copied the omp_mm binary to the container /tmp directory

I installed a few RPMs apparently needed to run, 'dnf install procps-ng stdlibc++ findutils'

I ran the setvars.sh top level script to set the environment

Then I ran 'advisor --collect survey ./omp_mm'

That crashed as already noted

I started another container using the Fedora31 base image and repeated all of the above. This time there were no error messages

Then I ran 'advisor --report survey .omp_mm' in the Fedora 31  and it ran with no errors. There was likely nothing to report since omp_mm is so simple, but advisor seems function.

The difference seems to be that on Fedora 31 /lib64/libc.so.6 is linked to libc-2.30.so and on Fedora 33 linked to libc-2.32.so.

I don't know if there are more differences, but this seems to be one that matters

[Ruslan_M_Intel]

According to oneAPI System Requirements Fedora 33 is not supported yet. Probably incompatibility between different versions of 'libc' is the reason.

[DH123]

Hi,

I have a xeon workstation and an nuc both with fedora 33. However, neither has a clean install. But as pointed out by another user, DRW, who did a clean install of fedora 33, he encountered exactly the same problem.

Thanks.

DH

[Johny_P_Intel]

I managed to reproduce this issue using Fedora 33 (version: Workstation Edition with Linux Kernel 5.8.15-301.fc33.x86_64) and libc (version 2.32). The error message does not exactly resemble the errors posted above. But I can see that the behavior is very similar and leading to a crash related to libc. The errors I observed before the crash/hang are listed below.

advisor: Warning: [2021.01.11 20:33:20] /lib64/libc.so.6 _init() instrumentation failed. Profiling data may be missing.

AMPLXE_TPSSCOLLECTOR: tool:939: g_instr_global.is_libc_so_initialized == 1 : g_instr_global.is_libc_so_initialized == 0

Assertion failed: tool:939: g_instr_global.is_libc_so_initialized == 1 : g_instr_global.is_libc_so_initialized == 0

. Please contact the technical support. advisor: Error: Assertion failed: tool:939: g_instr_global.is_libc_so_initialized == 1 : g_instr_global.is_libc_so_initialized == 0

. Please contact the technical support. 

The issue has been reported to our engineering team and they are taking a look to identify the root-cause of this issue. We will get back to you as soon as possible.

[Johny_P_Intel]

Hi,

I can now confirm that the issue is related to the glibc (version v2.32). We hope that a fix can be incorporated in the next release. I will provide further updates when a fix is available.

Regards,

Johny

[FelixBrakel]

I get the exact same error with glibc version 2.33. One thing to note is that it only happens when sampling in user mode. I also noticed that when running threading analysis my program using pthreads only executes on a single thread and this one thread is pinned at 100% usage. The program never seems to finish.

[GeorgeBisbas]

I also get the exact same error with glibc version 2.34. Reproduced with oneapi versions 2021.1.1 and 2021.1.2 on Ubuntu 20.10.

Similar symptoms with above.
I have experienced the same issue last year with a similar glibc interoperability problem.
LInk: https://community.intel.com/t5/Analyzers/Assertion-failed-in-tpss-deepbind-notify-on-pthread-loaded/m-p/1191886#M19036

Best regards,
George

[GeorgeBisbas]

Just an update:

The issue seems to disappear after the latest update to Intel(R) Advisor 2021.2 (build 609067).

Best regards,
George Bisbas

[Johny_P_Intel · Accepted solution]

David,

This issue has been resolved in the latest release (e.g. One API Base Toolkit 2021.2).

George,

Thanks for the confirmation that this issue has been resolved in the latest release.

This issue has been resolved and we will no longer respond to this thread. If you require additional assistance from Intel, please start a new thread. Any further interaction in this thread will be considered community only.

Regards,

Johny