Analyzers
Talk to fellow users of Intel Analyzer tools (Intel VTune™ Profiler, Intel Advisor)
4974 Discussions

libtpsstool.so is causing Java application to crash with SIGV_MAPERR

FJ_Shen
Beginner
3,090 Views

I am trying to profile a Java application with vTune in Linux Ubuntu 20.04. The application consists of multiple iterations which I would like to profile as separate "frames". I wrote my own JNI class so that I can make function calls from my Java application to the Intel ITT (Instrumentation and Tracing Technology) native library - specifically to the frame APIs (and domain APIs).

The eerie thing is that once I profile my Java code sprinkled with these JNI calls to ITT library, the application crashes in random occurrences.  I analyzed the fatal error log files generated by Java, and it turned out that although the exact thread that caused the crash varies from case to case,  all of these fatal errors are SIGSEGV errors - to be more precise, SEGV_MAPERR errors. These errors only happen in User-mode sampling, but never show up when I'm using Hardware event-based sampling. 

Furthermore, the offending pc values all point to Intel's libtpsstool.so library.  Please see the excerpts from Java's fatal error log:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f46df7cf1f5, pid=1540960, tid=0x00007f46d4f03700
#
# JRE version: OpenJDK Runtime Environment (8.0) (build 1.8.0-internal-fastdebug-shen449_2021_07_16_15_01-b00)
# Java VM: OpenJDK 64-Bit Server VM (25.71-b00-fastdebug mixed mode linux-amd64 )
# Problematic frame:
# C 0x00007f46df7cf1f5
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#

[...many lines omitted...]

Dynamic libraries:

[...many lines omitted...]

7f46deecf000-7f46deee3000 r--p 002cc000 08:02 6300535 /opt/intel/oneapi/vtune/2021.5.0/lib64/pinruntime/libpin3dwarf.so
7f46deee3000-7f46deee4000 rw-p 002e0000 08:02 6300535 /opt/intel/oneapi/vtune/2021.5.0/lib64/pinruntime/libpin3dwarf.so
7f46deee4000-7f46deee7000 rw-p 00000000 00:00 0
7f46deee7000-7f46df8bc000 r-xp 00000000 08:02 6300526 /opt/intel/oneapi/vtune/2021.5.0/lib64/libtpsstool.so
7f46df8bc000-7f46dfabb000 ---p 00000000 00:00 0
7f46dfabb000-7f46dfabe000 r--p 009d4000 08:02 6300526 /opt/intel/oneapi/vtune/2021.5.0/lib64/libtpsstool.so

[...many lines omitted...]

 

My vTune version is 2021.5.0

Does anyone have experienced similar issues, or have any suggestion/insight on what might be the cause of the problem? (i.e. if the crash is due to a bug in the library or I just toppled over some pitfalls)

0 Kudos
1 Solution
clevels
Moderator
1,926 Views

I will need to close this case if there are no updates on reproducing this issue on the latest version of VTune. Please feel free to file a new case if this issue persists.


View solution in original post

0 Kudos
12 Replies
JaideepK_Intel
Moderator
3,062 Views

Hi,

Thank you for posting in Intel Analyzers Forum.

Could you please share the below information for better understanding of the issue and to debug it from our end:

  1. The vtune command used to run the analysis
  2. Log file after running self-checker Script self_check.py which will be available in <vtune-install-dir>/bin64

Also, let us know whether you are getting the same errors while running the application without vtune.

If yes, please provide a sample reproducer so that we can try it out from our end.

Regards,

jaideep


0 Kudos
FJ_Shen
Beginner
3,046 Views

Here's the command that I used (I invoked it from python3 subprocess.run):

/opt/intel/oneapi/vtune/2021.5.0/bin64/vtune -result-dir=/home/shen449/intel/vtune/projects/rapids_tpch/2021_8_18_10_37_TPCH_Q1_IT6_1thrd_gc_noXcomp_fastdebugJVM -collect hotspots -knob enable-stack-collection=true -target-duration-type=medium -data-limit=0 -finalization-mode=full -app-working-dir /home/shen449/rapids-bench --app-working-dir=/home/shen449/rapids-bench -- /home/shen449/rapids-bench/wrapper.sh benchmark.py --benchmark tpch --template template.txt --input ./tpch-tables/1_none/ --input-format parquet --configs cpu --gc-between-runs --iterations 6 --query q1

 

The self_check log file is attached

0 Kudos
FJ_Shen
Beginner
3,040 Views

Here is the sampler reproducer that you asked for. You may try it. And no, without vTune the crash does not happen. 

 

The (majority of the) project is attached in this .tar.gz file. Before the first time you use it, extract the tarball to <project_dir> then call 

source setup_env.sh

 ./gen_parquet_tables.sh -s 1 -c none -m 8G

this will get the data files ready and download the jar files for execution. It can take a while.

 

Besides, to reproduce what I was doing you will need to build the fastdebug version of JDK8 on your machine and use that as the JVM for this application. I use fastdebug for better profiling "resolution". 

To direct the Spark application to use fastdebug JDK8, run the following commands:

cd <project_dir>/spark-3.0.0-bin-hadoop3.2/conf    #this folder won't appear until you source the setup_env script

cp spark-env.sh.template spark-env.sh

echo "JAVA_HOME=<your fastdebug JDK8 location>"  >>  spark-env.sh 

 

Finally go to <project_dir> and run the vtune command that I gave you. Please let me know if you need any more info or help to reproduce to crash.

0 Kudos
SreejithS_Intel
Moderator
2,946 Views

Hi,


We were able to reproduce the issue. We are checking on it internally and will get back to you soon with updates.


Thanks



0 Kudos
clevels
Moderator
2,920 Views

Hello! thank you for your ticket. I have accepted this case and will also be investigating this issue. Thanks.


0 Kudos
FJ_Shen
Beginner
2,908 Views

Thanks for your attention to this issue! Please let me know what you find out.

0 Kudos
clevels
Moderator
2,762 Views

Hello! Thank you for your patience. I have gone ahead and filed a ticket directly with the Intel VTune development team. I will provide an update just as soon as they are able to review this issue. Thanks!


0 Kudos
FJ_Shen
Beginner
2,691 Views
0 Kudos
clevels
Moderator
2,536 Views

Hello- I would like to check in. I have reviewed the status of the issue I filed with the Intel VTune Development team and it has not yet been reviewed. I will provide updates as soon as I can. Thank you.


0 Kudos
clevels
Moderator
1,946 Views

Hello - I have reached out to the development team regarding this issue filed some time ago. In the meantime, can you please see if this is able to be reproduced on the latest version of VTune (2022.3.0)? You can install the latest standalone VTune here : https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler-download.html


0 Kudos
clevels
Moderator
1,927 Views

I will need to close this case if there are no updates on reproducing this issue on the latest version of VTune. Please feel free to file a new case if this issue persists.


0 Kudos
FJ_Shen
Beginner
1,917 Views

Thanks for the update. Lately I've been working on other things, but I will try out the newer version when the opportunity comes.

0 Kudos
Reply