Vtune runtime failure

[dkokron]

Vtune fails with the following after the application has finished. 

"vcs/collectunits1/tmu/src/tmu.c:421 write_trace: Assertion 'Cannot write magic record to trace' failed."

I am running an OpenFOAM application across 20 nodes with a total of 800 cores.  My launch command is

mpiexec -np 800 vtune -collect memory-access -r MemAcc.${NCPUS}p.${CPU}.${PBS_JOBID} -finalization-mode=full -data-limit=8192 sonicFoam -parallel

OpenFOAM is compiled and run under the following environment

module purge

source /nas/sw/sdev/intel/oneapi/2022/v2/compiler/latest/env/vars.sh
source /nas/sw/sdev/intel/oneapi/2022/v2/mkl/latest/env/vars.sh
source /nas/sw/sdev/intel/oneapi/2021/v2/mpi/latest/env/vars.sh
source /nas/sw/sdev/intel/oneapi/2022/v2/vtune/latest/env/vars.sh

I have run the vtune-self-checker.sh on all of the nodes associated with the above failure.  Many though not all of the nodes hang on the "context switches" test.

Some system info:

Intel(R) VTune(TM) Profiler 2022.2.0 (build 623516) Command Line Tool

NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"

Processors: 2 x 40 Intel(R) Xeon(R) Gold 6242R CPU @ 3.10GHz

[VaradJ_Intel]

Hi,

Good day to you.

Thanks for posting in Intel Communities.

We are extremely sorry that the Intel VTune Profiler is causing issues with your workflow.

To clearly assist you with your issue, can you share the output of the following commands:

$ sudo <vtune-install-dir>/sepdk/src/insmod-sep -r

$ sudo <vtune-install-dir>/sepdk/src/insmod-sep -q

$ sudo <vtune-install-dir>/bin64/vtune-self-checker.sh

Eg.:

sudo /opt/intel/oneapi/vtune/latest/sepdk/src/insmod-sep -r

sudo /opt/intel/oneapi/vtune/latest/sepdk/src/insmod-sep -q

sudo /opt/intel/oneapi/vtune/latest/bin64/vtune-self-checker.sh

To replicate the issue from our side, can you share a sample reproducer code along with the exact steps or commands that you followed to do the analysis.

Thank You.

[VaradJ_Intel]

Hi,

We have not heard back from you. Please could you give us an update?

Thank You.

[VaradJ_Intel]

Hi,

We assume that your issue is resolved. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Thank You