Since last week I have been experiencing a mixture of Signal 7 (SIGBUS) error and failure to load a data file.
In our environment VTune (ver. 2020.0.0.605129) connects over SSH to remote target which is a Skylake server. The Skylake server runs a SEP collector which samples our uplink layer 5G simulation module. The process of collection always completes without any error and successfully creates a samples data file *.tb7.
VTune GUI which is installed on my development machines during parsing phase of .tb7 file displays a following error:
vtune: Error: Cannot load data file `/home/workspace/VRAN/vl1/build_rcp_avx/artifacts/bts_artifacts/PRACH_profiling/profiling/data/data.0/sep7ffff1183700.20200714T151424.068574.tb7' (tbrw call "TBRW_INMEM_initialize_data(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).
At random times VTune receives a SIGBUS error which might be sent due to accessing or touching probably undefined portion of memory mapped file(s).
I'm attaching the Ubuntu .crash file and vtune log file.
Thank you for your help.
Yes, at least for now.
The issue happens in a combination of running Vtune collector in non-root privilege and some JIT activities inside container in my understanding. So, it may not happen all the times as you experience.
Thanks for reaching out.
Could you provide the following information to help us debug on this.
1)Are you using the same version of vtune on your development machines and the remote server.
2)Are you trying to open the full vtune result using vtune or are you just importing the Tb7 file. If you are importing the full report could you try with importing the tb7 alone.To import Tb7 alone you can select import option from vtune gui and select your tb7. Let us know if this works.
Hi Arun Jose,
Thank you very much for quick reply!
1) The remote server uses only SEP driver collector, it does not have any VTune installation.
2) From the vtune-gui I open data.vtune file (size of 138 bytes) and in this moment the error materializes. I can not import the .tb7 file because the "import" option is inactive and is greyed out, hence I'm forced to open directly the *vtune.data file.
*We are not creating project, just only communicate with SEP driver and receive samples from remote server.
FYI -- this is the version of VTune installed on remoter server
What I meant was to download the tb7 file to your development machine and use the Import results option in the GUI. I am also attaching a screenshot on how to select and import the tb7 file.
Yes I tried to apply your workaround unfortunately the "import" option is still inactive. The tb7 file is stored on my local machine.
I will try the command line switch --import.
I will let you know.
I'm using Linux version of VTune
As a an additional information I would like to provide a top callstack functions active during the SIGBUS error, which sometimes VTune receives.
I strongly suspect, that somehow referencing the undefined section of memory mapped file (tb7?) might result in Signal 7 error.
The relevant captured register context
The faulting IP and surrounding it assembly.
Hi Min Yeol,
As per your request I attached two zipped files a .tb7 and smaller .sc file. These files do not belong to profiling session plagued by SIGBUS signal.
We observed two different situation which caused directly or indirectly to .tb7 file parsing failure.
1) An aforementioned rare condition of SIGBUS signal
2) Most frequently present failure of "TBRW_INMEM_initialize_data" function
(tbrw call "TBRW_INMEM_initialize_data(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).
In case of #1 there is always created a .crash file which I partly analyzed unfortunately I was not able to inspect the arguments (memory addresses) because their content obviously was not collected during the crash data collection phase.
I can hypothesize than somehow a memory reference into the undefined portion of memory mapped file could have triggered the SIGBUS error, I do not suspect a hardware error.
In case of #2 I suspect a possibility of some kind of data/metadata corruption. I suppose that SEP collector might be sending a packets over the network to its receiving end and somehow part of those packets might have been corrupted
Thank you for your help
Thanks for sharing the result files. It seems the issue is similar to what we experienced before. To confirm it, I have two questions for you.
1. Can you confirm your target machine runs a docker container?
2. Is the vtune collection through ssh is under the same user as docker container is launched? Or are they under different users?
Thank you for quick response.
I will contact the person responsible for managing our profiling environment and provide the answers in regards to your questions.
I have the answers:
1) Our target machine runs Docker container
2) The same user account is used for docker container and for VTune collection session.
Can you try two things?
1. Run VTune collection without docker container.
2. Run Vtune collection AND docker container with root privilege if you need to run a docker container.
1) Our simulation environment relies on Docker and we can not drop the usage of it
2) VTune process has a root privilege.
Could you explain (if allowed of course) the meaning of invalid string (error 97) failure?
Thank you for your help