Intel VTune loading data file error (failed: invalid string (97))

[Bernard]

Hello,

Since last week I have been experiencing a mixture of Signal 7 (SIGBUS) error and failure to load a data file.

In our environment VTune (ver. 2020.0.0.605129) connects over SSH to remote target which is a Skylake server. The Skylake server runs a SEP collector which samples our uplink layer 5G simulation module. The process of collection always completes without any error and successfully creates a samples data file *.tb7.

VTune GUI which is installed on my development machines during parsing phase of .tb7 file displays a following error:

vtune: Error: Cannot load data file `/home/workspace/VRAN/vl1/build_rcp_avx/artifacts/bts_artifacts/PRACH_profiling/profiling/data/data.0/sep7ffff1183700.20200714T151424.068574.tb7' (tbrw call "TBRW_INMEM_initialize_data(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).

At random times VTune receives a SIGBUS error which might be sent due to accessing or touching probably undefined portion of memory mapped file(s).

I'm attaching the Ubuntu .crash file and vtune log file.

Thank you for your help.

[ArunJ_Intel]

Hey iliyapolak,

Thanks for reaching out.

Could you provide the following information to help us debug on this. 

1)Are you using the same version of vtune on your development machines and the remote server.

2)Are you trying to open the full vtune result using vtune or are you just importing the Tb7 file. If you are importing the full report could you try with importing the tb7 alone.To import Tb7 alone you can select import option from vtune gui and select your tb7. Let us know if this works.

Thanks

Arun Jose

[Bernard]

Hi Arun Jose,

Thank you very much for quick reply!

1) The remote server uses only SEP driver collector, it does not have any VTune installation.

2) From the vtune-gui I open data.vtune file (size of 138 bytes) and in this moment the error materializes. I can not import the .tb7 file because the "import" option is inactive and is greyed out, hence I'm forced to open directly the *vtune.data file.

*We are not creating project, just only communicate with SEP driver and receive samples from remote server.

P.s.

      FYI -- this is the version of VTune installed on remoter server

       /opt/intel/vtune_profiler_2020.0.0.605129/bin64/vtune

[ArunJ_Intel]

Hey iliyapolak,

What I meant was to download the tb7 file to your development machine and use the Import results option in the GUI. I am also attaching a screenshot on how to select and import the tb7 file.

[Bernard]

Now I understood what you have ment.

I will do that.

Thank you

[ArunJ_Intel]

Hey iliyapolak,

Have you tried out the solution provided ?. Does that help.

Thanks

Arun Jose

[Bernard]

Hi Arun,

Yes I tried to apply your workaround unfortunately the "import" option is still inactive. The tb7 file is stored on my local machine.

P.s.

        I will try the command line switch --import.

       I will let you know.

I'm using Linux version of VTune

[Kirill_U_Intel]

Hi.

Create New Project and make the import inside it

Kirill

[Bernard]

Thanks Kirill for pointing it out.

I will try it and let you know the outcome.

Thanks

[Bernard]

I opened the raw samples file (tb7) and unfortunately the error persists.

Here is the screenshot.

[Bernard]

As a an additional information I would like to provide a top callstack functions active during the SIGBUS error, which sometimes VTune receives.

I strongly suspect, that somehow referencing the undefined section of memory mapped file (tb7?) might result in Signal 7 error.

The relevant captured register context

The faulting IP and surrounding it assembly.

[MinYeol_L_Intel]

Hi,

Can you share the tb7 file with us?

- Min

[Bernard]

Hi,

Yes of course I will share, that file tomorrow with you.

Thank you.

[Bernard]

Hi Min Yeol,

As per your request I attached two zipped files a .tb7 and smaller .sc file. These files do not belong to profiling session plagued by SIGBUS signal.

We observed two different situation which caused directly or indirectly to .tb7 file parsing failure.

1) An aforementioned rare condition of SIGBUS signal

2) Most frequently present failure of "TBRW_INMEM_initialize_data" function

(tbrw call "TBRW_INMEM_initialize_data(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).

In case of #1 there is always created a .crash file which I partly analyzed unfortunately I was not able to inspect the arguments (memory addresses) because their content obviously was not collected during the crash data collection phase.

I can hypothesize than somehow a memory reference into the undefined portion of memory mapped file could have triggered the SIGBUS error, I do not suspect a hardware error.

In case of #2 I suspect a possibility of some kind of data/metadata corruption. I suppose that SEP collector might be sending a packets over the network to its receiving end and somehow part of those packets might have been corrupted

Thank you for your help

[MinYeol_L_Intel]

Thanks for sharing the result files. It seems the issue is similar to what we experienced before. To confirm it, I have two questions for you.

1. Can you confirm your target machine runs a docker container? 

2. Is the vtune collection through ssh is under the same user as docker container is launched? Or are they under different users?

Regards,

Min

[Bernard]

Hello Min,

Thank you for quick response.

I will contact the person responsible for managing our profiling environment and  provide  the answers in regards to your questions.

Thank you

[Bernard]

Hi,

I have the answers:

1) Our target machine runs Docker container

2) The same user account is used for docker container and for VTune collection session.

Thank you

[MinYeol_L_Intel]

Thanks.

Can you try two things?

1. Run VTune collection without docker container.

2. Run Vtune collection AND docker container with root privilege if you need to run a docker container.

Thanks

[Bernard]

Hi Min,

I will test those 2 options and let you know the results.

Thank you for your help

[Bernard]

Hi Min

1) Our simulation environment relies on Docker and we can not drop the usage of it

2) VTune process has a root privilege.

P.s.

        Could you explain (if allowed of course) the meaning of invalid string (error 97) failure?

Thank you for your help

[MinYeol_L_Intel]

I didn't understand the second answer. Was it working with root privilege? If not, did you see the same error or different?