Analyzers
Talk to fellow users of Intel Analyzer tools (Intel VTune™ Profiler, Intel Advisor)
4975 Discussions

Intel VTune loading data file error (failed: invalid string (97))

Bernard
Valued Contributor I
34,257 Views

Hello,

Since last week I have been experiencing a mixture of Signal 7 (SIGBUS) error and failure to load a data file.

In our environment VTune (ver. 2020.0.0.605129) connects over SSH to remote target which is a Skylake server. The Skylake server runs a SEP collector which samples our uplink layer 5G simulation module. The process of collection always completes without any error and successfully creates a samples data file *.tb7.

VTune GUI which is installed on my development machines during parsing phase of .tb7 file displays a following error:

vtune: Error: Cannot load data file `/home/workspace/VRAN/vl1/build_rcp_avx/artifacts/bts_artifacts/PRACH_profiling/profiling/data/data.0/sep7ffff1183700.20200714T151424.068574.tb7' (tbrw call "TBRW_INMEM_initialize_data(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).

At random times VTune receives a SIGBUS error which might be sent due to accessing or touching probably undefined portion of memory mapped file(s).

I'm attaching the Ubuntu .crash file and vtune log file.

Thank you for your help.

 

 

Labels (1)
0 Kudos
1 Solution
MinYeol_L_Intel
Employee
34,077 Views

Yes, at least for now.

The issue happens in a combination of running Vtune collector in non-root privilege and some JIT activities inside container in my understanding. So, it may not happen all the times as you experience. 

View solution in original post

25 Replies
ArunJ_Intel
Moderator
33,279 Views

Hey iliyapolak,


Thanks for reaching out.


Could you provide the following information to help us debug on this. 


1)Are you using the same version of vtune on your development machines and the remote server.

2)Are you trying to open the full vtune result using vtune or are you just importing the Tb7 file. If you are importing the full report could you try with importing the tb7 alone.To import Tb7 alone you can select import option from vtune gui and select your tb7. Let us know if this works.



Thanks

Arun Jose



0 Kudos
Bernard
Valued Contributor I
33,276 Views

Hi Arun Jose,

Thank you very much for quick reply!

1) The remote server uses only SEP driver collector, it does not have any VTune installation.

2) From the vtune-gui I open data.vtune file (size of 138 bytes) and in this moment the error materializes. I can not import the .tb7 file because the "import" option is inactive and is greyed out, hence I'm forced to open directly the *vtune.data file.

*We are not creating project, just only communicate with SEP driver and receive samples from remote server.

P.s.

      FYI -- this is the version of VTune installed on remoter server

       /opt/intel/vtune_profiler_2020.0.0.605129/bin64/vtune

       

 

 

0 Kudos
ArunJ_Intel
Moderator
33,271 Views

Hey iliyapolak,

 

What I meant was to download the tb7 file to your development machine and use the Import results option in the GUI. I am also attaching a screenshot on how to select and import the tb7 file.

 

 

step1.PNGstep 2.PNGstep 3.PNG

 

 

0 Kudos
Bernard
Valued Contributor I
33,267 Views

Now I understood what you have ment.

I will do that.

Thank you

0 Kudos
ArunJ_Intel
Moderator
33,259 Views

Hey iliyapolak,


Have you tried out the solution provided ?. Does that help.


Thanks

Arun Jose


0 Kudos
Bernard
Valued Contributor I
33,248 Views

Hi Arun,

Yes I tried to apply your workaround unfortunately the "import" option is still inactive. The tb7 file is stored on my local machine.

P.s.

        I will try the command line switch --import.

       I will let you know.

I'm using Linux version of VTune

iliyapolak_0-1596016813186.png

 

 

0 Kudos
Kirill_U_Intel
Employee
33,242 Views

Hi.

Create New Project and make the import inside it

Kirill

Bernard
Valued Contributor I
33,237 Views

Thanks Kirill for pointing it out.

I will try it and let you know the outcome.

Thanks

 

0 Kudos
Bernard
Valued Contributor I
33,236 Views

I opened the raw samples file (tb7) and unfortunately the error persists.

Here is the screenshot.

iliyapolak_0-1596021029022.png

 

0 Kudos
Bernard
Valued Contributor I
33,221 Views

As a an additional information I would like to provide a top callstack functions active during the SIGBUS error, which sometimes VTune receives.

I strongly suspect, that somehow referencing the undefined section of memory mapped file (tb7?) might result in Signal 7 error.

 

iliyapolak_0-1596027264341.png

The relevant captured register context

iliyapolak_1-1596027386569.png

 

The faulting IP and surrounding it assembly.

 

iliyapolak_2-1596027548582.png

 

0 Kudos
MinYeol_L_Intel
Employee
33,203 Views

Hi,

Can you share the tb7 file with us?

- Min

0 Kudos
Bernard
Valued Contributor I
33,193 Views

Hi,

Yes of course I will share, that file tomorrow with you.

Thank you.

0 Kudos
Bernard
Valued Contributor I
33,182 Views

Hi Min Yeol,

As per your request I attached two zipped files a .tb7 and smaller .sc file. These files do not belong to profiling session plagued by SIGBUS signal.

We observed two different situation which caused directly or indirectly to .tb7 file parsing failure.

1) An aforementioned rare condition of SIGBUS signal

2) Most frequently present failure of "TBRW_INMEM_initialize_data" function

(tbrw call "TBRW_INMEM_initialize_data(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).

In case of #1 there is always created a .crash file which I partly analyzed unfortunately I was not able to inspect the arguments (memory addresses) because their content obviously was not collected during the crash data collection phase.

I can hypothesize than somehow a memory reference into the undefined portion of memory mapped file could have triggered the SIGBUS error, I do not suspect a hardware error.

 

In case of #2 I suspect a possibility of some kind of data/metadata corruption. I suppose that SEP collector might be sending a packets over the network to its receiving end and somehow part of those packets might have been corrupted

 

Thank you for your help

0 Kudos
MinYeol_L_Intel
Employee
33,154 Views

Thanks for sharing the result files. It seems the issue is similar to what we experienced before. To confirm it, I have two questions for you.

1. Can you confirm your target machine runs a docker container? 

2. Is the vtune collection through ssh is under the same user as docker container is launched? Or are they under different users?

Regards,

Min

0 Kudos
Bernard
Valued Contributor I
33,148 Views

Hello Min,

Thank you for quick response.

I will contact the person responsible for managing our profiling environment and  provide  the answers in regards to your questions.

Thank you

 

0 Kudos
Bernard
Valued Contributor I
33,141 Views

Hi,

I have the answers:

1) Our target machine runs Docker container

2) The same user account is used for docker container and for VTune collection session.

Thank you

0 Kudos
MinYeol_L_Intel
Employee
33,131 Views

Thanks.

Can you try two things?

1. Run VTune collection without docker container.

2. Run Vtune collection AND docker container with root privilege if you need to run a docker container.

 

Thanks

0 Kudos
Bernard
Valued Contributor I
33,124 Views

Hi Min,

 

I will test those 2 options and let you know the results.

 

Thank you for your help

0 Kudos
Bernard
Valued Contributor I
33,104 Views

Hi Min

1) Our simulation environment relies on Docker and we can not drop the usage of it

2) VTune process has a root privilege.

P.s.

        Could you explain (if allowed of course) the meaning of invalid string (error 97) failure?

Thank you for your help

0 Kudos
MinYeol_L_Intel
Employee
32,669 Views

I didn't understand the second answer. Was it working with root privilege? If not, did you see the same error or different?

0 Kudos
Reply