Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Bernard
Black Belt
1,445 Views

Intel VTune loading data file error (failed: invalid string (97))

Jump to solution

Hello,

Since last week I have been experiencing a mixture of Signal 7 (SIGBUS) error and failure to load a data file.

In our environment VTune (ver. 2020.0.0.605129) connects over SSH to remote target which is a Skylake server. The Skylake server runs a SEP collector which samples our uplink layer 5G simulation module. The process of collection always completes without any error and successfully creates a samples data file *.tb7.

VTune GUI which is installed on my development machines during parsing phase of .tb7 file displays a following error:

vtune: Error: Cannot load data file `/home/workspace/VRAN/vl1/build_rcp_avx/artifacts/bts_artifacts/PRACH_profiling/profiling/data/data.0/sep7ffff1183700.20200714T151424.068574.tb7' (tbrw call "TBRW_INMEM_initialize_data(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).

At random times VTune receives a SIGBUS error which might be sent due to accessing or touching probably undefined portion of memory mapped file(s).

I'm attaching the Ubuntu .crash file and vtune log file.

Thank you for your help.

 

 

Labels (1)
0 Kudos
1 Solution
MinYeol_L_Intel
Employee
1,265 Views

Yes, at least for now.

The issue happens in a combination of running Vtune collector in non-root privilege and some JIT activities inside container in my understanding. So, it may not happen all the times as you experience. 

View solution in original post

25 Replies
ArunJ_Intel
Moderator
1,170 Views

Hey iliyapolak,


Thanks for reaching out.


Could you provide the following information to help us debug on this. 


1)Are you using the same version of vtune on your development machines and the remote server.

2)Are you trying to open the full vtune result using vtune or are you just importing the Tb7 file. If you are importing the full report could you try with importing the tb7 alone.To import Tb7 alone you can select import option from vtune gui and select your tb7. Let us know if this works.



Thanks

Arun Jose



Bernard
Black Belt
1,167 Views

Hi Arun Jose,

Thank you very much for quick reply!

1) The remote server uses only SEP driver collector, it does not have any VTune installation.

2) From the vtune-gui I open data.vtune file (size of 138 bytes) and in this moment the error materializes. I can not import the .tb7 file because the "import" option is inactive and is greyed out, hence I'm forced to open directly the *vtune.data file.

*We are not creating project, just only communicate with SEP driver and receive samples from remote server.

P.s.

      FYI -- this is the version of VTune installed on remoter server

       /opt/intel/vtune_profiler_2020.0.0.605129/bin64/vtune

       

 

 

ArunJ_Intel
Moderator
1,162 Views

Hey iliyapolak,

 

What I meant was to download the tb7 file to your development machine and use the Import results option in the GUI. I am also attaching a screenshot on how to select and import the tb7 file.

 

 

step1.PNGstep 2.PNGstep 3.PNG

 

 

Bernard
Black Belt
1,158 Views

Now I understood what you have ment.

I will do that.

Thank you

ArunJ_Intel
Moderator
1,150 Views

Hey iliyapolak,


Have you tried out the solution provided ?. Does that help.


Thanks

Arun Jose


Bernard
Black Belt
1,139 Views

Hi Arun,

Yes I tried to apply your workaround unfortunately the "import" option is still inactive. The tb7 file is stored on my local machine.

P.s.

        I will try the command line switch --import.

       I will let you know.

I'm using Linux version of VTune

iliyapolak_0-1596016813186.png

 

 

Kirill_U_Intel
Employee
1,133 Views

Hi.

Create New Project and make the import inside it

Kirill

Bernard
Black Belt
1,128 Views

Thanks Kirill for pointing it out.

I will try it and let you know the outcome.

Thanks

 

Bernard
Black Belt
1,127 Views

I opened the raw samples file (tb7) and unfortunately the error persists.

Here is the screenshot.

iliyapolak_0-1596021029022.png

 

Bernard
Black Belt
1,112 Views

As a an additional information I would like to provide a top callstack functions active during the SIGBUS error, which sometimes VTune receives.

I strongly suspect, that somehow referencing the undefined section of memory mapped file (tb7?) might result in Signal 7 error.

 

iliyapolak_0-1596027264341.png

The relevant captured register context

iliyapolak_1-1596027386569.png

 

The faulting IP and surrounding it assembly.

 

iliyapolak_2-1596027548582.png

 

MinYeol_L_Intel
Employee
1,094 Views

Hi,

Can you share the tb7 file with us?

- Min

Bernard
Black Belt
1,084 Views

Hi,

Yes of course I will share, that file tomorrow with you.

Thank you.

Bernard
Black Belt
1,073 Views

Hi Min Yeol,

As per your request I attached two zipped files a .tb7 and smaller .sc file. These files do not belong to profiling session plagued by SIGBUS signal.

We observed two different situation which caused directly or indirectly to .tb7 file parsing failure.

1) An aforementioned rare condition of SIGBUS signal

2) Most frequently present failure of "TBRW_INMEM_initialize_data" function

(tbrw call "TBRW_INMEM_initialize_data(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).

In case of #1 there is always created a .crash file which I partly analyzed unfortunately I was not able to inspect the arguments (memory addresses) because their content obviously was not collected during the crash data collection phase.

I can hypothesize than somehow a memory reference into the undefined portion of memory mapped file could have triggered the SIGBUS error, I do not suspect a hardware error.

 

In case of #2 I suspect a possibility of some kind of data/metadata corruption. I suppose that SEP collector might be sending a packets over the network to its receiving end and somehow part of those packets might have been corrupted

 

Thank you for your help

Tags (1)
MinYeol_L_Intel
Employee
1,045 Views

Thanks for sharing the result files. It seems the issue is similar to what we experienced before. To confirm it, I have two questions for you.

1. Can you confirm your target machine runs a docker container? 

2. Is the vtune collection through ssh is under the same user as docker container is launched? Or are they under different users?

Regards,

Min

Bernard
Black Belt
1,039 Views

Hello Min,

Thank you for quick response.

I will contact the person responsible for managing our profiling environment and  provide  the answers in regards to your questions.

Thank you

 

Bernard
Black Belt
1,032 Views

Hi,

I have the answers:

1) Our target machine runs Docker container

2) The same user account is used for docker container and for VTune collection session.

Thank you

MinYeol_L_Intel
Employee
1,022 Views

Thanks.

Can you try two things?

1. Run VTune collection without docker container.

2. Run Vtune collection AND docker container with root privilege if you need to run a docker container.

 

Thanks

Bernard
Black Belt
1,015 Views

Hi Min,

 

I will test those 2 options and let you know the results.

 

Thank you for your help

Bernard
Black Belt
995 Views

Hi Min

1) Our simulation environment relies on Docker and we can not drop the usage of it

2) VTune process has a root privilege.

P.s.

        Could you explain (if allowed of course) the meaning of invalid string (error 97) failure?

Thank you for your help

MinYeol_L_Intel
Employee
560 Views

I didn't understand the second answer. Was it working with root privilege? If not, did you see the same error or different?

Reply