Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Beginner
135 Views

Intel Vtune is not loading data

Jump to solution

Hello,

I am running an application on Intel Vtune (vtune_amplifier_xe_2013) on the linux system (ubuntu).

using command-lin: /opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -collect snb-bandwidth -app-working-dir /usr/local/hadoop -- /usr/local/hadoop/projwc

However, intel vtune is not loading data into the target file.

amplxe: Collection stopped.
amplxe: Using result path `/usr/local/hadoop/r025ge'
amplxe: Executing actions  0 %                                                 
amplxe: Warning: The result contains a lot of raw data. Finalization may take a long time to complete.
amplxe: Executing actions 14 % Loading data files                              
amplxe: Warning: Cannot load data file `/usr/local/hadoop/r025ge/data.0/tbs1546692947.tb6' (tbrw call "TBRW_dobind(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).
amplxe: Executing actions 50 % Generating a report 

The result file r025ge is empty.

If I run the application with small data set, vtune runs smoothly however I am getting above error with big data set.

I have actually posted this question couple of weeks back. I was suggested to check the heap size and it solved my problem. But now I am using different machine and I am facing the same error again. I have increase the heap size. Moreover, it also has been suggested by the one of the intel experts to use "target-duration-type=long" on the command-line. But problem remains the same.

I will appreciate your response.

 

 

0 Kudos

Accepted Solutions
Highlighted
Employee
103 Views

@Maria M.

If you ran small data set, it can work. I doubt if system resource was exhausted after long run. Maybe you can check:

1. amplxe-cl -collect bandwidth -result /tmp/r000bw -app-working-dir /usr/local/hadoop -duration 120 -- /usr/local/hadoop/projwc

2. Manually invoke /usr/local/hadoop/projwc, then do - amplxe-cl -collect bandwidth -result /tmp/r000bw  -duration 600

View solution in original post

0 Kudos
20 Replies
Highlighted
Black Belt
103 Views

>>>I have actually posted this question couple of weeks back. I was suggested to check the heap size and it solved my problem>>>

Not always what is solving the problem on one system will be a solution on the second system.

I think that you need try to tweak the setting of Java VM probably those related to the heap.

http://pic.dhe.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=%2Fcom.ibm.websphere.express.doc%2Fin...

0 Kudos
Highlighted
Employee
103 Views

Are you using U16, and suggest to generate results in /tmp since I don't know if you have enough space to store data in /local/usr for long run.

For example,

amplxe-cl -collect bandwidth -result /tmp/r000bw -app-working-dir /usr/local/hadoop -- /usr/local/hadoop/projwc

 

 

0 Kudos
Highlighted
Beginner
103 Views

@Peter Wang can you please explain what is U16. I have changed the result directory to home directory. I have enough space available but still I am getting the same error.

0 Kudos
Highlighted
Black Belt
103 Views

 

 U16 = Update number 16.

0 Kudos
Highlighted
Employee
104 Views

@Maria M.

If you ran small data set, it can work. I doubt if system resource was exhausted after long run. Maybe you can check:

1. amplxe-cl -collect bandwidth -result /tmp/r000bw -app-working-dir /usr/local/hadoop -duration 120 -- /usr/local/hadoop/projwc

2. Manually invoke /usr/local/hadoop/projwc, then do - amplxe-cl -collect bandwidth -result /tmp/r000bw  -duration 600

View solution in original post

0 Kudos
Highlighted
Black Belt
103 Views

 

>>>I doubt if system resource was exhausted after long run>>>

Maybe there is a memory leak after long run?I tend to suspect hadoop implementation,but  @Maria is using Linux I do not know how to monitor heap usage on Linux.

0 Kudos
Highlighted
Employee
103 Views

> but  @Maria is using Linux I do not know how to monitor heap usage on Linux.

Use "cat /proc/meminfo" to know?

My prior post to investigate: 1) If big data set with short run is OK? 2) System wide profiling (bandwidth) is OK? Application ran separately from VTune

0 Kudos
Highlighted
Black Belt
103 Views

Thanks Peter.

0 Kudos
Highlighted
Beginner
103 Views

Thanks everyone to help me solve this issue.
I appreciate your help.

 

 

0 Kudos
Highlighted
Beginner
103 Views

Hi,

I am facing the similar problem and tried all the way arounds described in this forum but still no success. Here is a log

amplxe: Executing actions 14 % Loading data files                              
amplxe: Warning: Cannot load data file `/home/ajawan/Spark/kmeans/r00/data.0/tbs1526946420.tb6' (tbrw call "TBRW_dobind(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).
amplxe: Executing actions 33 % Precomputing frequently used data               
amplxe: Warning: Cannot find data to precompute. Skipping the precomputation step.
amplxe: Executing actions 50 % Generating a report                             

Collection and Platform Info
----------------------------
Parameter                 r00                                                 
------------------------  ----------------------------------------------------
Application Command Line                                                      
Operating System          2.6.32-504.8.1.el6.x86_64 CentOS release 6.6 (Final)
Computer Name             valarauko.sics.se                                   
Result Size               121704299                                           
Collection start time     18:10:02 11/03/2015 UTC                             
Collection stop time      18:12:38 11/03/2015 UTC                             

CPU
---
Parameter          r00                                
-----------------  -----------------------------------
Name               Intel(R) Xeon(R) E5/E7 v2 processor
Logical CPU Count  24                                 

Summary
-------
Elapsed Time:  0.000
amplxe: Executing actions 100 % done        

0 Kudos
Highlighted
Employee
103 Views

Can you please tell which VTune version do you use?

0 Kudos
Highlighted
Beginner
103 Views

I am using Intel vtune_amplifier_2013.

Problem occurs with the duration of measurement is larger than 1 min

0 Kudos
Highlighted
Employee
103 Views

Can you upgrade to 2015 version?

0 Kudos
Highlighted
Beginner
103 Views

let me try  the free version and see if it solves the problem

0 Kudos
Highlighted
Beginner
103 Views

it does not help either

/opt/intel/vtune_amplifier_xe_2015.2.0.393444/bin64/amplxe-cl -collect general-exploration -knob enable-stack-collection=false -knob collect-memory-bandwidth=true -follow-child -mrte-mode=auto -target-duration-type=short -no-allow-multiple-runs -no-analyze-system -data-limit=1024 -slow-frames-threshold=40 -fast-frames-threshold=100 -r /home/ajawan/Spark/wordcount/r00 -target-pid $vm_id

I have tried changing short to long and also increased the heap size as well but no success. 

0 Kudos
Highlighted
Beginner
103 Views

Have you tried increasing the "-data-limit" size.

0 Kudos
Highlighted
Beginner
103 Views

yes, I have also tried to set it to 0 as well. As a work around, I am collecting the data with -no-auto-finalize and then finalizing the result in the next command. It seems to to work for me

0 Kudos
Highlighted
Employee
103 Views

Ahsan Javed A. wrote:

yes, I have also tried to set it to 0 as well. As a work around, I am collecting the data with -no-auto-finalize and then finalizing the result in the next command. It seems to to work for me

Setting data-limit as zero, it means that you can collect raw data unlimited (for long run application). It might cause finalizing time extra long, my opinion is to use duration option that can reduce raw data and shorten finalizing time (also don't use stack sampling, which has more raw data to be processed)  

0 Kudos
Highlighted
Beginner
103 Views

I am trying to run concurrency analysis with latest version of Apache Spark, but it is failed. I run the concurrency analysis in paused mode and resume it manually after some time, but in the paused mode, Spark based java process drop the cpu utilization to 0.3 and the logfiles generated by spark stop being updated that suggests the process get hanged. Outside Vtune, Spark applications work well and finish properly, so I suppose that It has something to do with Vtune. I have also tried attaching to process option, in that case, when I attach to the process, afore-mentioned behaviour tends to appear. I also tried  changing the duration but no success. 

 

FInd below the command, I am using

/opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -collect concurrency -no-auto-finalize -start-paused -knob sampling-interval=10 -knob enable-user-tasks=false -knob enable-user-sync=false -follow-child -mrte-mode=auto -target-duration-type=medium -no-allow-multiple-runs -no-analyze-system -data-limit=5120 -slow-frames-threshold=40 -fast-frames-threshold=100 -app-working-dir /home/ajawan/Spark/kmeans -result-dir /home/ajawan/Spark/kmeans/r00 -- /home/ajawan/Spark/kmeans/kmeans $1

/opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -finalize -r /home/ajawan/Spark/kmeans/r00

 

Any help or clue, where to look for will be deeply appreciated

0 Kudos