Intel Vtune is not loading data

[Ayam]

Hello,

I am running an application on Intel Vtune (vtune_amplifier_xe_2013) on the linux system (ubuntu).

using command-lin: /opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -collect snb-bandwidth -app-working-dir /usr/local/hadoop -- /usr/local/hadoop/projwc

However, intel vtune is not loading data into the target file.

amplxe: Collection stopped.
amplxe: Using result path `/usr/local/hadoop/r025ge'
amplxe: Executing actions  0 %                                                 
amplxe: Warning: The result contains a lot of raw data. Finalization may take a long time to complete.
amplxe: Executing actions 14 % Loading data files                              
amplxe: Warning: Cannot load data file `/usr/local/hadoop/r025ge/data.0/tbs1546692947.tb6' (tbrw call "TBRW_dobind(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).
amplxe: Executing actions 50 % Generating a report 

The result file r025ge is empty.

If I run the application with small data set, vtune runs smoothly however I am getting above error with big data set.

I have actually posted this question couple of weeks back. I was suggested to check the heap size and it solved my problem. But now I am using different machine and I am facing the same error again. I have increase the heap size. Moreover, it also has been suggested by the one of the intel experts to use "target-duration-type=long" on the command-line. But problem remains the same.

I will appreciate your response.

[Bernard]

>>>I have actually posted this question couple of weeks back. I was suggested to check the heap size and it solved my problem>>>

Not always what is solving the problem on one system will be a solution on the second system.

I think that you need try to tweak the setting of Java VM probably those related to the heap.

http://pic.dhe.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=%2Fcom.ibm.websphere.express.doc%2Finfo%2Fexp%2Fae%2Furun_rconfproc_jvm.html

[Peter_W_Intel]

Are you using U16, and suggest to generate results in /tmp since I don't know if you have enough space to store data in /local/usr for long run.

For example,

amplxe-cl -collect bandwidth -result /tmp/r000bw -app-working-dir /usr/local/hadoop -- /usr/local/hadoop/projwc

[Ayam]

@Peter Wang can you please explain what is U16. I have changed the result directory to home directory. I have enough space available but still I am getting the same error.

[Bernard]

 U16 = Update number 16.

[Peter_W_Intel · Accepted solution]

@Maria M.

If you ran small data set, it can work. I doubt if system resource was exhausted after long run. Maybe you can check:

1. amplxe-cl -collect bandwidth -result /tmp/r000bw -app-working-dir /usr/local/hadoop -duration 120 -- /usr/local/hadoop/projwc

2. Manually invoke /usr/local/hadoop/projwc, then do - amplxe-cl -collect bandwidth -result /tmp/r000bw  -duration 600

[Bernard]

>>>I doubt if system resource was exhausted after long run>>>

Maybe there is a memory leak after long run?I tend to suspect hadoop implementation,but  @Maria is using Linux I do not know how to monitor heap usage on Linux.

[Peter_W_Intel]

> but  @Maria is using Linux I do not know how to monitor heap usage on Linux.

Use "cat /proc/meminfo" to know?

My prior post to investigate: 1) If big data set with short run is OK? 2) System wide profiling (bandwidth) is OK? Application ran separately from VTune

[Bernard]

Thanks Peter.

[Ayam]

Thanks everyone to help me solve this issue.
I appreciate your help.

[Ahsan_Javed_A_]

Hi,

I am facing the similar problem and tried all the way arounds described in this forum but still no success. Here is a log

amplxe: Executing actions 14 % Loading data files                              
amplxe: Warning: Cannot load data file `/home/ajawan/Spark/kmeans/r00/data.0/tbs1526946420.tb6' (tbrw call "TBRW_dobind(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).
amplxe: Executing actions 33 % Precomputing frequently used data               
amplxe: Warning: Cannot find data to precompute. Skipping the precomputation step.
amplxe: Executing actions 50 % Generating a report                             

Collection and Platform Info
----------------------------
Parameter                 r00                                                 
------------------------  ----------------------------------------------------
Application Command Line                                                      
Operating System          2.6.32-504.8.1.el6.x86_64 CentOS release 6.6 (Final)
Computer Name             valarauko.sics.se                                   
Result Size               121704299                                           
Collection start time     18:10:02 11/03/2015 UTC                             
Collection stop time      18:12:38 11/03/2015 UTC                             

CPU
---
Parameter          r00                                
-----------------  -----------------------------------
Name               Intel(R) Xeon(R) E5/E7 v2 processor
Logical CPU Count  24                                 

Summary
-------
Elapsed Time:  0.000
amplxe: Executing actions 100 % done        

[Vitaly_S_Intel]

Can you please tell which VTune version do you use?

[Ahsan_Javed_A_]

I am using Intel vtune_amplifier_2013.

Problem occurs with the duration of measurement is larger than 1 min

[Vitaly_S_Intel]

Can you upgrade to 2015 version?

[Ahsan_Javed_A_]

let me try  the free version and see if it solves the problem

[Ahsan_Javed_A_]

it does not help either

/opt/intel/vtune_amplifier_xe_2015.2.0.393444/bin64/amplxe-cl -collect general-exploration -knob enable-stack-collection=false -knob collect-memory-bandwidth=true -follow-child -mrte-mode=auto -target-duration-type=short -no-allow-multiple-runs -no-analyze-system -data-limit=1024 -slow-frames-threshold=40 -fast-frames-threshold=100 -r /home/ajawan/Spark/wordcount/r00 -target-pid $vm_id

I have tried changing short to long and also increased the heap size as well but no success. 

[Ayam]

Have you tried increasing the "-data-limit" size.

[Ahsan_Javed_A_]

yes, I have also tried to set it to 0 as well. As a work around, I am collecting the data with -no-auto-finalize and then finalizing the result in the next command. It seems to to work for me

[Peter_W_Intel]

Ahsan Javed A. wrote:

yes, I have also tried to set it to 0 as well. As a work around, I am collecting the data with -no-auto-finalize and then finalizing the result in the next command. It seems to to work for me

Setting data-limit as zero, it means that you can collect raw data unlimited (for long run application). It might cause finalizing time extra long, my opinion is to use duration option that can reduce raw data and shorten finalizing time (also don't use stack sampling, which has more raw data to be processed)  

[Ahsan_Javed_A_]

I am trying to run concurrency analysis with latest version of Apache Spark, but it is failed. I run the concurrency analysis in paused mode and resume it manually after some time, but in the paused mode, Spark based java process drop the cpu utilization to 0.3 and the logfiles generated by spark stop being updated that suggests the process get hanged. Outside Vtune, Spark applications work well and finish properly, so I suppose that It has something to do with Vtune. I have also tried attaching to process option, in that case, when I attach to the process, afore-mentioned behaviour tends to appear. I also tried  changing the duration but no success. 

FInd below the command, I am using

/opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -collect concurrency -no-auto-finalize -start-paused -knob sampling-interval=10 -knob enable-user-tasks=false -knob enable-user-sync=false -follow-child -mrte-mode=auto -target-duration-type=medium -no-allow-multiple-runs -no-analyze-system -data-limit=5120 -slow-frames-threshold=40 -fast-frames-threshold=100 -app-working-dir /home/ajawan/Spark/kmeans -result-dir /home/ajawan/Spark/kmeans/r00 -- /home/ajawan/Spark/kmeans/kmeans $1

/opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -finalize -r /home/ajawan/Spark/kmeans/r00

Any help or clue, where to look for will be deeply appreciated

[Peter_W_Intel]

Several comments:

1. Current version is 2015 U4, but you used 2013. Can you please verify if the problem cab be reproduced in latest version?

2. <... I run the concurrency analysis in paused mode and resume it manually after some time...>  do you have insert resume API in code? Or you run "amplxe-cl -command resume -r r???cc. I know that you target process is java, but what will happen if you set "-mrte-mode=auto" option? You can filter result from profiling in whole period.

3. Is there any problem if you run advanced-hotspots or general-exploration instead of concurrency analysis? 

amplxe-cl -collect advanced-hotspots -no-auto-finalize -follow-child -mrte-mode=auto -no-allow-multiple-runs -result-dir /home/ajawan/Spark/kmeans/r00 -- /home/ajawan/Spark/kmeans/kmeans $1

4. As another workaround, you can manually launch app, then use "amplxe-cl -collect advanced-hotspots -analyze-system -duration 30"