topic Are you using U16, and in Analyzers

Intel Vtune is not loading data

Ayam — Thu, 10 Apr 2014 15:59:17 GMT

Hello,

I am running an application on Intel Vtune (vtune_amplifier_xe_2013) on the linux system (ubuntu).

using command-lin: /opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -collect snb-bandwidth -app-working-dir /usr/local/hadoop -- /usr/local/hadoop/projwc

However, intel vtune is not loading data into the target file.

amplxe: Collection stopped.
amplxe: Using result path `/usr/local/hadoop/r025ge'
amplxe: Executing actions 0 %
amplxe: Warning: The result contains a lot of raw data. Finalization may take a long time to complete.
amplxe: Executing actions 14 % Loading data files
amplxe: Warning: Cannot load data file `/usr/local/hadoop/r025ge/data.0/tbs1546692947.tb6' (tbrw call "TBRW_dobind(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).
amplxe: Executing actions 50 % Generating a report

The result file r025ge is empty.

If I run the application with small data set, vtune runs smoothly however I am getting above error with big data set.

I have actually posted this question couple of weeks back. I was suggested to check the heap size and it solved my problem. But now I am using different machine and I am facing the same error again. I have increase the heap size. Moreover, it also has been suggested by the one of the intel experts to use "target-duration-type=long" on the command-line. But problem remains the same.

I will appreciate your response.

>>>I have actually posted

Bernard — Fri, 11 Apr 2014 06:12:32 GMT

>>>I have actually posted this question couple of weeks back. I was suggested to check the heap size and it solved my problem>>>

Not always what is solving the problem on one system will be a solution on the second system.

I think that you need try to tweak the setting of Java VM probably those related to the heap.

http://pic.dhe.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=%2Fcom.ibm.websphere.express.doc%2Finfo%2Fexp%2Fae%2Furun_rconfproc_jvm.html

Are you using U16, and

Peter_W_Intel — Fri, 11 Apr 2014 06:24:56 GMT

Are you using U16, and suggest to generate results in /tmp since I don't know if you have enough space to store data in /local/usr for long run.

For example,

amplxe-cl -collect bandwidth -result /tmp/r000bw -app-working-dir /usr/local/hadoop -- /usr/local/hadoop/projwc

@Peter Wang can you please

Ayam — Fri, 11 Apr 2014 13:09:37 GMT

@Peter Wang can you please explain what is U16. I have changed the result directory to home directory. I have enough space available but still I am getting the same error.

U16 = Update number 16.

Bernard — Fri, 11 Apr 2014 17:25:14 GMT

U16 = Update number 16.

@Maria M.

Peter_W_Intel — Sat, 12 Apr 2014 01:24:31 GMT

@Maria M.

If you ran small data set, it can work. I doubt if system resource was exhausted after long run. Maybe you can check:

1. amplxe-cl -collect bandwidth -result /tmp/r000bw -app-working-dir /usr/local/hadoop -duration 120 -- /usr/local/hadoop/projwc

2. Manually invoke /usr/local/hadoop/projwc, then do - amplxe-cl -collect bandwidth -result /tmp/r000bw -duration 600

Bernard — Sat, 12 Apr 2014 07:26:54 GMT

>>>I doubt if system resource was exhausted after long run>>>

Maybe there is a memory leak after long run?I tend to suspect hadoop implementation,but @Maria is using Linux I do not know how to monitor heap usage on Linux.

> but @Maria is using Linux

Peter_W_Intel — Sat, 12 Apr 2014 09:08:35 GMT

> but @Maria is using Linux I do not know how to monitor heap usage on Linux.

Use "cat /proc/meminfo" to know?

My prior post to investigate: 1) If big data set with short run is OK? 2) System wide profiling (bandwidth) is OK? Application ran separately from VTune

Thanks Peter.

Bernard — Sat, 12 Apr 2014 15:19:21 GMT

Thanks Peter.

Thanks everyone to help me

Ayam — Mon, 14 Apr 2014 13:29:00 GMT

Thanks everyone to help me solve this issue.
I appreciate your help.

Hi,

Ahsan_Javed_A_ — Wed, 11 Mar 2015 18:30:50 GMT

Hi,

I am facing the similar problem and tried all the way arounds described in this forum but still no success. Here is a log

amplxe: Executing actions 14 % Loading data files
amplxe: Warning: Cannot load data file `/home/ajawan/Spark/kmeans/r00/data.0/tbs1526946420.tb6' (tbrw call "TBRW_dobind(tbrwFile->getHandle(), streamIndex)" failed: invalid string (97)).
amplxe: Executing actions 33 % Precomputing frequently used data
amplxe: Warning: Cannot find data to precompute. Skipping the precomputation step.
amplxe: Executing actions 50 % Generating a report

Collection and Platform Info
----------------------------
Parameter                 r00
------------------------ ----------------------------------------------------
Application Command Line
Operating System          2.6.32-504.8.1.el6.x86_64 CentOS release 6.6 (Final)
Computer Name             valarauko.sics.se
Result Size               121704299
Collection start time     18:10:02 11/03/2015 UTC
Collection stop time      18:12:38 11/03/2015 UTC

CPU
---
Parameter          r00
----------------- -----------------------------------
Name               Intel(R) Xeon(R) E5/E7 v2 processor
Logical CPU Count 24

Summary
-------
Elapsed Time: 0.000
amplxe: Executing actions 100 % done

Can you please tell which

Vitaly_S_Intel — Wed, 11 Mar 2015 18:59:13 GMT

Can you please tell which VTune version do you use?

I am using Intel vtune

Ahsan_Javed_A_ — Fri, 13 Mar 2015 13:00:24 GMT

I am using Intel vtune_amplifier_2013.

Problem occurs with the duration of measurement is larger than 1 min

Can you upgrade to 2015

Vitaly_S_Intel — Fri, 13 Mar 2015 15:01:22 GMT

Can you upgrade to 2015 version?

let me try the free version

Ahsan_Javed_A_ — Fri, 13 Mar 2015 15:33:27 GMT

let me try the free version and see if it solves the problem

it does not help either

Ahsan_Javed_A_ — Fri, 13 Mar 2015 17:10:20 GMT

it does not help either

/opt/intel/vtune_amplifier_xe_2015.2.0.393444/bin64/amplxe-cl -collect general-exploration -knob enable-stack-collection=false -knob collect-memory-bandwidth=true -follow-child -mrte-mode=auto -target-duration-type=short -no-allow-multiple-runs -no-analyze-system -data-limit=1024 -slow-frames-threshold=40 -fast-frames-threshold=100 -r /home/ajawan/Spark/wordcount/r00 -target-pid $vm_id

I have tried changing short to long and also increased the heap size as well but no success.

Have you tried increasing the

Ayam — Fri, 13 Mar 2015 17:17:26 GMT

Have you tried increasing the "-data-limit" size.

yes, I have also tried to set

Ahsan_Javed_A_ — Fri, 13 Mar 2015 18:28:02 GMT

yes, I have also tried to set it to 0 as well. As a work around, I am collecting the data with -no-auto-finalize and then finalizing the result in the next command. It seems to to work for me

Quote:Ahsan Javed A. wrote:

Peter_W_Intel — Mon, 16 Mar 2015 03:24:46 GMT

Ahsan Javed A. wrote:

yes, I have also tried to set it to 0 as well. As a work around, I am collecting the data with -no-auto-finalize and then finalizing the result in the next command. It seems to to work for me

Setting data-limit as zero, it means that you can collect raw data unlimited (for long run application). It might cause finalizing time extra long, my opinion is to use duration option that can reduce raw data and shorten finalizing time (also don't use stack sampling, which has more raw data to be processed)

I am trying to run

Ahsan_Javed_A_ — Wed, 10 Jun 2015 10:20:51 GMT

I am trying to run concurrency analysis with latest version of Apache Spark, but it is failed. I run the concurrency analysis in paused mode and resume it manually after some time, but in the paused mode, Spark based java process drop the cpu utilization to 0.3 and the logfiles generated by spark stop being updated that suggests the process get hanged. Outside Vtune, Spark applications work well and finish properly, so I suppose that It has something to do with Vtune. I have also tried attaching to process option, in that case, when I attach to the process, afore-mentioned behaviour tends to appear. I also tried changing the duration but no success.

FInd below the command, I am using

/opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -collect concurrency -no-auto-finalize -start-paused -knob sampling-interval=10 -knob enable-user-tasks=false -knob enable-user-sync=false -follow-child -mrte-mode=auto -target-duration-type=medium -no-allow-multiple-runs -no-analyze-system -data-limit=5120 -slow-frames-threshold=40 -fast-frames-threshold=100 -app-working-dir /home/ajawan/Spark/kmeans -result-dir /home/ajawan/Spark/kmeans/r00 -- /home/ajawan/Spark/kmeans/kmeans $1

/opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -finalize -r /home/ajawan/Spark/kmeans/r00

Any help or clue, where to look for will be deeply appreciated