Problems with VTune Memory Consumption finalisation

[Towie__Ewan]

Hi all,

I'm trying to complete the Memory Consumption analysis using VTune 2019U5 and I keep hitting the same problem. For reference, I'm running on CentOS 7 using gcc 7.3.0 on Intel64 machine with 10cores, 760GB of RAM but only 280GB of disk space.

The problem I'm encountering is that I run out of disk space when finalising the results. I've tried reducing the test case size, and also increasing the min memory object size to 128KB. Neither of these seem to have helped much.

The test case I am trying to profile peaks at around 10GB of RAM (this is relatively small in my world), and the result folder (r00XX folder) produced before finalisation is ~37GB. However, when I try to finalise the results folder balloons to in excess of 280GB filling the drive.

Can anyone give me some tips on how to resolve the issue?

Are there are further configuration options I can adjust to help reduce the results size?

Any help appreciated!

[AthiraM_Intel]

Hi,

Thanks for reaching out to us.

The VTune Profiler saves the raw collector data after finalization. You could choose to remove these data to reduce the size of the result file if you do not plan to re-finalize this result in the future. 

To remove the raw collector data in the standalone interface, click the menu button and select Options... > General and select the Remove raw collector data after resolving the result option.

For more information, please refer the below link:

https://software.intel.com/content/www/us/en/develop/documentation/vtune-help/top/analyze-performance/control-data-collection/finalization.html

Also we would always recommend to use the latest versions ( VTune 2020 update 2).

Hope this helps.

[Towie__Ewan]

Hi,

That is a good suggestion, however I am not able to finalise the result before the available disk space is exceeded.

Do you have any suggestions on how to reduce the disk space required during finalisation?

I typically select a Full finalisation as I will only intend to finalise once, and this tends to be the more efficient option for viewing results at a later date. In my use case, I will copy the results from the sampling machine to my workstation after finalisation to perform my analysis.

Thanks!

[Kirill_U_Intel]

>>In my use case, I will copy the results from the sampling machine to my workstation after finalisation to perform my analysis.

You could try 'Remote Linux(SSH)' connection type from the host with sufficient disk space. VTune copy profiling results (37GB in your case) automatically and finalize result on the host.

Kirill

[Towie__Ewan]

Hi Kirill,

That may be a solution!

I've not tried a Remote SSH session with VTune before. Can this be run using the command-line tool?

Also, where do you locate the binary, symbols, and source, and the test case inputs in the Remote SSH case?

Would they need to be available on both machines?

Thanks

[Kirill_U_Intel]

Hi.

>>Can this be run using the command-line tool?

Sure. You could generate the command line form the VTune GUI. In general, difference will be in target options like  -target-system=ssh:user1@172.16.254.1; --target-install-dir=vtune_path_on_the target

https://software.intel.com/content/www/us/en/develop/documentation/vtune-help/top/command-line-interface/running-command-line-analysis/configuring-analysis-options-from-the-command-line/collecting-data-on-remote-linux-systems-from-the-command-line.html 

>>Also, where do you locate the binary, symbols, and source, and the test case inputs in the Remote SSH case?

VTune copy binaries from the target automatically for resolving.

Symbols and sources must be on the host in search dir (--search-dir/--source-search-dir). https://software.intel.com/content/www/us/en/develop/documentation/vtune-help/top/set-up-analysis-target/linux-targets/remote-linux-target-setup/search-directories-for-remote-linux-targets.html

Test case inputs should be on the target.

[AthiraM_Intel]

Hi,

Could you please give us an update? Is your issue resolved?

Thanks

[Towie__Ewan]

Hi,

Good timing, my test just finished this morning!

Sadly the issue isn't fully resolved. I managed to run a Remote SSH session from a machine that had 550GB of free disk space but still VTune finalisation of the memory consumption analysis filled the disk before completing.

So, I may be able to use a machine that has >1TB of free disk space but I now wonder if VTune will be functional with a finalised results size that large?

I come back to the question about what options there are for reducing the results size with memory consumption analysis?

Does adjusting the minimum dynamic memory object size have any impact?

Is it possible to use ITT to reduce the areas profiled?

Additionally, I checked the option to remove the raw results after finalisation in the GUI but I don't think the option has any impact if the results are finalised via the command line VTune profiling tool. I tend to only use the GUI to analyse the finalised results. Is there a way to remove the raw results via the command line tool, or can you tell me which results are deleted?

Thanks!

[Kirill_U_Intel]

Hi

Is it possible to reduce the time of profiled application?

Also you could try to start collection in pause mode and add itt resume/pause calls for interested code like https://www.cism.ucl.ac.be/Services/Formations/ICS/ics_2013.0.028/vtune_amplifier_xe/documentation/en/help/GUID-CF6D9CFF-E9E1-4C8D-8900-0D150FA0D210.htm

int main(int argc, char* argv[])
{
  // Do initialization work here
  __itt_resume();
 	// Do profiling work here
  __itt_pause();
  // Do finalization work here
  return 0;
}

Usually, raw collector data are located in RESULT_DIR/data.0 path

Kirill

[Towie__Ewan]

Ok, I will try and use ITT to reduce the range of code profiled. Reducing the runtime might be a bit tricky as I've cut down the test case quite heavily already.

And thanks, will do a bit of a clean up of my existing results to recover some disk space.

[AthiraM_Intel]

Hi,

Could you please let us know whether the issue resolved?

Thanks

[Towie__Ewan]

Hi,

I've not made any progress with VTune Memory Consumption analysis I'm afraid. I'm struggling to understand why it consumes so much disk space during finalisation.

As I needed to make progress on this task, I've reverted to Valgrind Massif memory analysis tool which has completed the same test case and produces a result file of <1MB. It may not have the resolution of results of VTune in it's default setting, but I have at least been able to get some starting results from it.

If I have any free time I may try to add some ITT calls to pause/resume collection around the key areas in the code. However, cutting the simulation runtime any shorter isn't possible (with VTune the runtime is roughly 1.5hrs). The reason I require memory consumption analysis is because the code I am running is consuming more RAM than we expected when completing large tasks and I'm not sure which part of the code isn't scaling well with size.

If anyone has any other suggestions for controlling the Memory Consumption analysis to produce results that are physically manageable, then I am happy to hear them!

Thanks,

Ewan

[AthiraM_Intel]

Hi,

We are forwarding your case to Subject matter Experts.

Thanks