Finalizing results hangs with large .pdb file

[Brent_R_]

I am experiencing an issue with Intel VTune Amplifier.

My executable has a rather large .pdb file (~390MB). After collecting data, the resolve step hangs while resolving symbols. I can fix the problem by removing un-referenced functions from the .exe and .pdb files (Visual Studio Linker option /OPT:REF). This shrinks the .pdb file to ~140MB and the .exe file from ~22MB to ~9MB.

Is this a known issue with V-Tune? Even with very small samples; V-Tune will hang if the symbol file is large.

Additional Details:

I have tried using the command-line client as well as the GUI.

Compiler: VS2013

V-TUNE Version: Tried 2013u17 and 2015u1

[Bernard]

Try to run Process Explorer when VTune hangs. In PE you should see high percentage usage of CPU cycle which could be related to aforementioned hung.  Next step will be checking for hung process threads and inspecting their stacks. Usually top-of-the-stack function will be the one which caused the hung. Feel free to post the screenshot.

[Brent_R_]

Interestingly, the problem isn't presenting itself today(after about a week of occurring). It may be a race condition without a 100% repro. When I looked in PE before while debugging I saw ~3% cpu usage and multiple threads running something like: "trylockbyte".

I will post the full stack here if I find the issue reproducing again.

[Bernard]

Here is link to TBB header file for SunOS where some function is named trylockbyte. I do not know how it is relevant to Windows OS and x86-64 architecture. Was PE able to resolve that "trylockbyte" function?

http://panthema.net/2013/parallel-string-sorting/parallel-string-sorting-0.6/minitbb/tbb/machine/sunos_sparc.h.html

[Vitaly_S_Intel]

Hi Brent!

Does VTune actually hangs or just needs more time to process? Did you try to wait e.g. 1 hour? Do you have your binaries and .pdb file on local drive or they are on mounted network drive?

[Bernard]

>>>I will post the full stack here if I find the issue reproducing again.>>>

Is it possible to post call stack as it was collected by windbg?

[Peter_W_Intel]

Vitaly Slobodskoy (Intel) wrote:

Hi Brent!

Does VTune actually hangs or just needs more time to process? Did you try to wait e.g. 1 hour? Do you have your binaries and .pdb file on local drive or they are on mounted network drive?

1. You might set "-d 60" to know if VTune really hangs, or performance issue on finalizing stage?

2. You can copy .pdb files to local driver, then do "amplxe-cl -finalize -search-dir=dir -r00??". VTune will re-resolve your raw data with pdb files. 

[Brent_R_]

@iliyapolak: Yes that looks like the call that was the current in the two threads using CPU.

@Vitaly: Yes it hangs. I let it run overnight one day when I left work and it still wasn't resolved when I came in. The pdbs and exes were all local on my SSD.

@iliyapolak(2nd response): The issue still hasn't reproduced. Though I have only taken about 3 new captures since I made this thread.

@Peter: Is -d documented? I thoroughly reviewed the help for possibly useful. I ran a very similar command-line when testing the vtune-CLI thinking it may be a GUI only problem.

As I've stated the issue hasn't reliably reproduced for me lately. If it does, I'll try to include more exhaustive information.

[Peter_W_Intel]

> @Peter: Is -d documented? I thoroughly reviewed the help for possibly useful. I ran a very similar command-line when testing the vtune-CLI thinking it may be a GUI only problem.

You can use "-d" option to control data collection, for example:

amplxe-cl -collect -search-dir=$your-module-src-dir -duration 60 --- application

[Vitaly_S_Intel]

Thanks Brent!

When you reproduce it again, can you please do the following steps which will allow us to find the root cause of hang:

1) Run the following command _from the directory you're able to write_:

<VTune install dir>\bin64\amplxe-feedback.exe -create-bug-report=hang -no-system-info

Please run it when VTune GUI hung, the command will capture stacks from the running VTune process.

2) Send me hang.zip file via private message

[Bernard]

>>>Yes that looks like the call that was the current in the two threads using CPU>>>

Do you mean that trylockbyte was on the top of the stack and was currently executed by the CPU?

[Bernard]

@Brent

You can also use ProcDump -h command to create dump of hung process.