Analyzers
Talk to fellow users of Intel Analyzer tools (Intel VTune™ Profiler, Intel Advisor)

VTune driver preventing xperf sampling?

Tristan_J_1
Beginner
1,267 Views

Hello,

I use multiple tools for performance analysis (e.g. Windows Performance Toolkit(xperf), ANTS).  I recently started evaluating VTune as well.  I was particularly interested in using it to analyze memory bandwidth usage.

Unfortunately, after installing Intel VTune Amplifier XE 2013 Update 2 and running some tests with it I found that I could no longer go back to using xperf from the Windows Performance Toolkit to gather CPU sampling data.  I typically would run xperf with this command line...

xperf -on PROC_THREAD+LOADER+INTERRUPT+DPC+PROFILE -stackwalk profile -minbuffers 16 -maxbuffers 1024 -flushtimer 0 -f c:\perf\profile.temp.etl

And then I would dump the profiling session with this command line...

xperf -d c:\perf\profile.final.etl

That used to generate a file that contained CPU samples that I could use for analysis...  but now I no longer have that sampling data.  As an experiment, I tried this on another machine and verified that this worked before the installation of VTune, and stopped working immediately afterward.  I noticed that one component of the VTune suite is an "Event-based sampling driver".

I was wondering if this could be disrupting the ability of xperf to do CPU sampling?  Unfortunately, even uninstalling VTune doesn't seem to fix this... I had to have one of our servers completely re-imaged in order to get CPU sampling functional again.

I'd be very interested in any workaround for this as  I like to have the flexiblity to use multiple tools.

Any help much appreciated!

0 Kudos
15 Replies
David_A_Intel1
Employee
1,267 Views

So, I checked with our developers and they have tested compatibility with WPA (Windows* Performance Analyzer): there were no issues.  We are using ETW just like they are and neither should cause problems with the ETW subsystem.

At this point, I would ask you to submit an issue at Intel® Premier Support and include the zip file created by executing the command 'amplxe-feedback -create-bug-report report.zip' from a command prompt window opened via the Intel Parallel Studio XE Start Menu entry and while in a writeable directory.  This command will collect system information that will help us troubleshoot your problem.  Please execute it on a system that 1) has VTune Amplifier XE installed and 2) is demonstrating the problem you are reporting.  We may need to ask for follow-up info, but this is always a good way to start when submitting an issue.

0 Kudos
Bernard
Valued Contributor I
1,267 Views

@Tristan

Before running Xperf please disable VTune.Only one profilling program can access Kernel Logger.

0 Kudos
Tristan_J_1
Beginner
1,267 Views

@iliyapolak

Thanks for the reply. What do you mean by disabling VTune?  I've tried closing and uninstalling VTune and even rebooting the system afterward but I still see the same problem.

0 Kudos
Bernard
Valued Contributor I
1,267 Views

Tristan J. wrote:

@iliyapolak

Thanks for the reply. What do you mean by disabling VTune?  I've tried closing and uninstalling VTune and even rebooting the system afterward but I still see the same problem.

Simply do not run VTune and any other software which calls Kernel Logger before you start capture session with Xperf.Do you see this message "xperf: error: NT Kernel Logger: Cannot create a file when that file already exists. (0xb7)".

0 Kudos
Tristan_J_1
Beginner
1,267 Views

I'm not running VTune (it has been uninstalled) or any other software that calls the NT Kernel logger.  I don't receive the error message you mention.  I have even rebooted after uninstalling to confirm it isn't still running.

Yet when I open up the generated ETL file, no CPU sampling data is present.

0 Kudos
Bernard
Valued Contributor I
1,267 Views

Sorry Tristan I must admit that I did not read all your post and I answered automatically.Maybe mentioned by you VTune driver is still present on your system other possibility is the data corruption which could have been done by the same driver.Can you enable boot logging and issue driverquery command you should also look in registry for any occurrence of rhe VTune driver

0 Kudos
Tristan_J_
Beginner
1,267 Views

Thanks for the suggestion.

I enabled bootlogging and have attached the results that were output.

0 Kudos
Bernard
Valued Contributor I
1,267 Views

>>>I enabled bootlogging and have attached the results that were output.>>>

Can you investigate this driver "BOOTLOG_LOADED \SystemRoot\System32\drivers\msrpc.sys".Because it containes the string "msr" it can be related to some VTune installation leftovers.I think about enabling so called NT Kernel Logging maybe this will help us to find the culprit.Please use this command in reged "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\WMI\GlobalLogger" set start to 1 and look for etl file it should be called "NT Kernel Logger.etl ".Other options which came recently to my mind and can be used to troubleshoot Xperf are to use Sysinternals tools like regmon to intercept registry calls you must to do it when you start Xperf session  you can also use process monitor to search throught the any VTune leftovers.Please issue also driverquery command there is possibility that some Vtune driver is loaded later and not during a bootup.The more advanced option is to use windbg as a kernel debugger and put a breakpoint on the MSR registers accesses and search through the code wich writes/reads those registers.

0 Kudos
Tristan_J_1
Beginner
1,267 Views

I filed a support ticket with Intel and wound up getting a workaround for this issue.

From C:\Program Files (x86)\Intel\VTune Amplifier XE 2013\bin32 (or wherever you installed VTune to), run this command:

amplxe-sepreg.exe -u pax

Then reboot your machine.  You will then be able to get xperf sampling data again.  However, you won't be able to use VTune until installing it again.

Unfortunately, the core problem of not being able to use both VTune and xperf still exists.  The workaround above makes it possible to recover from this, but ideally, installing VTune wouldn't prevent xperf sample collection to begin with.

0 Kudos
Bernard
Valued Contributor I
1,267 Views

@Tristan

Thanks for providing an update on your case.On my machine I can use VTune and Xperf , but not at the same time because of Kernel Logger dependency.

0 Kudos
Tristan_J_
Beginner
1,267 Views

@iliyapolak

Perhaps this issue is OS-related.  I'm experiencing the problem on a Windows Server 2012 machine.  However, I just tried this on my Windows 7 desktop and didn't run into the issue.

And to be clear, I'm not trying to use xperf and VTune at the same time.  I'm just trying to alternate between the tools.  The problem I'm experiencing on Windows Server 2012 is that even when VTune isn't running, xperf can't be used to gather CPU sampling data.  Only after the Vtune drivers are uninstalled does this work again.

0 Kudos
Bernard
Valued Contributor I
1,267 Views

>>>And to be clear, I'm not trying to use xperf and VTune at the same time.>>>

Yes I know that.I agree that the issue decribed by you is very strange and it is probably related to low-level driver operation and/or the usage of  new OS.

Afaik performance counters are accessed through the registry functions.The counters data itself is not stored in registry.I suppose that VTune driver simply hooks those functions and thus prevents from Xperf to function properly.Few posts earlier I advised you to use regmon in order to trace VTune registry accesses and it was crucial to understand the root cause of the problem.

0 Kudos
Tristan_J_
Beginner
1,267 Views

Thanks iliyapolak,  I tried using procmon (which includes regmon functionality) to do this, but I was getting massive amounts of data collected.  Do you have any idea what sort of things I should filter for?  I tried filtering for just xperf registry calls but even then there were large numbers of them.

I will play around with this more and get back to you.

0 Kudos
Bernard
Valued Contributor I
1,267 Views

Hi Tristan!

Sorry for beign late with the answer.You need to filter only xperf calls and you will get a large number of the intercepted calls.You need to filter mostly this key HKEY_PERFORMANCE_DATA and values queries.Look for this registry call "RegQueryValueEx"  and look for "SUCCESS" return value.

0 Kudos
Bernard
Valued Contributor I
1,267 Views

Hi Tristan,

do you have any updates?

0 Kudos
Reply