A bug in vtss.sys

[Slava_I_]

There is a bug in vtss.sys - an attempt to close an invalid handle from the driver, the bug reveales itself only when the driver verifier is active. Mostly it is a nuisance as this bug should not have any impact on the system but the driver verifier must be disabled to use VTune 2013 as Microsoft considers this bug as a fatal error that should be fixed so the driver verifier crashes the system. The following is a crash analysis

0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

INVALID_KERNEL_HANDLE (93)
This message occurs if kernel code (server, redirector, other driver, etc.)
attempts to close a handle that is not a valid handle.
Arguments:
Arg1: 0000000000000000, The handle that NtClose was called with.
Arg2: fffff8a0000018b0,
Arg3: 0000000000000000
Arg4: 0000000000000001

Debugging Details:
------------------

DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT

BUGCHECK_STR: 0x93

PROCESS_NAME: System

CURRENT_IRQL: 2

LAST_CONTROL_TRANSFER: from fffff80003bbc3c2 to fffff80003abd620

STACK_TEXT:
fffff880`02fd2da8 fffff800`03bbc3c2 : 00000000`00000000 fffffa80`03d1e040 00000000`00000065 fffff800`03b03b10 : nt!RtlpBreakWithStatusInstruction
fffff880`02fd2db0 fffff800`03bbd1ae : 00000000`00000003 00000000`00000000 fffff800`03b006d0 00000000`00000093 : nt!KiBugCheckDebugBreak+0x12
fffff880`02fd2e10 fffff800`03ac56c4 : 00000000`0000001c fffff980`1288efe0 00000000`00000000 00000000`00000000 : nt!KeBugCheck2+0x71e
fffff880`02fd34e0 fffff800`03d2261b : 00000000`00000093 00000000`00000000 fffff8a0`000018b0 00000000`00000000 : nt!KeBugCheckEx+0x104
fffff880`02fd3520 fffff800`03ac4813 : fffff880`02fd3600 00000000`00000000 00000000`00000000 00000000`00000000 : nt! ?? ::NNGAKEGL::`string'+0x51ce4
fffff880`02fd3620 fffff800`03ac0db0 : fffff880`1fe0d3ff 00000000`00000000 fffff800`03c54880 00000000`00240024 : nt!KiSystemServiceCopyEnd+0x13
fffff880`02fd3828 fffff880`1fe0d3ff : 00000000`00000000 fffff800`03c54880 00000000`00240024 fffffa80`0509d4a0 : nt!KiServiceLinkage
fffff880`02fd3830 fffff880`1fe10502 : 00000000`00000000 fffffa80`05184db0 00000000`746c6600 fffff880`02fd3970 : vtss+0x73ff
fffff880`02fd38a0 fffff800`03eadeb7 : fffffa80`05184db0 ffffffff`80001bf0 fffff980`1288efe0 00000000`00000001 : vtss+0xa502
fffff880`02fd39a0 fffff800`03eae2b5 : 00000000`00000010 00000000`00000000 00000000`00000010 00000000`00010202 : nt!IopLoadDriver+0xa07
fffff880`02fd3c70 fffff800`03ad27e1 : fffff880`00000000 ffffffff`80001bf0 fffff800`03eae260 00000000`00000000 : nt!IopLoadUnloadDriver+0x55
fffff880`02fd3cb0 fffff800`03d656fa : ffffffff`ffffffff fffffa80`03d1e040 00000000`00000080 fffffa80`03d065a0 : nt!ExpWorkerThread+0x111
fffff880`02fd3d40 fffff800`03aa3b46 : fffff880`009e6180 fffffa80`03d1e040 fffff880`009f0f40 01e09a41`0c0a3590 : nt!PspSystemThreadStartup+0x5a
fffff880`02fd3d80 00000000`00000000 : fffff880`02fd4000 fffff880`02fce000 fffff880`02fd28b0 00000000`00000000 : nt!KiStartSystemThread+0x16

[Peter_W_Intel]

Also please run "amplxe-feedback -create-bug-report <report archive>", and post zipped log file for investigating. Thank you.

Have you tried VTune(TM) Amplifier XE 2015 Update 1? 

[Bernard]

@marcel

Do you have any updates on your issue?

[Tronster_H_]

Same issue on Windows 8, 64-bit with "Intel VTune Amplifier XE 2015" installed.  I randomly blue screen via vtss.sys

Is there an ETA on when this will be fixed?

[Peter_W_Intel]

Not sure whether the problem still persists on VTune(TM) Amplifier XE 2015 Update 2.

Are there other users who work on Windows 8, 64-bit and have sane issue, with Update 2?

[Bernard]

@Tronster

Is the BSOD code the same as in previous versions of Windows?

[steve_z_1]

@Peter

I am running Intel Vtune Amplifier XE 2015 Update 2 (license) on Windows Server 2012 r2 64-bit. I just had BSOD when I tried to collect advanced hotspots analysis with Hotspots, call counts, stack and context switches, CPU sampling interval, ms by 1. It happened twice when I tried twice. I am afraid that it would happen every time if I keep trying. It works o.k. with basic hotspots and concurrency. I may not be able to upload the dump file to this site.

[Peter_W_Intel]

@Steve

This was a known issue about vtune drivers conflict on Windows* 8, thank you of reporting this on Windows Sever 2012.

Basic hotspots should have no problem since there is no vtune drivers used.

You may check if no stack sampling, call counts used - it still has this BSOD? If you change sample interval ms to 10, what happens? I just want to reduce time cost on ISR (interrupt service routine).

[Bernard]

@Steve

What is the BSOD code?

[steve_z_1]

@Peter

I tried to collect Advanced Hotspots with "Hotspots", 10 ms sampling, it's o.k. However, it crashed when using "Hotspots, call stack and context switches" at 10 ms sampling. The attached file vtuneBSOD_Stack.PNG is the BSOD's information for this crash. The 2nd attachment vtuneBSOD_Count.PNG is BSOD's information for yesterday's crash with "Hotspots, call counts, stacks and context switches" option.

[Peter_W_Intel]

@ steve z

Thank you for this update. That was why I asked to change sample interval to bigger, and don't use stack sampling or stack sampling with call counts.

I also reported this to engineering.  

[Bernard]

First BSOD "vtuneBSOD_Stack" was probably casued by prolonged execution of VTune ISR/DPC routine.

Second BSOD was probably caused by corrupting some important kernel structure "GDT"?

[dirk_s]

Hi all,

I am using "Intel VTune Amplifier XE 2015 Update 3" (installed as part of "Intel Parallel Studio XE 2015 Update 3 Professional Edition") and I am seeing a similar issue.

It crashes sporadically when just using the computer for normal office usage ... sometimes even when not doing something with the comuter but just looking at the screen.

System Configuration:

- Windows 7 SP1 64bit with all current updates and security fixes installed
- Dell Precision T3610
- Intel Xeon E5-1650 v2
- 48 GB RAM

Thanks for your help in advance!

[Peter_W_Intel]

Thank you for data. Currently please don't use stack sampling with/without call counts, as a workaround (I hope).

[dirk_s]

@Peter: thanks a lot for your quick answer but somehow I don't understand how this could help in my case as the crash does not happen during working with VTune but when doing something else (normal office work). Did I get something wrong?

[Bernard]

>>.thanks a lot for your quick answer but somehow I don't understand how this could help in my case as the crash does not happen during working with VTune but when doing something else (normal office work).>>>

Probably vtss.sys is still loaded and still is causing indirectly PageFault. What is interesting that vtss.sys runs in the context of BrokerAgent.exe process.

I would like to advise you simply kiling BrokerAgent process and observer what will happen.

[Bernard]

It seems that BrokerAgent.exe is Citrix program so in your case simply vtss.sys triggered BSOD while running in context of BrokerAgent.exe unless there was some interaction between vtss.sys and aferomentioned program. In such a case on the call stack of BrokerAgent.exe sholud exist and leftovers from calling vtss.sys.

[Sergey_F_]

I encounter SYSTEM_SERVICE_EXCEPTION in VTSS.SYS every time i print lots of information to console from CMD.EXE. Like, TYPE SOMEBIG.EXE. VTSS.SYS does not provide any version information, but i upgraded to the latest vtune. The only thing i found is date when it was signed: Monday, ‎May ‎18, ‎2015 2:13:21 PM

[dkokron]

Vtune 2016u1 becomes unresponsive on my fedora 21 system when doing hardware analysis (Mem Aanalysis and GenExp).

Dual socket E5-2643 running
Linux XXXXXXX 4.1.13-100.fc21.x86_64 #1 SMP Tue Nov 10 13:13:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[  270.916048] sep3_15: PMU collection driver v3.15.5 (EMON) has been loaded.
[  270.916052] sep3_15: Chipset support is enabled.
[  270.916053] sep3_15: IDT vector 0x21 will be used for handling PMU interrupts.
[  273.089979] vtss++ kernel module ("v1.8.10-434111 Intel(R) VTune(TM) Amplifier") registered
[ 1099.531531] ------------[ cut here ]------------
[ 1099.531542] WARNING: CPU: 7 PID: 6404 at kernel/smp.c:292 smp_call_function_single+0x133/0x150()
[ 1099.531544] Modules linked in: vtsspp(OE) sep3_15(OE) fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack pax(OE) ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw nvidia(POE) intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_ca0132 snd_hda_intel kvm snd_hda_controller snd_hda_codec iTCO_wdt crct10dif_pclmul crc32_pclmul snd_hda_core iTCO_vendor_support ppdev
[ 1099.531574]  crc32c_intel snd_hwdep snd_seq snd_seq_device snd_pcm parport_pc parport dcdbas drm i8k snd_timer snd sb_edac edac_core mei_me tpm_tis tpm lpc_ich mei i2c_i801 soundcore shpchp mfd_core nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc e1000e serio_raw megaraid_sas ptp pps_core [last unloaded: vtsspp]
[ 1099.531594] CPU: 7 PID: 6404 Comm: vtsspp_nmiwd_0 Tainted: P           OE   4.1.13-100.fc21.x86_64 #1
[ 1099.531595] Hardware name: Dell Inc. Precision T5600/0Y56T3, BIOS A06 10/25/2012
[ 1099.531597]  0000000000000000 00000000dcb1f38b ffff880fc931fad8 ffffffff81793645
[ 1099.531599]  0000000000000000 0000000000000000 ffff880fc931fb18 ffffffff810a165a
[ 1099.531600]  ffff880fc931fb28 0000000000000000 ffffffff81197c70 ffff88082fa1bca0
[ 1099.531602] Call Trace:
[ 1099.531609]  [<ffffffff81793645>] dump_stack+0x45/0x57
[ 1099.531613]  [<ffffffff810a165a>] warn_slowpath_common+0x8a/0xc0
[ 1099.531617]  [<ffffffff81197c70>] ? cpu_clock_event_start+0x40/0x40
[ 1099.531619]  [<ffffffff810a178a>] warn_slowpath_null+0x1a/0x20
[ 1099.531620]  [<ffffffff81120b13>] smp_call_function_single+0x133/0x150
[ 1099.531622]  [<ffffffff81196b43>] cpu_function_call+0x43/0x60
[ 1099.531624]  [<ffffffff8119b0c0>] ? group_sched_out+0xc0/0xc0
[ 1099.531626]  [<ffffffff8119a492>] _perf_event_disable+0xc2/0xd0
[ 1099.531628]  [<ffffffff8119a4ba>] perf_event_disable+0x1a/0x30
[ 1099.531631]  [<ffffffff811584e6>] watchdog_nmi_disable+0x36/0x70
[ 1099.531633]  [<ffffffff81158a40>] watchdog_enable_all_cpus+0x70/0xe0
[ 1099.531635]  [<ffffffff81158b05>] proc_watchdog_update+0x55/0x60
[ 1099.531637]  [<ffffffff81158bad>] proc_watchdog_common+0x9d/0x100
[ 1099.531639]  [<ffffffff81158e01>] proc_nmi_watchdog+0x21/0x30
[ 1099.531643]  [<ffffffff812a1003>] proc_sys_call_handler+0xd3/0xf0
[ 1099.531644]  [<ffffffff812a1034>] proc_sys_write+0x14/0x20
[ 1099.531647]  [<ffffffff812281a7>] __vfs_write+0x37/0x110
[ 1099.531650]  [<ffffffff8122b078>] ? __sb_start_write+0x58/0x120
[ 1099.531653]  [<ffffffff813325e3>] ? security_file_permission+0x23/0xa0
[ 1099.531655]  [<ffffffff81228919>] vfs_write+0xa9/0x1b0
[ 1099.531656]  [<ffffffff8122884e>] ? vfs_read+0x11e/0x140
[ 1099.531663]  [<ffffffffa0f72920>] vtss_kernel_write.constprop.1+0x50/0x80 [vtsspp]
[ 1099.531666]  [<ffffffffa0f72b88>] vtss_nmi_watchdog_disable_thread+0x118/0x220 [vtsspp]
[ 1099.531669]  [<ffffffffa0f72a70>] ? vtss_nmi_watchdog_enable_thread+0x120/0x120 [vtsspp]
[ 1099.531672]  [<ffffffff810c06d8>] kthread+0xd8/0xf0
[ 1099.531673]  [<ffffffff810c0600>] ? kthread_create_on_node+0x1b0/0x1b0
[ 1099.531676]  [<ffffffff8179a322>] ret_from_fork+0x42/0x70
[ 1099.531677]  [<ffffffff810c0600>] ? kthread_create_on_node+0x1b0/0x1b0
[ 1099.531678] ---[ end trace 0dcfb12189433c32 ]---
[ 1099.531689] ------------[ cut here ]------------

Edit;

This issue went away after upgrading my system to Fedora 22 and doing a fresh install of Parallel Studio.

[Bernard]

Sergey F. wrote:

I encounter SYSTEM_SERVICE_EXCEPTION in VTSS.SYS every time i print lots of information to console from CMD.EXE. Like, TYPE SOMEBIG.EXE. VTSS.SYS does not provide any version information, but i upgraded to the latest vtune. The only thing i found is date when it was signed: Monday, ‎May ‎18, ‎2015 2:13:21 PM

That can explain some weird interaction between vtss.sys and display driver and/or Dxgkernel.sys

Do not you have any dump file to upload?

[Bernard]

Hi @dkokron

To me it looks like vtss module was trying to disable NMI watchdog handler.

[Peter_W_Intel]

It seemed that this bug has been identified and solved in future releases, good news.

I would like to invite all of you, who ever met this problem (Sorry for that) to test new VTune driver (I attached).

Exact steps to verify:
1. Install VTune(TM) Amplifier AXE 2016 Update 2, first.
2. Go to <VTune_install_dir>\bin64: 1. amplxe-sepreg.exe -u 2. Copy the new vtss.sys to <VTune_install_dir>\bin64\sepdrv
3. amplxe-sepreg.exe -i

Please let me know if it works or not after doing above. Thanks for your cooperation.

Regards, Peter