BSOD with Advanced Analysis

Marián__VooDooMan__M · ‎05-24-2015

I have 100% reproducible BugCheck with the latest VTune update to date. Though stack trace is not very meaningful.

1: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 0000000000000018, Clock interrupt time out interval in nominal clock ticks.
Arg2: 0000000000000000, 0.
Arg3: ffffd0017f5a7180, The PRCB address of the hung processor.
Arg4: 0000000000000002, 0.

Debugging Details:
------------------


BUGCHECK_STR:  CLOCK_WATCHDOG_TIMEOUT_8_PROC

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

PROCESS_NAME:  Recorder7_Proc

CURRENT_IRQL:  d

ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) amd64fre

STACK_TEXT:  
ffffd001`7f566c88 fffff803`e97f4ecf : 00000000`00000101 00000000`00000018 00000000`00000000 ffffd001`7f5a7180 : nt!KeBugCheckEx
ffffd001`7f566c90 fffff803`e96c5f67 : 00000000`00000000 00000000`00000000 00000000`00000001 00000000`00000000 : nt! ?? ::FNODOBFM::`string'+0xad0f
ffffd001`7f566d20 fffff803`e962167f : 00000000`00000002 00000000`00000000 fffff803`e966deb0 00000000`00000002 : nt!KeClockInterruptNotify+0x787
ffffd001`7f566f40 fffff803`e9760143 : fffff803`e966de00 fffff803`e9788057 ffffd001`00000000 00000000`00000000 : hal!HalpTimerClockInterrupt+0x4f
ffffd001`7f566f70 fffff803`e97db12a : fffff803`e966de00 00000000`00000000 00000000`00000000 00000000`00000002 : nt!KiCallInterruptServiceRoutine+0xa3
ffffd001`7f566fb0 fffff803`e97db50f : 00000000`00000000 00000000`67198f47 00000000`00000000 00000000`00000000 : nt!KiInterruptSubDispatchNoLockNoEtw+0xea
ffffd000`27ae0460 fffff803`e96e98a8 : 00000000`00000046 fffff800`7a06b8ed 00000000`80000005 ffffe001`051b7080 : nt!KiInterruptDispatchLBControl+0x11f
ffffd000`27ae05f0 fffff803`e96ec6d6 : 00000000`00000000 00000000`00000002 00000000`001f4604 00000000`00000001 : nt!MiFlushTbList+0x2d8
ffffd000`27ae0730 fffff803`e96c9b2a : 00000000`00000006 ffffd000`27ae0909 00000000`00000006 00000000`00000006 : nt!MiFlushTbAsNeeded+0xe6
ffffd000`27ae0870 fffff803`e96cb1df : fffff901`00000000 fffff960`0027973c 00000000`00000002 00000000`00000000 : nt!MiAllocatePagedPoolPages+0x17a
ffffd000`27ae0970 fffff803`e96ca590 : 00000000`00000000 00000000`00000021 00000000`000054a0 000000a1`8fb3ecc0 : nt!MiAllocatePoolPages+0xc3
ffffd000`27ae09c0 fffff803`e992d973 : 00000000`00000001 00007ff6`9ee8e9c0 00007ffe`00000021 fffff803`e973a226 : nt!ExpAllocateBigPool+0xd0
ffffd000`27ae0ab0 fffff960`0048bcd7 : ffffd000`27ae0c21 00000000`00000001 00000000`00000000 fffff960`0029ba0f : nt!ExAllocatePoolWithTag+0xa83
ffffd000`27ae0b80 fffff960`0048c436 : fffff901`45ba8000 00000000`00000000 00000000`00000001 00000000`00000000 : win32k!NSInstrumentation::CLeakTrackingAllocator::AllocateCommon<<lambda_16c8fa6c1365df5cd1a45d38bb6ccfe2>,<lambda_278f24766fae20cbf0c48ddfc336e8e8> >+0x2f
ffffd000`27ae0c50 fffff960`003d2f51 : ffffe001`00000001 00000000`00000021 00000000`00000000 ffffe001`6c777355 : win32k!NSInstrumentation::CLeakTrackingAllocator::Allocate+0x46
ffffd000`27ae0ca0 fffff960`001733e4 : fffff901`45ba8000 00000000`00000000 fffff901`45ba8000 00000000`00000000 : win32k!InternalRebuildHwndListForIMEClass+0x31
ffffd000`27ae0d20 fffff960`00295293 : 00000000`00000001 ffffd000`27ae0ec0 00000000`00000000 00000000`00000006 : win32k!BuildHwndList+0xb4
ffffd000`27ae0d50 fffff803`e97e54b3 : ffffe001`051b7080 00000000`00000000 000000a1`91467b48 00000083`00000000 : win32k!NtUserBuildHwndList+0x183
ffffd000`27ae0dd0 00007ffe`ac1dbcca : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
000000a1`91467b28 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffe`ac1dbcca


STACK_COMMAND:  kb

SYMBOL_NAME:  ANALYSIS_INCONCLUSIVE

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: Unknown_Module

IMAGE_NAME:  Unknown_Image

DEBUG_FLR_IMAGE_TIMESTAMP:  0

IMAGE_VERSION:  

FAILURE_BUCKET_ID:  CLOCK_WATCHDOG_TIMEOUT_8_PROC_ANALYSIS_INCONCLUSIVE

BUCKET_ID:  CLOCK_WATCHDOG_TIMEOUT_8_PROC_ANALYSIS_INCONCLUSIVE

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:clock_watchdog_timeout_8_proc_analysis_inconclusive

FAILURE_ID_HASH:  {f60c102e-9b2c-017b-9309-e4af1999e360}

Followup: MachineOwner
---------

best, vdm.

Peter_W_Intel · ‎05-25-2015

I reported your data to development team. Thank you.

Bernard · ‎05-27-2015

Yes I agree that call stack is not very informative in your case.

I would advise to begin investigating by issueing following command !prcb ffffd0017f5a7180. You should search for TIB exception and definitaly have a look at currently running thread on CPU which was hung or inresponsive before BSOD.

Marián__VooDooMan__M · ‎05-27-2015

iliyapolak wrote:

Yes I agree that call stack is not very informative in your case.

I would advise to begin investigating by issueing following command !prcb ffffd0017f5a7180. You should search for TIB exception and definitaly have a look at currently running thread on CPU which was hung or inresponsive before BSOD.

Too bad I have accidentally deleted MEMORY.DMP, so I will try later to get BSOD again, and save informations, and give you more data.

Marián__VooDooMan__M · ‎05-27-2015

I have reproduced BugCheck, but now I have in stack trace that kernel driver for my HW (USB device for home studio recording) was there. I will try to upgrade driver.

It seems that VTune and this HW don't like each other.

Marián__VooDooMan__M · ‎05-27-2015

I upgraded driver for audiosystem, but BugCheck is still reproducible.

It seems to me that there is some deadlock in VTune kernel-mode driver.

7: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 0000000000000018, Clock interrupt time out interval in nominal clock ticks.
Arg2: 0000000000000000, 0.
Arg3: ffffd000cf6dc180, The PRCB address of the hung processor.
Arg4: 0000000000000006, 0.

Debugging Details:
------------------


BUGCHECK_STR:  CLOCK_WATCHDOG_TIMEOUT_8_PROC

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

PROCESS_NAME:  System

CURRENT_IRQL:  d

ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) amd64fre

STACK_TEXT:  
ffffd000`cf794c88 fffff801`51d6decf : 00000000`00000101 00000000`00000018 00000000`00000000 ffffd000`cf6dc180 : nt!KeBugCheckEx
ffffd000`cf794c90 fffff801`51c3ef67 : 00000000`00000000 00000000`00000000 00000000`00000001 00000000`00000000 : nt! ?? ::FNODOBFM::`string'+0xad0f
ffffd000`cf794d20 fffff801`5239e67f : 00000000`00000000 00000000`00000000 ffffe000`ff201cb0 00000000`00000000 : nt!KeClockInterruptNotify+0x787
ffffd000`cf794f40 fffff801`51cd9143 : ffffe000`ff201c00 fffff801`51d01057 ffffd000`00000000 00000000`00000000 : hal!HalPerformEndOfInterrupt+0x6f
ffffd000`cf794f70 fffff801`51d5412a : ffffe000`ff201c00 00000000`00000000 00000000`00000000 00000000`00000001 : nt!KiCallInterruptServiceRoutine+0xa3
ffffd000`cf794fb0 fffff801`51d5450f : 00000000`00000000 00000000`6545cb88 00000000`00000000 ffffe001`051ffffa : nt!KiInterruptSubDispatchNoLockNoEtw+0xea
ffffd000`26d0d460 fffff801`51c628b0 : fffff801`b6caae40 fffff801`b6cabc80 00000000`80000005 00000000`00000000 : nt!KiInterruptDispatchLBControl+0x11f
ffffd000`26d0d5f0 fffff801`51ccaa83 : e9c00003`71edf921 00000000`00000001 ffffb001`b2b01000 00000000`00000000 : nt!MiFlushTbList+0x2e0
ffffd000`26d0d730 fffff801`51c6ad95 : 00000000`00000000 00000000`00000000 ffffd000`26d0d911 00000000`00001000 : nt!MmSetAddressRangeModified+0xc7
ffffd000`26d0d860 fffff801`51c6b57b : ffffe001`04b6f4c8 00000000`00000000 ffffe001`00000001 00000000`00000000 : nt!CcFlushCachePriv+0x3cd
ffffd000`26d0d970 fffff801`51c8c465 : ffffe001`00737760 fffff801`51c02000 ffffd000`00000000 ffffe001`00168040 : nt!CcWriteBehindInternal+0x17b
ffffd000`26d0da00 fffff801`51c8c81d : 00000000`00000000 ffffe000`fefc3bb0 ffffe001`02bcfdd0 00000000`00000000 : nt!CcWriteBehind+0x95
ffffd000`26d0daa0 fffff801`51caf6bc : fffff801`b38f3500 ffffe001`00168100 fffff801`51eeaa20 00000000`00000000 : nt!CcWorkerThread+0x22d
ffffd000`26d0db50 fffff801`51d0236c : ffffe000`ff0dd880 ffffe001`00168040 00000000`00000080 ffffe001`00168040 : nt!ExpWorkerThread+0x28c
ffffd000`26d0dc00 fffff801`51d592c6 : ffffd000`cf6dc180 ffffe001`00168040 ffffe000`ff0dd880 00000000`00000000 : nt!PspSystemThreadStartup+0x58
ffffd000`26d0dc60 00000000`00000000 : ffffd000`26d0e000 ffffd000`26d08000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16


STACK_COMMAND:  kb

SYMBOL_NAME:  ANALYSIS_INCONCLUSIVE

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: Unknown_Module

IMAGE_NAME:  Unknown_Image

DEBUG_FLR_IMAGE_TIMESTAMP:  0

IMAGE_VERSION:  

FAILURE_BUCKET_ID:  CLOCK_WATCHDOG_TIMEOUT_8_PROC_ANALYSIS_INCONCLUSIVE

BUCKET_ID:  CLOCK_WATCHDOG_TIMEOUT_8_PROC_ANALYSIS_INCONCLUSIVE

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:clock_watchdog_timeout_8_proc_analysis_inconclusive

FAILURE_ID_HASH:  {f60c102e-9b2c-017b-9309-e4af1999e360}

Followup: MachineOwner
---------

7: kd> !prcb ffffd000cf6dc180
Cannot get PRCB address
7: kd> !prcb 0xffffd000cf6dc180
Cannot get PRCB address

Bernard · ‎06-03-2015

>>>It seems to me that there is some deadlock in VTune kernel-mode driver.>>>

Probably yes I think so, but without seeing prcb record is hard to tell what has happend.

Bernard · ‎06-03-2015

Regarding that error where prcb cannot be read. Did you try to load full memory dump or at least kernel memory dump and then run !prcb command?

I wonder why this command failed.

Can you try and manually inspect prcb address?

Bernard · ‎06-03-2015

You can read this excellent blog post related directly to debugging BSOD 0x101

http://blogs.msdn.com/b/ntdebugging/archive/2011/10/26/debugging-a-clock-watchdog-timeout-bugcheck.aspx

Bernard · ‎06-03-2015

>>>I wonder why this command failed.

Can you try and manually inspect prcb address?>>>

@Marian

Please disregard my quoted post and run prcb command on each CPU core untill you find matching address of hung CPU prcb.

For example:

!prcb 0

!prcb 1

.....//.....//....

!prcb 7

Bernard · ‎06-30-2015

@Marian

Any updates on your issues?

Marián__VooDooMan__M · ‎07-01-2015

I sent bug check dump to Microsoft for analysis (via "problems and solutions").

Meanwhile, the Microsoft's updates were installed, and I am not able to reproduce the bug check any more, so I guess problem was in Windows kernel.

I will update this topic in case another bug check I would encounter.