Solved: Re: Major problems w/ VTune Profiler (2024.0) on Windows 11 x64

aremmell · ‎11-29-2023

I have a fresh install of Windows 11 Pro x64 with all updates installed, and VTune 2024.0, and am experiencing the following issues:

When I attempt to profile using the 'Threading' model using attach to process, one of two things happens:
1. In user mode, profiling stops almost immediately after starting, and I see "Error 0x40000024 (No data)". My application is compiled with Microsoft CL in release mode with /Zi and /O2 per the documentation's suggestions, and all relevant binary/symbol/source directories are appropriately set.
2. In hardware mode, the profiling appears to but successful, but VTune gets stuck on 'finalizing results' at exactly 25% every time. I have tried canceling the finalization and re-loading the raw profiling data, but that process also gets stuck and never completes.
Running 'vtune-self-checker.bat' gets past the first step of "HW-based event analysis (counting mode) (Intel driver)", but then proceeds to hang on the next step of "Instrumentation based analysis check". I have observed that 'matrix.exe' is hung, and terminating that process frees up the hang and ends the self check.
Running 'amplxe-sepreg -i -v' fails with 'Unsupported windows version' error code 50. Windows Defender active scanning is not running.

I just noticed that I have a relevant entry in Event Viewer: an application crash of 'C:\Program Files (x86)\Intel\oneAPI\vtune\2024.0\bin64\pin.exe': exception code 0xc0000005 (access violation, iirc). The time stamp seems to coincide with the stalling of the finalization process. In fact, now that I looked, there are 16 crashes of pin.exe, so I am operating on the assumption that that is where all my problems lie.

It seems like something is fundamentally very wrong with my installation, and I am running out of ideas. I have Googled till my fingertips were raw, and I am getting nowhere.

Any help would be greatly appreciated. Thanks in advance.

aremmell · ‎11-29-2023

I figured out the problem. The TLDR is that VTune is entirely incompatible with CrowdStrike Falcon Sensor.

The DLL in the WinDbg output, 'CsXumd64_17605.dll' is digitally signed by CrowdStrike, Inc. so it's no mystery where it came from. I suspect it is a global hook, and simply does not work when injected into binaries doing low-level operations such as VTune's.

So there it is. Now it's online for posterity: you cannot use VTune on a system that has CrowdStrike Falcon installed.

View solution in original post

aremmell · ‎11-29-2023

As a follow-up, I set WinDbg to postmortem debugging, then ran `pin.exe -- some.exe` and let it crash. Here is the output from WinDbg analysis:

0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 374

    Key  : Analysis.Elapsed.mSec
    Value: 8388

    Key  : Analysis.IO.Other.Mb
    Value: 7

    Key  : Analysis.IO.Read.Mb
    Value: 0

    Key  : Analysis.IO.Write.Mb
    Value: 34

    Key  : Analysis.Init.CPU.mSec
    Value: 61

    Key  : Analysis.Init.Elapsed.mSec
    Value: 143173

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 77

    Key  : Failure.Bucket
    Value: APPLICATION_HANG_80000007_pinipc.dll!Unknown

    Key  : Failure.Hash
    Value: {067aafd9-c91a-8d2d-76ed-da65e99781db}

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 9580

    Key  : Timeline.Process.Start.DeltaSec
    Value: 145

    Key  : WER.OS.Branch
    Value: ni_release

    Key  : WER.OS.Version
    Value: 10.0.22621.1


NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 0000000000000000
   ExceptionCode: 80000007 (Wake debugger)
  ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD:  000047c0

PROCESS_NAME:  pin.exe

ERROR_CODE: (NTSTATUS) 0x80000007 - {Kernel Debugger Awakened}  the system debugger was awakened by an interrupt.

EXCEPTION_CODE_STR:  80000007

DERIVED_WAIT_CHAIN:  

Dl Eid Cid     WaitType
-- --- ------- --------------------------
   0   1540.47c0 Unknown                

WAIT_CHAIN_COMMAND:  ~0s;k;;

BLOCKING_THREAD:  00000000000047c0

STACK_TEXT:  
000000a2`269cf298 00007ff8`5176e0d0     : 00000000`0000027f 00000000`00000000 00000000`00000000 00000000`00780076 : ntdll!NtWaitForMultipleObjects+0x14
000000a2`269cf2a0 00007ff8`51771c8a     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pinipc!LEVEL_BASE::IPC_CLIENT::IPC_CLIENT+0x120
000000a2`269cf2e0 00007ff8`5177196d     : 000000a2`269cf580 00000000`00000000 00000000`000047c0 000001ed`b74201c0 : pinipc!LEVEL_BASE::IPC_CONNECTION::SendMessageA+0x44a
000000a2`269cf3f0 00007ff8`5176f162     : 00000000`00000016 00000000`000022c4 000000a2`269cf6f0 00000000`00000030 : pinipc!LEVEL_BASE::IPC_CONNECTION::SendMessageA+0x12d
000000a2`269cf4d0 00007ff8`5176f9ca     : 000001ed`b7230700 00007fff`00000001 00000000`00000000 000001ed`b59d0130 : pinipc!LEVEL_BASE::IPC_CLIENT_CONNECTION::RemoteProcedureCall+0x82
000000a2`269cf620 00007ff6`2201204b     : 00000000`00000001 00000000`00000000 000001ed`b7220c40 00007fff`b4d43660 : pinipc!LEVEL_BASE::IPC_CLIENT::Init+0x12a
000000a2`269cf860 00007ff6`22011f9a     : 00000000`00000190 00000000`ffffffff 000001ed`b7240700 00007fff`b4d43660 : pin!Ordinal0+0x204b
000000a2`269cf970 00007ff6`22011a06     : 000000a2`269cfa98 000000a2`269cfa30 00007fff`b4bc1bf0 00007ff6`2205e812 : pin!Ordinal0+0x1f9a
000000a2`269cf9e0 00007ff6`2201180d     : 00007ff6`22081a00 00007ff6`22035278 00000000`00000000 00000000`00000000 : pin!Ordinal0+0x1a06
000000a2`269cfa70 00007ff6`2205e9cd     : 00000000`00007f80 00007fff`b4ca59ed 000000a2`269cfbfc 00000000`00000000 : pin!Ordinal0+0x180d
000000a2`269cfb20 00007ff6`2205f0b3     : 00000000`00000001 000000a2`269cfc96 00000000`00000000 00007ff6`2205f510 : pin!PinCommitHashSizeC+0x2e69d
000000a2`269cfc20 00007ff6`2205fb5c     : 000001ed`b7410054 61737365`6d006465 00000000`00006567 6c6c6163`7379731c : pin!PinCommitHashSizeC+0x2ed83
000000a2`269cfcd0 00007fff`b4cafc4b     : 00000000`00000000 000001ed`b73d0000 000000a2`269cfe60 000001ed`b7330000 : pin!PinCommitHashSizeC+0x2f82c
000000a2`269cfdc0 00007ff6`2206cd85     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : PINCRT!_my_libc_init+0x15b
000000a2`269cfe40 00007ff8`6244257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x3ca55
000000a2`269cfeb0 00007ff8`6352aa58     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1d
000000a2`269cfee0 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28


STACK_COMMAND:  ~0s ; .cxr ; kb

SYMBOL_NAME:  pinipc+120

MODULE_NAME: pinipc

IMAGE_NAME:  pinipc.dll

FAILURE_BUCKET_ID:  APPLICATION_HANG_80000007_pinipc.dll!Unknown

OS_VERSION:  10.0.22621.1

BUILDLAB_STR:  ni_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {067aafd9-c91a-8d2d-76ed-da65e99781db}

Followup:     MachineOwner
---------

c0000005 Exception in ext.analyze debugger extension.
      PC: 00007fff`acc75668  VA: 00000000`00000000  R/W: 0  Parameter: 00000000`00000000

and call stacks for all threads:

0:006> ~* kb

#  0  Id: 1540.47c0 Suspend: 1 Teb: 000000a2`26b0c000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ff8`5176e0d0     : 00000000`0000027f 00000000`00000000 00000000`00000000 00000000`00780076 : ntdll!NtWaitForMultipleObjects+0x14
01 00007ff8`51771c8a     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pinipc!LEVEL_BASE::IPC_CLIENT::IPC_CLIENT+0x120
02 00007ff8`5177196d     : 000000a2`269cf580 00000000`00000000 00000000`000047c0 000001ed`b74201c0 : pinipc!LEVEL_BASE::IPC_CONNECTION::SendMessageA+0x44a
03 00007ff8`5176f162     : 00000000`00000016 00000000`000022c4 000000a2`269cf6f0 00000000`00000030 : pinipc!LEVEL_BASE::IPC_CONNECTION::SendMessageA+0x12d
04 00007ff8`5176f9ca     : 000001ed`b7230700 00007fff`00000001 00000000`00000000 000001ed`b59d0130 : pinipc!LEVEL_BASE::IPC_CLIENT_CONNECTION::RemoteProcedureCall+0x82
05 00007ff6`2201204b     : 00000000`00000001 00000000`00000000 000001ed`b7220c40 00007fff`b4d43660 : pinipc!LEVEL_BASE::IPC_CLIENT::Init+0x12a
06 00007ff6`22011f9a     : 00000000`00000190 00000000`ffffffff 000001ed`b7240700 00007fff`b4d43660 : pin!Ordinal0+0x204b
07 00007ff6`22011a06     : 000000a2`269cfa98 000000a2`269cfa30 00007fff`b4bc1bf0 00007ff6`2205e812 : pin!Ordinal0+0x1f9a
08 00007ff6`2201180d     : 00007ff6`22081a00 00007ff6`22035278 00000000`00000000 00000000`00000000 : pin!Ordinal0+0x1a06
09 00007ff6`2205e9cd     : 00000000`00007f80 00007fff`b4ca59ed 000000a2`269cfbfc 00000000`00000000 : pin!Ordinal0+0x180d
0a 00007ff6`2205f0b3     : 00000000`00000001 000000a2`269cfc96 00000000`00000000 00007ff6`2205f510 : pin!PinCommitHashSizeC+0x2e69d
0b 00007ff6`2205fb5c     : 000001ed`b7410054 61737365`6d006465 00000000`00006567 6c6c6163`7379731c : pin!PinCommitHashSizeC+0x2ed83
0c 00007fff`b4cafc4b     : 00000000`00000000 000001ed`b73d0000 000000a2`269cfe60 000001ed`b7330000 : pin!PinCommitHashSizeC+0x2f82c
0d 00007ff6`2206cd85     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : PINCRT!_my_libc_init+0x15b
0e 00007ff8`6244257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x3ca55
0f 00007ff8`6352aa58     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1d
10 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28

   1  Id: 1540.49e0 Suspend: 1 Teb: 000000a2`26b0e000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ff8`6350537e     : 000001ed`b57c79b0 000001ed`b57c79b0 00000000`00000001 000001ed`b57ca810 : ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00007ff8`6244257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x2ee
02 00007ff8`6352aa58     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1d
03 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28

   2  Id: 1540.2554 Suspend: 1 Teb: 000000a2`26b10000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ff8`6350537e     : 000001ed`b57c79b0 000001ed`b57c79b0 00000000`00000001 000001ed`b57cb3b0 : ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00007ff8`6244257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x2ee
02 00007ff8`6352aa58     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1d
03 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28

   3  Id: 1540.4afc Suspend: 1 Teb: 000000a2`26b12000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ff8`6350537e     : 000001ed`b57c79b0 000001ed`b57c79b0 00000000`00000001 000001ed`b57cb890 : ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00007ff8`6244257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x2ee
02 00007ff8`6352aa58     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1d
03 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28

   4  Id: 1540.22c4 Suspend: 1 Teb: 000000a2`26b14000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ff6`22034849     : 00000000`00000000 00000040`00000000 00007ff6`22081ae0 00007ff6`22081ae0 : ntdll!NtQueryVirtualMemory+0x14
01 00007ff6`2204ef8c     : 00000000`00000000 00000000`00000000 00000000`00000000 00007ff8`634d0000 : pin!PinCommitHashSizeC+0x4519
02 00007ff6`22046ce7     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x1ec5c
03 00007ff6`220466ff     : 000000a2`26fffcc0 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x169b7
04 00007ff6`2204965b     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x163cf
05 00007ff6`22049290     : 00007ff6`220403c0 000001ed`b7290020 00000000`00000000 000000a2`26fffee8 : pin!PinCommitHashSizeC+0x1932b
06 00007ff6`2203bed9     : 00000000`00000000 ffffffff`ffffffff 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x18f60
07 00007ff6`220403ee     : 00000000`00000000 00000000`00000000 000001ed`b7290020 000001ed`b7330000 : pin!PinCommitHashSizeC+0xbba9
08 00007ff8`6244257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x100be
09 00007ff8`6352aa58     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1d
0a 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28

   5  Id: 1540.2320 Suspend: 1 Teb: 000000a2`26b16000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 000000a2`00000000     : 000000a2`270ff6c0 00000000`00000000 000000a2`270ff688 000000a2`270ff950 : 0x000001ed`b747958b
01 000000a2`270ff6c0     : 00000000`00000000 000000a2`270ff688 000000a2`270ff950 000001ed`b7469556 : 0x000000a2`00000000
02 00000000`00000000     : 000000a2`270ff688 000000a2`270ff950 000001ed`b7469556 00000000`00000000 : 0x000000a2`270ff6c0

   6  Id: 1540.3194 Suspend: 1 Teb: 000000a2`26b18000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ff6`2203510c     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!NtSignalAndWaitForSingleObject+0x14
01 00007ff6`22042227     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x4ddc
02 00007ff6`2204208e     : 00007ff6`22089a58 00000000`00000000 00000000`00000001 000000a2`271fea10 : pin!PinCommitHashSizeC+0x11ef7
03 00007ff8`6355f197     : 000000a2`271fea10 00000000`00000000 000000a2`271fe9c8 000000a2`271fefb0 : pin!PinCommitHashSizeC+0x11d5e
04 00007ff8`6357441f     : 00000000`00000000 000000a2`271fef10 000000a2`271ff5f0 000000a2`271ff5f0 : ntdll!_C_specific_handler+0x97
05 00007ff8`634ee466     : 000000a2`271ff5f0 00007ff6`22010000 00007ff6`22041b4c 00007ff6`22089a58 : ntdll!RtlpExecuteHandlerForException+0xf
06 00007ff8`6357340e     : 00000000`00000000 00000000`00000000 00000000`00000000 ffffffff`ffffffff : ntdll!RtlDispatchException+0x286
07 00007ff6`22042034     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!KiUserExceptionDispatch+0x2e
08 00007ff6`22041b4c     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x11d04
09 00007ff8`6244257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x1181c
0a 00007ff8`6352aa58     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1d
0b 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28

aremmell · ‎11-29-2023

When I attempted to debug the frozen 'matrix.exe', I got this:

The application was unable to start correctly (0xc0000142). Click OK to close the application.

From err.exe:

# for hex 0xc0000142 / decimal -1073741502
STATUS_DLL_INIT_FAILED ntstatus.h
# {DLL Initialization Failed}
# Initialization of the dynamic link library %hs failed. The
# process is terminating abnormally.
# as an HRESULT: Severity: FAILURE (1), FACILITY_NULL (0x0), Code 0x142
# for hex 0x142 / decimal 322
ERROR_DEVICE_NO_RESOURCES winerror.h
# The target device has insufficient resources to complete
# the operation.
# 2 matches found for "0xc0000142"

aremmell · ‎11-29-2023

I apologize; the WinDbg output above is inaccurate. I should have hit Go in the debugger to hit the exception. Here is the actual exception analysis:

0:005> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


KEY_VALUES_STRING: 1

    Key  : AV.Dereference
    Value: NullClassPtr

    Key  : AV.Fault
    Value: Read

    Key  : Analysis.CPU.mSec
    Value: 452

    Key  : Analysis.Elapsed.mSec
    Value: 525

    Key  : Analysis.IO.Other.Mb
    Value: 0

    Key  : Analysis.IO.Read.Mb
    Value: 0

    Key  : Analysis.IO.Write.Mb
    Value: 0

    Key  : Analysis.Init.CPU.mSec
    Value: 234

    Key  : Analysis.Init.Elapsed.mSec
    Value: 55215

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 72

    Key  : Failure.Bucket
    Value: NULL_CLASS_PTR_READ_c0000005_CsXumd64_17605.dll!Unknown

    Key  : Failure.Hash
    Value: {96ccba9a-04d8-6cfb-1909-271a247a6db9}

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 522

    Key  : Timeline.Process.Start.DeltaSec
    Value: 58

    Key  : WER.OS.Branch
    Value: ni_release

    Key  : WER.OS.Version
    Value: 10.0.22621.1


NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00000283bb17958b
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 0000000000000098
Attempt to read from address 0000000000000098

FAULTING_THREAD:  00003a48

PROCESS_NAME:  pin.exe

READ_ADDRESS:  0000000000000098 

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  0000000000000000

EXCEPTION_PARAMETER2:  0000000000000098

IP_ON_HEAP:  000000d800000000
The fault address in not in any loaded module, please check your build's rebase
log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may
contain the address if it were loaded.

ADDITIONAL_DEBUG_TEXT:  Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]

IP_IN_FREE_BLOCK: d800000000

FRAME_ONE_INVALID: 1

STACK_TEXT:  
000000d8`ac4fe428 00007ffe`8e1e7b9c ntdll!RtlpxVirtualUnwind+0x9720c
000000d8`ac4fe440 00007ffe`8e2b5350 ntdll!RtlpDynamicFunctionTableLock+0x0
000000d8`ac4fe458 00007ffe`8e1f8fb6 ntdll!RtlpLookupDynamicFunctionEntry+0x5be3e
000000d8`ac4fe468 00007ffe`8e150732 ntdll!RtlpxLookupFunctionTable+0xd2
000000d8`ac4fe498 00007ffe`8e150635 ntdll!RtlLookupFunctionEntry+0x355
000000d8`ac4fe4e8 00007ffe`8e14e577 ntdll!RtlDispatchException+0x397
000000d8`ac4fe818 00007ffe`8e19a88a ntdll!RtlInsertElementGenericTableFullAvl+0xda
000000d8`ac4fe858 00007ffe`8e19a788 ntdll!RtlInsertElementGenericTableAvl+0x48
000000d8`ac4fe8a8 00000283`baf71ec8 CsXumd64_17605+0x1ec8
000000d8`ac4fe918 00000283`baf7196a CsXumd64_17605+0x196a
000000d8`ac4fe920 00000283`baf71940 CsXumd64_17605+0x1940
000000d8`ac4fe940 00007ffe`8e2cb408 ntdll!LdrpVectorHandlerList+0x0
000000d8`ac4fe948 00007ffe`8e1a7ae6 ntdll!RtlpCallVectoredHandlers+0x16e
000000d8`ac4fe998 00007ffe`8e18467d ntdll!RtlGetExtendedContextLength2+0x3d
000000d8`ac4fe9e8 00007ffe`8e14e2cf ntdll!RtlDispatchException+0xef
000000d8`ac4feae8 00007ffe`8e17423a ntdll!RtlpFindEntry+0x3a
000000d8`ac4feb28 00007ffe`8e16f476 ntdll!RtlpAllocateHeap+0x826
000000d8`ac4febd8 00007ffe`8e16d2d9 ntdll!RtlpLowFragHeapAllocFromContext+0x2e9
000000d8`ac4fec38 00007ffe`8e1d3433 ntdll!KiUserExceptionDispatch+0x53
000000d8`ac4feee8 00007ffe`8e1586a8 ntdll!LdrpCallInitRoutine+0x74
000000d8`ac4fef78 00000283`baf712a0 CsXumd64_17605+0x12a0
000000d8`ac4ff2d8 00007ffe`8e1c512b ntdll!_cpu_features_init+0x2f
000000d8`ac4ff308 00007ffe`8e1a3e23 ntdll!LdrpInitialize+0x2b
000000d8`ac4ff310 00007ffe`8e130000 ntdll!LdrpGetModuleName <PERF> +0x0
000000d8`ac4ff338 00007ffe`8e1a3de8 ntdll!LdrInitializeThunk+0x18
000000d8`ac4ff818 00007ffe`8e18aa58 ntdll!RtlUserThreadStart+0x28


STACK_COMMAND:  ** Pseudo Context ** Pseudo ** Value: 3 ** ; kb

SYMBOL_NAME:  CsXumd64_17605+1ec8

MODULE_NAME: CsXumd64_17605

IMAGE_NAME:  CsXumd64_17605.dll

FAILURE_BUCKET_ID:  NULL_CLASS_PTR_READ_c0000005_CsXumd64_17605.dll!Unknown

OS_VERSION:  10.0.22621.1

BUILDLAB_STR:  ni_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

IMAGE_VERSION:  7.4.17605.0

FAILURE_ID_HASH:  {96ccba9a-04d8-6cfb-1909-271a247a6db9}

Followup:     MachineOwner
---------

and all threads' call stacks:

0:005> ~* kb

   0  Id: 135c.3ca8 Suspend: 1 Teb: 000000d8`abeb8000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ffd`e216e0d0     : 00000000`0000027f 00000000`00000000 00000000`00000000 00000000`00780076 : ntdll!NtWaitForMultipleObjects+0x14
01 00007ffd`e2171c8a     : 00000000`00000000 00000000`00000000 00000000`00000000 00007ffd`e2277fa0 : pinipc!LEVEL_BASE::IPC_CLIENT::IPC_CLIENT+0x120
02 00007ffd`e217196d     : 000000d8`abd8f130 00000000`00000000 00000000`00003ca8 00000283`bb0401c0 : pinipc!LEVEL_BASE::IPC_CONNECTION::SendMessageA+0x44a
03 00007ffd`e216f162     : 00000000`00000016 00000000`000025bc 000000d8`abd8f2a0 00000000`00000030 : pinipc!LEVEL_BASE::IPC_CONNECTION::SendMessageA+0x12d
04 00007ffd`e216f9ca     : 00000283`bae407c0 00007ffd`00000001 00000000`00000000 00000283`badb0130 : pinipc!LEVEL_BASE::IPC_CLIENT_CONNECTION::RemoteProcedureCall+0x82
05 00007ff6`a121204b     : 00000000`00000001 00000000`00000000 00000283`bae30c40 00007ffd`e2273660 : pinipc!LEVEL_BASE::IPC_CLIENT::Init+0x12a
06 00007ff6`a1211f9a     : 00000000`00000188 00000000`ffffffff 00000283`bae50840 00007ffd`e2273660 : pin!Ordinal0+0x204b
07 00007ff6`a1211a06     : 000000d8`abd8f648 000000d8`abd8f5e0 00007ffd`e20c1bf0 00007ff6`a125e812 : pin!Ordinal0+0x1f9a
08 00007ff6`a121180d     : 00000000`0000003e 00000283`bae50840 00000000`00000000 00000000`00000000 : pin!Ordinal0+0x1a06
09 00007ff6`a125e9cd     : 00000000`00008aa0 00007ffd`e21d59ed 000000d8`abd8f7ac 00000000`00000000 : pin!Ordinal0+0x180d
0a 00007ff6`a125f0b3     : 00000000`00000001 000000d8`abd8f846 00000000`00000000 00007ff6`a125f510 : pin!PinCommitHashSizeC+0x2e69d
0b 00007ff6`a125fb5c     : 00000283`bb0102a2 61737365`6d006465 00000000`00006567 6c6c6163`7379731c : pin!PinCommitHashSizeC+0x2ed83
0c 00007ffd`e21dfc4b     : 00000000`00000000 00000283`bafd0000 000000d8`abd8fa10 00000283`baf40000 : pin!PinCommitHashSizeC+0x2f82c
0d 00007ff6`a126cd85     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : PINCRT!_my_libc_init+0x15b
0e 00007ffe`8d5c257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x3ca55
0f 00007ffe`8e18aa58     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1d
10 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28

   1  Id: 135c.14ec Suspend: 1 Teb: 000000d8`abeba000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ffe`8e16537e     : 00000283`b93f96d0 00000283`b93f96d0 00000000`00000001 00000283`b93fc790 : ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00007ffe`8d5c257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x2ee
02 00007ffe`8e18aa58     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1d
03 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28

   2  Id: 135c.2d40 Suspend: 1 Teb: 000000d8`abebc000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ffe`8e16537e     : 00000283`b93f96d0 00000283`b93f96d0 00000000`00000001 00000283`b93fd800 : ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00007ffe`8d5c257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x2ee
02 00007ffe`8e18aa58     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1d
03 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28

   3  Id: 135c.23e8 Suspend: 1 Teb: 000000d8`abebe000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ffe`8e16537e     : 00000283`b93f96d0 00000283`b93f96d0 00000000`00000001 00000283`b93fe080 : ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00007ffe`8d5c257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x2ee
02 00007ffe`8e18aa58     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1d
03 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28

   4  Id: 135c.25bc Suspend: 1 Teb: 000000d8`abec0000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ff6`a1241953     : 000000d8`ac3ffb20 00007ff6`a12855e0 00000000`00000000 00000000`00000190 : KERNELBASE!WaitForSingleObjectEx+0x95
01 00007ff6`a1245cd6     : 000000d8`ac3ffb20 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x11623
02 00007ff6`a124965b     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x159a6
03 00007ff6`a1249290     : 00007ff6`a12403c0 00000283`baea0020 00000000`00000000 000000d8`ac3ffd48 : pin!PinCommitHashSizeC+0x1932b
04 00007ff6`a123bed9     : 00000000`00000000 ffffffff`ffffffff 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x18f60
05 00007ff6`a12403ee     : 00000000`00000000 00000000`00000000 00000283`baea0020 00000283`baf40000 : pin!PinCommitHashSizeC+0xbba9
06 00007ffe`8d5c257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pin!PinCommitHashSizeC+0x100be
07 00007ffe`8e18aa58     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x1d
08 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28

#  5  Id: 135c.3a48 Suspend: 1 Teb: 000000d8`abec2000 Unfrozen
 # RetAddr               : Args to Child                                                           : Call Site
00 000000d8`00000000     : 000000d8`ac4ff500 00000000`00000000 000000d8`ac4ff4c8 000000d8`ac4ff790 : 0x00000283`bb17958b
01 000000d8`ac4ff500     : 00000000`00000000 000000d8`ac4ff4c8 000000d8`ac4ff790 00000283`bb169556 : 0x000000d8`00000000
02 00000000`00000000     : 000000d8`ac4ff4c8 000000d8`ac4ff790 00000283`bb169556 00000000`00000000 : 0x000000d8`ac4ff500

   6  Id: 135c.42d4 Suspend: 1 Teb: 000000d8`abec4000 Unfrozen
WARNING: Stack overflow detected. The unwound frames are extracted from outside normal stack bounds.
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ffe`8e15908b     : 00000000`00000000 000000d8`abec4000 00000000`00000000 000000d8`abec4000 : ntdll!LdrpGetNewTlsVector+0x2c
01 00007ffe`8e158387     : 00000000`00000000 00000000`00000008 00000000`00000000 00000000`00000000 : ntdll!LdrpAllocateTls+0x4f
02 00007ffe`8e1a3f7f     : 00000000`00000000 00000000`00000000 00000000`00000001 00000000`00000000 : ntdll!LdrpInitializeThread+0x6f
03 00007ffe`8e1a3eb3     : 000000d8`ac5ff520 00007ffe`8e130000 00000000`00000000 000000d8`abec57ee : ntdll!_LdrpInitialize+0x93
04 00007ffe`8e1a3dde     : 000000d8`ac5ff520 00000000`00000000 000000d8`ac5ff520 00000000`00000000 : ntdll!LdrpInitializeInternal+0x6b
05 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!LdrInitializeThunk+0xe

aremmell · ‎11-29-2023

It would appear that pin.exe attempts to LoadLibrary("KERNELBASE.DLL") which returns FALSE:

DllMain(0x00007FFE8B8F0000, DLL_PROCESS_ATTACH, 0x0000000000000000) in "KERNELBASE.DLL" called.
DllMain(0x00007FFE8B8F0000, DLL_PROCESS_ATTACH, 0x0000000000000000) in "KERNELBASE.DLL" returned 0 (0x0).
DllMain(0x00007FFE8B8F0000, DLL_PROCESS_DETACH, 0x0000000000000000) in "KERNELBASE.DLL" called.
DllMain(0x00007FFE8B8F0000, DLL_PROCESS_DETACH, 0x0000000000000000) in "KERNELBASE.DLL" returned 1 (0x1).
Unloaded "KERNELBASE.DLL" at address 0x00007FFE8B8F0000.
Unloaded "KERNEL32.DLL" at address 0x00007FFE8D5B0000.
Exited "PIN.EXE" (process 0x698) with code -1073741502 (0xC0000142).

aremmell · ‎11-29-2023

I figured out the problem. The TLDR is that VTune is entirely incompatible with CrowdStrike Falcon Sensor.

The DLL in the WinDbg output, 'CsXumd64_17605.dll' is digitally signed by CrowdStrike, Inc. so it's no mystery where it came from. I suspect it is a global hook, and simply does not work when injected into binaries doing low-level operations such as VTune's.

So there it is. Now it's online for posterity: you cannot use VTune on a system that has CrowdStrike Falcon installed.

RahulC_Intel · ‎11-30-2023

Hi Ryan, thank you for your confirmation. We are glad that the issue has been resolved.

If you have any further queries, please post a new question as this thread will no longer be monitored by Intel®.