Mismatched tpsstool.dll and .pdb

[Shannon_P_]

I'm using VTune Amp XE 2015, build 367959 and am running into some problems instrumenting our code with tasks.

When we run the application under VTune, it crashes with a stack overflow in pinvm.dll. Without VTune, it runs fine.

The actual overflow stack has a bunch of tpsstool.dll entries in it. When I attempt to load symbols for  C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\tpsstool.dll (in the Visual Studio 2012 debugger) and point it at the tpsstool.pdb found in C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\runtime, I am told that "A matching symbol file was not found in this folder."

This leads me to believe that the .dll and .pdb aren't from the same build.

shannon

P.S. I don't blame tpsstool for the crash, I just thought you should be notified that the builds do not match. It would be convenient to see what's going on in there while debugging. It would also be nice to have the symbols for pinvm.dll for the same reason.

[Peter_W_Intel]

I have two questions:

1. Was it possible that you didn't uninstall prior version completely, and just simply copy new version to cause this...

2. Can you use utility named symchk.exe (from WinDbg package) to check if PDB file is consistent of EXE? For example, "symchk.exe test.exe /v /s path_for_pdb"

[Shannon_P_]

1. As far as I know, this is on a clean install.

2. Results of symchk indicate that  tpsstool.pdb might be wanting some other pdbs?

C:\>"C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x86\symchk" "C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\tpsstool.dll" /v /s "C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\runtime\tpsstool.pdb"

[SYMCHK] Searching for symbols to C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\tpsstool.dll in path C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\runtime\tpsstool.pdb
DBGHELP: Symbol Search Path: C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\runtime\tpsstool.pdb
[SYMCHK] Using search path "C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\runtime\tpsstool.pdb"
DBGHELP: No header for C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\tpsstool.dll.  Searching for image on disk
DBGHELP: C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\tpsstool.dll - OK
DBGHELP: C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\runtime\tpsstool.pdb\amplxe-tpss-collector-tpsstool.pdb - file not found
DBGHELP: C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\runtime\tpsstool.pdb\dll\amplxe-tpss-collector-tpsstool.pdb - file not found
DBGHELP: C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\runtime\tpsstool.pdb\symbols\dll\amplxe-tpss-collector-tpsstool.pdb - file not found
DBGHELP: tpsstool - no symbols loaded
[SYMCHK] MODULE64 Info ----------------------
[SYMCHK] Struct size: 1680 bytes
[SYMCHK] Base: 0x55000000
[SYMCHK] Image size: 4509696 bytes
[SYMCHK] Date: 0x53e154c1
[SYMCHK] Checksum: 0x004247a4
[SYMCHK] NumSyms: 0
[SYMCHK] SymType: SymNone
[SYMCHK] ModName: tpsstool
[SYMCHK] ImageName: C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\tpsstool.dll
[SYMCHK] LoadedImage: C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\tpsstool.dll
[SYMCHK] PDB: ""
[SYMCHK] CV: RSDS
[SYMCHK] CV DWORD: 0x53445352
[SYMCHK] CV Data:  C:\bb\INNLphep2w6r\b\b\tmpec6bkc\build\build_release_win32-x86_icl_13.1_mstools_9.0\tpss.collector.tpsstool\amplxe-tpss-collector-tpsstool.pdb
[SYMCHK] PDB Sig:  0
[SYMCHK] PDB7 Sig: {D464B20C-392A-4DB1-B87B-1D8610FAEA4B}
[SYMCHK] Age: 1
[SYMCHK] PDB Matched:  TRUE
[SYMCHK] DBG Matched:  TRUE
[SYMCHK] Line nubmers: FALSE
[SYMCHK] Global syms:  FALSE
[SYMCHK] Type Info:    FALSE
[SYMCHK] ------------------------------------
SymbolCheckVersion  0x00000002
Result              0x00010001
DbgFilename         tpsstool.dbg
DbgTimeDateStamp    0x00000000
DbgSizeOfImage      0x00000000
DbgChecksum         0x00000000
PdbFilename         C:\bb\INNLphep2w6r\b\b\tmpec6bkc\build\build_release_win32-x86_icl_13.1_mstools_9.0\tpss.collector.tpsstool\amplxe-tpss-collector-tpsstool.pdb
PdbSignature        {D464B20C-392A-4DB1-B87B-1D8610FAEA4B}
PdbDbiAge           0x00000001
[SYMCHK] [ 0x00000000 - 0x00010001 ] Checked "C:\Program Files (x86)\Intel\VTune Amplifier XE 2015\bin32\tpsstool.dll"
SYMCHK: tpsstool.dll         FAILED  - amplxe-tpss-collector-tpsstool.pdb mismatched or not found

SYMCHK: FAILED files = 1
SYMCHK: PASSED + IGNORED files = 0

[Robert_L_Intel1]

Hi Peter, Shannon - Does the faulting stack actually have pinvm.dll on it?  Is the overflow getting reported at the same address every time?

Is there any way you could get the Process Explorer tool from Microsoft* and upload a full memory dump (right-click, full dump)?

Note, this is not related to the question about the symbols.  I'm not trying to hijack that, just would like to dig into the overflow issue in pinvm.dll if you are willing.

[Robert_L_Intel1]

Addendum to my previous post:  The reason I'm asking is because I think you may be misreading the faulting stack.  The faulting stack should have pinvm.dll on it.  It's possible that if you are trying to debug this, you are looking at the debug break thread and not the overflow thread.

In any case, if the IP address is the same in the error message and the problem is repeatable, it may be a recursion problem.  Sometimes these can be difficult to diagnose, so I would love to have a dump of that if possible.

Thanks!

[Shannon_P_]

Yes, the actual crash is in pinvm.dll. The tpsstool.pdb thing was just something I noticed along the way. I wasn't sure that the stack overflow wasn't caused by some multithreading problems of our own (though everything seems fine when not running in VTune), but if recursion issues are not uncommon then I will gladly provide a dump.

It happens consistently during our DirectX 11 startup. I can get a dump for you, but it will likely be pretty large. Is there a good place for me to provide that? 

UPDATE: The zipped dump is ~102MB.

shannon

[Robert_L_Intel1]

Hi - Yes, you can upload to me as a private message.  Put it in a compressed archive to save space.  Thanks!

[Peter_W_Intel]

Although it reported that tpsstool.dll mismatched tpsstool.pdb (amplxe-tpss-collector-tpsstool.pdb not found) – I doubt it was due to dll’s dependencies.

The crash happened in pinvm.dll, so symbol mismatching is not a major issue, the developer may need log files to investigate error(s) in ping tool. I think that Bob will tell you how to set environment variables before running VTune(TM) Amplifier, to generate log files.

[Bernard]

>>>The actual overflow stack has a bunch of tpsstool.dll entries in it>>>

Can you post the call stack data? Windbg is better suited for this kind of troubleshooting.

[Bernard]

>>>When we run the application under VTune, it crashes with a stack overflow in pinvm.dll. Without VTune, it runs fine>>>

You can set Windbg as default post- mortem debugger with this command: <windbg directory> windbg -I

Do not forget to associate windbg with various dump files <windbg dir> windbg -IA.

You should also enable breaking on exception.

http://msdn.microsoft.com/en-us/library/windows/hardware/ff558822(v=vs.85).aspx

[Robert_L_Intel1]

Hi Shannon - Things are still a little garbled.  The exception record is corrupt, but the dump is also unreliable I think because procmon created a 64-bit dump.  I was able to dig out the exception code, though, so I at least know that we have a stack allocation failure.  You can see how the code is actually in the record link slot.

0:077> dt ntdll!_EXCEPTION_RECORD 29cb5608
   +0x000 ExceptionCode    : 0n701191700
   +0x004 ExceptionFlags   : 0x29cb5664
   +0x008 ExceptionRecord  : 0xc00000fd`00000000 _EXCEPTION_RECORD
   +0x010 ExceptionAddress : (null) 
   +0x018 NumberParameters : 0x54206fb7
   +0x020 ExceptionInformation : [15] 0x29cb2000`00000000
0:077> !error c00000fd
Error code: (NTSTATUS) 0xc00000fd (3221225725) - A new guard page for the stack cannot be created.

I have two tools I would like you to use instead:  Application Verifier (appverif.exe) -> dirty-stacks," just in case the stack is corrupt, coupled with the procdump tool (for 32-bit dump on 64-bit OS).

Here is ProcDump (just a handy tool to have around anyway):

http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

Applicaion Verifier should be installed already, but if not, you can also download it here:

http://www.microsoft.com/en-us/download/details.aspx?id=20028

I'm attaching an image to show you where "dirty-stacks" is located.  You need to add your application from file menu, then deselect "Basics" and open up the misc node to select "dirty stacks."  Do this first before you attempt another repro.  Then, use procdump to dump the process.  This is going to be a command-line thing, like:  procdump notepad.exe.

Then, please upload to me in private as before.

And, WRT Peter's logging, I would like to wait on that until I have a chance verify we have a solid dirty-stacks dump.

[Shannon_P_]

VTune says:

Cannot start analysis because Image File Execution Options (IEFO) are enabled for this application. Suggestion: Use the Global Flags Editor (GFlags) to disable IFEO for this application.

Rather than having VTune launch the app, I launched the app and attached VTune. When I did so, it instantly crashed. Regardless of where I pause the startup it crashes as soon as I attach VTune. So it seems that option and VTune are incompatible.

shannon

[Robert_L_Intel1]

Based on my analysis of your last dump, I would like to see if we can rule out an infinite or recursive loop to be sure.  I think this would help chances for a more timely workaround or fix, if it's found to be a defect in our product.

I thought a way to do that might be to use editbin to change the default stack size.  You can see below that the last allocation hit the limit and the last several 1000h-sized blocks of stack space is all zeros.

If you are amenable to this, both editbin and dumpbin can be found in the Visual Studio* toolset.

EditBin
http://msdn.microsoft.com/en-us/library/35yc2tc3.aspx

DumpBin:
http://msdn.microsoft.com/en-us/library/9ha900t8.aspx

What you could do is check the stack size in the exe using dumpbin /headers 

In my test case, they were set to 1000 commit and 100000 reserve like so:
          100000 size of stack reserve
            1000 size of stack commit
          100000 size of heap reserve
            1000 size of heap commit 

Using the C-notation, I changed it to this:
          200000 size of stack reserve
            2000 size of stack commit

With this command:  editbin /STACK:0x200000,0x2000

I think the problem line is the sub 1000h allocation just above and not the current IP.

0:077> !teb
TEB at ffed1000
    ExceptionList:        361aed58
    StackBase:            361b0000
    StackLimit:           361a1000
    
0:077> ub eip l10
54206fa4 3bc8            cmp     ecx,eax <<<<<----- begin loop
54206fa6 720a            jb      pinvm!(54206fb2) <<<<<-----
54206fa8 8bc1            mov     eax,ecx
54206faa 59              pop     ecx
54206fab 94              xchg    eax,esp
54206fac 8b00            mov     eax,dword ptr [eax]
54206fae 890424          mov     dword ptr [esp],eax
54206fb1 c3              ret
54206fb2 2d00100000      sub     eax,1000h <<<<<---------- (problem is here, I think, but would like to confirm)
0:077> r
eax=361a2000 ebx=00000000 ecx=361a29f8 edx=00000005 esi=06c49160 edi=00000001
eip=54206fb7 esp=361a5ee8 ebp=00000000 iopl=0         nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206

54206fb7 8500            test    dword ptr [eax],eax  ds:002b:361a2000=00000000
0:077> u

54206fb7 8500            test    dword ptr [eax],eax
54206fb9 ebe9            jmp     pinvm!(54206fa4) <<<<<----- end loop

0:077> !address esp
                                  
Usage:                  Stack
Allocation Base:        361a0000
Base Address:           361a1000
End Address:            361b0000
Region Size:            0000f000
Type:                   00020000    MEM_PRIVATE
State:                  00001000    MEM_COMMIT
Protect:                00000004    PAGE_READWRITE
More info:              ~77k

[Shannon_P_]

Used editbin as you indicated:

        200000 size of stack reserve
          2000 size of stack commit
        100000 size of heap reserve
          1000 size of heap commit

And it crashed in the same code, it seems. If you want the dump I can provide that as well.

0:076> !teb
TEB at ffed4000
    ExceptionList:        382fe988
    StackBase:            38300000
    StackLimit:           382f1000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 0067c8b8
    Self:                 ffed4000
    EnvironmentPointer:   00000000
    ClientId:             00005aac . 000058d4
    RpcHandle:            00000000
    Tls Storage:          346d6130
    PEB Address:          fffde000
    LastErrorValue:       0
    LastStatusValue:      c000000d
    Count Owned Locks:    0
    HardErrorMode:        0
0:076> ub eip l10
pinvm!CrtEnableThreadCallbacks+0xb1bf1:
54206f91 8d4c2404        lea     ecx,[esp+4]
54206f95 2bc8            sub     ecx,eax
54206f97 1bc0            sbb     eax,eax
54206f99 f7d0            not     eax
54206f9b 23c8            and     ecx,eax
54206f9d 8bc4            mov     eax,esp
54206f9f 2500f0ffff      and     eax,0FFFFF000h
54206fa4 3bc8            cmp     ecx,eax
54206fa6 720a            jb      pinvm!CrtEnableThreadCallbacks+0xb1c12 (54206fb2)
54206fa8 8bc1            mov     eax,ecx
54206faa 59              pop     ecx
54206fab 94              xchg    eax,esp
54206fac 8b00            mov     eax,dword ptr [eax]
54206fae 890424          mov     dword ptr [esp],eax
54206fb1 c3              ret
54206fb2 2d00100000      sub     eax,1000h
0:076> u
pinvm!CrtEnableThreadCallbacks+0xb1c17:
54206fb7 8500            test    dword ptr [eax],eax
54206fb9 ebe9            jmp     pinvm!CrtEnableThreadCallbacks+0xb1c04 (54206fa4)
54206fbb 8bff            mov     edi,edi
54206fbd 55              push    ebp
54206fbe 8bec            mov     ebp,esp
54206fc0 51              push    ecx
54206fc1 51              push    ecx
54206fc2 57              push    edi

0:076> !address esp
                                     
Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
SYMSRV:  C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\sym\amplxe-tpss-collector-tpsstool.pdb\D464B20C392A4DB1B87B1D8610FAEA4B1\amplxe-tpss-collector-tpsstool.pdb not found
SYMSRV:  C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\sym\amplxe-tpss-collector-tpsstool.pdb\D464B20C392A4DB1B87B1D8610FAEA4B1\amplxe-tpss-collector-tpsstool.pdb not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/amplxe-tpss-collector-tpsstool.pdb/D464B20C392A4DB1B87B1D8610FAEA4B1/amplxe-tpss-collector-tpsstool.pdb not found
DBGHELP: C:\bb\INNLphep2w6r\b\b\tmpec6bkc\build\build_release_win32-x86_icl_13.1_mstools_9.0\tpss.collector.tpsstool\amplxe-tpss-collector-tpsstool.pdb - file not found
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for tpsstool.dll - 
DBGHELP: tpsstool - export symbols
DBGHELP: KERNELBASE - public symbols  
        C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\sym\wkernelbase.pdb\5ED1648668CE459B9C9C4093002426EF1\wkernelbase.pdb
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...


Usage:                  Stack
Base Address:           382f1000
End Address:            38300000
Region Size:            0000f000
State:                  00001000	MEM_COMMIT
Protect:                00000004	PAGE_READWRITE
Type:                   00020000	MEM_PRIVATE
Allocation Base:        382f0000
Allocation Protect:     00000004	PAGE_READWRITE
More info:              ~76k

[Robert_L_Intel1]

Okay, thanks for doing that.  It's what I expected, but it's nice to be sure.  That is all I need for now.  

[Bernard]

@Bob

What does this ecx=361a29f8 value point to?  It seems strange that loop counter is decremented by 1000h or maybe that is some kind of allocation of 4096 bytes per every loop iteration?

[Robert_L_Intel1]

Yeah, I'm not sure.  I'm not really attempting to determine exactly what's happening, only to gather a preponderance of evidence to demonstrate a likely-hood.  It seems to be a recursive or iterative 4k stack allocation that's causing the error.  We will have to wait and see what the engineering team says.

[Bernard]

>>>It seems to be a recursive or iterative 4k stack allocation that's causing the error>>>

I also suspect iterative stack allocation to be a reason for the stack overflow error.

[Robert_L_Intel1]

I guess I would want to caution that we not read too much into this.  It's possible we are just seeing the obvious.  The error message says that it can't allocate a guard page.  The default page size on Windows is 4k.  The fact that pin is an injector/interceptor kind of application means that we could just be seeing the system failing to allocate another page or enough pages for something entirely unknown.  It may well be reported in the debugger as pin code, or a fault in the pin DLL, but that could be simply because pin has placed itself in the middle of the call chain in order to do what it does.

[Bernard]

It could be interesting to see which export/imports are hooked or intercepted by pin tool. I think that it can be a good starting point for the investigation of the problem.

[Robert_L_Intel1]

Hi Shannon - It could be a coincidence, but I've discovered another issue with the exact same problem on Windows 7 x64.  In both cases, the NVIDIA driver is on the faulting stack and is of the same version.  It might be worth upgrading the driver as a possible resolution while we continue to investigate the issue internally.  In your case, the base pointer is being used as a general-purpose register, so I can't make much sense of the stack other than to say that there are addresses which fall in the range of the driver.

Here, I believe, would be the latest driver for your system.

http://www.nvidia.com/download/driverResults.aspx/79890/en-us