System crash when using H.264 HW encoder on Atom E3845

[Philipp_S_]

Hi, we're using the Media SDK to encode two 1920x1080p30 fps raw video streams into two H.264 bitstreams on our Intel Atom E3845 hardware. Each encoder pipeline runs in its own dedicated Windows process (on Windows 7 Professional 64 Bit) and we're using Media SDK's HW encoding support. This works fine, but after some non-deterministic amount of time (sometimes after a few seconds, sometimes after approx. 10 minutes) the whole system just freezes and stops working entirely. We do not get a BSOD but a frozen Windows Desktop, no keyboard or mouse input is possible in this crashed state. The only way to recover the system is a reset or a power cycle. We desperately tried to hunt the issue but had no luck so far. As the whole operating system crashes normal code debugging and also the Media SDK tracer tool did not do the trick. The system configuration found by MSDK's SysArch tool is attached to the post. Thanks a lot, Philipp

[Anthony_P_Intel]

Hi Philipp,

The drivers and HW you are using are 'embedded' version and I have contacted that team.  The issue sounds similar to an issue that was recently seen (and fixed) on similar tablet platforms.  For the best support of embedded driver, I would recommend using Intel Premier Support, but I will also try to connect the two teams.

The two drivers are different, however there is a good chance that both teams had the same bug, and new embedded driver has not been built yet.

There is also the possibility that issue is due to physical design (thermal, etc.).

If you could capture a log of MSDK API activity by using the 'tracer' tool, we might be able to identify if it is indeed a similar issue.

[Philipp_S_]

Hi Tony,

thanks for your fast response.

We tried to produce a log via the MSDK "tracer" tool. Unfortunately in case of this error the whole OS hangs, hence the tracer tool can't successfully write a log file to the disk (or at least did not do it on our system).

Are there any other ways we could help your driver / SDK teams in identifying the issue? E.g. running the system with an instrumented version of the embedded driver and sending back the kernel mode logs?

Thanks,

Philipp

[Philipp_S_]

Hi Tony, we finally managed to get a Windows crash dump just in the moment when the system starts to hang and just before keyboard and mouse input also stops to work. The crash dump was manually initiated by a keystroke as described in http://msdn.microsoft.com/en-us/library/windows/hardware/ff545499%28v=vs.85%29.aspx . Using the crash dump file and WinDbg we could extract the stack traces for all 4 cores (!analyze -v) and attach them to this post. Core 0 shows the stack trace of the manually initiated crash in the USB keyboard driver. Core 2 actually seems to show stack trace entries deep in the igdkmd64.sys graphics driver. Thanks, hth, Philipp

[Philipp_S_]

Seems that the file download doesn't work, so here is the !analyze -v output for core 2 (Graphics driver: Intel(R) Atom(TM) Processor E3800 Series/Intel(R) Celeron(R) Processor N2920/J1900 37.15.0.1064): ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* MANUALLY_INITIATED_CRASH (e2) The user manually initiated this crash dump. Arguments: Arg1: 0000000000000000 Arg2: 0000000000000000 Arg3: 0000000000000000 Arg4: 0000000000000000 Debugging Details: ------------------ BUGCHECK_STR: MANUALLY_INITIATED_CRASH DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT PROCESS_NAME: StreamServer.e CURRENT_IRQL: 0 ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre LAST_CONTROL_TRANSFER: from fffff88003fb56d7 to fffff88003fb5862 STACK_TEXT: fffff880`073ad4e8 fffff880`03fb56d7 : 00000000`00000000 fffffa80`06410000 00000000`00000000 00000000`00000000 : igdkmd64!hybDriverEntry+0xe7e92 fffff880`073ad4f0 fffff880`03f3a404 : fffffa80`06410000 fffffa80`06410000 00000000`00000000 fffff800`02e19bc0 : igdkmd64!hybDriverEntry+0xe7d07 fffff880`073ad530 fffff880`03f39a72 : fffffa80`04937a01 fffff800`00000000 00000000`00000000 00000000`00000000 : igdkmd64!hybDriverEntry+0x6ca34 fffff880`073ad590 fffff880`03ece619 : 00000000`00000001 00000000`00000100 fffff8a0`028a7940 00000000`c7c7c017 : igdkmd64!hybDriverEntry+0x6c0a2 fffff880`073ad710 fffff880`03eeb6d0 : 00000000`00000000 fffffa80`058d9af0 00000000`00000000 00000000`00000000 : igdkmd64!hybDriverEntry+0xc49 fffff880`073ad740 fffff880`03f4e7ae : 00000001`1747c94e 00000001`1747c94e fffff880`073ad9c0 fffffa80`06b76000 : igdkmd64!hybDriverEntry+0x1dd00 fffff880`073ad770 fffff880`03f280ee : 00000000`00000000 00000000`00000000 fffff8a0`028a7950 4b677844`029d7230 : igdkmd64!hybDriverEntry+0x80dde fffff880`073ad7a0 fffff880`03e93678 : 00000000`737a2450 fffffa80`06b76000 fffff880`073ad9c0 00000000`00000001 : igdkmd64!hybDriverEntry+0x5a71e fffff880`073ad7d0 fffff880`042c5f50 : 00000000`00000000 00000000`00000000 fffff880`073ad950 00000000`00010286 : igdkmd64+0x23678 fffff880`073ad840 fffff880`042b9093 : 00000000`00000000 00000000`00000000 fffff880`073adca0 00000000`00000003 : dxgkrnl!DXGADAPTER::DdiEscape+0x50 fffff880`073ad870 fffff960`0027c2f6 : 00000000`0392e760 fffffa80`058df060 00000000`7ef9e000 00000000`00000020 : dxgkrnl!DxgkEscape+0x7af fffff880`073adbf0 fffff800`02c948d3 : 00000000`06e0d918 fffff880`073adca0 00000000`044ce934 00000000`7ef9e000 : win32k!NtGdiDdDDIEscape+0x12 fffff880`073adc20 00000000`737f127a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 00000000`0392e768 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x737f127a STACK_COMMAND: kb FOLLOWUP_IP: igdkmd64!hybDriverEntry+e7e92 fffff880`03fb5862 4185c1 test r9d,eax SYMBOL_STACK_INDEX: 0 SYMBOL_NAME: igdkmd64!hybDriverEntry+e7e92 FOLLOWUP_NAME: MachineOwner MODULE_NAME: igdkmd64 IMAGE_NAME: igdkmd64.sys DEBUG_FLR_IMAGE_TIMESTAMP: 52cd9727 FAILURE_BUCKET_ID: X64_MANUALLY_INITIATED_CRASH_VRF_igdkmd64!hybDriverEntry+e7e92 BUCKET_ID: X64_MANUALLY_INITIATED_CRASH_VRF_igdkmd64!hybDriverEntry+e7e92 ANALYSIS_SOURCE: KM FAILURE_ID_HASH_STRING: km:x64_manually_initiated_crash_vrf_igdkmd64!hybdriverentry+e7e92 FAILURE_ID_HASH: {a4d40e47-fa5b-bbb9-50cc-76d6429168da} Followup: MachineOwner ---------

[Philipp_S_]

To be complete, the traces of the three remaining cores: Core 0: fffff800`00b9c1d8 fffff880`05929ddf : 00000000`000000e2 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KeBugCheckEx fffff800`00b9c1e0 fffff880`0592991b : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kbdhid!KbdHidProcessCrashDump+0x1df fffff800`00b9c220 fffff880`05921629 : fffffa80`0689df20 fffffa80`06892b00 fffffa80`00000000 00000000`0000000e : kbdhid!KbdHid_InsertCodesIntoQueue+0xbb fffff800`00b9c280 fffff880`0592187c : fffffa80`000000c6 fffffa80`06892b00 00000000`0000000e 00000000`00000001 : HIDPARSE!HidP_UsageAndPageListDifference+0x161 fffff800`00b9c2b0 fffff880`059217d2 : fffffa80`06892b00 fffff880`0591f9d6 00000000`00000007 00000000`00000000 : HIDPARSE!HidP_TranslateUsageAndPagesToI8042ScanCodes+0x188 fffff800`00b9c2f0 fffff880`0592961e : 00000000`00000000 fffffa80`0689deec fffffa80`0689de10 00000000`00000000 : HIDPARSE!HidP_TranslateUsageAndPagesToI8042ScanCodes+0xde fffff800`00b9c350 fffff800`0313a5d6 : fffff980`06792fb0 fffffa80`00000001 fffff800`00000000 fffff980`06792ca0 : kbdhid!KbdHid_ReadComplete+0x222 fffff800`00b9c3d0 fffff800`02c98a91 : fffff980`06792fb3 fffff800`00000001 00000000`00000000 fffff800`00000005 : nt!IovpLocalCompletionRoutine+0x166 fffff800`00b9c430 fffff800`0313219f : fffff980`06792ca0 fffffa80`0683bc06 fffffa80`0688fc06 00000000`00000000 : nt!IopfCompleteRequest+0x3b1 fffff800`00b9c510 fffff880`059099c4 : fffffa80`0688fce0 fffffa80`0688fcd0 fffffa80`0688fce0 fffffa80`0688fcd0 : nt!IovCompleteRequest+0x19f fffff800`00b9c5e0 fffff880`05909c2d : fffffa80`0688f902 fffffa80`0688f910 fffffa80`0683bca0 00000000`00000008 : HIDCLASS!HidpDistributeInterruptReport+0x130 fffff800`00b9c670 fffff800`0313a5d6 : fffff980`0670cfb0 fffffa80`0683bca0 fffff800`00b9c801 fffff980`0670cd30 : HIDCLASS!HidpInterruptReadComplete+0x235 fffff800`00b9c700 fffff800`02c98a91 : fffff980`0670cfb3 fffff800`00000001 00000000`00000000 fffff800`00000005 : nt!IovpLocalCompletionRoutine+0x166 fffff800`00b9c760 fffff800`0313219f : fffff980`0670cd30 fffff980`0670cd00 00000000`00000000 00000000`00000000 : nt!IopfCompleteRequest+0x3b1 fffff800`00b9c840 fffff880`0453af97 : 00000000`00000002 00000000`00000000 fffffa80`0654cb20 00000000`00000000 : nt!IovCompleteRequest+0x19f fffff800`00b9c910 fffff880`0452aef4 : fffffa80`06cf43f0 00000000`00000000 00000000`00000005 fffff980`05994e90 : iusb3xhc+0x63f97 fffff800`00b9c940 fffff880`045231ed : fffff980`02de0f00 fffffa80`0654cb20 fffff880`04553f60 fffff980`06702e02 : iusb3xhc+0x53ef4 fffff800`00b9c9c0 fffff880`0450ec2a : fffffa80`06cf43f0 fffffa80`06418000 fffff980`06702ed0 fffff980`066dcce0 : iusb3xhc+0x4c1ed fffff800`00b9ca00 fffff880`04504850 : fffff980`06702ed0 fffffa80`06418000 fffffa80`06418000 fffff980`066e8ea0 : iusb3xhc+0x37c2a fffff800`00b9ca70 fffff880`044ee5c2 : fffffa80`06418000 fffff880`03c20930 fffff980`066eaa02 00000000`00000001 : iusb3xhc+0x2d850 fffff800`00b9cba0 fffff880`044eda47 : 00000000`00000000 fffffa80`06418000 fffff980`066eafa0 00000000`00000010 : iusb3xhc+0x175c2 fffff800`00b9cca0 fffff800`02ca0b1c : fffff800`02e07e80 00000000`25e79135 00000000`25e78d41 fffffa80`05ee5118 : iusb3xhc+0x16a47 fffff800`00b9ccd0 fffff800`02c8d36a : fffff800`02e07e80 fffff800`02e15cc0 00000000`00000000 fffff880`044ed93c : nt!KiRetireDpcList+0x1bc fffff800`00b9cd80 00000000`00000000 : fffff800`00b9d000 fffff800`00b97000 fffff800`00b9cd40 00000000`00000000 : nt!KiIdleLoop+0x5a Core 1: fffff880`02f1bc98 fffff800`02c9ef09 : 00000000`001c8fd2 fffffa80`06781d48 00000000`00000000 00000000`00000000 : intelppm+0x2c61 fffff880`02f1bca0 fffff800`02c8d33c : fffff880`009ef180 fffff880`00000002 00000000`00000002 fffff800`00000000 : nt!PoIdle+0x52a fffff880`02f1bd80 00000000`00000000 : fffff880`02f1c000 fffff880`02f16000 fffff880`02f1bd40 00000000`00000000 : nt!KiIdleLoop+0x2c Core 3: fffff880`02ffdc98 fffff800`02c9ef09 : 00000000`001c8fd2 fffffa80`06789dd8 00000000`00000000 00000000`00000000 : intelppm+0x2c61 fffff880`02ffdca0 fffff800`02c8d33c : fffff880`02fd5180 fffff880`00000002 00000000`00000002 fffff800`00000000 : nt!PoIdle+0x52a fffff880`02ffdd80 00000000`00000000 : fffff880`02ffe000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiIdleLoop+0x2c

[Philipp_S_]

Hi Tony, meanwhile we could record two additional crash dumps. Maybe it's just a coincidence and still not the right place to look at, but in all of our three recorded dump files (after the system starts to hang) the following stack trace repeats itself in the exact same manner: igdkmd64!hybDriverEntry+0xe7e92 igdkmd64!hybDriverEntry+0xe7d07 igdkmd64!hybDriverEntry+0x6ca34 igdkmd64!hybDriverEntry+0x6c0a2 igdkmd64!hybDriverEntry+0xc49 igdkmd64!hybDriverEntry+0x1dd00 igdkmd64!hybDriverEntry+0x80dde igdkmd64!hybDriverEntry+0x5a71e igdkmd64+0x23678 ... So maybe your embedded drivers team could have a look on this issue, we would really appreciate that. Thanks a lot, Philipp

[Philipp_S_]

Hi Carsten, yes, this actually might be a related issue. I also understand from Tony's messages that a team at Intel is able to reproduce the bug (or a similar behaviour) and that the drivers team might have a possible fix for it. So what would really help in this situation is to get a concrete release date for the new driver version and / or a possibility to early-access the experimental driver. It's clear that this (experimental) driver version might not have production quality. But it might help to verify if the mentioned bugfixes really address the problem on the target hardware and, last but not least, it would help to regain confidence in the entire solution. A solution which is, by the way, very attractive for us in our system environment if it works reliably. Best regards, Philipp