Exception at encoding core, MFXVideoCORE_SyncOperation permanently returns MFX_ERR_UNKNOWN after it

[OTorg]

Application pseudo code:
1. create session
2. create h264 encoder on that session, init it, prepare buffers
3. encode frames from raw nv12-file
4. flush encoder, wait for last output frame
5. release encoder and all resources, except session
6. goto 2

Input file and encoding parameters are the same on every cycle execution. Debug compillation, x86. MFX_IMPL_SOFTWARE only. imsdk 1.7.
After some time after application start (from several minutes to several days, in different runs) i got at debugger output:
"First-chance exception at 0x595798f3 in imsdk_h264enc.exe: 0xC0000005: Access violation reading location 0x000000c0."
And MFXVideoCORE_SyncOperation returns MFX_ERR_UNKNOWN after that, subsequent MFXVideoCORE_SyncOperation calls (with the same arguments) permanently return MFX_ERR_UNKNOWN.

Exception point details:
C:\PLANG\Intel\Media SDK 2013 R2\bin\win32\libmfxsw32.dll, offset 0x001988F3 from section ".text" begin
595798D7 mov ecx,dword ptr [edi+1D34h]
595798DD push edx
595798DE lea edx,[ecx+eax*2]
595798E1 mov eax,dword ptr [esi+20h]
595798E4 lea ecx,[eax+edx*2]
595798E7 mov edx,dword ptr [edi+34h]
595798EA mov ecx,dword ptr [edx+ecx*4]
595798ED mov eax,dword ptr [edi+1B94h]
595798F3 add ecx,dword ptr [eax+0C0h] *********** it is here *********
595798F9 mov edx,dword ptr [esi+3Ch]
595798FC push ecx
595798FD push edx
595798FE push edi
595798FF lea ecx,[ebp+0Bh]
59579902 mov edx,esi

PS:
Now, I realized that there is a little chance that problem is provoked by crt versions conflict (i got "LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other libs" at build, because of debug application build).
I'll make own mfx_dispatch build with the same settings as at application (debug/mt-dll-crt/vs2010), use it to rebuild application, and run tests again.
I'll write the results here in a couple of days.

[OTorg]

Hi Peter

I ran the tests (imsdk_h264enc_v2.zip) on the weekend:
- Got exception at debug-builded application.
- Got exception with SyncOperation timeout = 1000.

[Bernard]

>>>I think it is much more prosaic. And error lies somewhere in libmfxsw32 code. Stake on inaccurate critical sections usage:)>>>

For now it could be anyhing which modifies the registers.You wrote that exception occures when some workload is present (clicking start button) hence my answer was more in-depth.I tend to agree that error could lie somewhere inside  libmfxsw32.dll mainly because in close vicinity of the faulting IP there is no present any far call.

[Petter_L_Intel]

Hi,

A quick update. We have been able to reproduce same issue using internal test suite. Issue is being debugged with hopefully a solution very soon. Thanks for your patience.

Regards,
Petter 

[OTorg]

>> hopefully a solution very soon
Glad to hear it

[Bernard]

Hi Petter

Will you shed some light on the issue?

Thanks in advance.

[Zach_J_1]

This is excellent news!  I'm eagerly awaiting a response.

[OTorg]

Hi Petter

Can you take a look at another topic: http://software.intel.com/en-us/forums/topic/475621 ?
Does "increasing number of forward-prediction reference frames" idea deserve attention or not?
Could it take a place in a future imsdk releases?

[Petter_L_Intel · Accepted solution]

Hi dj_alek,

I've asked my colleague to get back to on this since he explored thie question at an earlier stage.

Regarding the SW DLL exception issue. We plan to resolve the issue in Media SDK 2014, which will be released at the end of this year.

Regards,
Petter

[Petter_L_Intel]

Hi dj_alek,

I did not hear anything back from you regarding this topic. 

Were you able to try the new Media SDK 2014 release? Do you still see the issue using the new SW DLL?

Regards,
Petter

[OTorg]

Hi Peter!

I didn't test imsdk2014 yet. I'll run tests in a few days and share results. Thank you!

[OTorg]

I performed a series of runs of test application (single instance) for a few hours/days on i7-3770/i7-4770/i5-3330.
All ok, exceptions were not observed.

But when I run 2 applications simultaneously, the exceptions still occur.
And MFXVideoCORE_SyncOperation permanently returns MFX_WRN_IN_EXECUTION after it.
It happens after a few minutes/hours after application start.
Observed at i7-3770/i7-4770, win7x64, x86 application, MFX_IMPL_SOFTWARE, imsdk 1.8, msvs2010 (with latest updates).

Test application source and exception dumps are attached.

PS:
I'll test several encoder instances within one process also and share results.

[OTorg]

So, tests of several encoder instances within one process are completed.

2 sessions (joined) and 2 encoders were used in all tests.
Environment: i7-3770 and i7-4770, win7x64, x86 application, MFX_IMPL_SOFTWARE, imsdk 1.8, msvs2010 (with latest updates).

1.
Sessions/encoders are created once, at application startup (infinite files are encoded).
Several-hours/days-runnings work well, exceptions were not observed.
Test application attached: imsdk_h264enc_multi_v1.zip

2.
Sessions are created once, at application startup. Encoders are recreated after each 1000-frames-file.
Got AV-exceptions.
Dumps are at imsdk_h264enc_multi_v2_av.zip, test application: imsdk_h264enc_multi_v2.zip.

3.
Parent session is created once, at application startup. Child session and encoders are recreated after each 1000-frames-file.
Got handles leak and AV-exceptions. Some AVs have interesting addresses: 0xabababab, 0xbaadf00d.
Exceptions are raised very quickly, after several/tens minutes after application start.
Dumps are at imsdk_h264enc_multi_v3_av.zip, test application: imsdk_h264enc_multi_v3.zip.

[OTorg]

Nothing has changed in the new imsdk 1.9.
AV-exceptions and IGFXKMDNotifyBatchBuffersComplete-handle leaks continue to occur.
You are implementing new features all the time, but can not ensure the normal operation of even basic functionality.
Sad.

[Anthony_P_Intel]

Hi,

Very sorry for slow response. We are looking at this and appreciate the excellent description of issue.

[maquemo]

Hello, I have a similar problem.

I create three session for enconding three H264 streams in the same process but different threads and an internal exception is thrown in the library libmfxsw32.dll after 1 or 2 hours.

The issue seems that does not show up when only a session is created.

Thank you and regards,

[OTorg]

And I saw exceptions within even 1process/1session during recent tests. Both sw and hw implementations, imsdk 1.8/1.9.

I'm tired of waiting for fixes and I aim to use encoders from other developers.

It's a pity, I used to be a better opinion about Intel.

[maquemo]

Hello, I have noticed, that you have also handle leaks. I had detected some handle leaks too, but I thought that the problem came from another source,maybe I was wrong.

Ok, we will have to wait.

Thank you dj_alek.

[Darrell_S_]

I'm still seeing this error with the 2014 R2 version of the Media SDK.  

I turned on exception handling and got this message:

First-chance exception at 0x0000000000000030 in EncoderTest.exe: 0xC0000005: Access violation executing location 0x0000000000000030.

Here are the threads in play

Not Flagged        3348    0    Main Thread    Main Thread    libmfxsw64.dll!00000001800164c7    Normal
Not Flagged        3144    0    Worker Thread    ntdll.dll thread    ntdll.dll!000007fb059945db    Normal
Not Flagged        3716    0    Worker Thread    ntdll.dll thread    ntdll.dll!000007fb059945db    Normal
Not Flagged        2380    0    Worker Thread    combase.dll thread    combase.dll!000007fb0561196a    Normal
Not Flagged        1692    0    Worker Thread    ntdll.dll thread    ntdll.dll!000007fb059945db    Normal
Not Flagged        2132    0    Worker Thread    ntdll.dll thread    ntdll.dll!000007fb059945db    Normal
Not Flagged        1156    0    Worker Thread    CIndexer::ReadIndex    msvcr110.dll!malloc    Normal
Not Flagged        3968    0    Worker Thread    CIndexer::ReadIndex    msvcr110.dll!malloc    Normal
Not Flagged        424    0    Worker Thread    libmfxsw64.dll thread    libmfxsw64.dll!000000018001a62b    Normal
Not Flagged    >    3524    0    Worker Thread    libmfxsw64.dll thread    0000000000000030    Normal   <-- current thread
Not Flagged        2512    0    Worker Thread    libmfxsw64.dll thread    libmfxsw64.dll!000000018001a62b    Normal
Not Flagged        92    0    Worker Thread    libmfxsw64.dll thread    libmfxsw64.dll!000000018001a62b    Normal
Not Flagged        4504    0    Worker Thread    libmfxsw64.dll thread    libmfxsw64.dll!000000018001a811    Normal

After this, the encoder SyncOperation(syncPoint, 60000) returns either a MFX_WRN_IN_EXECUTION(1) error or MFX_ERR_UNKNOWN (-1).  

I am trying to use the JPEG Compressor.  

An interesting point is that I've found is that my content (720x368) needs to have the width/height adjusted as shown in the example code to make it aligned.  I've done this, but still get the problem in the sync operation.  

For the encoder surfaces, I'm using ippiMalloc_8u_C2(width, height *2, &stride) since my content is 422 YUY2.  If my content is  RGB4, then I call ippiMalloc_8u_C4(width, height * 4, &stride);  (width=736, height=384).

I have content that is already aligned (1920x1080) and there is no problems with the algorithm or the sync operation.

Any suggestions on how to debug this since the exception is coming from within the Media SDK?

[Darrell_S_]

As a follow up to the exception that I'm seeing from the SyncOperation:

First-chance exception at 0x0000000180312AD0 (libmfxsw64.dll) in EncoderTest.exe: 0xC0000005: Access violation reading location 0x000000031DA27000.

This is the disassembly of where the problem is occurring

0000000180312A8A  mov         rax,r11  
0000000180312A8D  vmovdqu     xmm0,xmmword ptr [r10]  
0000000180312A92  vmovdqu     xmmword ptr [r11],xmm0  
0000000180312A97  and         rax,0Fh  
0000000180312A9B  sub         rax,10h  
0000000180312A9F  sub         r10,rax  
0000000180312AA2  sub         r11,rax  
0000000180312AA5  add         rbx,rax  
0000000180312AA8  sub         rbx,40h  
0000000180312AAC  jl          0000000180312AEA  
0000000180312AAE  vmovdqu     xmm0,xmmword ptr [r10]  
0000000180312AB3  vmovdqa     xmmword ptr [r11],xmm0  
0000000180312AB8  vmovdqu     xmm0,xmmword ptr [r10+10h]  
0000000180312ABE  vmovdqa     xmmword ptr [r11+10h],xmm0  
0000000180312AC4  vmovdqu     xmm0,xmmword ptr [r10+20h]  
0000000180312ACA  vmovdqa     xmmword ptr [r11+20h],xmm0  
0000000180312AD0  vmovdqu     xmm0,xmmword ptr [r10+30h]      <--- exception is here
0000000180312AD6  add         r10,40h  
0000000180312ADA  vmovdqa     xmmword ptr [r11+30h],xmm0  
0000000180312AE0  add         r11,40h  
0000000180312AE4  sub         rbx,40h  
0000000180312AE8  jge         0000000180312AAE  
0000000180312AEA  add         rbx,40h  
0000000180312AEE  je          0000000180312B1F  
0000000180312AF0  test        rbx,20h  


The value for the registers are:

RAX = 0000000000000B40 
RBX = 00000000000006C0 
RCX = 0000000000000168 
RDX = 000000031D8A2FC0 
RSI = 0000000000000168 
RDI = 000000031DA26B90 
R8  = 0000000000000B40 
R9  = 0000000000000001 
R10 = 000000031DA26FD0 
R11 = 000000031D8A3400 
R12 = 0000000000000000 
R13 = 0000000000000002 
R14 = 0000000000000008 
R15 = 0000000000000018 
RIP = 0000000180312AD0 
RSP = 0000000311D7F900 
RBP = 000000000000002D 
EFL = 00010206 

The output bitstream going into the Encode Frame Async function is as follows and if I do the math : data+maxlength = 0x313c04040 + 0x652a70 = 0x314256ab0 so I’m guessing that this is not within the Access violation area listed above (0x31DA27000).  This is making me guess that the output data that I allocated with ippiMalloc_8u is not the cause of the  problem.

-mfxOutputBitStream
{EncryptedData=0x0000000000000000 <NULL> ExtParam=0x0000000000000000 {???} NumExtParam=0x0000 ...}mfxBitstream
EncryptedData 0x0000000000000000 <NULL>_mfxEncryptedData *
+ExtParam     0x0000000000000000 {???}mfxExtBuffer * *
NumExtParam   0x0000             unsigned short
+reserved     0x0000000311579030 {0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000}unsigned int[0x00000006]
DecodeTimeStamp    0x0000000000000000  __int64
TimeStamp     0x0000000000000000       unsigned __int64
+Data         0x0000000313c04040 "ÿØÿà"unsigned char *
DataOffset    0x00000000               unsigned int
DataLength    0x00000000               unsigned int
MaxLength     0x00652a70               unsigned int
PicStruct     0x0000                   unsigned short
FrameType     0x0000                   unsigned short
DataFlag      0x0000                   unsigned short
reserved2     0x0000                   unsigned short


The input surface looks like this – and it too doesn’t seem to be in the area of the exception.

-surface  0x0000000312d303c0 {reserved=0x0000000312d303c0 {0x00000000, 0x00000000, 0x00000000, 0x00000000} Info=...}mfxFrameSurface1 *
+reserved    0x0000000312d303c0 {0x00000000, 0x00000000, 0x00000000, 0x00000000}unsigned int[0x00000004]
-Info        {reserved=0x0000000312d303d0 {0x00000000, 0x00000000, 0x00000000, 0x00000000} reserved4=0x0000 BitDepthLuma=...}   mfxFrameInfo
+reserved      0x0000000312d303d0 {0x00000000, 0x00000000, 0x00000000, 0x00000000}unsigned int[0x00000004]
reserved4      0x0000     unsigned short
BitDepthLuma   0x0000     unsigned short
BitDepthChroma 0x0000     unsigned short
Shift          0x0000     unsigned short
+FrameId      {TemporalId=0x0000 PriorityId=0x0000 DependencyId=0x0000 ...}mfxFrameId
FourCC         0x32595559 unsigned int
Width          0x02d0     unsigned short
Height         0x0170     unsigned short
CropX          0x0000     unsigned short
CropY          0x0000     unsigned short
CropW          0x02d0     unsigned short
CropH          0x0170     unsigned short
FrameRateExtN  0x00005f90 unsigned int
FrameRateExtD  0x00000eaa unsigned int
reserved3      0x0000     unsigned short
AspectRatioW   0x0000     unsigned short
AspectRatioH   0x0000     unsigned short
PicStruct      0x0001     unsigned short
ChromaFormat   0x0002     unsigned short
reserved2      0x0000     unsigned short
-Data         {reserved=0x0000000312d30418 {0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, ...} ...}mfxFrameData
+reserved     0x0000000312d30418 {0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000}unsigned int[0x00000007]
reserved1     0x0000      unsigned short
PitchHigh     0x0000      unsigned short
TimeStamp     0x0000000000000000 unsigned __int64
FrameOrder    0x00000000  unsigned int
Locked        0x0001      unsigned short
Pitch         0x0b80      unsigned short
PitchLow      0x0b80      unsigned short
+Y            0x00000003137ac080 "%z%‚%|$„%|%‚%~%‚$}$$#$}$%~&‚'~'‚'~'‚'~'€(}((~'&}&€'~&‚&|&‚&|&‚&|&‚&{&‚&{%‚&{&ƒ'|'„'}(ƒ(|(„(|(„)|)ƒ)|)ƒ)|)„)|)„)|)„){(…)z+†-z.…-z,‡,{-‡.{/‡0z0†0y1†1x2†2x2‡2x2‡2x1‡1x1‡0y0‡0y0‡.y.‡.y,‡*y*†*y,…-z-ƒ...unsigned char *

+U
0x00000003137ac081 "z%‚%|$„%|%‚%~%‚$}$$#$}$%~&‚'~'‚'~'‚'~'€(}((~'&}&€'~&‚&|&‚&|&‚&|&‚&{&‚&{%‚&{&ƒ'|'„'}(ƒ(|(„(|(„)|)ƒ)|)ƒ)|)„)|)„)|)„){(…)z+†-z.…-z,‡,{-‡.{/‡0z0†0y1†1x2†2x2‡2x2‡2x1‡1x1‡0y0‡0y0‡.y.‡.y,‡*y*†*y,…-z-ƒ-...unsigned char *
+V
0x00000003137ac082 "%‚%|$„%|%‚%~%‚$}$$#$}$%~&‚'~'‚'~'‚'~'€(}((~'&}&€'~&‚&|&‚&|&‚&|&‚&{&‚&{%‚&{&ƒ'|'„'}(ƒ(|(„(|(„)|)ƒ)|)ƒ)|)„)|)„)|)„){(…)z+†-z.…-z,‡,{-‡.{/‡0z0†0y1†1x2†2x2‡2x2‡2x1‡1x1‡0y0‡0y0‡.y.‡.y,‡*y*†*y,…-z-ƒ-z...unsigned char *
+A
0x00000003137ac083 "‚%|$„%|%‚%~%‚$}$$#$}$%~&‚'~'‚'~'‚'~'€(}((~'&}&€'~&‚&|&‚&|&‚&|&‚&{&‚&{%‚&{&ƒ'|'„'}(ƒ(|(„(|(„)|)ƒ)|)ƒ)|)„)|)„)|)„){(…)z+†-z.…-z,‡,{-‡.{/‡0z0†0y1†1x2†2x2‡2x2‡2x1‡1x1‡0y0‡0y0‡.y.‡.y,‡*y*†*y,…-z-ƒ-z/...unsigned char *
MemId     0x0000000000000000  void *
Corrupted 0x0000              unsigned short
DataFlag  0x0000              unsigned short

Any suggestion of the root cause would be appreciated.

Thanks

[OTorg]

IMSDK 1.11 (part of INDE 2015).
Exceptions problem isn't solved...

[OTorg]

IMSDK 1.15 (INDE 2015 Update 2).
Still no improvements :((