Mpeg4 decoding crash in 5.3.3 Dll:s

[nipp]

We suddenly encounterd crashes in the 5.3.3 for cameras that not always produce clean code(it seems anyway). When heavily loaded it creates garbage pixels.

And we cannot control how endusers use the cameras or which loads they have on each camera.

Most of the time error images will be discarded with help of error return codes from Ipp and we reinitialize decoder and it runs fine again.

But with crashes right out with win32 exceptions we are lost.

We run UMC classes and thread is max 1(due to known problem in Ipp 5 where cpu can suddenly go 100%)

This is a major problem since we only have Ipp to do the parsing and application cannot under any circumstances crash.

Crash is i one of the threads created by Ipp itself and thereby no c++ exceptions are catched and you end up in Windows!!!!!!

Normal access violation 0xc00000005 is what we get.

Very often it's in mp4_DecodeVop_P and similar routines like mp4_DecodeIntraMB.

Is this a known issue?

Is there a way to catch all exceptions in a controlled manner from Ipp?

Normal try-catch does not work as mentioned.

We run native windows C++.

We would really appreciate some guidance here how to proceed.

If any more info is needed we will be glad to supply it.

Thanks.

[Vladimir_Dudnik]

Hello,

thanks for reporting the issue. Did you already try IPP 5.3.4 ? If this does not fix the issue could you please provide us sample test stream which cause the issue. This will help us to reproduce and investigate the problem

Regards,
Vladimir

[nipp]

Quoting - vdudnik

Hello,

thanks for reporting the issue. Did you already try IPP 5.3.4 ? If this does not fix the issue could you please provide us sample test stream which cause the issue. This will help us to reproduce and investigate the problem

Regards,
Vladimir

Thanks.

We have collected video that should cover such krasch incidents and we try to recreate crash with these logged video as well. That would really narrow it down and make it easy for you to look closer at this.

No, we have not tried 5.3.4 yet, but am about to. A little reluctant because update from 5.3.2 to 5.3.3 created some memory leaks due to different treatment of repeated initto VideoRenderer. It's a little bit scary to do updates and shipwithout fully testing every aspect again.

If there are any tips how to capture any deep exceptions inside Ipp it would be swell?

Will be back with sample code and video hopefully.

Best regards

Lars

[nipp]

Quoting - vdudnik

Hello,

thanks for reporting the issue. Did you already try IPP 5.3.4 ? If this does not fix the issue could you please provide us sample test stream which cause the issue. This will help us to reproduce and investigate the problem

Regards,
Vladimir

Now version 5.3.4 is tested with crashes the same way as 5.3.3.

Part of callstack for xxxServer.exe which is our app:
ippvct7-5.3.dll!01bdd2ec()
[Frames below may be incorrect and/or missing, no symbols loaded for ippvct7-5.3.dll]
ippvct7-5.3.dll!01b73bf5()
xxxServer.exe!_mp4_DecodeVOP_P() + 0x4dbb bytesC
xxxServer.exe!_mp4_DecodeVideoObjectPlane() + 0x4f7 bytesC
xxxServer.exe!UMC::MPEG4VideoDecoder::GetFrame() + 0x484 bytesC++
>xxxServer.exe!CDecoderIppMpeg4::Decode(unsigned char * pcPacket=0x0905d040, unsigned long dwPacketLen=3982, UMC::FrameType ft=P_PICTURE, tagBITMAPINFOHEADER * pbih=0x01f932d0, bool bDoCopy=true) Line 263C++
xxxServer.exe!CImg::ReadMPEG4(unsigned char * pPicData=0x0905d040, unsigned long dwDataSize=3982, unsigned long dwPicNo=7876, unsigned long dwSubNo=12, bool bDoCopy=true) Line 454 + 0x2e bytesC++
xxxServer.exe!CImg::ReadMPEG4(CPicMPEG4 * pPic=0x08b43a20, bool bDoCopy=true) Line 356 + 0x4c bytesC++
xxxServer.exe!CImg::Decompress(CPic * pPic=0x08b43a20) Line 337 + 0xa bytesC++

Further....

We discovered comparing source code from umc_mpeg4_video_decoder.cpp with earlier release that a lot of interesting addons are made in 5.3.4.

Ability to enable

m_decInfo->stopOnErr = 1;

m_decInfo->strictSyntaxCheck = 1;

like this when building libs might give some more useful info?

#1. Can you guide is in this strictSyntax thing?

#2. Best way to use a logging feature from mp4_Error for your sake?

a) your vm_debug_trace is providing everything needed?

b) add an own externalfunction for a extended logg to a file together with something more, like the mpeg4 frame?

Tip of best way to do this would be great.

#3. We also see that code for H264 has plenty try-catch blocks to catch exceptions, but mpeg4 code is not?

Is there a remake of mpeg4 classes coming that might be able to catch exceptions better, and not wind up in Windows?

#4. I'd like to repeat question from before about catching all exception from whatever threads that Ipp is starting on it's own. Can you please guide in doing this?

As always -***** happends - and handling exceptions in a controlled manner is mandatory I would say.

#5. Are all threads created by Ipp about to get exception treatment and ability to catch such events from user applications?

This issue is of absolute vital importance to us since there are a lot of our installations around the world using Ipp now and some are not runningat all until this is solved(those using certain Mpeg4 camera brands).

p.s. we have compiled Ipp libs with Intel compilerto get the same binary code as you do.

With a hope of a quick reply.

Best regards

Lars

[nipp]

Is there an answer to the last posted questions?

[nipp]

Quoting - vdudnik

Hello,

thanks for reporting the issue. Did you already try IPP 5.3.4 ? If this does not fix the issue could you please provide us sample test stream which cause the issue. This will help us to reproduce and investigate the problem

Regards,
Vladimir

As the main issue is shifted from eliminating crash to be able to handle crashes with Ipp as failing link it would be great to have some assistance in doing this.

Cameras from allover may produce garbage of all kinds from time to time and ability to handle it and recover seems like the best approach.

We have not been able to collect code from main application down to a level that we can ship to you.

Can you help us, or is the fact that no reply in three weeks will mean no further help ever?

[Vladimir_Dudnik]

Somehow I've overlooked this issue. I would recommend you to submit bug reports to Intel Premier Support where they are tracked on regular base

Vladimir

[nipp]

Quoting - fc

Hi Lars,

MPEG-4 frame can consists of VideoPackets (when resync_marker_disable = 0). If m_decInfo->stopOnErr = 0 than decoder tries to find next videopacket if error was detected in current videopacket

When m_decInfo->strictSyntaxCheck = 0 the decoder can ignore some errors in headers which don't affect on decoding process.

In Ipp 6.0 MPEG-4 decoder uses try-catch. BTW could you use your own try-catch when calling GetFrame()?

try {

mp4Dec->GetFrame(...);

}

catch (...) {

...

}

One thing you can do is next - do not pass bitstream directly to GetFrame() but copy bitstream from camera in temporal buffer and place at the and four zero bytes. In this case the decoder should not read after the buffer. If bitstream for current frame will be erroneous and decoder will reach the end of buffer it will return error

Nikolay

Thanks.

We have crowded our own code with try-catch blocks but it still ended up in Windows as unhandled exception. Looking at a memory dump carefully it was because the crash was in code in another thread created by Ipp.

Explanation of strict syntax was good. Thank you.

Best regards

Lars

[nipp]

Quoting - vdudnik

Somehow I've overlooked this issue. I would recommend you to submit bug reports to Intel Premier Support where they are tracked on regular base

Vladimir

Thanks, I never thought of that.

/Lars

[goyorl]

Quoting - nipp

Thanks, I never thought of that.

/Lars

Hi all,

We are having a crash which looks very similar to what is described in this thread. It would be very helpful to know if someone hasreached a conclusion about this. Is it an IPP bug? Has it been fixed? Do the provisional solutions above work?

Thanks for helping,

Goyo.

[Vladimir_Dudnik]

Quoting - goyorl

Hi all,

We are having a crash which looks very similar to what is described in this thread. It would be very helpful to know if someone hasreached a conclusion about this. Is it an IPP bug? Has it been fixed? Do the provisional solutions above work?

Thanks for helping,

Goyo.

Goyo,

there was suggestion provided by Nikolay (f_c), who is IPP MPEG4 and H.264codec developer, that you may need to insert 4 zero bytes at the end of buffer with bitstream you get from camera to avoid buffer overrun problem

Although, it was not proven as a bug in IPP. From the nipp message:

Looking at a memory dump carefully it was because the crash was in code in another thread created by Ipp.

it is not clear what the code and what the thread cause exception.

By the way, you need to enable C style exceptions (synchronous mode) to catch Win32 exceptions.

If you may provide test case which reproduce your issue it would help us to investigate this. Although it is question do you use the latest IPP 6.0 version?

Regards,
Vladimir

[goyorl]

Quoting - vdudnik

Goyo,

there was suggestion provided by Nikolay (f_c), who is IPP MPEG4 and H.264codec developer, that you may need to insert 4 zero bytes at the end of buffer with bitstream you get from camera to avoid buffer overrun problem

Although, it was not proven as a bug in IPP. From the nipp message:

Looking at a memory dump carefully it was because the crash was in code in another thread created by Ipp.

it is not clear what the code and what the thread cause exception.

By the way, you need to enable C style exceptions (synchronous mode) to catch Win32 exceptions.

If you may provide test case which reproduce your issue it would help us to investigate this. Although it is question do you use the latest IPP 6.0 version?

Regards,
Vladimir

Hi and thanks for the reply,

We have a core dump file of the crash, would it be helpful sending this to you? That could be useful to determine why it happens.

Regards,

Goyo.

[Vladimir_Dudnik]

I do not think crash dump file will be useful. We do not have sources of your application. If you can reproduce that issue with IPP sample and provide as test bitstream which cause crash in IPP sample then we can investigate the issue.

Vladimir

[goyorl]

Quoting - goyorl

Hi and thanks for the reply,

We have a core dump file of the crash, would it be helpful sending this to you? That could be useful to determine why it happens.

Regards,

Goyo.

Hi,

If you think it can help we can also sen pdbs...

Anyway, we have tried using a temporal buffer ended with zero bytes and the crash has repeated. I have to say that we don't know what makes the decoder crash it just happens from time to time while it is working properly, so we can not send an example that reproduces it. Any suggestion?

Thanks for helping,

Goyo.

[Vladimir_Dudnik]

We provide an example of how to use IPP media codecs. If you can reproduce the issue with IPP sample we canhelp. Otherwise it is unknown how do you use MPEG4 codec, what did you change (if you do), what things you add (if any) and so on.

If you did not change anything then it should be possible to reproduce the issue with IPP sample, isn'it?

Vladimir

[goyorl]

Hi all,

We cannot reproduce easily the crash but it is still happening from time to time. We are trying to catch the excetion as a workaround, but as mentioned in previous posts it is not working. We have enabled all the exceptions in our code in fact we usually are able to catch IPP exceptions but we can not catch this one. So the question is, How can we catch this exception?

Thanks,

Goyo.