decoder bugs

[madshi_net]

Using MSDK 3.0 beta 2 I've run into the following two bugs:

(1) The "DecodeHeader" API returns MFX_ERR_MORE_DATA when being called with the following h264 header sequence:

00 00 00 01 67 64 00 33 AC 34 E6 02 C0 49 FB 84 00 00 0F A0 00 03 0D 42 3C 60 C6 68
00 00 00 01 68 EE BC B0

However, if I just add an AUD (00 00 00 01 09 10) to the end of that data, "DecodeHeader" is happy to do its job. Your own "MP4/AVC Decode using the Intel Media SDK" article/demo stumbles over the same problem. In that demo DecodeHeader always fails.

(2) After seeking in a VC-1 video stream, the very first frame output by your (software) decoder often contains block artifacts. The blocks are usually in areas where there's strong motion. The very next frame is clean. It's only the first frame which shows these artifacts. And it doesn't always occur, but very often. It occurs with and without "injected headers". If you need a sample to reproduce this problem, just let me know. I've seen the same problem with 2 very different files, though (one SD interlaced, the other Blu-Ray 24p), so I'm pretty sure it will happen with all files.

Here's a screenshot of the problem:

[madshi_net]

Unfortunately it's not as easy as that. After a seek, DirectShow splitters provide preroll frames with negative timestamps and since MSDK doesn't tell me which frame is an I-frame and which is not I don't really know at which decoded frame exactly to start with presentation timestamp 0. If I simply guess, audio/video sync could be off by more than 1 frame. One proper solution to this problem would be to trust in I-frame timestamps and interpolate the timestamps for B- and P-frames, which is what the well known open source DirectShow decoder filter "ffdshow" does. But that's only possible if I know which decoded frame is an I-frame and which is not. libav/ffmpeg tells me that, the MSDK does not.

[Markus_Pingel]

For seeking, the DirectShow splitter needs the container time stamp. Does seeking work correct on a stream with bogus time stamps? If it does, the first frame to send to the renderer is the one with timestamp greater or equal 0. It does not need to be an I-frame.

[madshi_net]

Markus, I appreciate that you're trying to help. But you seem to misunderstand the purpose of my thread. I'm not looking for advice/help here. I know the problem, I know what possible solutions there are. Everything you told me so far I already knew before. The purpose of this thread is to give feedback to Intel, so that they can improve the MSDK.

In case you want to know: Your latest suggestion would work "ok", but not really well. If the timestamps are borked, then the first timestamp >= 0 could already be a bogus one. If you sync presentation start to the first decoded frame with a >= 0 timestamp, regardless of whether it's an I-, P- und B-frame, audio/video sync will be all over the place after every seek. If you sync presentation only to I-frames, there might still be a certain audio/video desync, but at least it should be constant, during playback and after every seek. Which is crucial to the end user.

[Markus_Pingel]

I understand the purpose of the thread. I just wanted to make clear that maybe except from VC-1, there is no decoder bug and the timestamp handling of MSDK is correct. All fixes/workarounds of wrong containers is a matter of the shim around it. We are already using MSDK 2in production for decoding ts and mov containers and seeking works even for damaged files (like ts from interrupted DVB).

[madshi_net]

The MSDK timestamp handling works as intended, as long as the container timestamps are correct. All I'm saying is that in certain situations there's a need to work around badly muxed video files and the MSDK doesn't make that as easy as libav/ffmpeg does. I think it's not a bad idea for the MSDK to learn from libav/ffmpeg, after all libav/ffmpeg is the defacto standard decoding library, used by the broad majority of media related software. What makes timestamp fixing so much easier with libav/ffmpeg is that libav/ffmpeg reports which decoded frame is which type (I, P, B, ...), which the MSDK does not seem to do yet. So my main suggestion was for the MSDK to also report which decoded frame is which type.

BTW, the TS files with bad timestamps are not "damaged". They are just muxed with bad timestamps. Playing damaged/corrupted files is a completely different thing compared to playing files with consistently bad container timestamps.

[IDZ_A_Intel]

The MSDK decode/encodestages generallyjust passtimestampsalong, as mentioned previously.VPP frame rate conversion might alter some timestamps, but there are currently no plans to support any other form of interpolation or cleanup.

Theidea behind the design, which we've verified with the MSDK architect,is that timestampprocessing should be at thecontainer level.

In terms of the DirectShow filters this putsa lot of responsibilityon the splitter.We are currently discussing plans to improve the sample splitters for cases like those mentioned in this thread. This may also include distributing the source code so that things like custom timestamp cleanup algorithms can be added. Unfortunately we can't make any guarantees about the contents of future releases, or when these changesmight be available, but this does give us a lot of good things to consider.

The forum is definitely a great way to provide feedback.Conversations like this are very helpful as we work toward a better product and better documentation.

Regards,

Jeff

[madshi_net]

@Jeff, thanks for your reply.

I think it's a good concept to have the MSDK just pass timestamps along. I also agree that in an ideal world the splitter would take care of fixing broken timestamps. Unfortunately I'm not aware of a single existing DirectShow splitter filter which fixes broken timestamps. In real life fixing broken timestamps is either not done at all, or it's done by the decoder filter. E.g. the most important open source video decoders "ffdshow", "DScaler" and "LAV Video Decoder" are all doing timestamp fixing internally, but none of the splitters (neither open source nor commercial) do that, as far as I can tell.

Practically that means that a DirectShow decoder filter (regardless of whether it's using the MSDK or not) must be able to internally fix the timestamps, if it wants to play problematic video files smoothly with the existing DirectShow splitter filters. That doesn't mean that the MSDK has to do any sort of timestamp fixing itself. It just needs to provide the developer with enough information to perform the necessary fixes. I think reporting which decoded frame is which type (I, P, B) would already do the job.

[IDZ_A_Intel]

You've provided some excellent reasons to consider providing frame type as an output from the decoders. We're exploring options we might have to include this feature in the future. This information could potentially be useful in a lot of cases beyond fixing timestamps.

Unfortunately we can't make any guarantees, but we'll do our best to make sure youruse case isrepresented.

In the meantime, how significantly islack offrame typeinformation affecting your projects? If this needs more attention in the short term we'll be glad to help with finding other workarounds.

Thanks for bringing this up.

Jeff

[madshi_net]

Thanks Jeff, for the excellent support.

In the short term I can live with the way things are.

[Wilczynski__Mark]

Sorry to bump this old thread but I seem to have run into a similar problem with 1080i VC1 content. Using the sample_decode.exe application (SW decode on Win7 x64) from "MSDK-2012_pu_3.0.015_R2", I get strange block artifacts at the start of playback (see attached screenshot). This is a sample form the popular HQV Benchmark Blu-Ray so I assume there are no encoding errors. It also shows up correctly with other decoders I have tried. Here is a short clip for you to test:http://www.filefactory.com/file/151ea2u29gkb/n/FilmRes_vc1 (select slow download at the bottom of page).

-Mark

[IDZ_A_Intel]

Hi Mark,

We were able to reproduce the issue but as you point out the artifacts are only visible during SW decode. The content decodes fine using recent graphics drivers.

Thanks for reporting the issue, we will work on fixing this SW decode bug as soon as possible.

Regards,

Petter