Quicksync AVC H.264 vs Quicktime player

[RyanS]

When I use quicksync to encode to an h.264 file, then mux to mp4 using mp4box, Quicktime (Windows version) generally will not play the file correctly. Other players play it fine. The mux is apparently not the problem because the same mux command works on h.264 files from another h.264 encoder. I am encoding using a modified sample_encode from the media sdk examples. I am on a Core i7 3770k (HD 4000), Windows 7 Pro SP1 x64, with latest graphics driver (8.15.10.2761).

It doesn't matter whether I use hw or sw encode. I have tried baseline, main, and high profiles.

I can make it more or less work by setting m_mfxEncParams.mfx.GopPicSize = 1. However even in this case Quicktime comes up showing a green or black first frame, which it doesn't do normally, but at least in that case when it starts playing it shows good frames. When I set GopPicSize to higher numbers, Quicktime seems to show about that many green frames before showing good frames. 

I can't believe that Quicktime doesn't support B frames. Is there some other parameter I am missing?

[Petter_L_Intel]

Hi, Mp4box is likely not compatible with regards to creating containers that Quicktime accepts. At least not with the default set of muxing parameters. You could try FFmpeg instead, which is known to produce containers that can be played by Quicktime. Regards, Petter

[Petter_L_Intel]

Hi Ryan, can you share your H.264 stream so that we can explore the issue on our side? Regards, Petter

[RyanS]

Here is the quicksync h.264 file.

[Petter_L_Intel]

Hi Ryan, I see the same thing for Quicktime when using command line ffmpeg muxing of H.264 streams. I 'm pretty sure the issue is related to timestamp handling. Since no meaningful timestamps are provided, FFmpeg will estimate PTS and DTS. Quicktime is probably confused with the generated time stamp values, while VLC is more intelligent and adjust accordingly. For a real-life muxing implementation PTS and DTS should be set accordingly. There is a white paper detailing how to use FFmpeg to mux streams from Media SDK into containers. http://software.intel.com/en-us/articles/integrating-intel-media-sdk-with-ffmpeg-for-muxdemuxing-and-audio-encodedecode-usages But unfortunately, this paper does not go into detail on how to implement PTS and DTS handling, instead a rough shortcut is taken, leading to the same results as when using command line FFmpeg or mp4box muxing. If you're interested I can send you a hacked version of the src code from the paper that does a better (does not cover all scenarios) job at setting PTS and DTS values. Regards, Petter

[RyanS]

Yes, please, I would like to try the hacked source you mentioned. I did notice that ffmpeg complains about missing PTS on every frame ("pts has no value"). I did try to hack in timestamps, using constant incrementing 90KHz units into mfxBitstream.TimeStamp just before CSmplBitstreamWriter::WriteNextFrame. And also by setting the same timestamp into mfxFrameSurface1.Data.TimeStamp. But neither of those fixed the problem. I did not try to account for frame reordering. mp4box has an option to dump pts values. (mp4box -std -dts fq.mp4) I do not see any substantive difference between a working and non-working mp4 file. Can you recommend any other (inexpensive) tool to give visibility into the h.264 stream to look for differences? I think I need to see frame type and dts/pts. Thanks, Ryan

[Petter_L_Intel]

Yes, the frame reordering, for the case when B-frames is encoded, has an impact on PTS and DTS handling. Timestamp dumping from FFmpeg or mp4box should be enough, but I have not looked at it in details recently. Regards, Petter

[RyanS]

In CSmplBitstreamWriter::WriteNextFrame I printed the TimeStamp of each mfxBitstream as it is written and found that the frames are written in decode order (not presentation order). The decode order looks good (B frames come after the future frames they reference). Then I rendered frame numbers onto the frames themselves, and now I see that after ffmpeg mux Quicktime is presenting the frames in the order in which they are in the h.264 stream (file), which is the decode order, not the presentation order. Hence the temporal jitter. So it seems like the presentation order is represented in the h.264 stream in a way that works for other decoders but not a way that Quicktime honors (after muxing by ffmpeg).

[RyanS]

Ffmpeg doesn't quite work either. The resulting mp4 file plays back in quicktime player with a severe time stutter for the quicksync h.264 input (not for the x264 h.264 input). It looks like it plays 5 frames them jumps back a couple frames, then maybe forward again several frames. It makes forward progress through the clip at about the right rate, but is jumping forwards and backwards, maybe due to B frame decode order. I'm using: ffmpeg -i q.264 -vcodec copy fq.mp4 Am I missing mfxExtPictureTimingSEI or something like that? Thanks, Ryan EDIT: Correction, Quicktime also plays the x264-produced stream in decode order, not presentation order. It is just less noticeable because B frames are more rare with default settings.

[RyanS]

Note my correction above. The x264/ffmpeg file also plays in the incorrect order in quicktime. So the frame order issue is not a media sdk sample problem (or at least not exclusively a media sdk sample problem). It is most likely an ffmpeg problem. I am using the Oct 10 2012 build of ffmpeg. So for quicktime playback, the ffmpeg mux doesn't work due to B-frame decode vs presentation order problem for both the media sdk sample encoded stream and the x264 stream. And mp4box mux works for x264 streams but not media sdk streams.

[RyanS]

This may be the relevant ffmpeg bug: https://ffmpeg.org/trac/ffmpeg/ticket/502

[RyanS]

(In this post I am disregarding your integrated ffmpeg mux, I am still just dealing with my code that writes a raw h.264 file.) There seem to be several things I don't know about dts/pts handling. You are setting TimeStamp values to successive integers, so I tried that too. But ffmpeg -dump -debug_ts still shows 90KHz dts/pts values. Where are the 90KHz values coming from? Note that ffmpeg displays very similar dts/pts values, and also the "pts has no value" message, for the x264 mux and that one plays correctly in quicktime. Ryan EDIT: Correction, the x264 version also plays incorrectly. So the timestamps are not the problem.

[Petter_L_Intel]

Hi Ryan, Using the hack I provided which sets Quicktime required DTS and PTS I do not see any jitter issues. We have not encountered any FFmpeg bugs related to this usage. The Media SDK encoder output is definitely in decoder order, thus the monotonically increasing DTS. PTS timestamps will be out of order in case B-frames are encoded. Regards, Petter

[RyanS]

Which ffmpeg version are you using? Can you provide a sample (working) mp4 that I can use for comparison?

[Petter_L_Intel]

Hi, I used the FFmpeg version that is refereed to in the white paper: build "2012-08-27" of FFmpeg from: http://ffmpeg.zeranoe.com/builds/ I'll send you encoded mp4 file in a private message. Regards, Petter

[RyanS]

EDIT: One confusion is talking about ffmpeg.exe vs the libs associated with it (libavformat etc). Another is using ffprobe to look at packets vs frames. This post is looking at ffprobe frames display, not packets. Second edit: fixed the explanation below. The command-line ffmpeg.exe (as opposed to your code that uses the libs that compose ffmpeg) does not mux correctly for quicktime. When looking at frames using ffprobe, it looks like ffmpeg.exe sets PTS to the decode order, which is not correct. I tried 3 versions of ffmpeg.exe and they all give different PTS/DTS values, but the PTS values are always in decode order, so it never works for quicktime. Your sample code produces an mp4 that plays correctly in quicktime. Except quicktime swaps frames 2 and 3, maybe due to both having DTS=0? I noticed that for the other two muxers (ffmpeg.exe and mp4box), when looking at frames using ffprobe -show_frames, the dts is always greater than the pts, but not in your mux.

[Petter_L_Intel]

Hi Ryan, yes, FFmpeg and the other command line tools you used clearly does not set PTS/DTS correctly, thus playback issues in Quicktime. Not much to say about this fact. For a real life implementation/integration you have control of the timestamps for each frame yourself and therefore the ability to set PTS/DTS appropriately, similar to the simplified code I provided. Regards, Petter

[jaybo_nomad]

It looks like the behavior of the encoder has changed in the 2013 SDK release.  The 2013 encoder seems to make QuickTime happy when dealing with B frames by reordering the output stream into display order. 

2012 encoder ordering

pkt_pts_time=0.000000

pict_type=I

pkt_pts_time=0.066625

pict_type=B

pkt_pts_time=0.099937

pict_type=B

pkt_pts_time=0.033313

pict_type=P

pkt_pts_time=0.166563

pict_type=B

pkt_pts_time=0.199875

pict_type=B

pkt_pts_time=0.133250

pict_type=P

2013 Ordering

pkt_pts_time=0.033313

pict_type=I

pkt_pts_time=0.066625

pict_type=B

pkt_pts_time=0.099958

pict_type=P

pkt_pts_time=0.133292

pict_type=B

pkt_pts_time=0.166625

pict_type=P

pkt_pts_time=0.199958

pict_type=B

pkt_pts_time=0.233292

pict_type=P

[Petter_L_Intel]

Please refer to the following post for a response.

http://software.intel.com/en-us/forums/topic/311726