Media (Intel® Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools like Intel® oneAPI Video Processing Library and Intel® Media SDK
Announcements
The Intel Media SDK project is no longer active. For continued support and access to new features, Intel Media SDK users are encouraged to read the transition guide on upgrading from Intel® Media SDK to Intel® Video Processing Library (VPL), and to move to VPL as soon as possible.
For more information, see the VPL website.

Timestamp issues in h264 decoding

Nicolas_S_
Beginner
1,040 Views

Hi,

I am using Intel Media Server Studio 2015 – Graphics Driver, version 16.4.2.39163 and Intel Media Server Studio 2015 – SDK, version 6.0.16043166.166,  running on CentOS 7.0 with a Intel(R) Core(TM) i7-4770R.

I am decoding a h264 stream, but the decoded output pts are sometimes decreasing, which messes up with my muxer. My program is based on the simple_decode tutorial, and I added the decode and presentation timestamps in the bitstream:

mfxStatus PushBitStreamData(mfxBitstream* pBS, Packet* pkt)
{
    if(pBS->DataOffset) {
        memmove(pBS->Data, pBS->Data + pBS->DataOffset, pBS->DataLength);
        pBS->DataOffset = 0;
    }                                                                                                                 
    memcpy(pBS->Data + pBS->DataLength, pkt->data, pkt->size);               
    pBS->DataLength += pkt->size;        

    pBS->DecodeTimeStamp = pkt->dts != AV_NOPTS_VALUE ? pkt->dts : pkt->pts;
    pBS->TimeStamp = pkt->pts;
}

And decoding:

while(1){
    ctx->sts = ctx->mfxDEC->DecodeFrameAsync(&ctx->mfxBS, ctx->pmfxSurfaces[nIndex], &pmfxOutSurface, &syncp);
    if(ctx->sts == MFX_WRN_VIDEO_PARAM_CHANGED)
        continue;

    if(ctx->sts == MFX_WRN_DEVICE_BUSY){
        usleep(800);
        continue;
    }
    break;
}

This produces the following output:

Packet:: pts:5239528686 dts:5239521486
Packet:: pts:5239525086 dts:5239525086
Packet:: pts:5239532286 dts:5239528686
Packet:: pts:5239550286 dts:5239532286
Packet:: pts:5239543086 dts:5239535886
Packet:: pts:5239539486 dts:5239539486
Packet:: pts:5239546686 dts:5239543086
Decoded:: frame order:0 timestamp:5239528686
Packet:: pts:5239564686 dts:5239546686
Decoded:: frame order:1 timestamp:5239535886
Packet:: pts:5239557486 dts:5239550286
Decoded:: frame order:2 timestamp:5239525086
Packet:: pts:5239553886 dts:5239553886
Decoded:: frame order:3 timestamp:5239535886
Packet:: pts:5239561086 dts:5239557486
Decoded:: frame order:4 timestamp:5239543086
Packet:: pts:5239579086 dts:5239561086
Decoded:: frame order:5 timestamp:5239550286
Packet:: pts:5239571886 dts:5239564686
Decoded:: frame order:6 timestamp:5239546686

Where the frame 2 has lower pts than the frame 1, this behaviour is shown in many other frames (ie. 5 to 6). The output is much longer, those are the first lines.

 

0 Kudos
5 Replies
Jeffrey_M_Intel1
Employee
1,040 Views

For Media SDK, except for the exceptions of frame rate conversion and deinterlace timestamps are simply passed along from decode to surfaces to encode.  There are no guarantees that an input stream will have rational timestamps. However, your application can play an active role in "cleaning" them where necessary.  This is up to your application and out of Media SDK's scope.  A simplistic starting strategy could be to take a starting value from the stream but replace the MFXFrameData TimeStamp (PTS) values before the surface goes to the encoder with a value your application calculates based on the specified FPS.  

Regards, Jeff

0 Kudos
Nicolas_S_
Beginner
1,040 Views

Hi Jeff, thanks for your comment.

My application takes an input stream which has video and audio and It is possible some frames are lost before the application gets them (network problems for example). So, I cannot calculate the TimeStamp based on the FPS, for example if the original frames are 1 2 3 and frame 2 is lost and I only receive frames 1 3, I must keep the original TimeStamp for both frames so I don't end up setting PTS 2 to frame 3, which in the long run will make the video not synchronized with the audio.

When I receive a packet with PTS and DTS, I want to send it to Intel, so when that packet is decoded the raw frame has the correct PTS.

I tried setting the timestamp directly to the surface 

ctx->pmfxSurfaces[nIndex]->Data.TimeStamp = packet->dts == NOPTS ? packet->pts : packet->dts;

just after calling DecodeFrameAsync(), and it looks like it works better, but I think the correct way of setting the TimeStamp was setting it to the Bitstream.

With the streams I am working, the input streams are fine and have correct PTS and DTS and when decoding by other means (like FFMpeg) the raw frame decoded has the correct PTS and the PTS is always greater than the PTS before.

So, how should I set the TimeStamps to intel so I can keep the original TimeStamps?

Thanks.

 

0 Kudos
Jeffrey_M_Intel1
Employee
1,040 Views

If you're getting timestamps from the container, the model Media SDK is designed around looks like this:

  1. demux, get packet timestamp
  2. put value in bitstream's Timestamp
  3. passed to decode output surface
  4. passed through frame processing steps (may need to be adjusted for FRC/DI)
  5. surface with passed timestamp goes to encoder
  6. timestamp value can be read from output bitstream and fed to muxer

For an example of how timestamps go through a Media SDK pipeline you can look at the simple_6_transcode_opaque_lowlatency tutorial.  In this case timestamps are artificial values used for tracking but the same idea could be used for working with timestamps from containers.

Is this example useful for your case?

 

0 Kudos
Nicolas_S_
Beginner
1,040 Views

I saw the example.

There is a difference which I think is the main problem, my input stream usually has I P and B frames, so I don't have monotonic increasing PTSs, but I have monotonic increasing DTSs. In the example, it assigns always monotonic increasing PTSs.

But, it was useful anyway, I saw it was setting MFX_BITSTREAM_COMPLETE_FRAME to the bitstream, I did some reading about it, and looks like I need it in cases like mine (with h264 with B frames), and so far so good, it is working as expected.

Thanks Jeffrey, I will post if I have any more news

0 Kudos
Nicolas_S_
Beginner
1,040 Views

I didn't check it correctly, now I'm getting corrupted frames. As the reference manual reads:

If this flag is set, but the bitstream buffer contains incomplete frame or pair of field, then decoder will produce corrupted output.

Now I'll see if I can provide the bitstream with complete frames or if there is any other way of getting the correct PTSs.

 

0 Kudos
Reply