H264 encoder behavior when frame timestamps has jumps

[Cuminato__Lucas]

Hi,

I have successfully created a transcoding pipeline using FFmpeg + Media-SDK. My UDP input stream comes in as either MPEG2 or H264 and I transcode the stream to H264. I'm also using VPPs before encoding to deinterlace, re-scale and frame rate convert the input to two different output profiles, 1280x720 and 850x480, both at 30fps. The only issue I'm having right now is avsync when I have dropped frames in the input.

My expectation was that the encoder would generate timestamps (PTS/DTS) according to the input frame given to the encoder, but that is not what I'm seeing when frames are dropped:

The "From encoder" line is duplicated because we output 2 different profiles. The timing information is in 1/framerate timebase (30fps both in and out) so it's essentially counting frames.

Aug 20 18:39:17 transcoded[30251]: Before decoder PTS: 1095407
Aug 20 18:39:17 transcoded[30251]: Out of decoder PTS: 1095405
Aug 20 18:39:17 transcoded[30251]: From encoder PTS: 1095400 DTS: 1095399
Aug 20 18:39:17 transcoded[30251]: From encoder PTS: 1095400 DTS: 1095399
Aug 20 18:39:17 transcoded[30251]: Before decoder PTS: 1095408
Aug 20 18:39:17 transcoded[30251]: Out of decoder PTS: 1095406
Aug 20 18:39:17 transcoded[30251]: From encoder PTS: 1095405 DTS: 1095400
Aug 20 18:39:17 transcoded[30251]: From encoder PTS: 1095405 DTS: 1095400
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095409
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095407
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095403 DTS: 1095401
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095403 DTS: 1095401
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095411                        <----- frame dropped here, should've been 1095410
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095408
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095402 DTS: 1095402
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095402 DTS: 1095402
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095412
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095409
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095404 DTS: 1095403
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095404 DTS: 1095403
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095413
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095411                        <----- good, expected this
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095409 DTS: 1095404
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095407 DTS: 1095405
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095409 DTS: 1095404
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095407 DTS: 1095405
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095414
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095412
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095406 DTS: 1095406
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095406 DTS: 1095406
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095415
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095413
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095408 DTS: 1095407
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095408 DTS: 1095407
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095416
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095414
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095413 DTS: 1095408
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095413 DTS: 1095408
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095417
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095415
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095411 DTS: 1095409
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095411 DTS: 1095409
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095418
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095416
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095410 DTS: 1095410   <--- bad, wasn't expecting 1095410 to be ever created
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095410 DTS: 1095410
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095419
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095417
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095412 DTS: 1095411
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095412 DTS: 1095411
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095420
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095418
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095417 DTS: 1095412
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095417 DTS: 1095412
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095421
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095418                        <--- bad, repeated value.
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095415 DTS: 1095413
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095415 DTS: 1095413
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095422
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095419
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095414 DTS: 1095414
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095414 DTS: 1095414

The more dropped frames I get the more my output timing is shifted in regards to the input. I was expecting the encoder to follow whatever timing information is provided to it. Maybe FFmpeg is doing something with the PTS/DTSs before calling MFXVideoENCODE_EncodeFrameAsync ?

Is this an expected behavior and I have to deal with avsync some other way ? By also dropping audio frames and adjusting its PTS/DTS accordingly ? 

Appreciate any advice given.

Regards.

[Cuminato__Lucas]

No one can help me here?

[Cuminato__Lucas]

Hi, 

I figured it out. Thanks.