Media (Intel® Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools like Intel® oneAPI Video Processing Library and Intel® Media SDK
Announcements
The Intel Media SDK project is no longer active. For continued support and access to new features, Intel Media SDK users are encouraged to read the transition guide on upgrading from Intel® Media SDK to Intel® Video Processing Library (VPL), and to move to VPL as soon as possible.
For more information, see the VPL website.
3058 Discussions

H264 encoder behavior when frame timestamps has jumps

Cuminato__Lucas
Beginner
‎08-20-2019 12:11 PM
561 Views

Hi,

I have successfully created a transcoding pipeline using FFmpeg + Media-SDK. My UDP input stream comes in as either MPEG2 or H264 and I transcode the stream to H264. I'm also using VPPs before encoding to deinterlace, re-scale and frame rate convert the input to two different output profiles, 1280x720 and 850x480, both at 30fps. The only issue I'm having right now is avsync when I have dropped frames in the input.

My expectation was that the encoder would generate timestamps (PTS/DTS) according to the input frame given to the encoder, but that is not what I'm seeing when frames are dropped:

The "From encoder" line is duplicated because we output 2 different profiles. The timing information is in 1/framerate timebase (30fps both in and out) so it's essentially counting frames.

Aug 20 18:39:17 transcoded[30251]: Before decoder PTS: 1095407
Aug 20 18:39:17 transcoded[30251]: Out of decoder PTS: 1095405
Aug 20 18:39:17 transcoded[30251]: From encoder PTS: 1095400 DTS: 1095399
Aug 20 18:39:17 transcoded[30251]: From encoder PTS: 1095400 DTS: 1095399
Aug 20 18:39:17 transcoded[30251]: Before decoder PTS: 1095408
Aug 20 18:39:17 transcoded[30251]: Out of decoder PTS: 1095406
Aug 20 18:39:17 transcoded[30251]: From encoder PTS: 1095405 DTS: 1095400
Aug 20 18:39:17 transcoded[30251]: From encoder PTS: 1095405 DTS: 1095400
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095409
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095407
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095403 DTS: 1095401
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095403 DTS: 1095401
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095411                        <----- frame dropped here, should've been 1095410
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095408
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095402 DTS: 1095402
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095402 DTS: 1095402
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095412
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095409
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095404 DTS: 1095403
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095404 DTS: 1095403
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095413
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095411                        <----- good, expected this
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095409 DTS: 1095404
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095407 DTS: 1095405
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095409 DTS: 1095404
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095407 DTS: 1095405
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095414
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095412
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095406 DTS: 1095406
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095406 DTS: 1095406
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095415
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095413
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095408 DTS: 1095407
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095408 DTS: 1095407
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095416
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095414
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095413 DTS: 1095408
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095413 DTS: 1095408
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095417
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095415
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095411 DTS: 1095409
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095411 DTS: 1095409
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095418
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095416
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095410 DTS: 1095410   <--- bad, wasn't expecting 1095410 to be ever created
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095410 DTS: 1095410
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095419
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095417
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095412 DTS: 1095411
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095412 DTS: 1095411
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095420
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095418
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095417 DTS: 1095412
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095417 DTS: 1095412
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095421
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095418                        <--- bad, repeated value.
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095415 DTS: 1095413
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095415 DTS: 1095413
Aug 20 18:39:18 transcoded[30251]: Before decoder PTS: 1095422
Aug 20 18:39:18 transcoded[30251]: Out of decoder PTS: 1095419
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095414 DTS: 1095414
Aug 20 18:39:18 transcoded[30251]: From encoder PTS: 1095414 DTS: 1095414

The more dropped frames I get the more my output timing is shifted in regards to the input. I was expecting the encoder to follow whatever timing information is provided to it. Maybe FFmpeg is doing something with the PTS/DTSs before calling MFXVideoENCODE_EncodeFrameAsync ?

Is this an expected behavior and I have to deal with avsync some other way ? By also dropping audio frames and adjusting its PTS/DTS accordingly ? 

 

Appreciate any advice given.

Regards.

0 Kudos

Link Copied

2 Replies
Cuminato__Lucas
Beginner
‎08-26-2019 10:28 AM
561 Views

No one can help me here?

0 Kudos
Copy link
Cuminato__Lucas
Beginner
‎08-29-2019 10:29 AM
561 Views

Hi, 

I figured it out. Thanks.

0 Kudos
Copy link
Reply

Community support is provided during standard business hours (Monday to Friday 7AM - 5PM PST). Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.