Media (Intel® Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools like Intel® oneAPI Video Processing Library and Intel® Media SDK
Announcements
The Intel Media SDK project is no longer active. For continued support and access to new features, Intel Media SDK users are encouraged to read the transition guide on upgrading from Intel® Media SDK to Intel® Video Processing Library (VPL), and to move to VPL as soon as possible.
For more information, see the VPL website.

Very high latency in the decoder side

Jack_Chimasera
Beginner
449 Views
Hello
I've been building a video conferencing application based on the encoder & decoder in the SDK (namely, H.264 codec, using 3.0 beta 2).
The encoder works wonders, with extremely low latency, However, the decoder appears to have extremely high latency ! I know it is the decoder (decoding to system memory, by the way), because when I connect another decoder to the encoder's output, I receive almost no latency (however, CPU utilization suffers).

I've gone over all the tricks listed in the manual and the videoconf samples, including setting AsyncDepth to 1 (on both encoder & decoder), setting the MaxDecFrameBuffering to 1, GopRefDist to 1...

When encoding 640x480 30fps, I see a latency of about 1/2 second. What I've noticed was that when I call
Status=MFXVideoDECODE_QueryIOSurf (m_EncodingSession,&m_DecodedVideoParameters,&AllocRequest);
on the decoder, I get a frame-count of 17, which almost exactly matches the perceived latency.
The only paramter I've changed which has any effect over it is the codec level, but this is tied to the resolution passed and framrate, which I can not change. I hardly understand why these parameters affect the latency so.

What I'm askin in short is : what am I missing here, in shortening the latency ? I doubt this latency is inherent to the decoder, which is otherwise very good !

regards

Jack Chimasera
0 Kudos
1 Solution
Nina_K_Intel
Employee
449 Views
Hi Jack,

That's right, the workaround is for the sw decoder only (I thought I've mentioned that). Unfortunately there's no such easy workaround for the current SNB decoder. MSDK 3.0 features (API version 1.3)are targeted for the next generation of Intel platforms code named Ivy Bridge. Its hardware decoders will fully support video conferencing features. Sandy Bridge version of Media SDK will hopefully add this support in future as well.

Regards,
Nina

View solution in original post

0 Kudos
5 Replies
Nina_K_Intel
Employee
449 Views
Hi Jack,

You are doing everything right, but unfortunately low latency mode of software H.294 decoder doesn't work properly as of MSDK 3.0 Beta2 version. Also, since we've discovered this limitation too late, the Beta3 version which is to be available pretty soon will suffer from the same defect.

So here is a workaround we suggest to use with Beta2 and Beta3: in addition to everything you did before set NumThread=1. Please note, that the MSDK manual lists this parameter as deprecated, so this is only a temporary solution we offer to let the customers go on with testing of VC features. You will need to drop this setting when MSDK gets fixed.

MSDK 3.0 Beta4 release will function without this hack, totally complying with the manual with regards to low latency configuration.

Best regards,
Nina
0 Kudos
Jack_Chimasera
Beginner
449 Views
Hello Nina
Setting ThreadNum=1 worked perfectly when it concerns the software decoder, lowering the latency to barely perceptible levels, thank you very much.
However, just note that this workaround does absolutely nothing when using SandyBridge hardware-based decoding (i3 2100 CPU). Latency remains in the realm of 500ms.

regards

Jack Chimasera
0 Kudos
Nina_K_Intel
Employee
450 Views
Hi Jack,

That's right, the workaround is for the sw decoder only (I thought I've mentioned that). Unfortunately there's no such easy workaround for the current SNB decoder. MSDK 3.0 features (API version 1.3)are targeted for the next generation of Intel platforms code named Ivy Bridge. Its hardware decoders will fully support video conferencing features. Sandy Bridge version of Media SDK will hopefully add this support in future as well.

Regards,
Nina
0 Kudos
applemax82
Beginner
449 Views
Hi Nina,

1)

Do you mean, MSDK 2.0 can't support video conferencing features? Thus, it'snot goodto use MSDK 2.0 to develop video conference application. Is that right?

2)

Another question:

As you said, the next generation of Intel Platforms is Ivy Bridge, and MSDK 3.0 is targeted at Ivy Bridge. At this time, Sandy Bridge version of Media SDK can't support video conferencing features. Is that right? And what is the Sandy Bridge version of Media SDK now, 2.0 or 3.0 beta2?

Thanks!

BR

Qiang
0 Kudos
Nina_K_Intel
Employee
449 Views
Hi Qiang,

That's correct. Media SDK 2.0 has the API version 1.1 which doesn't include many of important VC related features (low latency, dynamic bitrate change etc.). Sandy Bridge version corresponds to MSDK 2.0 and has the API 1.1.

So it's not recommended to program VC apps with MSDK 2.0 (maybe except for some very limited models) and with SandyBridge version as of now.

MSDK 3.0 and Ivy Bridge library have the API version 1.3 which fully supports VC features.

It's best to start prototyping using 1.3 API and SW library from MSDK 3.0 Beta (taking into account all the workarounds :)). When Ivy Bridge platforms with specific MSDK library are available your app will smoothly switch to using hardware acceleration.

Regards,
Nina
0 Kudos
Reply