Media (Intel® Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools like Intel® oneAPI Video Processing Library and Intel® Media SDK
The Intel Media SDK project is no longer active. For continued support and access to new features, Intel Media SDK users are encouraged to read the transition guide on upgrading from Intel® Media SDK to Intel® Video Processing Library (VPL), and to move to VPL as soon as possible.
For more information, see the VPL website.
3056 Discussions

Problems in decoding live stream form ip camerra with low latency


I'm trying to integrate Intel Media SDK decoder into an CCTV application wchich receives h.264 encoded frames from ip camera and expects raw frames as an output. Output frames are next presented on the screen, so decoding needs to be done with low latency. Input stream does not contain B frames.

I have met several problems:
- with AsyncDepth=1 QueryIoSurf sets the Requests.NumFramesSuggested to 7, wchich causes 7 frames latency in receiving decoded output frame (in CCTV application this is not acceptable); 
- when AsyncDepth is set to 1 many calls of DecodeFrameAsync (about 15-20%) returns staus MFX_WRN_DEVICE_BUSY (despite I am only using software implementation) and the output stream has very bad quality with lots of artifacts (especially when here is movement on camera). Setting AsyncDepth to 0 partialy resolves the problem but Requests.NumFramesSuggested is now 10.

Is there any possibility to reduce latency and increase quality of input stream?

The application previously was using IPP for decoding and didn't meet such problems.

0 Kudos
1 Reply


Some things to keep in mind, for low latency, beyond the requirement to configure the decoder for low latency. The encoded stream you receive must have no B-frames (as you stated), and the stream must also have DPB set to 1 to ensure decoder does not buffer frames (if you used Media SDK encode this would be controlled by mfxExtCodingOption::MaxDecFrameBuffering = 1).

Requests.NumFramesSuggested (in your case = 7) does not determine the decoder latency, it's reflecting the required # of internal decoder buffers, due to Media SDK and driver internal logic (# of buffers for decode may seem artificially high, but at this point they are required).

MFX_WRN_DEVICE_BUSY is a normal return code indicating that the underlying decoder is busy processing the frame. This use case is illustrated in the Media SDK samples.

If you are experiencing lack of low latency (essentially, the decoder not being able to deliver a frame directly after single successful DecodeFrameAsync + SyncOperation) then my guess is that there are issues with the received stream such as DPB not set to 1.

Also, assuming you receive a single h.264 frame at a time, note that you should set the decoded bitstream flag to MFX_BITSTREAM_COMPLETE_FRAME (in mfxBitStream::DataFlag) before calling DecodeFrameAsync().



0 Kudos