topic I didn't ever really get it in Intel® Integrated Performance Primitives

Decoding Streaming H.264 Video

Stephen_V_1 — Tue, 06 Nov 2012 14:48:31 GMT

I am looking to decode real-time streaming H.264 video that comes in via packets. My thought was to use the umc_video_dec_con example as a framework for implementing this functionality. Here are my thoughts that I'm currently stuck with that I would like some help:

1) I was going to use the MemoryReader as my DataReader class, but it seems to be setup such that you have all the data you want to decode in advance. There is no methods for "adding" data to the reader memory nor is there a method for telling the MemoryReader that more data is available. What is the method I should be using for adding video packets that are received so they can get decoded?

2) The splitter initialize functions again seems to need the whole video file available to the reader as it searches the video stream for information at initialize time. If I look at the InitSplitter function of the umc_dec_con example, the Init function for the MP4Splitter will return errors because it might not have all the data to properly initialize. My thought was to keep adding data to my DataReader and continually call Init until there is enough data there for the Splitter to properly initialize. Is this the proper way to setup the splitter?

3) I think once everything is initialized, I should be able to call GetNextData on the splitter until it stops returning UMC_ERR_NOT_ENOUGH_DATA. Once that happens, I would have enough data for a frame and call GetFrame on the decoder to get the decoded frame. I would just keep repeating this process as done in umc_video_dec_con example. Does this sound correct?

Is the approach outlined above the correct way for decoding real-time packet data of H.264 utilizing UMC? It seems like all the example code is geared around opening files which you already have all the data and are formed properly. I'm unsure how to use UMC when the data will come in chunks, some data may be lost if there is a bad ethernet connection, etc. How would UMC handle such scenarios? Any help would be greatly appreciated.

I am using Linux IPP7.1.0.011 samples

Update: I did some further

Stephen_V_1 — Tue, 06 Nov 2012 19:51:19 GMT

Update: I did some further debugging in the UMC classes as well as looking at the binary data in a H.264 file and discovered that the H.264 video file contains information about the size of the data which are made up of "atoms" (name in the MP4Splitter class). I looked at some raw binary H.264 streaming data that I will need to decode and I don't see any of the atom tags that the function "ParseMP4Header" is looking for (such as "mdat", "ftyp"). The streaming H.264 data is just NAL packets that I will have to be decoding. Do I need to do something completely different to decode H.264 NAL packets vs. the format of data in a H.264 video file? If so, does UMC have a module for decoding those?

I still am having issues

Stephen_V_1 — Thu, 08 Nov 2012 13:00:17 GMT

I still am having issues decoding these nal packets that I receive from a streaming H.264 packet rather then opening an H.264 video file. When I try to use a splitter like the umc_video_dec_con example, it doesn't work because some of the header information it is looking for doesn't seem to exist ("mdat" atom for example). I have been trying to do the decoding by not using a splitter and just use a decoder by itself, but it's currently crashing. I believe the problem is because some of the configuration the decoder needs, which it would normally receive from the splitter, I am not setting properly. Do you have any suggestions? I tried using the example here: http://software.intel.com/en-us/articles/getting-started-with-intel-ipp-unified-media-classes-sample but it doesn't work at all. I suspect it was written for an older IPP version then what I have and is no longer compatible. I am using 7.1.0.011

I have run into the same

Matthew_F_ — Fri, 07 Jun 2013 20:49:30 GMT

I have run into the same problem. Did you ever get this resolved? What approach did you take? Thanks.

I didn't ever really get it

Stephen_V_1 — Mon, 10 Jun 2013 12:06:24 GMT

I didn't ever really get it resolved. My solution ended up being buffering up a large amount of NAL packets received and then send them to the GetFrame function. I don't use the splitter, I found the crashing problem when using the decoder by itself. I know the video type being decoded and the general size so through testing I determined the smallest amount of data I needed to buffer that would ensure that I would always have enough data to extract a single good frame. The obvious downside to this approach is that it adds latency. In my case we were able to live with the delay, but were not happy about it. Hope that helps you and if you find a way to decode the streaming H.264 data without buffering, please let me know!