Media (Intel® Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools like Intel® oneAPI Video Processing Library and Intel® Media SDK
Announcements
The Intel Media SDK project is no longer active. For continued support and access to new features, Intel Media SDK users are encouraged to read the transition guide on upgrading from Intel® Media SDK to Intel® Video Processing Library (VPL), and to move to VPL as soon as possible.
For more information, see the VPL website.

How do I process a h264 byte array stream

nchannon
Beginner
1,231 Views
Hi I am new to the media sdk and I am looking to process encoded h264 byte array directly from a camera and return the decoded data as ayuv 420 byte array. is there any sample code that takes the h264 array directly from a buffered stream andnot from a file or can someone point me into the right direction on how to process the live h264 video stream array.
Thanks
0 Kudos
9 Replies
Petter_L_Intel
Employee
1,231 Views
Hi channon,

We do not provide such sample as part of the SDK but implementing direct buffer interface instead of reading data from file for your specific case should not be difficult at all. mfxBitstream carries the data that is fed to the encoder. mfxBitstream represents the buffer input to encoder and it does not matter where the data originates (file, other buffer etc...). Just keep in mind to insert appropriate timestamps in mfxBitstream for each frame. The sample_decode sample provides a good base for this purpose.

Other developers on this forum may also be able to share some thier best practices on this subject.

Regards,
Petter
0 Kudos
nchannon
Beginner
1,231 Views
HI Peter thanks for the reply I will work on this today. This might be a silly question but can you explain the high importance of the timestamp as you mentioned luckily my callback function from the camera includes a time stamp so I trust this should be passed to mfxBitstream but I do not have any information regarding the timestamp and its importance
Regards
Nigel
0 Kudos
Petter_L_Intel
Employee
1,231 Views
Hi Nigel,
Well it depends on what you are planning to do. Timestamp is important for the case when you want to present the decoded content with the same rate as it was encoded. I just mentioned this since Media SDK provides a transparent mechanism to tie a timestamp to a frame that is being decoded (or encoded).
Thanks,
Petter
0 Kudos
nchannon
Beginner
1,231 Views
Hi Petter,
I am having some difficulty decoding H264frame by frame can you tell me what SDK functions and in what order I must call them and delete them to decode one frame at a time.

Currently I can decode the first frame ok which is the keyframe and contains the header information. My camera sends each key frame every 10th frame. I did manage to decode the second frame as well but I got memory leaks so the decoding was short lived.

The order I am currently calling the sdk functions to decode are as follows and taken from the sample_decode.exe
1: I createone session
2: I create a MFX bitstream to hold the frame InitMfxBitstream(&m_mfxBS,1024 * 1024)
3: I iniit the decoder MFXVideoDECODE(session)
4: I DecodeHeader MFXVideoDECODE_DecodeHeader(..)
m_mfxBS.Data = pBuffer //H265 buffer from camera
m_mfxBS.DataLength = pSize //length of buffer
m_mfxBS.TimeStamp = pTimeline //timestamp from camera
5: I specify memory type which I set to SYSTEM MEMORY
6: I create a memoryallocator
7: I init the decoder MFXVideoDECODER_Init(..)
8: I find a working surface GetFreeSurfaceIndex(...)
9: I decodeframeAsync(&m_mfxBS,...); this always returns MFX_ERR_DATA then I recall GetFreeSurfaceIndex
10: Recall GetFreeSurfaceIndex
11: I recall decodeframeAsync(NULL,...) without m_mfxBS return is now MFX_ERR_MORE_SURFACE
12: I then call GetFreeSurfaceIndex(..) again nIndex = 1 now
13: I then call DecodeFrameAsync(NULL,..) return now MFX_ERR_NONE
14: I then call SyscOperation(..)
15: I now have the firstdecoded frame

Now for the second frame

16:I now just repeat from 8: ~ 14: except I pass the next frame buffer to m_mfxBSbefore I call DecodeFrameAsync(m_mfxBS,..)
m_mfxBS.Data = pBuffer //H264 buffer from camera
m_mfxBS.DataLength = pSize //size of video buffer
m_mfxBS.TimeStamp = pTimeline //Time stamp from camera

17: DecodeFrameAsync(m_mfxBS,...) returns MFX_ERR_MORE_DATA the same as the first frame did
18: I then call GetFreeSurfaceIndex again where nIndex = 0;
19: DecodeFrameAsync(NULL,...) return MFX_ERR_MORE_DATA so I added MFX_ERR_MORE_DATA to the while loop and call GetFreeSurface(..) again where mfxResponse.NumFrameActual = 9
problem is nIndex dosnt implement to 1 and the code just continously loops due to MFX_ERR_MORE_DATA being returned by DecodeFrameAsync(NULL,..)

Hop you can help me out here as im at a loose as what to do and Im new to the media sdk and still trying to understand how to use it.
Regards
Nigel








0 Kudos
Petter_L_Intel
Employee
1,231 Views
Hi Nigel,

Looking at your pseudocode thereare some things that stand out.

9 - 11: Calling DecodeFrameAsync with mfxBS=NULL will effectively start "draining" the decoder the retrieve the buffered frames. This is normally what is done after there are no more video stream data input to decoder.If you look at sample_decode you will see how this works. The first loop processes the input stream until it's empty (or ends). The second loop handles decoder buffer "draining as explained above.

Since (9)DecodeFrameAsync(mfxBS) returns MFX_ERR_MORE_DATA it means that the mfxBS does not have enough stream data to be able to process at least one frame. As can be seen in sample_decode, the mfxBS is then appended with more data from stream file. So my guess is that you do not have complete stream data to be able to start decoding of the fist frame in mfxBS.

In your case it seems you're using the SDK for a streaming or video conferencing usage scenario. If so please refer to this article:http://software.intel.com/en-us/articles/video-conferencing-features-of-intel-media-software-development-kit/


I recommend taking a look at the Media SDK decode sample again. It may help get some clarity on how to change your code. Note that sample_decode has low latency option which represents Media SDK usage when decoding common video conf or streaming streams.

Regards,
Petter
0 Kudos
nchannon
Beginner
1,231 Views
Hi Petter,
I have now sucessfully decoding each frame but I have two issues
1: I create an empty buffer for each new frame that is passed by my frame grabber using

InitBitstream(mfxBitstream* pBitstream, mfxU32 nSize) but when I try to delete it after the frame has been decoded I get an error from WipeMfxBitstream(&m_mfxBS) when it trys to delete MSDK_SAFE_DELETE_ARRAY(pBitstream->Data);
Any Ideas as I have a big memory leak from this.
2: my decoded frames are very pixelated meaning each decoded frame looks like a very compressed jpeg image im decoding via hardware and using the i7-3770K third generation CPU
what would cause the decoding to be this way
Thanks
Nigel

0 Kudos
Petter_L_Intel
Employee
1,231 Views
Hi Nigel,

1) I suspect your memory area has been corrupted somehow. Please compare with behavior of sample_decode

2) I have no clue to why this would happen. If you write the stream that is received from your camera to file then decode the file using sample_decode. Do you encounter the same results?

Regards,
Petter
0 Kudos
nchannon
Beginner
1,231 Views
Hi Petter,
The memory issue is solved but I seem to have a problem with hardware decoding and software decoding.
When I decode using software option I retreive from (mfxFrameSurface1* pmfxOutSurface) a NV12 color space which I then convert to RGB which works fine but when I decode using Hardware option pmfxOutSurface nolonger seems to be the same NV12 color space can you point me into the right direction as to what is going on here, as my frame is now all distorted but fine if I use software decode option.

Also when I use the option -r from the sample_decode.exe I only get the first frame in the rendered window but the console is telling me it is decoding each frame untill all frames are complete I added a small extra function to just see if each frame was being processed this function just copys the NV12 color space to RGB and saves a bmp image each frame is being processed and saved but the rendered surface remains with the first frame only.

Regards
Nigel
0 Kudos
Petter_L_Intel
Employee
1,231 Views
Hi Nigel,

Surfaces are referenced in different ways depending on if you use D3D or system memory surfaces. In the case of D3D, the surface handle (effectively a IDirect3DSurface pointer) is stored in Data.MemId. In the case of using system memory the surface is referenced via the memory plane pointers (Data.Y, Data.U, Data.V).
Note that in the sample code, D3D surfaces are forced in case rendering (-r option) is selected.

So, I suspect your issue may be related to not referencing the surface correctly.

I do not know the reason that you only see the first frame getting rendered. Have you verified that the input stream is not broken?

Regards,
Petter
0 Kudos
Reply