H264 NAL unit

[bendeguy]

Hi!

I use live555 to receive H264 video from a camera. As a result, I receive discrete NAL units with 1 byte headers
1 bit zero bit
2 bit nal_ref_idc
5 bit nal_unit_type
As I saw, the decoder searches for start codes. But there are no start codes in the NAL units. How can I feed these packets to the H264 decoder?
As I saw no PPS and SPS packets coming from the camera, but I get an SDP at the start of the streaming. How can I make PPS, SPS from that?

Thanks in advance,
Miki

[bendeguy]

I realized that, I only have to insert the delimitier(0x00000001) in front of every NAL unit ;-)

[trebo]

I have also encountered the same problem.

I have tried to insert the delimiter (0x00000001) in front of every NAL unit and when the frame is complete, I feed it to the decoder.

However, "m_pH264Decoder->Init(m_pDecParams);" works fine.
But "m_pH264Decoder->GetInfo(&params);" fails with -998 (UMC_ERR_NOT_INITIALIZED).

How can this be when the Init was successful?
What can be wrong?

Do I have to use different setting for the decoder compared to when I'm decoding Annex B format (which works fine)?
Do I have to do something different?
Should every NAL unit be sent to the decoder separately, or should they be sent together as a complete frame?

Br,
Robert

[Miklós_Szeles]

Hi Robert,

I put the NAL units separately to the decoder. I think you have the problem with GetInfo, because you haven't fed SPS/PPS to the decoder before you called GetInfo. I initialize the decoder the following way:
- Init
- GetFrame with SPS NAL Unit
- GetFrame with PPS NAL Unit
- GetInfo

BR,
Miki

[trebo]

Thanks for the help!

However, I have problem finding the SPS and PPS NAL units...
The only NAL units I get looks like following:

I-frame first packet:
0x7C, 0x85, 0x88, 0x84, 0x00...
I-frame middle packets:
0x7C, 0x05, 0x79...
I-frame last packet:
0x7C, 0x45, 0xFA...
P-frames:
0x41, 0x9A, 0x20...

As I understand it, the first byte is the header, and NAL UNIT HEADER & 0x1F = NAL Unit type.
In my case this is 0x7C & 0x1F = 28 decimal, which I don't know what it is.
And 0x41 & 0x1F = 1 decimal, which is NAL SLICE.

So, where do I find the SPS and PPS NAL Units?

Br,
Robert

[Miklós_Szeles]

How do you get the stream? For example for some cameras, you can get the SDP from the camera, which contains the SPS, PPS and also packetization mode. I think you should check the RFC 3894 documentation( http://www.rfc-editor.org/rfc/rfc3984.txt ). It says(and of course, you can read much more about this in it)

Fragmentation unit: Used to fragment a single NAL unit over multiple
   RTP packets.  Exists with two versions, FU-A and FU-B, identified
   with the NAL unit type numbers 28 and 29, respectively.  Specified in
   section 5.8.

I hope it helps.
BR,
Miki

[trebo]

Youre good Miki!
I get the SPS and PPS in the SDP and the NAL units are FU-A (non interleaved mode) and single NAL units.

GetInfo is ok now - I get the image resolution and so on.
Even though, GetFrame with SPS NAL Unit and GetFrame with PPS NAL Unit returns VM_NOT_ENOUGH_DATA.
But this is maybe ok? GetInfo returns OK.

However, I still have problems with the actual decoding. GetFrame for decoding the frame returns either VM_NOT_ENOUGH_DATA or UMC_ERR_INVALID_STREAM, depending on how I try to de-packetize and feed the NALUs to the decoder...

I haven't found in any spec how to do this. Neither in 3984 nor in Intel specs. Anyone who knows or can guide me?

I have tried several ways; feeding the NALUs separately to the decoder with the delimiter (0x00000001) in front of every one, sending the NALUs concatenated to the decoder (according to 3984 7.1) with delimiter in front of every NALU, and also with delimiter only in front of complete frame, and also with FU indicator and FU headers removed (3984 5.8)... and in some other ways too...

[Miklós_Szeles]

Hi,

I feed the NAL units with the decoder one-by-one. They are complete(depacketized) frames(or other NAL units) and they start with the delimiter. It"s absolutely okay, to get the VM_NOT_ENOUGH_DATA. H264 decoder stores previous frames and use them to decode frames. It doesn't matter, if your stream starts with an IDR frame, the decoder will say VM_NOT_ENOUGH_DATA. After some frames buffered, it will start to return decoded frames. Anyway, you can get the buffered frames from the decoder calling GetFrame (using NULL for the in parameter).
BR,
Miki

[trebo]

Ok, now it starts to work, but there still are problems!

If I set the resolution small enough (and compression high enough) so that I only get one NALU per frame, i.e. there are no fragmented frames, the decoder starts to decode the frames after a while and I can display the frames.

However, if the frames are too big for one NALU we get fragmentation and the decoder returns VM_NOT_ENOUGH_DATA forever.
Unless I make the image all black by covering the camera lens, then the frame size is smaller and fits in one NALU and the decoder starts decoding. After this "initiation" the decoder also works with fragmented frames. BUT the images are quite distorted!

Am I de-packetizing or defragmenting the frames wrongly?
3984 7.1 says: "If a decapsulated packet is an FU-A, all the fragments of the fragmented NAL unit are concatenated and passed to the decoder."
But there is no further information about how this is done. I have tried several methods; adding delimiters between each NALU or not, removing FU indicator and FU header or not etc. None works ok!

How should this be done,for the decoder to accept the fragmented frames?

Br,
Robert

[trebo]

I managed to solve it by changing the header of the first fragment to Single NAL Unit and then just concatenating the paylod of the following frames, without delimiters or headers.

Br,
Robert

[ggrosselohmann]

Hi,

I also started to use the intel h264 example decoder and got some problems here.

Fragmentated units are not accepted by the decoder, I always get the error message VM_NOT_ENOUGH_DATA,

also if Imergethe NALUs and send onlya singleNALU Header (created from the fragmentated headers)before the first fragmentated unit. (The two bytes of the fragmentation header are discarded for every fragmentated packet.)

If only unfragmentated packages are used no error occurs.

But how can I control nowwhether the decoding has been successfull?

If I set the output format to yuv420, doI get raw yuv data which I can color convert to, for example, bmp with the intel library?

Is one Frame one GOP for the function h264Decoder->GetFrame?

Or is one Frame one picture?

best regards,

Gerd Groelohmann

[ole_van_off]

Quoting - Mikls Szeles

Hi,

I feed the NAL units with the decoder one-by-one. They are complete(depacketized) frames(or other NAL units) and they start with the delimiter. It"s absolutely okay, to get the VM_NOT_ENOUGH_DATA. H264 decoder stores previous frames and use them to decode frames. It doesn't matter, if your stream starts with an IDR frame, the decoder will say VM_NOT_ENOUGH_DATA. After some frames buffered, it will start to return decoded frames. Anyway, you can get the buffered frames from the decoder calling GetFrame (using NULL for the in parameter).
BR,
Miki

I have pretty much the same situation - receive the stream using live555, trying to decode it with the H264VideoDecoder. I got the Init()/GetInfo() the way Miki suggested, also feed the NAL units one-by-one, complete units of types 5 and 1, and I put the delimiters in front of them. And all I get is VM_NOT_ENOUGH_DATA, no matter how long I run the decoder. If I try to get the buffered frames calling GetFrame using NULL for the in param, the data i get appears not good (can't play it). Could it be that i have to buffer some NALUs together before feeding them to the decoder - and if yes, why?

any help plz??

thanks
Oleg

[ole_van_off]

Quoting - ole.van.off

I have pretty much the same situation - receive the stream using live555, trying to decode it with the H264VideoDecoder. I got the Init()/GetInfo() the way Miki suggested, also feed the NAL units one-by-one, complete units of types 5 and 1, and I put the delimiters in front of them. And all I get is VM_NOT_ENOUGH_DATA, no matter how long I run the decoder. If I try to get the buffered frames calling GetFrame using NULL for the in param, the data i get appears not good (can't play it). Could it be that i have to buffer some NALUs together before feeding them to the decoder - and if yes, why?

any help plz??

thanks
Oleg

got it working, with the NULL for GetFrame() (it was my own bug that got in the way) - no need to do anything with the NALUs, just feed them one by one (after proper Init())

thanks!

O.