H264VideoDecoder Init() is OK but GetInfo() fails.

[mojonez1]

Im an IPP newbie, but ran out of things to try so thought posting here might help.

I'm encoding/decoding with the H264 encoder/decoder. Encoding seems to be working fine. Raw h264 bitstreams written to file playback correctly with the simple player sample.

My problem is in my encode routineI write the encoded samples to an AVI file using the DirectShow AVI file writer filter. The resulting AVI file plays back correctly in Windows Media Player using the Lead H264 codec. Using the IPP H264 decoder, however, with the simple player just plays back the audio:

C:Program FilesIntelIPPipp-samplesaudio-video-codecs\_binwin32_cl7>simple_
player Leader-transcoded.avi
Audio Render : DSOUND

Stream Type : AVI
Video Info :
-Video Type : UNDEF
-Resolution : 720x480
-Frame Rate : 29.97
Audio Info :
-Audio Type : PCM
-S.Frequency : 32000
-Num.Channel : 2
-BitPerSample: 16

Error in audio render

Using the IPP H264 decoder in my routine, the Init() function returns OK, but the subsequent GetInfo() fails.

If anyone has any ideas for me to try I sure appreciate it.

My code:

	UMC::Status ret;

	m_pDecoder = new UMC::H264VideoDecoder();
	m_pDecoderParams = new UMC::VideoDecoderParams();
	UMC::BaseCodec* pPointerCopier = new UMC::DataPointersCopy();
	UMC::VideoProcessing* color = new UMC::VideoProcessing();

	UMC::VideoProcessingParams postProcessingParams;
	postProcessingParams.m_DeinterlacingMethod = (UMC::DeinterlacingMethod)(0 & UMC::FLAG_CCNV_DEINTERLACE);
        postProcessingParams.InterpolationMethod = 0;
        color->SetParams(&postProcessingParams);
    
	m_pDecoderParams->pPostProcessing  = color;
	//m_pDecoderParams->pPostProcessing  = pPointerCopier;
	m_pDecoderParams->info.stream_type = UMC::H264_VIDEO;
        m_pDecoderParams->numThreads = 1;
        m_pDecoderParams->lFlags = 0;
        m_pDecoderParams->m_pData = &m_IppIn;

	ret = m_pDecoder->Init(m_pDecoderParams);    // <-- returns OK
	if (ret != UMC::UMC_OK)
	{
		switch(ret)
		{
		case UMC::UMC_ERR_ALLOC:
			MyMessage("[H264] Video decoder initialization failed. Decoder failed to allcoate memory for internal buffers.");
			break;
		case UMC::UMC_ERR_INVALID_STREAM:
			MyMessage("[H264] Video decoder initialization failed. Incorrect bitstream provided.");
			break;
		case UMC::UMC_ERR_NULL_PTR:
			MyMessage("[H264] Video decoder initialization failed. NULL pointer passed.");
			break;
		case UMC::UMC_ERR_UNSUPPORTED:
			MyMessage("[H264] Video decoder initialization failed. Unsuported flag specified.");
			break;
		default:
			MyMessage("[H264] Video decoder initialization failed. Unknown error.");
			break;
		}
		return false;
	}

	UMC::H264VideoDecoderParams params;
	ret
 = m_pDecoder->GetInfo(ms);        // <--- fails
        if (ret != UMC::UMC_OK)
	{
		MyMessage("[H264] Video decoder GetInfo failed");
		return false;
	}
	else
	{
         ...
         ...

        }

Thanks!

[Vladimir_Dudnik]

Hello,

might be there is some issue with how that stream was muxed. Could you please provide that stream for analis?

Regards,
Vladimir

[dchris]

mojonez,
I have found as I have worked with the H264VideoDecoder that if the decoder has not found valid Sequence Parameter and Picture Parameter Sets
Init will succeed but GetInfo will fail until the SPS and PPS have been decoded, so be sure that you decode valid SPS and PPS before calling anything
else.

Vladimir,
Is this the expected behavior? I know that we are not the only ones who have had problems with this behavior, this is not the first thread that has
discussed this issue. I think it would be more intuitive if the Init failed when the proper data is not passed to it rather than succeeding making you think
that everything is okay and later failing on the GetInfo call.


You can see a little more detailed description see this thread:
http://softwarecommunity.intel.com/isn/Community/en-US/forums/permalink/30239833/30239833/ShowThread...


Derek

[Vladimir_Dudnik]

Hi,

in our current design Init should return OK, it is expected behaviour. You can use Init without first frame. I think it is an issue of splitter, it should process stream until findSPS/PPS info at the first call, so the following GetInfo will be able to return meaningful info.

Regards,
Vladimir

[mojonez1]

Thanks Guys,

You werecorrect. The first frame delivered by the AVI splitter, which I was using to init the decoder, didn't contain the SPS/PPS info.

It's working correct now. Thanks for your help.

Morgan

[ole_van_off]

Hello!

Hopefully, this thread is still being monitored - my symptoms are the same, that's why i decided to revitalize it...

I'm new to this wonderful IPP stuff; I'm trying to use H264VideoDecoder class to decode a live video stream that I receive via RTSP/RTP. My video source is an AXIS Video encoder, that sends over the video signal as a *standard* RTSP/RTP- transported, h264-encoded stream that i can receive and play using their (AXIS) application and/or VLC player, which gives me hope that it can be played by any standard-compliant tool(s). My network interface is based on Live Streaming Media library (live555.com), which seems to handle the RTSP part well: its H264VideoRTPSource class is able to handle NAL units - making sure that if they were split across multiple packets, they get reassembled alright, and delivering me one frame at a time.

Before passing the NALUs/frames to the H264VideoDecoder, i prepend each of them with those starting 0x00 ... 0x01 (saw this in the VLC code) - but all to no avail, my H264VideoDecoder gets Init() OK, but GetInfo() fails - no matter how many packets I run through it, nor how big some of them are (assuming that they contain SPS/PPS data).

I used the provided sample code, basically -

void CVideoClass::OnGetVideoFrame(unsigned char* data,unsigned int i_size)
{

unsigned char* cYUVData = (unsigned char*)VirtualAlloc(NULL,DATA_BUFFER_SIZE*DEFAULT_PACKET_SIZE,MEM_RESERVE|MEM_COMMIT,PAGE_READWRITE);

int imgWidth; int imgHeight; int frameNumber;
UMC::Status status; UMC::MediaData DataIn; UMC::VideoData DataOut;
UMC::VideoDecoderParams Params;
UMC::H264VideoDecoder H264Decoder;


DataIn.SetBufferPointer(data,(int)i_size);
DataIn.SetDataSize((int)i_size);

Params.m_pData = &DataIn;
Params.lFlags=0;
Params.numThreads=1;
Params.info.stream_type = UMC::H264_VIDEO;


if(status = H264Decoder.Init(&Params)==UMC::UMC_OK)
{

H264Decoder.GetInfo(&Params);
imgWidth=Params.info.clip_info.width; imgHeight=Params.info.clip_info.height;

DataOut.Init(imgWidth,imgHeight,UMC::YV12,8);
DataOut.SetBufferPointer(cYUVData,imgWidth*imgHeight*3/2);

int exit_flag=0; frameNumber=0;
do{
status = H264Decoder.GetFrame(&DataIn, &DataOut);
if (status == UMC::UMC_OK)
{
cYUVData += (imgWidth*imgHeight*3/2);
DataOut.SetBufferPointer(cYUVData,imgWidth*imgHeight*3/2);
frameNumber++;
}
if((status !=UMC::UMC_OK)||(frameNumber >=MAXFRAME))
exit_flag = 1;

}while (exit_flag!=1);

do{
status = H264Decoder.GetFrame(NULL, &DataOut);
if (status == UMC::UMC_OK)
{
cYUVData += (imgWidth*imgHeight*3/2);
DataOut.SetBufferPointer(cYUVData,imgWidth*imgHeight*3/2);
frameNumber++;
}
}while(status == UMC::UMC_OK);


}

[...]

};

I understand that my question is kinda vague, but may be somebody can point me to the right direction - say, how can I make sure that i supply the needed SPS/PPS data - can i analyze it on the packet/frame level, and if yes - what would i look for ??

thanks, any help will be greatly appreciated

Oleg

[Emmanuel_W_]

Quoting - ole.van.off

Hello!

Hopefully, this thread is still being monitored - my symptoms are the same, that's why i decided to revitalize it...

I'm new to this wonderful IPP stuff; I'm trying to use H264VideoDecoder class to decode a live video stream that I receive via RTSP/RTP. My video source is an AXIS Video encoder, that sends over the video signal as a *standard* RTSP/RTP- transported, h264-encoded stream that i can receive and play using their (AXIS) application and/or VLC player, which gives me hope that it can be played by any standard-compliant tool(s). My network interface is based on Live Streaming Media library (live555.com), which seems to handle the RTSP part well: its H264VideoRTPSource class is able to handle NAL units - making sure that if they were split across multiple packets, they get reassembled alright, and delivering me one frame at a time.

Before passing the NALUs/frames to the H264VideoDecoder, i prepend each of them with those starting 0x00 ... 0x01 (saw this in the VLC code) - but all to no avail, my H264VideoDecoder gets Init() OK, but GetInfo() fails - no matter how many packets I run through it, nor how big some of them are (assuming that they contain SPS/PPS data).

I used the provided sample code, basically -

void CVideoClass::OnGetVideoFrame(unsigned char* data,unsigned int i_size)
{

unsigned char* cYUVData = (unsigned char*)VirtualAlloc(NULL,DATA_BUFFER_SIZE*DEFAULT_PACKET_SIZE,MEM_RESERVE|MEM_COMMIT,PAGE_READWRITE);

int imgWidth; int imgHeight; int frameNumber;
UMC::Status status; UMC::MediaData DataIn; UMC::VideoData DataOut;
UMC::VideoDecoderParams Params;
UMC::H264VideoDecoder H264Decoder;


DataIn.SetBufferPointer(data,(int)i_size);
DataIn.SetDataSize((int)i_size);

Params.m_pData = &DataIn;
Params.lFlags=0;
Params.numThreads=1;
Params.info.stream_type = UMC::H264_VIDEO;


if(status = H264Decoder.Init(&Params)==UMC::UMC_OK)
{

H264Decoder.GetInfo(&Params);
imgWidth=Params.info.clip_info.width; imgHeight=Params.info.clip_info.height;

DataOut.Init(imgWidth,imgHeight,UMC::YV12,8);
DataOut.SetBufferPointer(cYUVData,imgWidth*imgHeight*3/2);

int exit_flag=0; frameNumber=0;
do{
status = H264Decoder.GetFrame(&DataIn, &DataOut);
if (status == UMC::UMC_OK)
{
cYUVData += (imgWidth*imgHeight*3/2);
DataOut.SetBufferPointer(cYUVData,imgWidth*imgHeight*3/2);
frameNumber++;
}
if((status !=UMC::UMC_OK)||(frameNumber >=MAXFRAME))
exit_flag = 1;

}while (exit_flag!=1);

do{
status = H264Decoder.GetFrame(NULL, &DataOut);
if (status == UMC::UMC_OK)
{
cYUVData += (imgWidth*imgHeight*3/2);
DataOut.SetBufferPointer(cYUVData,imgWidth*imgHeight*3/2);
frameNumber++;
}
}while(status == UMC::UMC_OK);


}

[...]

};

I understand that my question is kinda vague, but may be somebody can point me to the right direction - say, how can I make sure that i supply the needed SPS/PPS data - can i analyze it on the packet/frame level, and if yes - what would i look for ??

thanks, any help will be greatly appreciated

Oleg

NAL type 7 and 8 are SPS and PPS respectively. The NAL header is very simple, you can find the details in the specs. The low 5 bits of the first byte of a NAL is the nal_unit_type.

Try to call getInfo after the first call to getframe. After getting the correct viedo size call getframe again.

Emmanuel

[ole_van_off]

Quoting - eweber

NAL type 7 and 8 are SPS and PPS respectively. The NAL header is very simple, you can find the details in the specs. The low 5 bits of the first byte of a NAL is the nal_unit_type.

Try to call getInfo after the first call to getframe. After getting the correct viedo size call getframe again.

Emmanuel

Thanks for your reply, Emmanuel

from what i see, i don't get SPS nor PPS units, only of type 1 (small packets, "Coded slice of a non-IDR picture
", according to specs) and 5 (huge packets, "Coded slice of an IDR picture"). Is it possible to decode such a stream?? And if not - how come i can decode/play it with other players??

that's how i check the unit types:

[...]
unsigned char newPacketNALUnitType = (tk->buffer[0] & 0x1f);

if (newPacketNALUnitType >= 24)
OutputDebugString("n unsupported NAL type for H264n" );
else if (newPacketNALUnitType == 7 || newPacketNALUnitType == 8)
OutputDebugString("n received SPS/PPS NAL unitn" );

[...]

thanks!

Oleg

[Emmanuel_W_]

Quoting - ole.van.off

Thanks for your reply, Emmanuel

from what i see, i don't get SPS nor PPS units, only of type 1 (small packets, "Coded slice of a non-IDR picture
", according to specs) and 5 (huge packets, "Coded slice of an IDR picture"). Is it possible to decode such a stream?? And if not - how come i can decode/play it with other players??

that's how i check the unit types:

[...]
unsigned char newPacketNALUnitType = (tk->buffer[0] & 0x1f);

if (newPacketNALUnitType >= 24)
OutputDebugString("n unsupported NAL type for H264n" );
else if (newPacketNALUnitType == 7 || newPacketNALUnitType == 8)
OutputDebugString("n received SPS/PPS NAL unitn" );

[...]

thanks!

Oleg

Hi Oleg,

Check the SDP that you get in RTSP.

It is possible that the parameter set are encoded in it.

If this is the case you need to extract them, decode them back to H.264 format and feed them to the decoder.

Checkrfc3984 (I believe sprop-parameter-sets is what you would be looking for).

Emmanuel

[ole_van_off]

Quoting - eweber

Hi Oleg,

Check the SDP that you get in RTSP.

It is possible that the parameter set are encoded in it.

If this is the case you need to extract them, decode them back to H.264 format and feed them to the decoder.

Checkrfc3984 (I believe sprop-parameter-sets is what you would be looking for).

Emmanuel

thanks so much Emmanuel - that's exactly where i found them, in the RTSP/SDP packet that I receive in response to "Describe" message! both NALUs 7 and 8 were there, parts of the sprop-parameter-sets. now at least it starts making sense to me :-) will try to figure out what to do next

thanks again for your help, i appreciate it!

O.

[ole_van_off]

Still not working...

I get the NAL units 7 and 8 from the SDP string, decode them, and supply to the H264VideoDecoder; I get it initialized, GetInfo is ok now - I get the image resolution and so on; but - I'm still having problems with the actual decoding.
I receive units type 5 (not too often) and type 1.

GetFrame for decoding the frame returns -996 (VM_NOT_ENOUGH_DATA).

And if i try to get the buffered frames from the decoder calling GetFrame with NULL for the in parameter, I get -882 (UMC_ERR_INVALID_STREAM)

am I missing something?

thank you

O.

[Emmanuel_W_]

Quoting - ole.van.off

Still not working...

I get the NAL units 7 and 8 from the SDP string, decode them, and supply to the H264VideoDecoder; I get it initialized, GetInfo is ok now - I get the image resolution and so on; but - I'm still having problems with the actual decoding.
I receive units type 5 (not too often) and type 1.

GetFrame for decoding the frame returns -996 (VM_NOT_ENOUGH_DATA).

And if i try to get the buffered frames from the decoder calling GetFrame with NULL for the in parameter, I get -882 (UMC_ERR_INVALID_STREAM)

am I missing something?

thank you

O.

When you call GetFrame, do you provide only one NAL unit (1 or 5) or do you provide the whole frame.
The frame might be composed of multiple NAL units (they will all have the same timestamp in RTP).
If this is the case you need to accumulate them in a buffer (separated with 0x00 0x000x01).
And call getframe with the accumulated buffer with the whole frame.

Emmanuel

[ole_van_off]

Quoting - eweber

When you call GetFrame, do you provide only one NAL unit (1 or 5) or do you provide the whole frame.
The frame might be composed of multiple NAL units (they will all have the same timestamp in RTP).
If this is the case you need to accumulate them in a buffer (separated with 0x00 0x000x01).
And call getframe with the accumulated buffer with the whole frame.

Emmanuel

I see a kinda opposite situation - sometimes a NAL unit comes in multiple packets, which have the same RTP timestamp, NAL unit type being 28 and the last packet in the sequence being marked. When I put them together (well, live555 does) it ends up as a NALU of type 5. All type 1 and 're-composed' type 5 units come with different, their own RTP timestamps (we're talking the RTP packet header's timestamp, not somewhere in the payload/NALU, right?)

but thanks for the pointing out, Emmanuel, now I'm trying to find out more about composing frames from multiple NALUs before feeding them into GetFrame ...

from what i understand, I have a "single nal unit mode" here (see below) , which means one NALU == one frame, right?

Received DESCRIBE response: RTSP/1.0 200 OK
[...]
Content-Length: 378
v=0
o=- 1241127301359907 1241127301359907 IN IP4 192.168.1.118
s=Media Presentation
e=NONE
c=IN IP4 0.0.0.0
b=AS:50000
t=0 0
a=control:*
a=range:npt=0.000000-
m=video 0 RTP/AVP 96
b=AS:50000
a=framerate:30.0
a=control:trackID=1
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1; profile-level-id=420029; sprop-parameter-sets=Z0IAKeKQLD9gasGAQbh4kRU=,aM48gA==
" char *

[ole_van_off]

Finally got it working - Init(), GetInfo() and all that; still have problems with the stream decoding, especially if the resolution is bigger than CIF, but will move it to the different thread.

[carmas]

Quoting - ole.van.off

Finally got it working - Init(), GetInfo() and all that; still have problems with the stream decoding, especially if the resolution is bigger than CIF, but will move it to the different thread.

Hi, sorry to open again this thread but I'm in the same situation and cannot understand what are the correct steps to make the decoder work.
Or maybe I'm stuck even before... I just have the stream coming from a h.264 Axis camera via rtp/rtsp and simply don't know how to handle it.
The decoder will fail at the GetInfo, because the Params.info.clip_info.width and height fields are set to 0, so that the next call to GetFrame will result in a -996 (VM_NOT_ENOUGH_DATA)error.

Looking at the SDP from my Describe request, I read the line:

a=fmtp:96; packetization-mode=1; profile-level-id=420029; sprop-parameters-set=Z0IAKeKQLD9gasGAQbh4kRU=, aM48gA==

AFAIK sprop-parameters-set are base64-encoded values, but when I try to decode them I get a string that makes no sense to me (I was expecting something more like a key=value pair).
How do I get these parameters and feed them to the decoder?

Anyone can help?
Basically my actual code is just like the example above... what am I missing?

[jenseb]

Quoting - carmas

Looking at the SDP from my Describe request, I read the line:

a=fmtp:96; packetization-mode=1; profile-level-id=420029; sprop-parameters-set=Z0IAKeKQLD9gasGAQbh4kRU=, aM48gA==

AFAIK sprop-parameters-set are base64-encoded values, but when I try to decode them I get a string that makes no sense to me (I was expecting something more like a key=value pair).
How do I get these parameters and feed them to the decoder?

Hello,

Maybe a bit late for the answer - but in case somebody else is crashing into the same problems ...
It is not a key-value-pair but the PPS and SPS as specified in the H.264-specification. I think this document will help you dealing with the Axis-H.264 stream: http://www.axis.com/files/manuals/MigrationGuide_v1_04.pdf

jens.