Real Time H.264 questions

[jstanton]

Hi, I'm creating a program that uses a sequence grabber to snag a bitmap, which I'd then like to encode to send it over an RTP stream and then decode and display it on the other end. It's a video conferencing like application. I've encoded a little class that seems to encode correctly (as best I can tell). However, decoding has proven more troublesome. I'm using the H.264 encoder/decoder from the media kit. All I'm seeing on my output is completely blue video. First question, I think I'm currently grabbing my picture from my sequence grabber as an RGB24 image, but I noticed that the H.264 decoder seems to by YUV only, is that true? Can I use the color converter to switch from one to the other, or do I need to change the grabbing code? Also, since I'm not reading from a file, what settings should I set on the decoder? Here's what I have so far:



------CODE------
ColorInit.SizeSource.width = 320;

ColorInit.SizeSource.height = 240;

vm_var32 uiFactor = 1;



ColorInit.FormatDest = UMC::RGB24; //UMC::YV12

ColorInit.SizeDest.width = 320;

ColorInit.SizeDest.height = 240;



ColorInit.lFlags = 1;

ColorInit.lDeinterlace = 0;

ColorInit.lInterpolation = UMC::FLAG_CCNV_CONVERT;



video_info = new VideoStreamInfo();

video_info->stream_type = H264_VIDEO;

video_info->stream_subtype = UNDEF_VIDEO_SUBTYPE;

video_info->color_format = UMC::RGB24;

video_info->framerate = 15;

video_info->clip_info.height = 240;

video_info->clip_info.width = 320;

video_info->aspect_ratio_height = 3;

video_info->aspect_ratio_width = 4;

video_info->bitrate = 80000;

video_info->streamPID = 0;

p_vidDecoderParams->info = *video_info;

p_vidDecoderParams->cformat = UMC::RGB24; p_vidDecoderParams->lFlags = 0;

p_vidDecoderParams->lpConverter = &pColorSpaceConverter

p_vidDecoderParams->lpConvertInit = &ColorInit

p_vidDecoderParams->uiLimitThreads = 1;



------/CODE------
Any help would be greatly appriciated. Thanks



Jamie

[jstanton]

Okay. I'm much closer now. I think all that's left is a formatting issue, but I can't figure it out. I'm now setting up my decoder like this:

ColorInit.SizeSource.width = 320;
ColorInit.SizeSource.height = 240;
vm_var32 uiFactor = 1;
ColorInit.FormatDest = UMC::YV12;
ColorInit.SizeDest.width = 320;
ColorInit.SizeDest.height = 240;
ColorInit.lFlags = 1;
ColorInit.lDeinterlace = 0; //0
ColorInit.lInterpolation = UMC::FLAG_CCNV_CONVERT;
video_info = new VideoStreamInfo();
video_info->stream_type = H264_VIDEO;
video_info->stream_subtype = UNDEF_VIDEO_SUBTYPE;
video_info->color_format = UMC::YV12;
video_info->framerate = 15;
video_info->clip_info.height = 240;
video_info->clip_info.width = 320;
video_info->aspect_ratio_height = 1;
video_info->aspect_ratio_width = 1;
video_info->bitrate = 80000;
video_info->streamPID = 0;
p_vidDecoderParams->info = *video_info;
p_vidDecoderParams->cformat = UMC::YV12;
p_vidDecoderParams->lFlags = 0;
p_vidDecoderParams->lpConverter = &pColorSpaceConverter;
p_vidDecoderParams->lpConvertInit = &ColorInit;
p_vidDecoderParams->uiLimitThreads = 1;

Also, I'm setting up my decoder to point to VideoData that is initialized as being RGB24 color format.

p_dataOut->SetVideoParameters(320, 240, RGB24);

This gives me a picture, but the picture has diagonal bars of blackness and some minor color issues. I've tried setting the decoder parameters to RGB24, but that just makes the picture worse. Also, I've noticed that the p_dataOut->GetDataSize() is always the same as what I put in, even if I pick something arbitrarily large. So, anyone have any idea how I can go RGB24 -> encoder -> decoder -> RGB24?

Thanks!

Jamie

[Vladimir_Dudnik]

It is difficult to understand what the issue without actual source, could you please provide a little test case?

Regards,
Vladimir

[jstanton]

vdudnik, thanks for replying. I've actually managed to get clean video in/out of the H264 encoder/decoder. The trick is that the encoder/decoder only can handle YV12 data, so you have to manually colorshift your data using the ippi routines that colorshift data before feeding it into the encoder/decoder. I just got a ping back from premier support that says that this is the only way of doing it. They did mention you can let the decoder switch the color back for you, I didn't get it to work that way, though, I had to shift it with the raw ippi routines.

I do still have one problem, though, the encoder/decoder seems to put in a solid 1-2 second delay. i.e. from inserting it into the encoder to getting a valid picture out of the decoder takes 1-2 seconds. The encoder immediately is outputing data, but the decoder waits about 16 frames before outputing data. (earlier frames all have size 0 on decoding) What encoder/decoder settings do you recommend for realtime encoding/decoding? Is something more then a settings change required? And help would be appriciated.

[sdhays]

I remember reading something about H.264 having the capability of having multiple reference frames, i.e. one frame can be the computed difference of several frames instead of just 1 or 2 as in P and B frames, respectively. Could this be why the decoder is holding back?

Scott

[Vladimir_Dudnik]

Hi Scott,

you are right, H.264 allow up to 16 refernece frames to be used. So, decoder may need to accumulate several frames inside, before it can provide first frame to output.

Regards,
Vladimir

[jstanton]

Thanks for the replies, everyone. So, is there any way I can set the number of reference frames to be quite low, say like 2, and would this make me basically realtime?

I see that there is a paramater on the encoder "num_ref_frames", and I've set that to 2, but I still see the delay on decoding.

I guess my question boils down to this: What settings do I need to have real/near real encoding/decoding with the IPP H.264 codec? Or is it simply not possible? Any help would be appriciated.