Using Intel H264 Encoder MFT with SourceReader/SinkWriter

sjbaran · ‎03-03-2011

related to: http://software.intel.com/en-us/forums/showthread.php?t=80307&o=a&s=lr

not finding a color converter to convert from YUY2 (or some other format output by the camera)to NV12
I used Intel VPP instead to do the conversion. I grabbed the IMFSample from the IMFSoureReader, got the buffer for the video frame and used VPP to convert it to NV12.

That all went smoothly and I can even save these NV12 frames to a file and play back using FFMpeg. When I try to wrap these frames into a IMFSample and pass to an encoder MFT (either Intel's or Microsoft's) they return an errorsuch asERROR_INVALID_PARAMETER.

What I'm wondering is how to properlyget the videoattributesfrom the VPP color conversionto a MFT.

Below are some pieces of code:

//-----------------------------
// Setup Sink Writer
//------------------------------

hr = m_pReader->GetCurrentMediaType((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, &pType);

hr = pType->SetGUID(MF_MT_SUBTYPE,MFVideoFormat_NV12);

hr=pType->GetGUID(MF_MT_SUBTYPE,&guidSubtype);
hr=MFGetAttributeSize(pType,MF_MT_FRAME_SIZE, &vidWidth, &vidHeight);
hr=MFGetStrideForBitmapInfoHeader(guidSubtype.Data1,vidWidth,&lStride);

hr=pType->SetUINT32(MF_MT_DEFAULT_STRIDE,UINT32(lStride));

vidSampleSize=(UINT32)(vidHeight*lStride+(vidHeight>>1)*lStride);
hr=pType->SetUINT32(MF_MT_SAMPLE_SIZE,vidSampleSize);

m_pWriter->AddStream(pType2,0);
//pType2 same as pType but with MF_MT_SUBTYPE= MFVideoFormat_H264 and a bitrate set

m_pWriter->SetInputMediaType(0, pType, NULL);

//-----------------------------
// VPP Color Conversion
//------------------------------

// Video Enhancement Algorithms
pColorConvertParams->denoiseParam.factor = 1;
pColorConvertParams->denoiseParam.mode=VPP_FILTER_DISABLED;
pColorConvertParams->detailParam.factor=1;
pColorConvertParams->detailParam.mode=VPP_FILTER_DISABLED;
pColorConvertParams->procampParam.brightness=0.0;
pColorConvertParams->procampParam.contrast=1.0;
pColorConvertParams->procampParam.saturation=1.0;
pColorConvertParams->procampParam.hue=0.0;
pColorConvertParams->procampParam.mode=VPP_FILTER_DISABLED;
pColorConvertParams->vaParam.mode=VPP_FILTER_DISABLED;
pColorConvertParams->impLib = MFX_IMPL_HARDWARE;
pColorConvertParams->bd3dAlloc = true;

//input frame

pColorConvertParams->frameInfo[VPP_IN].nWidth=vidWidth;
pColorConvertParams->frameInfo[VPP_IN].nHeight=vidHeight;
pColorConvertParams->frameInfo[VPP_IN].CropX=0;
pColorConvertParams->frameInfo[VPP_IN].CropY=0;
pColorConvertParams->frameInfo[VPP_IN].CropW=vidWidth;
pColorConvertParams->frameInfo[VPP_IN].CropH=vidHeight;
pColorConvertParams->frameInfo[VPP_IN].FourCC = MFX_FOURCC_YUY2;
pColorConvertParams->frameInfo[VPP_IN].PicStruct=MFX_PICSTRUCT_PROGRESSIVE;
pColorConvertParams->frameInfo[VPP_IN].dFrameRate=30.0; //same as source

//output frame

pColorConvertParams->frameInfo[VPP_OUT].nWidth=vidWidth;
pColorConvertParams->frameInfo[VPP_OUT].nHeight=vidHeight;
pColorConvertParams->frameInfo[VPP_OUT].CropX=0;
pColorConvertParams->frameInfo[VPP_OUT].CropY=0;
pColorConvertParams->frameInfo[VPP_OUT].CropW=vidWidth;
pColorConvertParams->frameInfo[VPP_OUT].CropH=vidHeight;
pColorConvertParams->frameInfo[VPP_OUT].FourCC = MFX_FOURCC_NV12;
pColorConvertParams->frameInfo[VPP_OUT].PicStruct=MFX_PICSTRUCT_PROGRESSIVE;
pColorConvertParams->frameInfo[VPP_OUT].dFrameRate=30.0;
pColorConvertParams->requestedFramesCount=1;

//...

sts = FrameProcessor.pmfxVPP->RunFrameVPPAsync( pSurf[VPP_IN], pSurf[VPP_OUT], (VPP_FILTER_DISABLED != Params.vaParam.mode)? &extVPPAuxData : NULL, &syncPoint );

//...

//------------------------------------------------------------------
// Create a new IMFMediaSample 'pConvertedSample'
// and copy attributes from 'pSample' returned by the Source Reader
//-------------------------------------------------------------------

hr = pSample->GetUINT32(MFSampleExtension_BottomFieldFirst,&unBottomFieldFirst);
hr = pSample->GetUINT32(MFSampleExtension_CleanPoint,&unCleanPoint);
hr = pSample->GetUINT32(MFSampleExtension_Interlaced,&unInterlaced);
hr = pSample->GetUINT32(MFSampleExtension_Discontinuity,&unDiscontinuity);
hr = pSample->GetSampleTime(&hnsSampleTime);
hr = pSample->GetSampleDuration(&hnsSampleDuration);

hr = pConvertedSample->SetUINT32(MFSampleExtension_BottomFieldFirst, unBottomFieldFirst);
hr = pConvertedSample->SetUINT32(MFSampleExtension_CleanPoint, unCleanPoint);
hr = pConvertedSample->SetUINT32(MFSampleExtension_Interlaced, unInterlaced);
hr = pConvertedSample->SetUINT32(MFSampleExtension_Discontinuity, unDiscontinuity);
hr = pConvertedSample->SetSampleTime(hnsSampleTime);
hr = pConvertedSample->SetSampleDuration(hnsSampleDuration);

any help would be appreciated
thanks

Nikolay_A_Intel · ‎03-04-2011

Could you please be more specific, what function returns error?

sjbaran · ‎03-04-2011

The Media Foundation Tranforms return errors as seen by MFTrace. The errors indicate that there is something wrong with the IMFSample I'm passing into the transform.

I get errors with the following functions:

IMFSinkWriter::WriteSample(0,my_pSample); //if I'm using a IMFSinkWriter to setup the encoder

or
IMFTransform::ProcessInput(0,my_pSample,0);
IMFTransform::ProcessOutput(0,1,&OutputSamples,0); //if I'm trying to use the encoder directly

Note: I allocate a buffer for the output in this case. SetInputType, SetOutPutType, and ProcessInput all pass - I get the failure at ProcessOutput.

my_pSample is the sample I get from IMFSourceReader (in the YUY2 format)and then convert to NV12 using VPP functions (I called it pConvertedSample above).

errors:
CMFTransformDetours::ProcessOutput @0068FB54 failed hr=0x0x80070057 ERROR_INVALID_PARAMETER
CMFSinkWriterDetours::WriteSample @00447920 failed hr=0x80070057 ERROR_INVALID_PARAMETER

I really have two questions:
1) how do I properly set the IMFMediaType when setting up the MFT(Encoder) and properly setup the IMFSample when passing to the MFT (I gave the code for what I'm doing, but I must be missing some values)

2) how do I match up the video frame properties from the VPPparams/functionsto the attributes ofIMFMediaType and IMFSample (there are some values that I don't know, like say pixel aspect ratio, that I'm just copying from theIMFMediaType from the input device using IMFSourceReader::GetCurrentMediaType)

sjbaran · ‎03-04-2011

Looks like the answer to my second question is in the mfxVideoParam structure. I'm not setting all the values in the structure so I suppose the VPP is making some assumptions, but I can fix that.mfxFrameInfo, for example, has the aspect ratio, and so if I need it I can get it.