Media SDK with IDirect3DSurface9 + h.264 encoder

[Lu_N_]

Hi,

  I want to do the screen capture(1920*1080) via DirectX function GetFrontBufferData, and use IDirect3DSurface9 with input for h.264 encoder,

but the encoder is without any output video file, here is the my code(modification by smaple code) and the log data by mediasdk_tracer.

Could you help to check which items I need to modify, thanks.

//Run Encoder

while (MFX_ERR_NONE <= sts || MFX_ERR_MORE_DATA == sts )
{
       sts = GetFreeTask(&pCurrentTask);
       MSDK_BREAK_ON_ERROR(sts);
       nEncSurfIdx = GetFreeSurface(m_pEncSurfaces, m_EncResponse.NumFrameActual);
       MSDK_CHECK_ERROR(nEncSurfIdx, MSDK_INVALID_SURF_IDX, MFX_ERR_MEMORY_ALLOC);
       pSurf = &m_pEncSurfaces[nEncSurfIdx];
       if (m_bExternalAlloc)
      {
          
           sts = m_pMFXAllocator->Lock(m_pMFXAllocator->pthis, pSurf->Data.MemId, &(pSurf->Data));
       }
        MSDK_BREAK_ON_ERROR(sts);
       pSurf->Info.FrameId.ViewId = currViewNum;

       g_pd3dDevice->GetFrontBufferData(0,pSurf->Data.MemId);
      
       if (m_bExternalAlloc)
       {
            sts = m_pMFXAllocator->Unlock(m_pMFXAllocator->pthis, pSurf->Data.MemId, &(pSurf->Data));
       }
       MSDK_BREAK_ON_ERROR(sts);
      for (;;)
      {
            sts = m_pmfxENC->EncodeFrameAsync(NULL, &m_pEncSurfaces[nEncSurfIdx], &pCurrentTask->mfxBS, &pCurrentTask->EncSyncP);
            if (MFX_ERR_NONE < sts && !pCurrentTask->EncSyncP)
            {
                 if (MFX_WRN_DEVICE_BUSY == sts)
                 MSDK_SLEEP(1);

            }

           else if (MFX_ERR_NONE < sts && pCurrentTask->EncSyncP)
          {
                sts = MFX_ERR_NONE; // ignore warnings if output is available
                break;
          }
          else if (MFX_ERR_NOT_ENOUGH_BUFFER == sts)
          {
               sts = AllocateSufficientBuffer(&pCurrentTask->mfxBS);
               MSDK_CHECK_RESULT(sts, MFX_ERR_NONE, sts);
          }
         else
        {
              MSDK_IGNORE_MFX_STS(sts, MFX_ERR_MORE_BITSTREAM);
              break;
         }
    }
}

[Petter_L_Intel]

Hi Lu,

GetFrontBufferData captures the front buffer in native format which is: D3DFMT_A8R8G8B8
http://msdn.microsoft.com/en-us/library/windows/desktop/bb174388(v=vs.85).aspx

But Media SDK EncodeFrameAsync can only process surface types of type NV12.So from the code it appears that you are trying to have the GetFrontBufferData call do an implicit color conversion from RGB32 type surface to an NV12. This is very likely not supported. If you check the return value from GetFrontBufferData you will probably see that it returns an error.

So, to achieve what you want you will have to do color conversion before feeding to EncodeFrameAsync. RGB32 to NV12 converison can be done using Media SDK VPP component or via other custom color converison routines you may have.

Media SDK  sample_encode showcases how to use VPP ahead of encode,

Regards,
Petter

[Lu_N_]

Hi Petter,

Thansk for your help.

So we may use VPP to do the surface color conversion(from RGB32 to NV12) , and use it as the input of EncodeFrameAsync,right?

As below:

IDirect3DSurface9* g_pSurface;

Surface_Lock();

g_pd3dDevice->GetFrontBufferData(0, g_pSurface); //RGB32 format

Vpp_color_convert(g_pSurface, NV12_Surface); //Vpp color convert function

EncodeFrameAsync(NV12_Surface); //Encode

Surface_UnLock();

But I have checked the sample_VPP, it seems not have the surface color conversion, could you give some suggstion or hint for the surface color conversion, thanks.

[Petter_L_Intel]

Hi Lu,

Correct, for RGB32 to NV12 color space conversion you can use VPP.

sample_vpp does showcase RGB32 to NV12 color converson.

Also sample_encode sample has functionality for VPP processing before encode. But the sample code only illustrates how to use it for resize. But it could easily be extended to support RGB32.

Regards,
Petter

[Lu_N_]

Hi Petter,

Thanks.

I see the partial code for the read RGB4 file to memory, as below:

 Due to I don't see any others ,does it do the conversion from file(RGB4) to video memory(NV12)? 

if (pInfo->FourCC == MFX_FOURCC_RGB4) 
{
    ptr = MSDK_MIN( MSDK_MIN(pData->R, pData->G), pData->B );

    ptr = ptr + pInfo->CropX + pInfo->CropY * pitch;

  for(i = 0; i < h; i++)
  {
     nBytesRead = (mfxU32)fread(ptr + i * pitch, 1, 4*w, m_fSrc);  //File reader
   }

}

[Petter_L_Intel]

Hi Lu,

RGB4 to NV12 color space conversion happen inside Media SDK VPP component.

The code section you refer to is just the file reader that reads a RGB4 surface from file into D3D or system memory buffer. The next step performed by the sample code is to feed the RGB4 surface to VPP for processing into NV12 surface.

By the way. Some things to keep in mind when using "GetFrontBufferData" in this context:
- Not only does GetFrontBufferData require output surface to be of type D3DFMT_A8R8G8B8. The surface must also be stored in system memory (D3DPOOL_SYSTEMMEM)
- This forces you to store the surface retrieved from "GetFrontBufferData" in a intermediate surface. Such surface can be created using "CreateOffscreenPlainSurface".
- You can then copy the intermediate surface into the Media SDK RGB4 surface using "UpdateSurface".

Regards,
Petter

[Lu_N_]

Hi Petter,

Thanks a lot, know how to do the color coversion.

And I have used "CreateOffscreenPlainSurface" to store the surface retrieved from "GetFrontBufferData".

But when I use the "UpdateSurface" for GetFrontBufferData-> VPP RGB4 surface, it will not succeed:

//************************************************************************************

Lock(VPP_surface);

GetFrontBufferData(0, Getfrontdata_surface); 

UpdateSurface(Getfrontdata_surface, NULL, (IDirect3DSurface9*)VPP_surface->Data.MemId, NULL);   // not succeed 

Unlock(VPP_surface);

RunFrameVPPAsync(&VPP_surface, &Encode_surface,NULL, &VppSyncPoint);  // Do VPP color conversion

EncodeFrameAsync(NULL, &Encode_surface, &pCurrentTask->mfxBS, &pCurrentTask->EncSyncP);  // Encode

//************************************************************************************

After check the Microsoft MSDN for UpdateSurface ,  this function has the some restriction, ex:

-The destination surface must have been created with D3DPOOL_DEFAULT => It seems the VPP_surface not created with  D3DPOOL_DEFAULT, does any other need to be modified, thanks.

[Petter_L_Intel]

Hi Lu,

"UpdateSurface" will likely fail since the surface is locked by the "Lock" call. For DirectX calls such as this you do not need to call the Media SDK locking operation.

Media SDK D3D surfaces are created using D3DPOOL_DEFAULT.

Also, I suggest you use DirectX control panel to enable DirectX debugging information. By enabling DirectX debug and singlestepping though your code you can get much more details on what and why specific DirectX calls fail.

Regards,
Petter 

[Seongjun_Kim]

Hi Petter and Lu.
I'm implementing IDirect3DSurface9 + h.264 program following your history.
I passed UpdateSurface() step successfully, but next step RunFrameVPPAsync() failed. This function returns MFX_ERR_INCOMPATIBLE_VIDEO_PARAM.
What is this error?
​
My pseudo code is below.
//************************************************************************************
CreateSurface Getfrontdata_surface (RGB4 format);
CreateSurface VPP_surface (RGB4 format);
CreateSurface Encode_surface (NV12 format);
GetFrontBufferData(0, Getfrontdata_surface); 
UpdateSurface(Getfrontdata_surface, NULL, (IDirect3DSurface9*)VPP_surface->Data.MemId, NULL);   // success!
RunFrameVPPAsync(&VPP_surface, &Encode_surface, NULL, &VppSyncPoint);  // Convert RGB4->NV12 by VPP, but failed!
EncodeFrameAsync(NULL, &Encode_surface, &pCurrentTask->mfxBS, &pCurrentTask->EncSyncP);  // Encode
//************************************************************************************

Thank you for your reply.

[Sravanthi_K_Intel]

Hello there -

"I passed UpdateSurface() step successfully, but next step RunFrameVPPAsync() failed. This function returns MFX_ERR_INCOMPATIBLE_VIDEO_PARAM." --> Can you please give us more details on how you are setting (and the values of )mfx and vpp params? If you could follow this post for rephrasing your question, it would be really helpful for us to understand your question much better.

[Seongjun_Kim]

Hi Sravanthi.
I posted Media SDK Analyzer's result below.

=====================================================
Intel Media SDK System Analyzer (64 bit)

The following versions of Media SDK API are supported by platform/driver:

        Version Target  Supported       Dec     Enc
        1.0     HW      Yes             X       X
        1.0     SW      Yes             X       X
        1.1     HW      Yes             X       X
        1.1     SW      Yes             X       X
        1.3     HW      Yes             X       X
        1.3     SW      Yes             X       X
        1.4     HW      Yes             X       X
        1.4     SW      Yes             X       X
        1.5     HW      Yes             X       X
        1.5     SW      Yes             X       X
        1.6     HW      Yes             X       X
        1.6     SW      Yes             X       X
        1.7     HW      Yes             X       X
        1.7     SW      Yes             X       X
        1.8     HW      Yes             X       X
        1.8     SW      Yes             X       X

Graphics Devices:
        Name                                         Version             State
        Intel(R) HD Graphics 4600                    10.18.10.3907       Active

System info:
        CPU:    Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
        OS:     Microsoft Windows 7 Professional K
        Arch:   64-bit

Installed Media SDK packages (be patient...processing takes some time):
        Intel?Media SDK Video Processing Sample 5.0.461.91752
        Intel?Media SDK Video Encoding Sample 5.0.461.91752
        Intel?Media SDK 2014 R2 for Clients (x64)

Installed Media SDK DirectShow filters:

Installed Intel Media Foundation Transforms:
  Intel?Hardware VC-1 Decoder MFT : {059A5BAE-5D7A-4C5E-8F7A-BFD57D1D6AAA}
  Intel?Hardware H.264 Decoder MFT : {45E5CE07-5AC7-4509-94E9-62DB27CF8F96}
  Intel?Hardware MPEG-2 Decoder MFT : {CD5BA7FF-9071-40E9-A462-8DC5152B1776}
  Intel?Quick Sync Video H.264 Encoder MFT : {4BE8D3C0-0515-4A37-AD55-E4BAE19AF471}
  Intel?Hardware Preprocessing MFT : {EE69B504-1CBF-4EA6-8137-BB10F806B014}
=====================================================

UpdateSurface() has the restriction that the source format must match the dest format.
So I changed VppParam's FourCC value from NV12 to RGB4 on InitMfxVppParams() of sample_encode sources.
I didn't touch the rest of VppParams setting except of FourCC value.

diff --git a/sample_encode/src/pipeline_encode.cpp b/sample_encode/src/pipeline_encode.cpp
index a6e0860..90d544a 100644
--- a/sample_encode/src/pipeline_encode.cpp
+++ b/sample_encode/src/pipeline_encode.cpp
@@ -426,7 +426,8 @@ mfxStatus CEncodingPipeline::InitMfxVppParams(sInputParams *pInParams)
     }

     // input frame info
-    m_mfxVppParams.vpp.In.FourCC    = MFX_FOURCC_NV12;
+    m_mfxVppParams.vpp.In.FourCC = MFX_FOURCC_RGB4;
     m_mfxVppParams.vpp.In.PicStruct = pInParams->nPicStruct;;
     ConvertFrameRate(pInParams->dFrameRate, &m_mfxVppParams.vpp.In.FrameRateExtN, &m_mfxVppParams.vpp.In.FrameRateExtD);

I printed the D3DSURFACE_DESC of each surface before RunFrameVPPAsync() for debug.

diff --git a/sample_encode/src/pipeline_encode.cpp b/sample_encode/src/pipeline_encode.cpp
index a6e0860..23149e7 100644
--- a/sample_encode/src/pipeline_encode.cpp
+++ b/sample_encode/src/pipeline_encode.cpp
@@ -1255,6 +1256,12 @@ mfxStatus CEncodingPipeline::Run()
             bVppMultipleOutput = false; // reset the flag before a call to VPP
             for (;;)
             {
+                D3DSURFACE_DESC pVppSurfaceDesc;
+                D3DSURFACE_DESC pEncSurfaceDesc;
+
+                ((IDirect3DSurface9*)(&m_pVppSurfaces[nVppSurfIdx])->Data.MemId)->GetDesc(&pVppSurfaceDesc);
+                ((IDirect3DSurface9*)(&m_pEncSurfaces[nEncSurfIdx])->Data.MemId)->GetDesc(&pEncSurfaceDesc);
+
                 sts = m_pmfxVPP->RunFrameVPPAsync(&m_pVppSurfaces[nVppSurfIdx], &m_pEncSurfaces[nEncSurfIdx],
                     NULL, &VppSyncPoint);

How can I convert colorspace from RGB4 to NV12 using RunFrameVPPAsync() ?

[Seongjun_Kim]

I found a solution from this link!
http://www.briancbecker.com/blog/2011/intel-quick-sync-hardware-encoding/

diff --git a/sample_encode/src/pipeline_encode.cpp b/sample_encode/src/pipeline_encode.cpp
index 352b27c..840da73 100644
--- a/sample_encode/src/pipeline_encode.cpp
+++ b/sample_encode/src/pipeline_encode.cpp
@@ -426,7 +426,7 @@ mfxStatus CEncodingPipeline::InitMfxVppParams(sInputParams *pInParams)
     }

     // input frame info
-    m_mfxVppParams.vpp.In.FourCC    = MFX_FOURCC_NV12;
+    m_mfxVppParams.vpp.In.FourCC = MFX_FOURCC_RGB4;
     m_mfxVppParams.vpp.In.PicStruct = pInParams->nPicStruct;;
     ConvertFrameRate(pInParams->dFrameRate, &m_mfxVppParams.vpp.In.FrameRateExtN, &m_mfxVppParams.vpp.In.FrameRateExtD);

@@ -444,6 +444,8 @@ mfxStatus CEncodingPipeline::InitMfxVppParams(sInputParams *pInParams)
     // fill output frame info
     MSDK_MEMCPY_VAR(m_mfxVppParams.vpp.Out,&m_mfxVppParams.vpp.In, sizeof(mfxFrameInfo));

+    m_mfxVppParams.vpp.Out.FourCC = MFX_FOURCC_NV12;
+
     // only resizing is supported
     m_mfxVppParams.vpp.Out.Width = MSDK_ALIGN16(pInParams->nDstWidth);
     m_mfxVppParams.vpp.Out.Height = (MFX_PICSTRUCT_PROGRESSIVE == m_mfxVppParams.vpp.Out.PicStruct)?

Then I won the success to convert colorspace from RGB4 to NV12 using RunFrameVPPAsync().
Thank you! :)

[Harshdeep_B_Intel]

That sounds great - glad your issue is fixed! 

[Sravanthi_K_Intel]

Hello there - Glad you found the workaround. Looking at your solution, it seems like all you were missing was explicitly setting the vpp.Out.FourCC to NV12. The current samples simply memcopy the input VPP params to out VPP params (and in mfx) - which does not work when input is non NV12, and causes issue (like yours) and we saw another case as well.

Thanks for reporting this behavior - the sample should reflect this change, and we hope to fix this. Thanks.

[jianfeng_s_]

Hi,

  I tried to convert color space by VPP(RGB4->NV12), initiated `pVPPSurfaceIn` like this:

g_pd3dDevice->CreateOffScreenPlainSurface(d3dDisplaymode.Width, d3dDisplaymode.Height, D3DFMT_A8R8G8B8, D3DPOOL_SYSTEMMEM, &g_pSurface, NULL);
g_pd3dDevice->GetFrontBufferData(0, g_pSurface);
g_pd3dDevice->UpdateSurface(g_pSurface, NULL, (IDirect3DSurface9*)pVPPSurfaceIn->Data.MemId, NULL);

but, I got a black screen after VPP. Did I miss something? Thank you.

[Surbhi_M_Intel]

Hi Jianfeng, 

The information you have provided is not sufficient to understand what's happening in your pipeline. Please send us the reproducer to debug the issue at our end, along with your system info(system analyzer logs, system analyzer is installed in your software directory under tools folder). If sending us a reproducer requires lot of code change then you can refer to sample_vpp which demonstrate color conversion pipeline as well or simple_6_encode_vmem_vpp_preproc, part of tutorial package has pipeline set for RGB32->NV12 color conversion in the most simplest manner. 
Please make sure you are setting correct mfx VPP in and out params. You can see above posts to confirm that or refer to simple_6_encode_vmem_vpp_preproc tutorial as well. 

Thanks,
Surbhi

[jianfeng_s_]

Hi Surbhi,

Thanks for your reply. I tested VPP pipeline with rgb32 file, and it's ok. Now my question is how to send surface(in DirectX) to VPP.

All samples get input from yuv file. Could you show me some example in which VPP get input from DirectX like GetFrontBufferData or UpdateSurface?

Look forward to your early reply. Thank you very much.