Media SDK decode MJPEG to RGB4

[James_B_9]

Hi,

I am trying to decode an mjpeg video to RGB4 color format.  In the mediasdkjpeg-man.pdf document it says:

"Unlike other SDK decoders, JPEG one supports three different output color formats - NV12, YUY2 and RGB32. This support sometimes requires internal color conversion and more complicated initialization. The color format of input bitstream is described by JPEGChromaFormat and JPEGColorFormat fields in mfxInfoMFX structure. The MFXVideoDECODE_DecodeHeader function usually fills them in. But if JPEG bitstream does not contains color format information, application should provide it. Output color format is described by general SDK parameters - FourCC and ChromaFormat fields in mfxFrameInfo structure."

I am assuming this means I can decode directly to RGB4 in hardware and do not need to use the VPP as with the h264 decoder.  If this is correct is their an example of how to do this?

I can successfully decode mjpeg by adapting the mediasdk-tutorials-0.0.3 example "simple_2_decode" to include the "mfxjpeg.h" header and MFX_CODEC_JPEG however I don't know how to change the output color format, specifically which mfxFrameInfo structure, described below, to change.

"Output color format is described by general SDK parameters - FourCC and ChromaFormat fields in mfxFrameInfo structure"

Thank you.

[Shaojuan_Z_Intel]

Hi James,

MediaSDK decoder only support YUV420 as output format. For further color conversion to RGB4, you can refer to sample_VPP which supports a list of color formats shown in mediasdk-man.pdf. Thanks!

[James_B_9]

Hi Shaojuan,

Thank you for your quick response.

If the MediaSDK does not support other output formats when decoding MJPEG, what does the following, taken from the mediasdkjpeg-man.pdf, mean?

"Unlike other SDK decoders, JPEG one supports three different output color formats - NV12, YUY2 and RGB32. This support sometimes requires internal color conversion and more complicated initialization."

Thank you,

James.

[Shaojuan_Z_Intel]

Hi James,

After further investigation, MediaSDK MJPEG decoder does support three output formats: NV12, YUY2 and RGB32. I was wrong in my previous comment. Sorry about the confusion. To use RGB32 as output, a couple things need to be modified: mfx.FrameInfo.FourCC = MFX_FOURCC_RGB4 and mfx.FrameInfo.ChromaFormat = MFX_CHROMAFORMAT_YUV444. Please let us know if it doesn't work. Thanks!

[James_B_9]

Hi Shaojuan,

Thanks again for your quick reply.

I have been trying to set both

mfx.FrameInfo.FourCC = MFX_FOURCC_RGB4

and

mfx.FrameInfo.ChromaFormat = MFX_CHROMAFORMAT_YUV444

in the "simple_2_decode" example but I am not sure exactly where to set them.  I assume that I should be setting the mfx inside the mfxVideoParam structure which I use to initialize the decoder? 

If this is correct then where do I set them.  If I do it before decoding the video header they get overwritten with MFX_FOURCC_NV12 and MFX_CHROMAFORMAT_YUV420, if I do it after that but before allocating the surfaces DecodeFrameAsync() returns MFX_ERR_UNDEFINED_BEHAVIOR and if I do it after allocating the surfaces but before initializing the decoder SyncOperation() returns MFX_ERR_UNKNOWN.

Thank you,

James.

[Shaojuan_Z_Intel]

Hi James,

The right place to set FourCC and ChromaFormat is after DecodeHeader but before Query and Init. I tried the setting changes in the simple_decode, and see the same "undefined behavior" as yours, but in sample_decode, which is part of sample package, with the same setting changes, there is no error in decoding from JPEG to RGB32. The current simple_decode only accept h264 as input codec, to decode JPEG to RGB32 format, in addition to change of MFX_CODEC_JPEG, there must be some other parameters that need to be changed. I will update once I have more information. Right now, you may try to use the sample_decode, make corresponding changes, and see if it works. Thanks!

[James_B_9]

Hi Shaojuan,

Thank you, it works with sample_decode, however I suspect this is because it is using directx surfaces.  To confirm this I made the same changes

mfx.FrameInfo.FourCC = MFX_FOURCC_RGB4

and

mfx.FrameInfo.ChromaFormat = MFX_CHROMAFORMAT_YUV444

to simple_2_decode_vmem (in addition to changing the codec to JPEG and including the mfxjpeg.h), and the decoder produces RGB4 output without any problems.

Is it possilble this feature is only available when using D3D surfaces, or do I need to fill some other fields when initializing the surface headers in system memory?

Currently I initialize the system memory surface headers by following the example code given in the tutorial, as below

    mfxU16 width = (mfxU16) MSDK_ALIGN32(Request.Info.Width);
    mfxU16 height = (mfxU16) MSDK_ALIGN32(Request.Info.Height);
    mfxU8 bitsPerPixel = 32;        // BGRX is a 32 bits per pixel format
    mfxU32 surfaceSize = width * height * bitsPerPixel / 8;
    mfxU8* surfaceBuffers = (mfxU8*) new mfxU8[surfaceSize * numSurfaces];

    // Allocate surface headers (mfxFrameSurface1) for decoder
    mfxFrameSurface1** pmfxSurfaces = new mfxFrameSurface1 *[numSurfaces];
    MSDK_CHECK_POINTER(pmfxSurfaces, MFX_ERR_MEMORY_ALLOC);
    for (int i = 0; i < numSurfaces; i++) {
        pmfxSurfaces = new mfxFrameSurface1;
        memset(pmfxSurfaces, 0, sizeof(mfxFrameSurface1));
        memcpy(&(pmfxSurfaces->Info), &(mfxVideoParams.mfx.FrameInfo), sizeof(mfxFrameInfo));
        pmfxSurfaces->Data.R = &surfaceBuffers[surfaceSize * i];
        pmfxSurfaces->Data.G = pmfxSurfaces->Data.R + 1;
        pmfxSurfaces->Data.B = pmfxSurfaces->Data.G + 2;
        pmfxSurfaces->Data.Pitch = width*4;
    }

Is this correct?

Thank you,

James.

[Shaojuan_Z_Intel]

Hi James,

The surface settings for RGB in MediaSDK should be:

pmfxSurfaces2->Data.B = &surfaceBuffers2[surfaceSize * i];
pmfxSurfaces2->Data.G = pmfxSurfaces2->Data.B + 1;
pmfxSurfaces2->Data.R = pmfxSurfaces2->Data.B + 2;
pmfxSurfaces2->Data.A = pmfxSurfaces2->Data.B + 3;

You are correct with the Pitch setting. Please let us know if you still saw error. Thanks!

[James_B_9]

Hi Shaojuan,

Thank you for you time,  it works.

It appears the errors were caused by my careless initialization of the surface headers.

James.