JPEG Encoder and aligned width/height

[Darrell_S_]

How does the JPEG encoder handle non aligned data?  Specifically, I have content that is 853x420, and 720x368 and even though I set the encoding parameters to be a multiple of 16 -

encoderParams.mfx.FrameInfo.Width = MSDK_ALIGN16(width);
encoderParams.mfx.FrameInfo.Height= MSDK_ALIGN16(height);

And I set the surfaces to be a multiple of 32

mfxU16 width = MSDK_ALIGN32(encoderRequest.Info.Width);
mfxU16 height = MSDK_ALIGN32(encoderRequest.Info.Height);

Using 422/YUY2 data I create a surface as follows:

encoderSurface->Data.Y = ippiMalloc_8u_C2(width, height * 2, &stride);

The  syncOperation gets an exception and returns UNKNOWN(-1).  
So how does the JPEG compressor handle odd/non-aligned data?

[Harshdeep_B_Intel · Accepted solution]

Hi,

The reason "syncOperation gets an exception and returns UNKNOWN(-1)." is because you are using 422/YUY2 data as input to the JPEG encoder. Only NV12 format input is support by JPEG encoder. Convert the input stream from YUY2 to NV12 format. Please refer to the https://software.intel.com/media-solutions-portal. Also please encode using one of the samples "sample_encode"  as we did not see issue with handling non-aligned data or cropping.

Thanks,

[Darrell_S_]

Thanks for clarifying this.  No where in your JPEG documentation can I find it saying that NV12 format is the only input supported by JPEG.  I do see this in the on-line docs - "The Intel Media SDK sample filters require the input data to be in NV12 format. " But I read this to mean that the SAMPLES are set to be NV12.  It wasn't clear that the compressor ONLY accepts NV12.  I'll give that a shot now.

[Darrell_S_]

Just to clarify, the input is always in NV12 format, but I can set the FOURCC to MFX_FOURCC_YUY2 and the Chroma format to MXF_CHROMAFORMAT_YUV422 since I want YUY2 for output.

[Harshdeep_B_Intel]

Hi,

Following are some updates regarding to the YUY2 and NV12 input  from our developer. JPEG encoder actually supports RGB32, YUY2 and YV12 in addition to NV12. But this is not documented in the manual. Apologize for the confusion. 

Also from the JPEG documentation: " The application should properly configured chroma sampling format and color format. FourCC and ChromaFormat fields in mfxFrameInfo structure are used for this. For example, to encode 4:2:2 vertically sampled YCbCr picture, the application should set FourCC to MFX_FOURCC_YUY2 and ChromaFormat to MFX_CHROMAFORMAT_YUV422V. To encode 4:4:4 sampled RGB picture, the application should set FourCC to MFX_FOURCC_RGB4 and ChromaFormat to MFX_CHROMAFORMAT_YUV444".

Also if you are using the sample " simple_encode", as said before it takes input only in NV12 format. If you have made changes to sample to perform JPEG encoding then NV12 will be its input format. Also if you can run "mediasdk_sys_analyzer.exe (found in <msdk_intall>/tools folder)" and send us in the output. This would help us a lot to dive in deeper into the issue. 

Also I see JPEG Compressor - GetVideoParam and MJPEG RGB Encoder threads in the forum. Are these same issues related to this? If yes, please post all your questions in a single thread, and rest assured all of us are best capable to answer your questions.

Thanks,

[Darrell_S_]

Thank you for clearing up the confusion.  Yes, those other questions in the forum are from me - even though the questions are all around JPEG compression - I thought they were separate questions.

So I'm back at the question about alignment and getting an exception from the Media SDK.  In my other posts, I gave a stack dump as well as what is in the registers.  Is there any way to determine what is going wrong?  For me, the SDK is a black box and I have no idea if the problem is with my input buffer, output buffer, or some internal buffers used by the SDK.  From reading the memory addresses, it looks to be something within the SDK, but it would be nice to get confirmation on that.

Hopefully the system analysis attached will help isolate why I'm seeing 0 being returned for the BufferSizeInKB when I perform a GetVideoParam() call.

I appreciate your help.  Thanks.

[Harshdeep_B_Intel]

Hi 

Thanks for the SysAnalysis and also can you please run the mediasdk_tracer (found in <msdk_install>/tools/mediasdk_tracer) as this will provide us more information in regarding to the input buffer, output buffer, or internal buffers used by the SDK and send us the output . I will be able to look into the issue further. 

Also we did not see 0 reported by BufferSizeInKB  in any of our scenarios. So the output from the tracer will shed more light into the issue in regard to your input and output by SDK. 

Thanks,

[Darrell_S_]

The tracer core has an access violation.  I'm running the simple_encode.cpp that I sent earlier.

Unhandled exception at 0x641025FA (tracer_core32.dll) in simple_encode.exe: 0xC0000005: Access violation reading location 0x00000005.

     tracer_core32.dll!641025fa()    Unknown
     [Frames below may be incorrect and/or missing, no symbols loaded for tracer_core32.dll]    
     tracer_core32.dll!6411ebfd()    Unknown
     tracer_core32.dll!64105731()    Unknown
     [External Code]    
     tracer_core32.dll!64109f97()    Unknown
     [External Code]    
     tracer_core32.dll!64128e15()    Unknown
     tracer_core32.dll!64128e15()    Unknown
     [External Code]    
>    simple_encode.exe!Initialize(int impl, mfxVersion ver, MFXVideoSession * pSession, mfxFrameAllocator * pmfxAllocator, bool bCreateSharedHandles) Line 34    C++
     simple_encode.exe!main(int argc, char * * argv) Line 150    C++

Not Flagged    >    3644    0    Worker Thread    libmfxsw32.dll thread    libmfxsw32.dll!60ff9527    Normal
Not Flagged        3668    0    Main Thread    Main Thread    tracer_core32.dll!641025fa    Normal
Not Flagged        3000    0    Worker Thread    libmfxsw32.dll thread    libmfxsw32.dll!60ff93a3    Normal
Not Flagged        2516    0    Worker Thread    libmfxsw32.dll thread    libmfxsw32.dll!60ff93a3    Normal
Not Flagged        3324    0    Worker Thread    libmfxsw32.dll thread    libmfxsw32.dll!60ff93a3    Normal
Not Flagged        1820    0    Worker Thread    libmfxsw32.dll thread    libmfxsw32.dll!60ff93a3    Normal

[Harshdeep_B_Intel]

Ok, this is new. As tracer is not able to locate the sw dll's. Before I go ahead and dive into this more. Can please go ahead and let me know:

1) Are you using a modified simple_encode code to get output in JPEG format. If yes, please send in the modified code.

2) Also can u please use the sample_encode (found at Video Encoding Sample) with the tracer and send me the tracer log. 

Thanks,

[Darrell_S_]

1) Yes, I'm using a modified simple_encode code to get output JPEG format. - see attached.

2) Running sample_encode with the source video samples as suggested also gives me the same access violation with tracer.exe.  It says the library can't be loaded.  

[Harshdeep_B_Intel]

Hi

After speaking to my team, please see my updates below:  

1) In your modified simple_encode.cpp code with "mfxEncParams.mfx.FrameInfo.FourCC = MFX_FOURCC_YUY2" as surface allocated to the encoder has to be NV12 to the encoder. Please go through the online JPEG documentation.

2) Also in regard to the sample_encode.cpp " [-nv12] - input is in NV12 color format, if not specified YUV420 is expected". Please go through the help section for Video Encoding Sample as we did not see any issue in regard to JPEG encoder. Please let us know if this helps.

Thanks,

[Darrell_S_]

I am so confused now.   You said that the input can be something other than NV12.

Harsh Jain (Intel) wrote:

Following are some updates regarding to the YUY2 and NV12 input  from our developer. JPEG encoder actually supports RGB32, YUY2 and YV12 in addition to NV12. But this is not documented in the manual. Apologize for the confusion. 

Now you're saying the input surface has to be NV12.  So what is it?  I am successful in passing in YUY2 and YV12 content and the JPEG encoder does a wonderful job when the image dimensions are a multiple of 32.  Only when the dimensions are not a multiple of 32 do I see an exception being raised by the Media SDK.  That's great to hear if you are not seeing any issues with the JPEG encoder - then that means my parameters must be wrong.  I'll triple check those.

The mediasdkjpeg-man.pdf that comes with the Media SDK states "...to encode 4:2:2 vertically sampled YCbCr picture, the application should set FourCC to MFX_FOURCC_YUY2 and ChromaFormat to MFX_CHROMAFORMAT_YUV422V."  No where does it say that the surface needs to be NV12.  I see in the mediasdk-man.pdf that it says if one is using Direct3D9, that the input color format is NV12 in table 6  on page 23, but I am not using Direct3D9 or Direct3D11 or VPP.   

[Harshdeep_B_Intel]

Hi,

So this clears it, " I am not using Direct3D9 or Direct3D11 or VPP". Also "I am successful in passing in YUY2 and YV12 content and the JPEG encoder does a wonderful job when the image dimensions are a multiple of 32" this is good. Please check with the parameter set up, and if they are rite then you will be able to successfully encode from YUY2 to JPEG.

Also I strongly suggest to take a look into one of our sample_encode "Video Encoding Sample" which has used surfaces for encode but will help in setting up of parameters suited for your environment. Hope this helps.

Thanks,

[Darrell_S_]

Hi Harsh,
Thanks for your suggestions.  I don't see any problems with my parameters, but if that is where the problem could originate, I'll review it further.  I have been reviewing the "Video Encoding Sample" but nothing is jumping out as the root cause to my exception.

Kindly,

Darrell

[Darrell_S_]

Harsh,

If I run the simple_vpp project from the video encoding sample without any modifications and using the input parameter of -sw -g 853x480, the session.SyncOperation() returns MFX_ERR_UNDEFINED_BEHAVIOR(-16).

If I run the simple_vpp project again without any modifications and set the input parameter to -sw -g 864x480, then the session.SyncOperation() returns MFX_ERR_NONE(0).

The same thing can be said for the simple_vpp_d3d project:  A size of 853x480 gives an error, and 864x480 doesn't.

This tells me that input that is not a multiple of 32 has problems within the Intel Media SDK.  Can you please investigate that.

Darrell

[Harshdeep_B_Intel]

Hi ,

So I further looked into the issue you are seeing and we are not able to see the same. So the code you are modifying or looking into are tutorials. The link to samples (http://software.intel.com/sites/default/files/IntelMediaSDK_VideoEncodingSample_5.0.461.91752.msi)  from here download and install the software and with sample_encode.cpp (found in "Intel Media SDK Samples/sample_encode" ) you will be able to perfectly encode any input format to a JPEG format. Please refer to documentation for help. Hope this helps.

Thanks,

[Darrell_S_]

 Harsh,

Thank you for all your guidance.  If I set the encoderParams.mfx.FrameInfo.FourCC to MFX_FOURCC_NV12 and set the encoderParams.mfx.FrameInfo.ChromaFormat to MFX_CHROMAFORMAT_YUV420, then I no longer see the exception being raised by the Intel Media SDK when the image dimensions are not a multiple of 32. 

This leads to the question of - how do I specify to the JPEG compressor that I wish to have YUV422/YUY2 output then if I set the input to NV12?  Do I need to pull in VPP to do that?

Wouldn't you agree that this also suggests that if the input is not NV12 and the dimensions aren't a multiple of 32, then there is still a problem within the JPEG compressor?

Darrell

[Harshdeep_B_Intel]

Hi,

So as you said " If I set the encoderParams.mfx.FrameInfo.FourCC to MFX_FOURCC_NV12 and set the encoderParams.mfx.FrameInfo.ChromaFormat to MFX_CHROMAFORMAT_YUV420, then I no longer see the exception being raised by the Intel Media SDK when the image dimensions are not a multiple of 32. " This is exactly how our samples work, and hence we did not see any issue on our side.

Also "how do I specify to the JPEG compressor that I wish to have YUV422/YUY2 output then if I set the input to NV12". We support only 420  format, But we have tools like Vpp, using them you can have YUV422/YUY2 output. 

Also, from my previous comment: " So the code you are modifying or looking into are tutorials. The link to samples (http://software.intel.com/sites/default/files/IntelMediaSDK_VideoEncodin...)  from here download and install the software and with sample_encode.cpp (found in "Intel Media SDK Samples/sample_encode" ) you will be able to perfectly encode any input format to a JPEG format. Please refer to documentation for help. Hope this gives you a solution."

Thanks,

[Darrell_S_]

Thanks for all your assistance.  I have the JPEG encoder working now.  I was just hoping to avoid taking a 422 YUV format clip and down sample it to 420 (NV12) only to have to up sample it again on the output side back to 422 YUV.  I see that the VPP will perform this for me, so am looking to integrate that into my program.

[Harshdeep_B_Intel]

Hi,

Great, thanks for the information.

On a side note - After you work out the implementations, if you would like your code to be useful for many more people, I would highly encourage you publish an article with your code. Once you publish and send the link to me, I can verify and point people to it on the forum as a resource. Here is the link - https://software.intel.com/node/add/article.

Glad we could help!

Thanks,

[Harshdeep_B_Intel]

Hi,

I have an update with the input formats supported by JPEG encoder and JPEG encode supports RGB32, YUY2 and YV12 in addition to NV12. Also this has been documented in the new version of the manual which is available for you to refer and you can avoid " 422 YUV format clip and down sample it to 420 (NV12) only to have to up sample it again on the output side back to 422 YUV". Please let me know if this helps.

Thanks,