H.264 decoder crashes when decoding certain streams

[Mike_D_1]

I've been working with the sample implementations of the DirectShow H.264 encoder and decoder filters from the Intel® Media SDK 2014 R2 for Clients and I noticed several issues.

The H.264 decoder filter crashes when attempting to decode the H.264 video streams in the two MP4 files available here: http://www.editorsean.com/blog/49-audiovideosynctest.

The media files play back fine using Windows Media Player (built-in H264 MF decoder) and using FFDshow decoders.

I also noticed that attempting to encode or decode video at 320x240 resolution seems to either hang the encoder and decoder, or simply generate very choppy files.

I should note that I'm mostly working with the 32bit versions of the filters but have tested with the 64bit and got the same results.

Also, the crash happens on Windows 7 x64 and I have also tested on Windows 8 x64 where the filter does not crash the video does not advance (ie. hangs on the frame where playback is started).

I would appreciate any insights.

[Sravanthi_K_Intel]

Regarding your comment - "The H.264 decoder filter crashes when attempting to decode the H.264 video streams in the two MP4 files available here:http://www.editorsean.com/blog/49-audiovideosynctest."

Are you passing MP4 file as input to the decoder? If so - The MSDK decoder expects video streams as input (.h264 etc), and not MP4 files. You could use ffmpeg or yamb software to extract the video stream from the MP4 file and pass that to the decoder.

Will get back to your other questions.

[Mike_D_1]

@SRAVANTHI K, no I'm not passing the MP4 to the decoder. I'm passing the H.264 video stream from the MP4 container in question.

I should have mentioned in the original post that this issue does not happen with other streams that I've come across. Only the two from the MP4s on that page.

[Sravanthi_K_Intel]

"I also noticed that attempting to encode or decode video at 320x240 resolution seems to either hang the encoder and decoder, or simply generate very choppy files." -- I tested the two video streams you mentioned on h264 encoder and decoder from the tutorials, and at: 720x576 resolution (native resolution) and 320x240 resolution. Both are working fine without any choppy-ness. Although, like I mentioned, I used the simple_encode and simple_decode tutorials instead of the DirectShow samples. Do you see choppy-ness on DirectShow encoder/decoder alone or the tutorials I tried on as well? It is a little unclear.

[Mike_D_1]

As I mentioned in the original post, I've been testing with the sample DirectShow filters. I haven't tried the tutorials you mentioned. Can you please include a donwload link to the specific tutorial you were using (so far I've only tried the DS samples).

Regarding choppiness, it's not with those media files but when attempting to record and encode H.264 from a web cam at 320x240 @30fps from a stream with input resolution of 320x240 @30fps. During encoding the picture appears choppy and the playback as well.

[Sravanthi_K_Intel]

Tutorials zip:http://software.intel.com/sites/default/files/mediasdk-tutorials-0.0.3.zip

Some information about each tutorial: https://software.intel.com/en-us/articles/media-sdk-tutorial-tutorial-samples-index

[Mike_D_1]

Thanks for posting the links for the tutorials. I will review those as well.

After playing with this some more to isolate the problem this is what I found.

Using GraphEdit, I've constructed a simple graph that takes input from the camera at 320x240 @30fps and runs that straight into the Intel Media SDK H.264 Encoder filter (configured for 320x240) which is then directly connected to the Intel Media SDK H264 Decoder filter which is connected directly to a Video Renderer. If I run this graph everything works as expected and there is no choppiness.

But, as soon as I try to insert a Infinite Tee or Smart Tee in-between the video input source (camera) and the encoder the graph fails to run (it hangs immediately after start).

I've tested on Windows 8 x64 with 32bit versions of the filters, running all encoding and decoding in hardware on an Intel i3 processor, as well as on Windows 7 x64 with 32bit filters with encoding in software and decoding in hardware (first generation i7 processor).

Note: This only happens with the 320x240 resolution, but not with 640x480 resolution.

Also: This only happens with the Intel Media SDK H.264 encoder filter. Not with other H.264 encoder (also tried MainConcept) and the Intel Media SDK H.264 decoder is not affected.

This seems to support your findings in that the issue is not in the encoder as much as in the direct-show filter encoder sample. Do you have any suggestions?

[Sravanthi_K_Intel]

Hello Mike,

Thanks for the follow-up. So, you were able to isolate the issue to DirectShow encoding filters (i.e., the behavior you are seeing happening with DirectShow encoding filters, and not in simple encoder). And this occurs with 320x240 only, and not with 640x480.

I will discuss this behavior with our dev team and colleagues and get back to you with an answer on this. In the meantime, if you can send us the output of running mediasdk_sys_analyzer.exe (found in <MSDK_install_path>/tools/), that will be good. Thanks!

[Mike_D_1]

Hi Sravanthi,

Yes, I've run multiple tests on Friday and have found that the problem seems to be related to the combination of filters present in the Graph. I have not had a chance yet to test the encoder outside of DirectShow but the reason I suspect the problem is not in the encoder per se is because I was able to create a filter graph where the input was coming in at 320x240 and encoding and decoding worked ok.

I will run the analyzer and post the output.

In the mean time, here is what to do to replicate the problem:

1. Start GraphEdit (or GraphStudioNext or similar tool). Create a new graph using the following filters:

1. Web Cam set to output video at 320x240 @30fps resolution, in YUV format
2. Infinite Tee (or Smart Tee) DirectShow filter
3. Intel Media SDK H.264 Encoder filter
4. Intel Media SDK H.264 Decoder (or other H.264 decoder filter)
5. Video Renderer filter (possibly 2 instances of this filter)

2. Connect the WebCam (1) capture output pin directly to the Tee filter (2) input pin. Connect one of the output pins of the Tee filter (2) to the input of pin of the Intel Media SDK H.264 Enoder filter (3). Connect the other output pin of the Tee filter (2) to the input pin of a Video Renderer filter (5) - although I was able to replicate this even without the second output pin of the Tee being connected. Connect the output pin of the Intel Media SDK H.264 Encoder filter (3) to the input pin of the Intel Media SDK H.264 decoder filter (4). Connect the output pin of the Intel Media SDK H.264 filter (4) to the input pin of the Renderer filter (5).

When attempting to run this graph the video will hang and the graph may stop.

If the Tee filter is removed from the graph (and the web cam output pin is connected directly to the Intel Media SDK H.264 encoder filter) then the problem does not happen.

Also, if the problem does not appear when the input video from the webcam is configured for 640x480 resolution.

[Mike_D_1]

Here is the output of running media_sdk_analyzer on my main development machine. This is the one running under Windows 7 x64 in software mode.  I will post the output of running the analyzer on the other machine, that has HW support as well.  Both machines exhibit the same behavior with respect to this problem.

Also, I've just run a few more tests that I think will prove useful:

- tested with two different web cams:  a Microsoft LifeCam Studio and a Logitech C920. The issue is the same regardless of camera used

- tested with a different set of encoder/decoder filters (using the open source CSiR VPP H264 encoder/decoder pair) and there were no issues recording at 320x240

- also tested with the Intel Media SDK encoder/decoder filters but using input at 640x480 and then down-converting the input in the encoder to 320x240 (via configuration) - that does not cause the problem.

This is the analyzer output:

Intel Media SDK System Analyzer (32 bit)

The following versions of Media SDK API are supported by platform/driver:

        Version Target  Supported       Dec     Enc
        1.0     HW      No
        1.0     SW      Yes             X       X
        1.1     HW      No
        1.1     SW      Yes             X       X
        1.3     HW      No
        1.3     SW      Yes             X       X
        1.4     HW      No
        1.4     SW      Yes             X       X
        1.5     HW      No
        1.5     SW      Yes             X       X
        1.6     HW      No
        1.6     SW      Yes             X       X
        1.7     HW      No
        1.7     SW      Yes             X       X
        1.8     HW      No
        1.8     SW      Yes             X       X

Graphics Devices:
        Name                                         Version             State
        VMware SVGA 3D                               7.14.1.5026         Active

System info:
        CPU:    Intel(R) Core(TM) i7 CPU       M 640  @ 2.80GHz
        OS:     Microsoft Windows 7 Ultimate
        Arch:   64-bit

Installed Media SDK packages (be patient...processing takes some time):
        Intel« Media SDK DirectShow* Sample 5.0.461.91752
        Intel« Media SDK 2014 R2 for Clients (x64)

Installed Media SDK DirectShow filters:
  Intel« Media SDK H.264 Encoder :
    D:\Work\Stoelting\_H264\Intel Media SDK\Samples\_bin\win32\h264_enc_filter.dll
  Intel« Media SDK H.264 Decoder :
    D:\Work\Stoelting\_H264\Intel Media SDK\Samples\_bin\win32\h264_dec_filter.dll
  Intel« Media SDK AAC Encoder :
    D:\Work\Stoelting\_H264\Intel Media SDK\Samples\_bin\win32\imc_aac_enc_ds.dll

Installed Intel Media Foundation Transforms:


Tips:
 - HW target does not work: If you expect it should, then make sure to install
   latest Intel gfx driver, and that Intel gfx is selected as primary driver
 - If Intel driver is associated with secondary adapter, make sure to
   initialize DirectX device used with Media SDK with corresponding adapter


Analysis complete... [press ENTER]

[Sravanthi_K_Intel]

Thanks for the quick update Mike, appreciate it. I have provided the information you gave above to the dev team and experts, and will update you with their response. In the meantime, it is worthwhile to mention that the DirectShow samples were (and are) intended to provide pointers on how-to use this feature in MSDK, and the sample applications were not intended to be used as-is for applications. There definitely is a possibility that these samples have some implementation issues or gaps that were not closed - thus the issues you are seeing. Having said that, I hope to get back to you with a workaround as soon as I hear back from the dev team. Thanks again.

[Mike_D_1]

Thanks for helping out. I realize the samples are not intended to be production grade and that Intel has no obligation to "make it work" but any help the developers can provide would be greatly appreciated. On our end we're working hard to get up to speed with the Intel Media SDK framework and to build up the necessary technical understanding but hopefully the experts on your team will realize quickly what the problem is and at least provide some hints as to what to look for (it will certainly be faster for them than it will be for me).

In the mean time, here is the output from the analyzer on the other system (Windows 8 x64 with HW support) that I'm testing on:

Intel Media SDK System Analyzer (32 bit)

The following versions of Media SDK API are supported by platform/driver:
        Version Target  Supported       Dec     Enc
        1.0     HW      Yes             X       X
        1.0     SW      Yes             X       X
        1.1     HW      Yes             X       X
        1.1     SW      Yes             X       X
        1.3     HW      Yes             X       X
        1.3     SW      Yes             X       X
        1.4     HW      Yes             X       X
        1.4     SW      Yes             X       X
        1.5     HW      No
        1.5     SW      Yes             X       X
        1.6     HW      No
        1.6     SW      Yes             X       X
        1.7     HW      No
        1.7     SW      Yes             X       X
        1.8     HW      No
        1.8     SW      Yes             X       X
Graphics Devices:
        Name                                         Version             State
        Intel(R) HD Graphics 4000                    9.17.10.2867        Active
System info:
        CPU:    Intel(R) Core(TM) i3-3120M CPU @ 2.50GHz
        OS:     Microsoft Windows 8
        Arch:   64-bit
Installed Media SDK packages (be patient...processing takes some time):
        Intel« Media SDK DirectShow* Sample 5.0.461.91752
        Intel« Media SDK 2014 R2 for Clients (x64)
Installed Media SDK DirectShow filters:
  Intel« Media SDK H.264 Encoder :
    C:\Work\_H264\Intel Media SDK\Samples\_bin\win32\h264_enc_filter.dll
  Intel« Media SDK H.264 Decoder :
    C:\Work\_H264\Intel Media SDK\Samples\_bin\win32\h264_dec_filter.dll
  Intel« Media SDK AAC Encoder :
    C:\Work\_H264\Intel Media SDK\Samples\_bin\win32\imc_aac_enc_ds.dll
Installed Intel Media Foundation Transforms:
  Intel« Quick Sync Video H.264 Encoder MFT : {4BE8D3C0-0515-4A37-AD55-E4BAE19AF471}
 
Analysis complete... [press ENTER]

I've also tried to capture a log using the trace tool but that seems pretty excessive (~15 MB) so please let me know if it would help to post, otherwise I'll just hold on to it.

Finally, I suspect the issue has something to do with memory allocation because as far as I understand the Tee filters duplicate the input frames on the output pins. So perhaps there is some incompatibility between the Intel MFX memory management and those filters.

[Sravanthi_K_Intel]

Hello Mike, appreciate the details you are posting.

"On our end we're working hard to get up to speed with the Intel Media SDK framework and to build up the necessary technical understanding " --> To get up to speed with Media SDK, we highly recommend using the tutorials I pointed to above. They are simple applications written with the purpose of easing developer learning curve. Apart from that, we have some technical articles that can help you as well - Here is one that explains the basic framework of developing applications with MSDK, and here is the link to the article of articles (This link contains all feature-specific articles as well). Hope this helps.

"I've also tried to capture a log using the trace tool but that seems pretty excessive (~15 MB) so please let me know if it would help to post, otherwise I'll just hold on to it." --> Yes, please hold on to it for now. I think the information you have provided so far should be enough. In the unlikely case, we will request the log.

[Sravanthi_K_Intel]

Hello Mike, What you are observing is a known limitation with height not aligned at 32. The Tee filter requests and allocated buffers of W*H, while the encoder tries to access data thinking the width is aligned at 16 and height at 32, and thus goes out of allocated memory boundary. (240 is not aligned at 32 while 480 is - thus 320x240 failing while 640x480 working for you).

This was the workaround suggested by our dev team if you want to make it work for 240 height - A workaround would be to either fix Tee filters or allocate mfxFrameSurfaces pool in encoder filter and introduce a copy from input MediaSample to mfxFrameSurface. For the former, not sure if the source code is accessible for the Tee fiters. You may find some leads on the web. 

Hope this helps.

[Mike_D_1]

Thanks for researching this and getting back to me.   I will investigate further.  However, I'm just curious, why does the encoder work when the surfaces are supplied by the camera as opposed to the filter? Wouldn't the miss-alignment happen with the camera as well?

Also, I realize that I should have started two separate threads but there is also the problem of playing back those files that I indicated in the original post. Did you mention those to the engineers?  Do you think it might be the same alignment issue (the resolution of those files is 720x576).

[Sravanthi_K_Intel]

Hello Mike - This extra information can help you understand the issue better. In connection Tee filter - Encoder filter, Tee filter allocates buffers for media samples. On filters connection, Encoder filter provides its requirements for those buffers through CEncoderInputPin::GetAllocatorRequirements method, where it request the buffer height to be aligned at 32.

This is done to avoid copy from MediaSample to mfxFrameSurface and map the surface directly to Media Sample memory buffer, and mfxFrameSurface must be have height aligned at 32 for interlaced content.

This could be the issue with the playback too. I will get back to you if there is more information I find out.