OS Type: Windows 10 Enterprise x64
CPU Type: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
Intel iGPU: Intel HD Graphics 4600
Intel Driver: 220.127.116.1180
What and how am I doing (note all below is on Hardware plugin):
I am using HEVC decoding of multiple streams at the same time. I am using following pattern in following sequence:
Create session/init/query/decoder and load plugin by GUID - MFXVideoUSER_Load(Session, HW_plugin_GUID, 1)
some parameters: MFX_IMPL_HARDWARE_ANY with MFX_IMPL_VIA_D3D11,
AsyncDepth = 4,
MFX_BITSTREAM_COMPLETE_FRAME is set,
Flush decoder (with decode calls with NULL)
Destruct decoder and unload plugin by GUID - MFXVideoUSER_UnLoad(session, HW_plugin_GUID)
Sometimes (I have not found a pattern) my application hangs forever when destroying the decoder. I have traced the CallStack:
So MFXVideoSession destructor calls MFXClose and this function never returns.
How to reproduce:
Well it can be reproduced even with one application running and trying to create a decoder (same sequence described above) -> decode one frame -> destroy decoder. do this 1000 times right after each other. Sometimes the application will hand forever.
It is easier for me to reproduce if I run multiple separate applications that do the same (4-5 app with 1000 iterations each), some of the applications will hang .. sometimes its 1 that hangs, sometimes 2..and sometimes all finish.
Is there something I am missing? Some call to release differently. I am basically using Procedure B according to https://software.intel.com/en-us/node/562446
Is there any workaround / fix you can recommend?
I have attached the one frame that I am decoding in the application.(but i can reproduce this with other streams as well). NOTE the file extension is txt(because i can not upload files without extension), but its a binary file, so just remove the extension.
Its basically not a question of IF this hang happens, but WHEN.
Please let me know if you need more info.
Doing the above using SOFTWARE plugin (15dd936825ad475ea34e35f3f54217a6 fully licensed) does NOT have the same hanging issue.(the code is exactly the same, the only difference is the loaded/unloaded plugin is different).
There is also a difference on how long the destructor takes :
on HW 30~ milliseconds
on SW 0~ milliseconds
so it seems the destructor of the hardware plugin does something differently.
Thank you for providing detailed description of this issue. We recently had a graphic driver release, can you update to the latest from https://downloadcenter.intel.com/ and rerun the test, and let me know if you observe the same behavior? From your description, I do not see anything steps missing from your application for HEVC decoding, but I did analyze the FrameI attached using our Intel(R) Video pro analyzer tool and observed a black frame with no picture (Is this expected?). If the issue is still seen after driver update, can you share with me a package (Zip it before uploading so, no files are missed) with a reproducer application, with executable, command line and the Input file.
FYI: HEVC decoding on 4th gen processors(Intel HD Graphics 4600) is a hybrid implementation and not a complete fixed function implementation. Hybrid implementation is not competitive compared to SW implementation. If you are interested achieving best quality/performance with HW (fixed function) implementation, I recommend upgrading to latest 6th generation processors which provides this feature support.
I just downloaded the driver from here:
and it seems it is not newer than the one i had, its the same driver as i tested on:
Intel Driver: 18.104.22.16880
so i think you can reproduce it with the current driver. The application i am using to trigger is a big UI and needs some backend installed.. i dont think i can zip this. Is there any other way you could reproduce this issue ? I think by starting multiple decoder threads you should be able to see this issue..
The version of driver installed on your machine is 22.214.171.12480 ( 3/8/2016) release. I suggest installing this latest driver version 126.96.36.19924 (4/22/2016) release link here: https://downloadcenter.intel.com/download/25948/Intel-Graphics-Driver-for-Windows-10-and-Windows-7-8-1-15-40-.
Reproducing the issue on our end is the key for further triage, for reproducer you can use sample_decode from package (https://software.intel.com/sites/default/files/managed/b7/7e/MediaSamples_Windows_2016.msi) which already includes complete end to end HEVC decoding pipeline and modify and build the sample to reproduce the issue observed. Please share the repro package, as reproducer will help looking into this issue further, as I ran HEVC multiple decoding threads session on my end on both the drivers and did not observe similar behavior.
I think you missed my second comment from previous message (below) which is another idea, a reproducer, so I can take a look at the application code, so we can traige if its is application or driver issue.
"Reproducing the issue on our end is the key for further triage, for reproducer you can use sample_decode from package (https://software.intel.com/sites/default/files/managed/b7/7e/MediaSample...) which already includes complete end to end HEVC decoding pipeline and modify and build the sample to reproduce the issue observed. Please share the repro package, as reproducer will help looking into this issue further, as I ran HEVC multiple decoding threads session on my end on both the drivers and did not observe similar behavior."
I didn't miss the code part,, its just as i wrote before, a big app and in any case i can not provide the code (legal stuff.. :) ), i can try to write the sequence of the calls i make if that can be usefull?
Let me clarify the request again for reproducer, so to avoid confusion. I understand your app code cannot he shared. Hence, please follow below steps for a reproducer:
1) Download and install https://software.intel.com/sites/default/files/managed/b7/7e/MediaSamples_Windows_2016.msi
2) Change sample_decode application to include sequence of calls made in your application. Build the sample_decode to reproduce the issue observed.
3) Share the sample_decode reproducer package.
I am still trying to find a way to provide you with my code (the legal stuff :) ).
But in the mean time i tried spawning 100 instances of sample_decode with a movie (H265) "Tears of steel" i found on the net:
the 1080p version. and i get some of the (2-5) instances crash, the closes to debug it i got was that it crashed in DecodeFrameAsync.
these are the parameters i use:
h265 -hw -d3d11 -i 1280x720_TearsOfSteel -p 33a61c0b4c27454ca8d85dde757c6f8e
I have 2 more questions:
1. We have an issue that when the RAM on the iGPU is maxed out we see a heavy load on the system that rises exponentially which seems to be paging file swaping activity. Is there any way to force MediaSDK to use RAM(on the machine) and not pagefile? Have you seen this behaviour ?
2. I downloaded the new MediaSDK 2016.0.2 and the new samples Intel Media SDK Samples 2016 188.8.131.52. And when i tried both my application and sample_secode with SW HEVC i get a crash in pipeline_decode.cpp line 389 :
sts = m_pmfxDEC->Init(&m_mfxVideoParams);
these are parameters I try with :
h265 -sw -i OneH265Frame -p 15dd936825ad475ea34e35f3f54217a6
I get this exception:
Unhandled exception at 0x00007FFE2AE97B8B (libmfxsw64.dll) in sample_decode.exe: 0xC0000005: Access violation reading location 0x0000000000000000.
Lets answer your question one by one, so we don't miss any:
1) i get some of the (2-5) instances crash -> I tested with sample_decode and could not reproduce this issue. Again, a reproducer with snapshot will help in further debugging this issue on our end.
2) Can you let me know which application is causing this issue. I have not observed this behavior before. In regard to configuring MediaSDK choosing path depending your scenario, take a look MediaSDK dev guide:https://software.intel.com/sites/default/files/Intel_Media_Developers_Guide_0.pdf, section 4.4.1 on IOPattern. Hope this helps!
3) Looks like you could be missing SW HEVC plugin here. HEVC SW plugin is distributed via MSS suite. Download plugin from here https://software.intel.com/en-us/intel-media-server-studio.
1 . just to make sure - you tried reproducing by spawning 100 sample_decode application at the same time? this is when i see this error, not of i run them sequentially.
2. This is our application that seems to do that. we are uding MFX_IOPATTERN_OUT_SYSTEM_MEMORY i suspect this may have influence on this. I will look again in the guide to make sure.
3. the plugin is not missing, we have a full licensed one, and if i debug i can see that it loads the plugin correctly, i even tried putting the plugin in the same location as sample_decode, but still get the crash.
1) Thanks for clarifying. Yes, I tried the same scenario on my end on 4th generation platform and no issue observed. Again, can you please specify, the error ID you are seeing and share a snapshot with reproducer.
3) Interesting, if plugin was loaded successfully. It should work. I tried both scenario 1) copying plugin to the same location as sample_decode and 2) complete path to plugin to command line and in both test cases with HEVC SW plugin passed. Just to isolate if its is not a VS config issue, can you try with the prebuilt sample_decode app at location (<installed directory/Intel Media SDK Samples 2016 184.108.40.206\_bin\x64). Let me know how it goes.
1) Can you clarify what you mean by "error ID" and "share a snapshot with reproducer". What i did was run this bat file:
for /l %%x in (1, 1, 100) do (
start sample_decode.exe h265 -hw -d3d11 -i "E:\TEMP\intel oveload test\1280x720_TearsOfSteel" -f 30 -p 33a61c0b4c27454ca8d85dde757c6f8e
and i saw that out of the 100 i spawned, randomly 2-5 instances crashed, and the dump pointed to DecodeFrameAsync.
How do i make the snapshot you need?
3) I have attached a picture of the command and failure on the prebuilt sample_decode, it is the same as with the one i built.
1) When a MSDK error occurs, then you will get a error value describing the issue status. Please refer to MSDK manual on error return values. Please note, MediaSDK samples are not production ready but are provided more as getting started samples to showcase MediaSDK features. In regard to your use case, when multiple decode process are launched few instances crash observed on DecodeFrameAsync; decoding process for that instance was interrupted by event such as there was need for additional working buffers. From MSDK manual : "decoding process can be interrupted for events such as if the decoder needs additional working buffers, is readying a frame for retrieval, or encountering a new header. In these cases, the function returns appropriate status code and moves the bitstream pointer to the remaining data." MediaSDK samples are opensourced https://github.com/Intel-Media-SDK/samples. I suggest modifying the sample to allocate enough memory for buffers to resolve this issue. Hope this helps!
2) Thank you for the screen shot. I was able to reproduce this issue on specific platform SKUs. This is a bug and we are investigating this issue.
If you look at my comment from Fri, 06/10/2016 - 01:22
you can see that i have tried to install 2016.0.2 version however there is a bug that prevents me from testing, so i have not been able to use/test 2016.0.2 version at all untill the bug is fixed.
Thanks for the reply,
I have asked the dev team and the bug is still unresolved. I am pushing them now to get it resolved.
Sorry I didn't look at the full thread of posts since I just take this one over.
The develop team is trying to reproduce the issue but they can not.
They have tried to use sample_decode as you suggested and their own method but they can't reproduce it. So could you give them more information about how to reproduce it or the environment information when the crash happened?
They are think this might be caused by the environment issue, for example, the other software component might interfere with the Media SDK sample. So some report like AIDA64 would be helpful.