MFXVideoCORE_SyncOperation failing with MFX_ERR_DEVICE_FAILED

[Andrey_T_]

Hello,

I'm using FFmpeg for my transcoding needs (to transcode multiple live streams) and I've been looking into a QSV-enabled encoder which was recently added there. I looked at the code and it's a fairly simplistic, basically something among the lines of:

Result encode()
{
   /* fill surface */
    do {
        ret = MFXVideoENCODE_EncodeFrameAsync(session, NULL, surface, &bs, &sync);
        if (ret == MFX_WRN_DEVICE_BUSY) sleep(1);
    } while (ret > 0);

    if (ret < 0) return ret;

    if (sync) {
        MFXVideoCORE_SyncOperation(session, sync, 60000);
       /* consume encoded data in bs */
    }
}

I know it is not the recommended way to use QSV because it tries to sync operations as often as possible (I guess this way it was easier to do within FFmpeg infrastructure) but it it should work nevertheless. And it usually works, until it doesn't. I've added some logs and it turns out MFXVideoCORE_SyncOperation would occasionally return -17 (MFX_ERR_DEVICE_FAILED) error. Sometimes it happens after just a few frames, sometimes it happens after several minutes of flawless work. Now, MediaSDK manual says it is a sort-of a fatal error and the application should reset and reinit everything; sample_encode basically does that. FFmpeg doesn't, that probably should be fixed.

My concern, however, is the fact it happens in the first place. Why would it fail? I can't think of any events that could lead to a failure. I'm also concerned that even if there is a recovery procedure to work around this fail, there can be some problems in the output bitstream (I think at least few latest video frames sent to EncodeFrameAsync would be lost, and also maybe there would be some damage to the bitstream)

Is there something I can do to fix that failure or to try to understand why it is happening? This all happens on a Windows 8.1 x64 with i7-4790K CPU and GPU driver version 10.18.14.4222

Thanks in advance

[Andrey_T_]

Any idea? Anyone?

[Harshdeep_B_Intel]

Hi Andrey,

It is recommended to avoid -17 (MFX_ERR_DEVICE_FAILED) error message, application has to initialize the SDK library right before actual encoding operations to determine the if hardware acceleration capabality (QSV) is supported.  What is the return query status from MFXInit? Usually if you sync operation without Init, it starts the operation but then as soon as Init function fails to initialize SDK library interruption on hw operation will occur as resulting in error. 

Thanks,

[Andrey_T_]

FFmpeg does MFXInit before doing any encoding; I don't think this is related to MFXInit.

Apparently it has something to do with buffer management though I cannot figure out what's wrong. There are no errors if I try to encode raw NV12 video; however, transcoding encoded stream eventually fails. I found out it doesn't fail if I force FFmpeg to copy video frame into another buffer prior to handing it off to Media SDK. I don't see anything wrong in code; I also tried to track buffer allocations and don't see anything fishy there either. Anyway, this might be a FFmpeg issue after all, so I'll write them. In meantime, maybe there is some way to get some insight why this error message occurs? Maybe there are some debug logs that can be switched on or something?

[Harshdeep_B_Intel]

Hi,

Anyway, this might be a FFmpeg issue after all, so I'll write them. In meantime, maybe there is some way to get some insight why this error message occurs? Maybe there are some debug logs that can be switched on or something? -> To isolate the issue, you can use MediaSDK debug tracer tool located in <install_folder>/tools/mediasdk_tracer/tracer.exe". Using this tool you can log all the SDK API calls and parameters to a log file. The tool can also generate full log of API calls b/w application and SDK library with per-frame processing.Please refer to "readme-mediasdk-tracer.pdf" for more details on utility usage.

Thanks,

[Andrey_T_]

I was a bit too hasty; the workaround doesn't always work. I tried transcoding FullHD file this time and it started to fail again. I tried tracer utility, and it is quite useful though it didn't gave me any new insights (since I already instrumented all API calls). Here is a log of a run where it failed particularly quickly: https://gist.github.com/tea/40b720a8868ee0601acd . I've no idea where to go from here (I'll try to do the same on Linux with Gen3 processor just to see if it works there or not...)

[Harshdeep_B_Intel]

Hello Andrey,

Can you send me your application code, so I reproduce the issue for further debug and follow up with suggestions. You share reproducer code directly with me via "Send Author A Message".   

Thanks, 

[Milen_Y_]

Hi,

I also experience exactly the same problem. Any news about this issue? Was it resolved?

My source code is very similar as structure to simple_3_encode from http://software.intel.com/sites/default/files/mediasdk-tutorials-0.0.3.zip

I use the H.264 QSV encoder (MFX_IMPL_HARDWARE_ANY). It usually works, until it doesn't. MFXVideoCORE_SyncOperation would occasionally return -17 (MFX_ERR_DEVICE_FAILED) error. It happens after several minutes of flawless work.

The MFX_ERR_DEVICE_FAILED happens on i7-4770K, windows 10, driver 15.40.10.64.4300.
The MFX_ERR_DEVICE_FAILED happens on i5-4690K, windows 8, driver 15.36.28.4332.

The H.264 QSV encoder runs flawlessly on i5-6600K and i5-6200U, Windows 10. No MFX_ERR_DEVICE_FAILED has been observed on these CPUs.

Thanks!

[Harshdeep_B_Intel]

Hi Milen,

As, you mentioned "experience exactly the same problem" are you using the same scenario? Are you using latest version of ffmpeg libraries? What do you mean by "It usually works, until it doesn't" ? Is issue visible sporadically? If you can share you application code and steps for repro, we will able to investigate this issue further.

Also, please consider starting a new thread on Media forum - https://software.intel.com/en-us/forums/intel-media-sdk, it will be easier for us to log, track and respond.

Thanks, 

[Milen_Y_]

Hi Harsh,

I'm not using ffmpeg. I'm using the sample simple_3_encode from http://software.intel.com/sites/default/files/mediasdk-tutorials-0.0.3.zip
The sample is compiled with VS 2015 in Release|Win32 mode. The input file is uncompressed YUV: https://bitbucket.org/myordanovps/qsvsamples/downloads/352x240.7z

I run a BAT file with the following code:
 
-----------
:loop_start
simple_encode.exe -g 352x240 -b 300 -f 25/1 352x240.yuv out.h264 && GOTO loop_start
-----------

The simple_encode.exe runs about 5 - 500 times successfully and then the encoding fails with MFX_ERR_DEVICE_FAILED:

The test machine is i7-4770K, 32GB RAM, Windows 10 Pro 64bit, driver 15.40.10.64.4300. It also has NVIDIA GTX 960 and NVIDIA Quadro K600.

Here is my command prompt log:
-------------------------------------------
E:\tmp\mediasdk-tutorials-0.0.3\_build\Win32\Release>run_test.bat

E:\tmp\mediasdk-tutorials-0.0.3\_build\Win32\Release>simple_encode.exe -g 352x240 -b 300 -f 25/1 352x240.yuv out.h264   && GOTO loop_start
Frame number: 3644
Execution time: 3.59 s (1013.70 fps)

E:\tmp\mediasdk-tutorials-0.0.3\_build\Win32\Release>simple_encode.exe -g 352x240 -b 300 -f 25/1 352x240.yuv out.h264   && GOTO loop_start
Frame number: 3644
Execution time: 3.55 s (1026.90 fps)

E:\tmp\mediasdk-tutorials-0.0.3\_build\Win32\Release>simple_encode.exe -g 352x240 -b 300 -f 25/1 352x240.yuv out.h264   && GOTO loop_start
Frame number: 3644
Execution time: 3.74 s (975.18 fps)

E:\tmp\mediasdk-tutorials-0.0.3\_build\Win32\Release>simple_encode.exe -g 352x240 -b 300 -f 25/1 352x240.yuv out.h264   && GOTO loop_start
Frame number: 3644
Execution time: 3.59 s (1016.45 fps)

E:\tmp\mediasdk-tutorials-0.0.3\_build\Win32\Release>simple_encode.exe -g 352x240 -b 300 -f 25/1 352x240.yuv out.h264   && GOTO loop_start
Frame number: 3644
Execution time: 3.57 s (1021.92 fps)

E:\tmp\mediasdk-tutorials-0.0.3\_build\Win32\Release>simple_encode.exe -g 352x240 -b 300 -f 25/1 352x240.yuv out.h264   && GOTO loop_start
Frame number: 1635
 Device operation failure. src\simple_encode.cpp 223
-------------------------------------------

[Harshdeep_B_Intel]

Hi Milen, 

Thanks for sharing the complete system details and as it is different from original topic on the thread and thank you for starting a new thread: https://software.intel.com/en-us/forums/intel-media-sdk/topic/606141 on this issue. It is been currently handled and issue further discussed on new thread. 

Thanks,   

[mahmut_şamil_s_]

I also got MFX_ERR_DEVICE_FAILED message from sync operation in a similar way. is there any solution about this issue?

[Harshdeep_B_Intel]

Hi Mahmut, 

Please start a new thread on the Media Forum so we can track your issue separately with system information,driver version, OS version etc with brief description. Take a look at suggested format (https://software.intel.com/en-us/forums/intel-media-sdk/topic/531083). Attach a log from system analyzer tool https://software.intel.com/en-us/articles/media-sdk-tools. This will help us understand your system environment better and provide our best inputs to resolve the issue. 

Thanks,

[Bundy__Al]

Has someone found out how to fix or workaround MFX_ERR_DEVICE_FAILED from MFXVideoENCODE_EncodeFrameAsync?
It seems that I have the same issue when using handbrake and according to the developers of handbrake this seems to be an issue with the driver or the sdk.
https://github.com/HandBrake/HandBrake/issues/1482

This are my hardware details according to handbrake:

[22:20:58] CPU: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz

[22:20:58]  - Intel microarchitecture Haswell

[22:20:58]  - logical processor count: 8

[22:20:58] Intel Quick Sync Video support: yes

[22:20:58]  - Intel Media SDK hardware: API 1.20 (minimum: 1.3)

[22:20:58]  - H.264 encoder: yes

[22:20:58]     - preferred implementation: hardware (any) via D3D11

[22:20:58]     - capabilities (hardware):  breftype icq+la+i+downs vsinfo opt1 opt2+mbbrc+extbrc+trellis+ib_adapt+nmpslice

[22:20:58]  - H.265 encoder: no