HW decoder always return green frames on Linux kernel 4.X

[Liu__Chao]

Hi,

I am trying to use QSV with ffmpeg on Linux to decode videos. The media SDK I use is from MediaServerStudio2016.

First I got it working on the golden system, CentOS 7.1. It works pretty well. The performance improvement compared to software is quite amazing. I am pretty happy with that.

Then I tried it on Ubuntu 16.04 (kernel is 4.4.0). Everything works well, except it always returns pure green frames..

I upgraded the kernel to 4.4.15, with the same problem. Same for kernel 4.1.28.

Eventually, it works on kernel 3.18.37. NOTE: this problem is only for hardware decoding. It works on all kernels if I remove libmfxhw64-p.so.1.17.

I wonder was there any bug introduced in the i915 driver on Linux 4?

[Liu__Chao]

Hmm, kernel 3.19.x also has this bug. Looks like only 3.18.x is fine to run QSV hardware decoder.

[Surbhi_M_Intel]

Hi there, 
I don't expect their should be issue on kernel 4.4.x. can you please try sample_decode on Ubuntu 16.04 and copy the output of your run, along with the o/p you receive. 

Thanks,
Surbhi

[Liu__Chao]

Thanks.

I run the sample_decode and it failed because SyncOutputSurface returns MFX_ERR_DEVICE_FAILED. I could repro the same failure on 3.18.37 and it worked on CentOS 7.1. So it might be related to that I didn't set things up correctly... However, the same environment works for ffmpeg on 3.18.37. I compared ffmpeg with sample_decode. ffmpeg only calls SyncOperation after DecodeFrameAsync returns non busy code. On the other hand, sample_decode calls SyncOperation after DecodeFrameAsync returns MFX_WRN_DEVICE_BUSY. Which one is correct? In case it might help, here is the ffmpeg code.

Following is the output of sample_decode

__bin/sample_decode h264 -i test_stream.264

libva info: VA-API version 0.99.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_0_32
libva info: va_openDriver() returns 0
Decoding Sample Version 0.0.000.0000

Input video     AVC
Output format   NV12
Input:
  Resolution    176x96
  Crop X,Y,W,H  0,0,0,0
Output:
  Resolution    176x96
Frame rate      30.00
Memory type             system
MediaSDK impl           hw
MediaSDK version        1.17

Decoding started
DecodeFrameAsync
DecodeFrameAsync
DecodeFrameAsync
DecodeFrameAsync
SyncOutputSurface err: -17
Frame number:    0, fps: 0.000, fread_fps: 0.000, fwrite_fps: 0.000
Return on error: error code -1, /4s/chao/projects/samples/samples/sample_decode/src/pipeline_decode.cpp 1782

Return on error: error code 1,  /4s/chao/projects/samples/samples/sample_decode/src/sample_decode.cpp   666

[Surbhi_M_Intel]

Hi Again,

To me it seems like an install error for the device failure. Is it possible to try it on a fresh OS? If you have any system with fresh OS then simply install Media Server Studio and check sample_decode before ffmpeg. If you see no issue, then install ffmpeg and test it otherwise please report back the issue. 

Couple of docs that might help (if you haven't seen them already)- 
FFMpeg install with MSDK - http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/quicksync-video-ffmpeg-install-valid.pdf ;
Generic Install - https://software.intel.com/en-us/articles/how-to-setup-media-server-studio-on-secondary-os-of-linux ;

We are taking in many parameters, just to get the problem right. Can you please tell in which scenario did it work and in which it didn't (Are below observations correct?)

OS - CentOS 7.1
Kernel - Default 
Sample_decode - no error
FFmpeg decode - no error 

OS - Ubuntu or CentOS?
Kernel - 3.18.37
Sample_decode - Device failure error
FFmpeg decode - Device failure error 

OS - Ubuntu 16.04
Kernel - 4.4.0 / 4.4.15 / 4.4.28
Sample_decode - Green frames?
FFmpeg decode - Green frames

-Surbhi

[Liu__Chao]

Hi Surbhi,

You are right that the sample_decode error was caused by install error on Ubuntu.

I installed 4.4.15 on CentOS 7.1 and reproduced the green frame issue:

./sample_decode h264 -i test_stream.264 -o test.yuv

libva info: VA-API version 0.99.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /4s/chao/bin/drivers/iHD_drv_video.so
libva info: Found init function __vaDriverInit_0_32
libva info: va_openDriver() returns 0
Decoding Sample Version 6.0.16043361.361

Input video     AVC
Output format   NV12
Resolution      176x96
Crop X,Y,W,H    0,0,0,0
Frame rate      30.00
Memory type             system
MediaSDK impl           hw
MediaSDK version        1.17

Decoding started
Frame number:  101, fps: 1242.221, fread_fps: 0.000, fwrite_fps: 2265.031
Decoding finished

517118

[Liu__Chao]

Not sure whether this is related:

On 4.4, metrics monitor doesn't work:

sudo ./run.sh
ERROR: Failed to initialize metrics monitor, error code -10 

[bordon_l_]

The current version of  ffmpeg(Version 3.1.1) has not yet fully realize QSV hardware decoding, please refer to the source code of ffmpeg. I think the fastest way is that Intel can help to achieve the function as soon as possible, So that we can use the QSV hardware decoding function on ffmpeg.  Perhaps Surbhi M can help us to convey this important proposal to Intel Develop Team, Let's cheer him on.

[Liu__Chao]

Hi Bordon,

I have been using QSV decoding from ffmpeg 3.0.1 for a while. The latest versions don't work because of some bugs, which is already (or going to be) fixed.

Could you give a bit details about what you said? Are you saying the performance could be further improved?

[Liu__Chao]

Looks like the green frame issue is also related to the GPU.

So far I could only repro it on HD Graphics 4400 on Linux kernel 3.19.x and above (CentOS and Ubuntu).

I have another machine with IRIS Pro 6200, I just installed Ubuntu 16.04 (kernel 4.4.0), it works!

Looks like HD Graphics 4400 + kernel >=3.19.x is the culprit...

[bordon_l_]

It couldn't work correctly if you carefully check the source code of the ffmpeg that QSV decoding, such as 'qsvdec_mpeg2.c' in ffmpeg:

static av_cold int qsv_decode_close(AVCodecContext *avctx)
{
    QSVMPEG2Context *s = avctx->priv_data;

    ff_qsv_decode_close(&s->qsv);

    return 0;
}

static av_cold int qsv_decode_init(AVCodecContext *avctx)
{
    return 0;
}

You can find the function return 0 directly.

[Liu__Chao]

Hi Bordon,

Almost all the init/decode things are done in libavcodec/qsvdec.c. You need to read the code more carefully.

Also, please don't hijack this thread. Some engineer from Intel already replied your thread. I also gave you some example command you could try in that thread. Let's discuss your questions there. Thanks.

[Liu__Chao]

Hi Surbhi,

Sorry for pushing on this. Just to let you know that I am still waiting for your help in case it already fell off your radar..

We are building a software depending on QSV hardware acceleration. For now, we could use 3.18.x to run it. However, it would be nice if we could also run it on latest kernels. Especially, new Ubuntu releases use 4.x by default. It sucks if we have to downgrade the kernel.

Thanks!

[Surbhi_M_Intel]

Hi, 

Sorry for late response. I am discussing this with other folks in the team that if this is a known issue or we have a work around available. Please allow some time for that since we tested extensively on the gold OS and 3.14 kernel for the Media Server Studio 2016 release and not on later kernel. However with our latest release Media Server Studio 2017 coming out in few weeks we have patches available for 4.4 kernel which should definitely resolve issue and let you update the kernel. Will keep you posted. 

Have a good weekend. 

Thanks,
Surbhi

[Liu__Chao]

FYI, I could repro the issue on IRIS 5100 on kernel 4.x.

[Surbhi_M_Intel]

HI there, 

Apologies for a super slow response. There has been new Media Server Studio release i.e. 2017 or 16.5 which support kernel 4.4.x as  stated before. I have tested your scenario on Ubuntu 16.04 and see no green frame issue. Can you please update to latest version and check if you are able to reproduce the issue?

Thanks,
Surbhi

[Liu__Chao]

Unfortunately, the CPUs I was complaining about are 4th gen CORE, which are not supported by 2017 if I understand it correctly...

[Surbhi_M_Intel]

Yes, you are right. I thought we were seeing it on 5th generation too. 
4th generation is not supported on Media Server Studio 2017 and the kernel 4.4.x is not supported by us through older Media Server Studio version.