topic It couldn't work correctly if in Media (Intel® Video Processing Library, Intel Media SDK)

HW decoder always return green frames on Linux kernel 4.X

Liu__Chao — Sat, 23 Jul 2016 00:26:49 GMT

Hi,

I am trying to use QSV with ffmpeg on Linux to decode videos. The media SDK I use is from MediaServerStudio2016.

First I got it working on the golden system, CentOS 7.1. It works pretty well. The performance improvement compared to software is quite amazing. I am pretty happy with that.

Then I tried it on Ubuntu 16.04 (kernel is 4.4.0). Everything works well, except it always returns pure green frames..

I upgraded the kernel to 4.4.15, with the same problem. Same for kernel 4.1.28.

Eventually, it works on kernel 3.18.37. NOTE: this problem is only for hardware decoding. It works on all kernels if I remove libmfxhw64-p.so.1.17.

I wonder was there any bug introduced in the i915 driver on Linux 4?

Hmm, kernel 3.19.x also has

Liu__Chao — Mon, 25 Jul 2016 00:53:35 GMT

Hmm, kernel 3.19.x also has this bug. Looks like only 3.18.x is fine to run QSV hardware decoder.

Hi there,

Surbhi_M_Intel — Mon, 25 Jul 2016 16:57:33 GMT

Hi there,
I don't expect their should be issue on kernel 4.4.x. can you please try sample_decode on Ubuntu 16.04 and copy the output of your run, along with the o/p you receive.

Thanks,
Surbhi

Thanks.

Liu__Chao — Mon, 25 Jul 2016 21:06:30 GMT

Thanks.

I run the sample_decode and it failed because SyncOutputSurface returns MFX_ERR_DEVICE_FAILED. I could repro the same failure on 3.18.37 and it worked on CentOS 7.1. So it might be related to that I didn't set things up correctly... However, the same environment works for ffmpeg on 3.18.37. I compared ffmpeg with sample_decode. ffmpeg only calls SyncOperation after DecodeFrameAsync returns non busy code. On the other hand, sample_decode calls SyncOperation after DecodeFrameAsync returns MFX_WRN_DEVICE_BUSY. Which one is correct? In case it might help, here is the ffmpeg code.

Following is the output of sample_decode

__bin/sample_decode h264 -i test_stream.264

libva info: VA-API version 0.99.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_0_32
libva info: va_openDriver() returns 0
Decoding Sample Version 0.0.000.0000

Input video AVC
Output format NV12
Input:
Resolution 176x96
Crop X,Y,W,H 0,0,0,0
Output:
Resolution 176x96
Frame rate 30.00
Memory type system
MediaSDK impl hw
MediaSDK version 1.17

Decoding started
DecodeFrameAsync
DecodeFrameAsync
DecodeFrameAsync
DecodeFrameAsync
SyncOutputSurface err: -17
Frame number: 0, fps: 0.000, fread_fps: 0.000, fwrite_fps: 0.000
Return on error: error code -1, /4s/chao/projects/samples/samples/sample_decode/src/pipeline_decode.cpp 1782

Return on error: error code 1, /4s/chao/projects/samples/samples/sample_decode/src/sample_decode.cpp 666

Hi Again,

Surbhi_M_Intel — Wed, 27 Jul 2016 05:30:00 GMT

Hi Again,

To me it seems like an install error for the device failure. Is it possible to try it on a fresh OS? If you have any system with fresh OS then simply install Media Server Studio and check sample_decode before ffmpeg. If you see no issue, then install ffmpeg and test it otherwise please report back the issue.

Couple of docs that might help (if you haven't seen them already)-
FFMpeg install with MSDK - http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/quicksync-video-ffmpeg-install-valid.pdf ;
Generic Install - https://software.intel.com/en-us/articles/how-to-setup-media-server-studio-on-secondary-os-of-linux ;

We are taking in many parameters, just to get the problem right. Can you please tell in which scenario did it work and in which it didn't (Are below observations correct?)

OS - CentOS 7.1
Kernel - Default
Sample_decode - no error
FFmpeg decode - no error

OS - Ubuntu or CentOS?
Kernel - 3.18.37
Sample_decode - Device failure error
FFmpeg decode - Device failure error

OS - Ubuntu 16.04
Kernel - 4.4.0 / 4.4.15 / 4.4.28
Sample_decode - Green frames?
FFmpeg decode - Green frames

-Surbhi

Hi Surbhi,

Liu__Chao — Wed, 27 Jul 2016 17:54:31 GMT

Hi Surbhi,

You are right that the sample_decode error was caused by install error on Ubuntu.

I installed 4.4.15 on CentOS 7.1 and reproduced the green frame issue:

./sample_decode h264 -i test_stream.264 -o test.yuv

libva info: VA-API version 0.99.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /4s/chao/bin/drivers/iHD_drv_video.so
libva info: Found init function __vaDriverInit_0_32
libva info: va_openDriver() returns 0
Decoding Sample Version 6.0.16043361.361

Input video AVC
Output format NV12
Resolution 176x96
Crop X,Y,W,H 0,0,0,0
Frame rate 30.00
Memory type system
MediaSDK impl hw
MediaSDK version 1.17

Decoding started
Frame number: 101, fps: 1242.221, fread_fps: 0.000, fwrite_fps: 2265.031
Decoding finished

517118

Not sure whether this is

Liu__Chao — Wed, 27 Jul 2016 22:13:10 GMT

Not sure whether this is related:

On 4.4, metrics monitor doesn't work:

sudo ./run.sh
ERROR: Failed to initialize metrics monitor, error code -10

The current version of

bordon_l_ — Fri, 29 Jul 2016 11:42:59 GMT

The current version of ffmpeg(Version 3.1.1) has not yet fully realize QSV hardware decoding, please refer to the source code of ffmpeg. I think the fastest way is that Intel can help to achieve the function as soon as possible, So that we can use the QSV hardware decoding function on ffmpeg. Perhaps Surbhi M can help us to convey this important proposal to Intel Develop Team, Let's cheer him on.

Hi Bordon,

Liu__Chao — Fri, 29 Jul 2016 18:04:20 GMT

Hi Bordon,

I have been using QSV decoding from ffmpeg 3.0.1 for a while. The latest versions don't work because of some bugs, which is already (or going to be) fixed.

Could you give a bit details about what you said? Are you saying the performance could be further improved?

Looks like the green frame

Liu__Chao — Sat, 30 Jul 2016 07:16:10 GMT

Looks like the green frame issue is also related to the GPU.

So far I could only repro it on HD Graphics 4400 on Linux kernel 3.19.x and above (CentOS and Ubuntu).

I have another machine with IRIS Pro 6200, I just installed Ubuntu 16.04 (kernel 4.4.0), it works!

Looks like HD Graphics 4400 + kernel >=3.19.x is the culprit...

It couldn't work correctly if

bordon_l_ — Mon, 01 Aug 2016 02:26:39 GMT

It couldn't work correctly if you carefully check the source code of the ffmpeg that QSV decoding, such as 'qsvdec_mpeg2.c' in ffmpeg:

static av_cold int qsv_decode_close(AVCodecContext *avctx)
{
QSVMPEG2Context *s = avctx->priv_data;

ff_qsv_decode_close(&s->qsv);

return 0;
}

static av_cold int qsv_decode_init(AVCodecContext *avctx)
{
return 0;
}

You can find the function return 0 directly.

Hi Bordon,

Liu__Chao — Mon, 01 Aug 2016 18:07:23 GMT

Hi Bordon,

Almost all the init/decode things are done in libavcodec/qsvdec.c. You need to read the code more carefully.

Also, please don't hijack this thread. Some engineer from Intel already replied your thread. I also gave you some example command you could try in that thread. Let's discuss your questions there. Thanks.

Hi Surbhi,

Liu__Chao — Thu, 04 Aug 2016 17:52:14 GMT

Hi Surbhi,

Sorry for pushing on this. Just to let you know that I am still waiting for your help in case it already fell off your radar..

We are building a software depending on QSV hardware acceleration. For now, we could use 3.18.x to run it. However, it would be nice if we could also run it on latest kernels. Especially, new Ubuntu releases use 4.x by default. It sucks if we have to downgrade the kernel.

Thanks!

Hi,

Surbhi_M_Intel — Fri, 05 Aug 2016 23:55:00 GMT

Hi,

Sorry for late response. I am discussing this with other folks in the team that if this is a known issue or we have a work around available. Please allow some time for that since we tested extensively on the gold OS and 3.14 kernel for the Media Server Studio 2016 release and not on later kernel. However with our latest release Media Server Studio 2017 coming out in few weeks we have patches available for 4.4 kernel which should definitely resolve issue and let you update the kernel. Will keep you posted.

Have a good weekend.

Thanks,
Surbhi

FYI, I could repro the issue

Liu__Chao — Sat, 20 Aug 2016 07:58:54 GMT

FYI, I could repro the issue on IRIS 5100 on kernel 4.x.

HI there,

Surbhi_M_Intel — Fri, 23 Sep 2016 23:13:21 GMT

HI there,

Apologies for a super slow response. There has been new Media Server Studio release i.e. 2017 or 16.5 which support kernel 4.4.x as stated before. I have tested your scenario on Ubuntu 16.04 and see no green frame issue. Can you please update to latest version and check if you are able to reproduce the issue?

Thanks,
Surbhi

Unfortunately, the CPUs I was

Liu__Chao — Sun, 25 Sep 2016 05:14:51 GMT

Unfortunately, the CPUs I was complaining about are 4th gen CORE, which are not supported by 2017 if I understand it correctly...

Yes, you are right. I thought

Surbhi_M_Intel — Mon, 26 Sep 2016 19:02:30 GMT

Yes, you are right. I thought we were seeing it on 5th generation too.
4th generation is not supported on Media Server Studio 2017 and the kernel 4.4.x is not supported by us through older Media Server Studio version.